This adds a new parameter to the quantized index mapping that allows
default oversampling and rescoring to occur.
This doesn't adjust any of the defaults. It allows it to be configured.
When the user provides `rescore_vector: {oversample: <number>}` in the
query it will overwrite it.
For example, here is how to use it with bbq:
```
PUT rescored_bbq
{
"mappings": {
"properties": {
"vector": {
"type": "dense_vector",
"index_options": {
"type": "bbq_hnsw",
"rescore_vector": {"oversample": 3.0}
}
}
}
}
}
```
Then, when querying, it will auto oversample the `k` by `3x` and rerank
with the raw vectors.
```
POST _search
{
"knn": {
"query_vector": [...],
"field": "vector"
}
}
```
Building on the work started in https://github.com/elastic/elasticsearch/pull/123904, we now want to auto-generate most of the small subfiles from the ES|QL functions unit tests.
This work also investigates any remaining discrepancies between the original asciidoc version and the new markdown, and tries to minimize differences so the docs do not look too different.
The kibana json and markdown files are moved to a new location, and the operator docs are a little more generated than before (although still largely manual).
* add max number of chunks
* wire merge function
* implement sparse merge function
* move tests to correct package/file
* float merge function
* bytes merge function
* more accurate byte average
* spotless
* Fix/improve EmbeddingRequestChunkerTests
* Remove TODO
* remove unnecessary field
* remove Chunk generic
* add TODO
* Remove specialized chunks
* add comment
* Update docs/changelog/123150.yaml
* update changelog
This creates a `pragma` you can use to request that fields load from a
stored field rather than doc values. It implements that pragma for
`keyword` and number fields.
We expect that, for some disk configuration and some number of fields,
that it's faster to load those fields from _source or stored fields than
it is to use doc values. Our default is doc values and on my laptop it's
*always* faster to use doc values. But we don't ship my laptop to every
cluster.
This will let us experiment and debug slow queries by trying to load
fields a different way.
You access this pragma with:
```
curl -HContent-Type:application/json -XPOST localhost:9200/_query?pretty -d '{
"query": "FROM foo",
"pragma": {
"field_extract_preference": "STORED"
}
}'
```
On a release build you'll need to add `"accept_pragma_risks": true`.
* SEARCH-802 - bug fixed - Query rules allows for creation of rules with invalid match criteria
* [CI] Auto commit changes from spotless
* Worked on the comments given in the PR
* [CI] Auto commit changes from spotless
* Fixed Integration tests
* [CI] Auto commit changes from spotless
* Made changes from the PR
* Update docs/changelog/122823.yaml
* [CI] Auto commit changes from spotless
* Fixed the duplicate code issue in queryRuleTests
* Refactored code to clean it up based on PR comments
* [CI] Auto commit changes from spotless
* Logger statements were removed
* Cleaned up the QueryRule tests
* [CI] Auto commit changes from spotless
* Update x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/EnterpriseSearchModuleTestUtils.java
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* Inline cast to date
* Update docs/changelog/123460.yaml
* New capability for `::date` casting
* More tests
* Update tests
---------
Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com>
Fixes https://github.com/elastic/elasticsearch/issues/123067
Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike
This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in https://github.com/elastic/elasticsearch/pull/123398, which I kept separated as it changes many files
Resolves#123053
This adds the thread name to the driver sleep profile output.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* ESQL: Lazy collection copying during node transform
A set of optimization for tree traversal:
1. perform lazy copying during children transform
2. use long hashing to avoid object creation
3. perform type check first before collection checking
Relates #124395
Inter-segment concurrency is disabled whenever sort by field, included script sorting, is used in a search request.
The reason why sort by field does not use concurrency is that there are some performance implications, given that the hit queue in Lucene is build per slice and the different search threads don't share information about the documents they have already visited etc.
The reason why script sort has concurrency disabled is that the script sorting implementation is not thread safe. This commit addresses such concurrency issue and re-enables search concurrency for search requests that use script sorting. In addition, missing tests are added to cover for sort scripts that rely on _score being available and top_hits aggregation with a scripted sort clause.
Currently, we use NamedWriteable for serializing blocks. While
convenient, it incurs a noticeable performance penalty when pages
contain thousands of blocks. Since block types are small and already
centered in ElementType, we can safely switch from NamedWriteable to
typed code. For example, the NamedWriteable alone of a small page with
10K fields would be 180KB, whereas the new method reduces it to 10KB.
Below are the serialization improvements with FROM idx | LIMIT 10000
where the target index has 10K fields:
- write_exchange_response executed 173 times took: 73.2ms -> 26.7ms
- read_exchange_response executed 173 times took: 49.4ms -> 25.8ms
Read dimension values once per tsid/bucket docid range instead of for each document being processed.
The dimension value within a bucket-interval docid range is always to same and this avoids unnecessary reads.
Latency of downsampling the tsdb track index into a 1 hour interval downsample index drop by ~16% (running on my local machine).
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
* Added check for blank string to skip generating embeddings with unit test
* Adding yaml tests for skipping embedding generation
* dynamic update not required if model_settings stays null
* Updating node feature for handling empty input name and description
* Update yaml tests with refresh=true
* Update unit test to follow more accurate behavior
* Added yaml tests for multu chunks
* [CI] Auto commit changes from spotless
* Adding highlighter yaml tests for empty input
* Update docs/changelog/123763.yaml
* Update changelog and test reason to have more polished documentation
* adding input value into the response source and fixing unit tests by reformating
* Adding highligher test for backward compatibility and refactor existing test
* Added bwc tests for empty input and multi chunks
* Removed reindex for empty input from bwc
* [CI] Auto commit changes from spotless
* Fixing yaml test
* Update unit tests helper function to support both format
* [CI] Auto commit changes from spotless
* Adding cluster features for bwc
* Centralize logic for assertInference helper
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
In the create-index-from-source action, we should set the cause of the create index request so that it is clear in the logs. Without setting the cause on the request, the default value of api is used.
Adds options to QSTR function.
#118619 added named function parameters. This PR uses this mechanism for allowing query string function parameters, so query string parameters can be used in ES|QL.
Closes#120933
When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false.
By keeping a list of all the rollover results in a rollover request
batch, we were keeping references to all the intermediate cluster states
that we built. We've seen this list take up ~1.4GB with 600 rollover
requests in one batch.
We only kept the list of results to compute the "reason" for the
allocation reroute, so we can easily drop the cluster state reference
from the list and only keep what we need.
Fixes#123893
This commit improves memory accounting by incorporating document
expansion during shard bulk execution. Additionally, it introduces a new
limit on the maximum document size, which defaults to 5% of the
available heap.
This limit can be configured using the new setting:
indexing_pressure.memory.max_operation_size
These changes help prevent excessive memory consumption and
improve indexing stability.
Closes ES-10777
This change introduces a cluster setting
`esql.query.allow_partial_results` that allows enabling or disabling
allow_partial_results in ES|QL at the cluster-wide level. Initially,
this setting defaults to false, but it will be switched to true soon.
The reason for not changing the default in this PR is that it requires
adjusting many tests, which would make the PR too large. Instead, we
will adjust the tests incrementally and switch the default when the
tests are ready. This cluster setting is useful for falling back to the
previous behavior (i.e., disabling allow_partial_results) if users
upgrade to the new version and haven't updated their queries.
Also, the default setting can be overridden on a per-request basis via a
URL parameter (allow_partial_results) (changed from request body to URL
parameter to conform to the proposal).
Relates #122802
