* Add index_options parameter to semantic_text field mapping
* Cleanup & tests
* Update docs
* Update docs/changelog/119967.yaml
* Addressed some PR feedbak
* Update yaml tests
* Refactoring
* Cleanup
* Fix some tests
* Hack in inferring text_embedding task type from index options
* [CI] Auto commit changes from spotless
* Fix error inferring model settings
* Update docs
* Update tests
* Update docs/reference/mapping/types/semantic-text.asciidoc
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* Address some minor PR feedback
* Remove partial model_settings with inferred task type
* Cleanup
* Remove unnecessary changes
* Fix errors from merge
* [CI] Auto commit changes from spotless
* Cleanup
* Checkpoint, saving changes before merge
* Update parsing
* [CI] Auto commit changes from spotless
* Stash changes
* Fix compile errors
* [CI] Auto commit changes from spotless
* Cleanup error
* fix test
* fix test
* Fix another test
* A bit of cleanup
* Fix tests
* Spotless
* Respect index options if set over defaults
* Cleanup
* [CI] Auto commit changes from spotless
* Support updating to compatible versions, add some cleanup and validation
* Remove test that can't be done here - needs to be unit test
* Add validation
* Cleanup
* Fix some yaml tests
* [CI] Auto commit changes from spotless
* Happy path early index validation works now; edge cases surrounding default BBQ remain
* Always emit index options, even when using defaults
* Minor cleanup
* Fix test compilation failures
* Fix some tests
* Continue to iterate on test failures
* Remove index options from inference field metadata as it is only needed at field creation time
* Fix some tests
* Remove transport version, no longer needed
* Fix yaml tests
* Add tests
* IndexOptions don't need to implement Writeable
* [CI] Auto commit changes from spotless
* Refactor - move SemanticTextIndexOptions
* Remove writeable
* Move index_options parsing to semantic text field mapper
* Cleanup
* Fix test compilation issue
* Cleanup
* Remove whitespace
* Remove writeables from index options
* Disable merging null options?
* Add docs
* [CI] Auto commit changes from spotless
* Revert "Disable merging null options?"
This reverts commit 2ef8b1dc29.
* Remove default serialization
* Include default index option type to defaults
* [CI] Auto commit changes from spotless
* Go back to allowing null updateS
* Cleanup
* Fix validation error
* Revert "Include default index option type to defaults"
This reverts commit b08e2a1d7e.
* Update tests
* Revert "Update tests"
This reverts commit aedfafe0e7.
* Better fix for null inputs
* Remove redundant merge validation
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* propgating retrievers to inner retrievers
* test feature taken care of
* Small changes in concurrent multipart upload interfaces (#128977)
Small changes in BlobContainer interface and wrapper.
Relates ES-11815
* Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054)
The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index.
The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers.
The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers.
This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage.
Closes#128541
* [Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
* ESQL: Check for errors while loading blocks (#129016)
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
* Make `PhaseCacheManagementTests` project-aware (#129047)
The functionality in `PhaseCacheManagement` was already project-aware,
but these tests were still using deprecated methods.
* Vector test tools (#128934)
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
* ES|QL: refactor generative tests (#129028)
* Add a test of LOOKUP JOIN against a time series index (#129007)
Add a spec test of `LOOKUP JOIN` against a time series index.
* Make ILM `ClusterStateWaitStep` project-aware (#129042)
This is part of an iterative process to make ILM project-aware.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078
* Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076)
The `ClusterState` parameter of the `asyncPredicate` is not used
anywhere.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098
* Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867
* Threadpool merge executor is aware of available disk space (#127613)
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
* Add option to include or exclude vectors from _source retrieval (#128735)
This PR introduces a new include_vectors option to the _source retrieval context.
When set to false, vectors are excluded from the returned _source.
This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely.
By default, vectors remain included unless explicitly excluded.
* Remove direct minScore propagation to inner retrievers
* cleaned up skip
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149
* Add transport version for ML inference Mistral chat completion (#129033)
* Add transport version for ML inference Mistral chat completion
* Add changelog for Mistral Chat Completion version fix
* Revert "Add changelog for Mistral Chat Completion version fix"
This reverts commit 7a57416bdc.
* Correct index path validation (#129144)
All we care about is if reindex is true or false. We shouldn't worry
about force merge. Because if reindex is true, we will create the
directory, if its false, we won't.
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148
* Implemented completion task for Google VertexAI (#128694)
* Google Vertex AI completion model, response entity and tests
* Fixed GoogleVertexAiServiceTest for Service configuration
* Changelog
* Removed downcasting and using `moveToFirstToken`
* Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses
* Added unit tests
* PR feedback
* Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion
* Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel
* Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel
* Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest
---------
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* ES|QL - kNN function initial support (#127322)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* [Inference API] Add "rerank" task type to "elastic" provider (#126022)
* Rename target destination for microbenchmarks (#128878)
* Include direct memory and non-heap memory in ML memory calculations (take #2) (#128742)
* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Throw better exception for unsupported aggregations over shape fields (#129139)
* Update Test Framework To Handle Query Rewrites That Rely on Non-Null Searchers (#129160)
* Update ReproduceInfoPrinter to correctly print a reproduction line for Lucene & build candidate upgrade tests (#129044)
* Increment inference stats counter for shard bulk inference calls (#129140)
This change updates the inference stats counter to include chunked inference calls performed by the shard bulk inference filter on all semantic text fields.
It ensures that usage of inference on semantic text fields is properly recorded in the stats.
* Synthetic source: avoid storing multi fields of type text and match_only_text by default. (#129126)
Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field.
Without this change, ES would store field otherwise twice in a multi-field configuration.
For example:
```
...
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
...
```
In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field.
This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field.
* Adding `scheduled_report_id` field to kibana reporting template (#127827)
* Adding scheduled_report_id field to kibana reporting template
* Incrementing stack template registry version
* ES|QL: Add FORK generative tests (#129135)
* ES|QL Completion command syntax change (#129189)
* Included pinned retriever in 9.1 docs
* reverted unnecessary change
* made the suggested changes
* Update retrievers.md
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Update retrievers.md
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Luigi Dell'Aquila <luigi.dellaquila@gmail.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Albert Zaharovits <email+github@zalbert.me>
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Leonardo Hoet <55866308+leo-hoet@users.noreply.github.com>
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Jan Kuipers <148754765+jan-elastic@users.noreply.github.com>
Co-authored-by: Tim Grein <tim.grein@elastic.co>
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Ignacio Vera <ignacio.vera@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Ying Mao <ying.mao@elastic.co>
Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com>
Co-authored-by: Aurélien FOUCRET <aurelien.foucret@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Adds support for LIKE function alternative syntax with a list of patterns.
Examples:
FROM foo
| WHERE bar LIKE ("A*","B*", "C?")
The new syntax is documented as part of the existing LIKE function documentation. We will use the existing WildcardLike java implementation for existing cases using the old syntax and one list argument case to improve mixed cluster compatibility.
This introduces a `none` chunking strategy that disables automatic chunking when using an inference endpoint.
It enables users to provide pre-chunked input directly to a `semantic_text` field without any additional splitting.
The chunking strategy can be configured either on the inference endpoint or directly in the `semantic_text` field definition.
**Example:**
```json
PUT test-index
{
"mappings": {
"properties": {
"my_semantic_field": {
"type": "semantic_text",
"chunking_settings": {
"strategy": "none" <1>
}
}
}
}
}
```
<1> Disables automatic chunking on `my_semantic_field`.
```json
PUT test-index/_doc/1
{
"my_semantic_field": ["my first chunk", "my second chunk", ...] <1>
...
}
```
<1> Pre-chunked input provided as an array of strings.
Each array element represents a single chunk that will be sent directly to the inference service without further processing.
This extends RENAME's grammar to allow a new syntax: `| RENAME new_name
= old_name` This is supported along the existing `... old_name AS
new_name` syntax.
Closes#129208
* Take match_phrase out of snapshot and make tech preview
* Update docs/changelog/128925.yaml
* PR feedback
* Adding regenerated test data
* Update docs/changelog/128925.yaml
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* [CI] Auto commit changes from spotless
* Checkstyle
* Correct docs
* Hopefully fix docs build
* Found one more bad docs link - here's hoping this now fixes the doc build
* OMG bitten by - vs _
---------
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Aurélien FOUCRET <aurelien.foucret@gmail.com>
* Initial commit of match_phrase
* Add MatchPhraseQueryTests
* First pass at CSV specs
* Update docs/changelog/127661.yaml
* Refactor so MatchPhrase doesn't use all fulltext test cases, just text only
* Fix tests
* Add some CSV test cases
* Fix test
* Update changelog
* Update tests
* Comment out MATCH_PHRASE in search-functions Markdown
* Minor PR feedback
* PR feedback - refactor/consolidate code
* Add some more tests
* Fix some tests
* [CI] Auto commit changes from spotless
* Fix tests
* PR feedback - add tests, support boost and numeric data
* Revert "PR feedback - add tests, support boost and numeric data"
This reverts commit 4e7a699e3e.
* Apply testing/PR feedback outside numeric support only
* Regenerate docs
* Add negative test
* Update x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-phrase-function.csv-spec
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* Update x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-phrase-function.csv-spec
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* Update x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-phrase-function.csv-spec
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* PR feedback
* Fix auto-commit error
* Regenerate docs
* Update x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/MatchPhrase.java
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Remove non text field types
* Fake test data
* Remove tests that no longer should pass without ip/date/version support
* Put real data in score tests now that I was able to engineer a failure
* Realized the scoring test might be flakey because how it was written, updated
* PR feedback
* PR feedback
* [CI] Auto commit changes from spotless
* Add check to MatchPhrase tests
* Fix merge errors
* [CI] Auto commit changes from spotless
* Test generated docs
* Add additional verifier tests
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Added support for the three primary scalar grid functions:
* `ST_GEOHASH(geom, precision)`
* `ST_GEOTILE(geom, precision)`
* `ST_GEOHEX(geom, precision)`
As well as versions of these three that take an optional `geo_shape` boundary (must be a `BBOX` ie. `Rectangle`).
And also supporting conversion functions that convert the grid-id from long to string and back to long.
This work represents the core of the feature to support geo-grid aggregations in ES|QL.
* New l2 normalizer added
* L2 score normaliser is registered
* test case added to the yaml
* Documentation added
* Resolved checkstyle issues
* Update docs/changelog/128504.yaml
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Score 0 test case added to check for corner cases
* Edited the markdown doc description
* Pruned the comment
* Renamed the variable
* Added comment to the class
* Unit tests added
* Spotless and checkstyle fixed
* Fixed build failure
* Fixed the forbidden test
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Begins adding support for running "tagged queries" to the compute
engine. Here, it's just the `LuceneSourceOperator` because that's
useful and contained.
Example time! Say you are running:
```
FROM foo
| STATS MAX(v) BY ROUND_TO(g, 0, 100, 1000, 100000)
```
It's *often* faster to run this as four queries:
* The docs that round to `0`
* The docs that round to `100`
* The docs that round to `1000`
* The docs that round to `100000`
This creates an ESQL operator that can run these queries, one after the
other and attach those tags.
Aggs uses this trick and it's *way* faster when it can push down count
queries, but it's still faster when it pushes doc loading things. This
implementation in `LuceneSourceOperator` is quite similar to the doc
loading version in _search.
I don't have performance measurements yet because I haven't plugged this
into the language. In _search we call this `filter-by-filter` and enable
it when each group averages to more than 5000 documents and when there
isn't an `_doc_count` field. It's faster in those cases not to push. I
expect we'll be pretty similar.
Creates a `ROUND_TO` function that rounds it's input to one of the
provided values. Like so:
```
ROUND_TO(v, 0, 5000, 10000, 20000, 40000, 100000)
v | ROUND_TO
0 | 0
100 | 0
6000 | 5000
45001 | 40000
999999 | 100000
```
For some sequences of numbers you could do this with the `/` operator -
but for arbitrary sequences of numbers you needed `CASE` which is quite
slow. And hard to read!
Rewriting the example above would look like:
```
CASE (
v < 5000, 0,
v < 10000, 5000,
v < 20000, 10000,
v < 40000, 20000,
v < 100000, 40000,
100000
)
```
Even better, this is *fast*:
```
(operation) Mode Cnt Score Error Units
round_to_4_via_case avgt 7 138.124 ± 0.738 ns/op
round_to_4 avgt 7 0.805 ± 0.011 ns/op
round_to_3 avgt 7 0.739 ± 0.011 ns/op
round_to_2 avgt 7 0.651 ± 0.009 ns/op
date_trunc avgt 7 2.425 ± 0.018 ns/op
```
I've included a comparison to `DATE_TRUNC` above because we should be
able to rewrite `DATE_TRUNC` into `ROUND_TO` when we know the date range
of the index. This doesn't do it now, but it should be possible.
Documents that the VALUES aggregate function returns unique documents
and points folks to the TOP aggregate function if they want to keep
dupes.
Closes#128091
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
