* Add recursive chunker
* Update docs/changelog/126866.yaml
* Clean up separator sets and add asMap function for RecrusiveChunkingSettings
* Add javadoc for chunker, add tests, reduce word counting operations
* Remove split merging and add long document unit test
* [CI] Auto commit changes from spotless
* Add markdown chunking tests and reduce substring calls
* Clean up matcher logic
* Add testing for not splitting after valid chunk is found
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Using a static `diff` or epsilon just doesn't work for this test as the
scores can be very large, but relatively close.
Maybe there is a simpler way, but my mind wasn't wanting to "math" very
much.
For example, the seed that this previously failed on had scores like
`1.726524E9` and `1.7265239E9`, which, given their size, are really
close together (within 128). But a static epsilon wouldn't capture that.
closes: https://github.com/elastic/elasticsearch/issues/128485
* Remove error parsing class
* Adding test for lack of error parsing logic
* Adding transport version check
* Wrapping string in try/catch and adding test
For legacy reasons, `Metadata.Builder` contains a map of
`ProjectMetadata.Builder`s instead of a map of `ProjectMetadata`s. While
the plan is still to change that, there is a significant amount of
refactoring that needs to be done before we can make that change.
We have seen regressions in benchmarks due to this extra conversion of
project metadata -> builder -> project metadata. We add a method that
allows updating a single project without having to convert all the
projects to builders first.
This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
https://github.com/elastic/elasticsearch/pull/120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.
* Add new option to allow full delegation of tests to gradle in IntelliJ plus docs
* Update CONTRIBUTING.md
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Update CONTRIBUTING.md
Fix typo in settings file name in docs
---------
Co-authored-by: Rene Groeschke <rene@breskeby.com>
A new bc-upgrade step in the intake pipeline invokes a bash script to get the commit hash of the latest BC (or snapshot if no BC is available yet) from https://docs.elastic.dev/release/stack/release-api#future-releases for the current branch and runs BWC / upgrade tests from this commit to the HEAD commit of the current branch.
If no matching BC / snapshot exists, the upgrade tests are skipped.
Relates to ES-11905
* [Profiling] Omit address in grouping for unknown function but known executable
This reduces the complexity of the flamegraph in unsymbolized areas.
Symbolized areas of the flamegraph are not affected.
* Fix tests
* Fix IT tests
It is possible to get all the way down to the knn format reader and
there be no vectors in the index.
This execution path is possible if utilizing nested queries (which
bypasses the higher level checks in
`KnnFloatVectorQuery#approximateSearch`).
bbq_flat should check for the existence of vectors before attempting to
create the scorer.
* Add index_options parameter to semantic_text field mapping
* Cleanup & tests
* Update docs
* Update docs/changelog/119967.yaml
* Addressed some PR feedbak
* Update yaml tests
* Refactoring
* Cleanup
* Fix some tests
* Hack in inferring text_embedding task type from index options
* [CI] Auto commit changes from spotless
* Fix error inferring model settings
* Update docs
* Update tests
* Update docs/reference/mapping/types/semantic-text.asciidoc
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* Address some minor PR feedback
* Remove partial model_settings with inferred task type
* Cleanup
* Remove unnecessary changes
* Fix errors from merge
* [CI] Auto commit changes from spotless
* Cleanup
* Checkpoint, saving changes before merge
* Update parsing
* [CI] Auto commit changes from spotless
* Stash changes
* Fix compile errors
* [CI] Auto commit changes from spotless
* Cleanup error
* fix test
* fix test
* Fix another test
* A bit of cleanup
* Fix tests
* Spotless
* Respect index options if set over defaults
* Cleanup
* [CI] Auto commit changes from spotless
* Support updating to compatible versions, add some cleanup and validation
* Remove test that can't be done here - needs to be unit test
* Add validation
* Cleanup
* Fix some yaml tests
* [CI] Auto commit changes from spotless
* Happy path early index validation works now; edge cases surrounding default BBQ remain
* Always emit index options, even when using defaults
* Minor cleanup
* Fix test compilation failures
* Fix some tests
* Continue to iterate on test failures
* Remove index options from inference field metadata as it is only needed at field creation time
* Fix some tests
* Remove transport version, no longer needed
* Fix yaml tests
* Add tests
* IndexOptions don't need to implement Writeable
* [CI] Auto commit changes from spotless
* Refactor - move SemanticTextIndexOptions
* Remove writeable
* Move index_options parsing to semantic text field mapper
* Cleanup
* Fix test compilation issue
* Cleanup
* Remove whitespace
* Remove writeables from index options
* Disable merging null options?
* Add docs
* [CI] Auto commit changes from spotless
* Revert "Disable merging null options?"
This reverts commit 2ef8b1dc29.
* Remove default serialization
* Include default index option type to defaults
* [CI] Auto commit changes from spotless
* Go back to allowing null updateS
* Cleanup
* Fix validation error
* Revert "Include default index option type to defaults"
This reverts commit b08e2a1d7e.
* Update tests
* Revert "Update tests"
This reverts commit aedfafe0e7.
* Better fix for null inputs
* Remove redundant merge validation
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
We currently fail to serialize this case. This change has been made so
it is backwards compatible. I label it as non an issue as this feature
is behind a feature flag and is not released.
fixes https://github.com/elastic/elasticsearch/issues/129551
This injects assertions in spec-based and YAML integration tests that the received results aren't partial.
This only happens in case the tests themselves don't set "allow_partial_results": true.
Closes#129256
Disallow partial quoting: individual quoting of constituent strings in an index pattern
---------
Co-authored-by: Pawan Kartik <pawankartik.chitrapu@elastic.co>
These usages had the potential of causing test failures when a data
stream was created before midnight and the backing index name generation
ran the next day - which would be millisecconds apart. To avoid these
failures, we update the tests to be robust to these time differences.
Resolves#123376
Updates the ILM retry action to resolve the correct project.
Running the async action currently isn't project-aware yet, but that
will be handled when that action class is made project-aware.
* propgating retrievers to inner retrievers
* test feature taken care of
* Small changes in concurrent multipart upload interfaces (#128977)
Small changes in BlobContainer interface and wrapper.
Relates ES-11815
* Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054)
The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index.
The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers.
The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers.
This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage.
Closes#128541
* [Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
* ESQL: Check for errors while loading blocks (#129016)
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
* Make `PhaseCacheManagementTests` project-aware (#129047)
The functionality in `PhaseCacheManagement` was already project-aware,
but these tests were still using deprecated methods.
* Vector test tools (#128934)
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
* ES|QL: refactor generative tests (#129028)
* Add a test of LOOKUP JOIN against a time series index (#129007)
Add a spec test of `LOOKUP JOIN` against a time series index.
* Make ILM `ClusterStateWaitStep` project-aware (#129042)
This is part of an iterative process to make ILM project-aware.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078
* Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076)
The `ClusterState` parameter of the `asyncPredicate` is not used
anywhere.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098
* Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867
* Threadpool merge executor is aware of available disk space (#127613)
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
* Add option to include or exclude vectors from _source retrieval (#128735)
This PR introduces a new include_vectors option to the _source retrieval context.
When set to false, vectors are excluded from the returned _source.
This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely.
By default, vectors remain included unless explicitly excluded.
* Remove direct minScore propagation to inner retrievers
* cleaned up skip
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149
* Add transport version for ML inference Mistral chat completion (#129033)
* Add transport version for ML inference Mistral chat completion
* Add changelog for Mistral Chat Completion version fix
* Revert "Add changelog for Mistral Chat Completion version fix"
This reverts commit 7a57416bdc.
* Correct index path validation (#129144)
All we care about is if reindex is true or false. We shouldn't worry
about force merge. Because if reindex is true, we will create the
directory, if its false, we won't.
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148
* Implemented completion task for Google VertexAI (#128694)
* Google Vertex AI completion model, response entity and tests
* Fixed GoogleVertexAiServiceTest for Service configuration
* Changelog
* Removed downcasting and using `moveToFirstToken`
* Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses
* Added unit tests
* PR feedback
* Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion
* Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel
* Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel
* Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest
---------
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* ES|QL - kNN function initial support (#127322)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* [Inference API] Add "rerank" task type to "elastic" provider (#126022)
* Rename target destination for microbenchmarks (#128878)
* Include direct memory and non-heap memory in ML memory calculations (take #2) (#128742)
* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Throw better exception for unsupported aggregations over shape fields (#129139)
* Update Test Framework To Handle Query Rewrites That Rely on Non-Null Searchers (#129160)
* Update ReproduceInfoPrinter to correctly print a reproduction line for Lucene & build candidate upgrade tests (#129044)
* Increment inference stats counter for shard bulk inference calls (#129140)
This change updates the inference stats counter to include chunked inference calls performed by the shard bulk inference filter on all semantic text fields.
It ensures that usage of inference on semantic text fields is properly recorded in the stats.
* Synthetic source: avoid storing multi fields of type text and match_only_text by default. (#129126)
Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field.
Without this change, ES would store field otherwise twice in a multi-field configuration.
For example:
```
...
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
...
```
In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field.
This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field.
* Adding `scheduled_report_id` field to kibana reporting template (#127827)
* Adding scheduled_report_id field to kibana reporting template
* Incrementing stack template registry version
* ES|QL: Add FORK generative tests (#129135)
* ES|QL Completion command syntax change (#129189)
* Included pinned retriever in 9.1 docs
* reverted unnecessary change
* made the suggested changes
* Update retrievers.md
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Update retrievers.md
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Luigi Dell'Aquila <luigi.dellaquila@gmail.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Albert Zaharovits <email+github@zalbert.me>
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Leonardo Hoet <55866308+leo-hoet@users.noreply.github.com>
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Jan Kuipers <148754765+jan-elastic@users.noreply.github.com>
Co-authored-by: Tim Grein <tim.grein@elastic.co>
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Ignacio Vera <ignacio.vera@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Ying Mao <ying.mao@elastic.co>
Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com>
Co-authored-by: Aurélien FOUCRET <aurelien.foucret@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
