There was a bug in the code for deleting unused and orphan ML data. When deletion using DBQ occurred, the bug caused the request to time out. This PR resolves the issue.
Keep better track of shard contexts using RefCounted, so they can be released more aggressively during operator processing. For example, during TopN, we can potentially release some contexts if they don't pass the limit filter.
This is done in preparation of TopN fetch optimization, which will delay the fetching of additional columns to the data node coordinator, instead of doing it in each individual worker, thereby reducing IO. Since the node coordinator would need to maintain the shard contexts for a potentially longer duration, it is important we try to release what we can eariler.
An even more advanced optimization is to delay fetching to the main cluster coordinator, but that would be more involved, since we need to first figure out how to transport the shard contexts between nodes.
Summary of main changes:
DocVector now maintains a RefCounted instance per shard.
Things which can build or release DocVectors (e.g., LuceneSourceOperator, TopNOperator), can also hold RefCounted instances, so they can pass them to DocVector and also ensure contexts aren't released if they can still be potentially used later.
Driver's main loop iteration (runSingleLoopIteration), now closes its operators even between different operator processing. This is extra aggressive, and was mostly done to improve testability.
Added a couple of tests to TopNOperator and a new integration test EsqlTopNShardManagementIT, which uses the pausable plugin framework to check that TopNOperator releases things as early as possible..
The local plan optimizer should not change the layout, as it has already
been agreed upon. However, CombineProjections can violate this when some
grouping elements refer to the same attribute. This occurs when
ReplaceFieldWithConstantOrNull replaces missing fields with the same
reference for a given data type.
Closes#128054Closes#129811
ES|QL index patterns validation: Ensure that the patterns in the query are syntactically and semantically valid
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
With the introduction of entitlements (#120243) and exclusive file
access (#123087) it is no longer safe to watch a whole directory.
In a lot of deployments, the parent directory for SSL config files
will be the main config directory, which also contains exclusive files
such as SAML realm metadata or File realm users. Watching that
directory will cause entitlement warnings because it is not
permissible for core/ssl-config to read files that are exclusively
owned by the security module (or other modules)
Introduces a new `RemoveBlock` API that complements the existing `AddBlock` API by allowing users to remove index blocks using `DELETE /{index}/_block/{block}`.
Resolves#128966
---------
Co-authored-by: Niels Bauman <nielsbauman@gmail.com>
The comment in `TransportHandshaker` indicates (correctly) that we emit
a warning when talking to a chronologically-newer-yet-numerically-older
version, but the wording of the warning message is inverted and says
that the remote is chronologically-older-yet-numerically-newer. This
commit straightens out the message to match the situation it is
describing.
Relates #123397
* Initial checkin of refactored index_options code
* [CI] Auto commit changes from spotless
* initial unit testing
* complete unit tests; add yaml tests
* [CI] Auto commit changes from spotless
* register test feature for sparse vector
* Update docs/changelog/129089.yaml
* update changelog
* add docs
* explicit set default index_options if null
* [CI] Auto commit changes from spotless
* update yaml tests; update docs
* fix yaml tests
* readd auth for teardown
* only serialize index options if not default
* [CI] Auto commit changes from spotless
* serialization refactor; pass index version around
* [CI] Auto commit changes from spotless
* fix transport versions merge
* fix up docs
* [CI] Auto commit changes from spotless
* fix docs; add include_defaults unit and yaml test
* [CI] Auto commit changes from spotless
* override getIndexReaderManager for SemanticQueryBuilderTests
* [CI] Auto commit changes from spotless
* cleanup mapper/builder/tests; index vers. in type
still need to refactor / clean YAML tests
* [CI] Auto commit changes from spotless
* cleanups to mapper tests for clarity
* [CI] Auto commit changes from spotless
* move feature into mappers; fix yaml tests
* cleanups; add comments; remove redundant test
* [CI] Auto commit changes from spotless
* escape more periods in the YAML tests
* cleanup mapper and type tests
* [CI] Auto commit changes from spotless
* rename mapping for previous index test
* set explicit number of shards for yaml test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
Send the Elastic API Payload to a SageMaker endpoint, and parse the
response as if it were an Elastic API response.
- SageMaker now supports all task types in the Elastic API format.
- Streaming is supported using the SageMaker client/server rpc,
rather than SSE. Payloads must be in a complete and valid JSON
structure.
- Task Settings can be used for additional passthrough settings, but
they will not be saved alongside the model. Elastic cannot make
guarantees on the structure or contents of this payload, so Elastic
will treat it like the other input payloads and only allow them during
inference.
#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in #129581 are likely better.
As a first step, rmove the spatial grid functions added in #125143 from release builds so they don't get released into 8.19/9.1.
---------
Co-authored-by: Craig Taverner <craig@amanzi.com>
Add a new logical plan optimization:
When there is a Project (KEEP/DROP/RENAME/renaming EVALs) in a LOOKUP JOIN's left child (the "main" side), perform the Project after the LOOKUP JOIN. This prevents premature field extractions when the lookup join happens on data nodes.
There is an issue where for Flattened fields with synthetic source, if there is a key with a scalar value, and a duplicate key with an object value, one of the values will be left out of the produced synthetic source. This fixes the issue by replacing the object with paths to each of its keys. These paths consist of the concatenation of all keys going down to a given scalar, joined by a period. For example, they are of the form foo.bar.baz. This applies recursively, so that every value within the object, no matter how nested, will be accessible through a full specified path.
* Add recursive chunker
* Update docs/changelog/126866.yaml
* Clean up separator sets and add asMap function for RecrusiveChunkingSettings
* Add javadoc for chunker, add tests, reduce word counting operations
* Remove split merging and add long document unit test
* [CI] Auto commit changes from spotless
* Add markdown chunking tests and reduce substring calls
* Clean up matcher logic
* Add testing for not splitting after valid chunk is found
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This deprecates the `indices.merge.scheduler.use_thread_pool` setting
that was introduced in
https://github.com/elastic/elasticsearch/pull/120869 because this
setting should not normally be used, unless instructed so by engineering
to get around temporary issues with the new threadpool-based merge
scheduler.
It is possible to get all the way down to the knn format reader and
there be no vectors in the index.
This execution path is possible if utilizing nested queries (which
bypasses the higher level checks in
`KnnFloatVectorQuery#approximateSearch`).
bbq_flat should check for the existence of vectors before attempting to
create the scorer.
* Add index_options parameter to semantic_text field mapping
* Cleanup & tests
* Update docs
* Update docs/changelog/119967.yaml
* Addressed some PR feedbak
* Update yaml tests
* Refactoring
* Cleanup
* Fix some tests
* Hack in inferring text_embedding task type from index options
* [CI] Auto commit changes from spotless
* Fix error inferring model settings
* Update docs
* Update tests
* Update docs/reference/mapping/types/semantic-text.asciidoc
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* Address some minor PR feedback
* Remove partial model_settings with inferred task type
* Cleanup
* Remove unnecessary changes
* Fix errors from merge
* [CI] Auto commit changes from spotless
* Cleanup
* Checkpoint, saving changes before merge
* Update parsing
* [CI] Auto commit changes from spotless
* Stash changes
* Fix compile errors
* [CI] Auto commit changes from spotless
* Cleanup error
* fix test
* fix test
* Fix another test
* A bit of cleanup
* Fix tests
* Spotless
* Respect index options if set over defaults
* Cleanup
* [CI] Auto commit changes from spotless
* Support updating to compatible versions, add some cleanup and validation
* Remove test that can't be done here - needs to be unit test
* Add validation
* Cleanup
* Fix some yaml tests
* [CI] Auto commit changes from spotless
* Happy path early index validation works now; edge cases surrounding default BBQ remain
* Always emit index options, even when using defaults
* Minor cleanup
* Fix test compilation failures
* Fix some tests
* Continue to iterate on test failures
* Remove index options from inference field metadata as it is only needed at field creation time
* Fix some tests
* Remove transport version, no longer needed
* Fix yaml tests
* Add tests
* IndexOptions don't need to implement Writeable
* [CI] Auto commit changes from spotless
* Refactor - move SemanticTextIndexOptions
* Remove writeable
* Move index_options parsing to semantic text field mapper
* Cleanup
* Fix test compilation issue
* Cleanup
* Remove whitespace
* Remove writeables from index options
* Disable merging null options?
* Add docs
* [CI] Auto commit changes from spotless
* Revert "Disable merging null options?"
This reverts commit 2ef8b1dc29.
* Remove default serialization
* Include default index option type to defaults
* [CI] Auto commit changes from spotless
* Go back to allowing null updateS
* Cleanup
* Fix validation error
* Revert "Include default index option type to defaults"
This reverts commit b08e2a1d7e.
* Update tests
* Revert "Update tests"
This reverts commit aedfafe0e7.
* Better fix for null inputs
* Remove redundant merge validation
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
Disallow partial quoting: individual quoting of constituent strings in an index pattern
---------
Co-authored-by: Pawan Kartik <pawankartik.chitrapu@elastic.co>
For APM logs, set event.dataset to data_stream.dataset if event.dataset is empty, to satisfy Anomaly Detection's requirement to have event.dataset in every logs-* data stream.
The backing indices of a time series data streams (TSDS) have time ranges (start_time & end_time) and they include documents that belong to these time ranges.
To ensure that we will not unfollow a leader TSDS index before the indexing is complete, we need to add a WaitUntilTimeSeriesEndTimePassesStep to the unfollow action. This will ensure that we will only unfollow after the end_time has passed.
This creates some weird semantics with the combination of the rollover and the unfollow. Because we need the rollover of the leader index to finalise the end_time but the unfollow action is injected before the rollover. However, this should be fine, because the leader index will skip the unfollow action so it will rollover and finalise the end_time and the follower index will wait the end_time to pass before it unfollows. Rolling over the follower index will have no effect since it’s already rolled over.
Adds support for LIKE function alternative syntax with a list of patterns.
Examples:
FROM foo
| WHERE bar LIKE ("A*","B*", "C?")
The new syntax is documented as part of the existing LIKE function documentation. We will use the existing WildcardLike java implementation for existing cases using the old syntax and one list argument case to improve mixed cluster compatibility.
This change introduces a new optional 'state' query parameter for the Get Snapshots API,
allowing users to filter snapshots by state. The parameter accepts comma-separated
values for states: SUCCESS, IN_PROGRESS, FAILED, PARTIAL, INCOMPATIBLE (case-insensitive).
A new 'snapshots.get.state_parameter' NodeFeature has been added with this change.
The new state query parameter will only be supported in clusters where all nodes support
this feature.
---------
Co-authored-by: Elena Stoeva <elenastoeva99@gmail.com>
We add the skip setting to prevent ILM from processing the shrunken
index before the execution state has been copied - which could happen if
the shards of the shrunken index take a long time to allocate.
Resolves#109206
Added min and max allocations as attributes to the telemetry for trained
models with adaptive allocations enabled.
Added telemetry for models with adaptive allocations disabled or never
set.
* This PR addresses #113535 - a confusing error message when the user attempts to update the password for the `elastic` superuser in a cloud deployment.
At the heart of the issue is the difference in how the `elastic` superuser is implemented on self-hosted deployments vs. managed cloud deployments. Elasticsearch has two distinct security realms: `file` and `native`. On a self-hosted deployment, the `elastic` superuser is represented as a document in the `.security` index, whereas in a cloud deployment `elastic` is defined in the `ES_PATH_CONF/users` and `ES_PATH_CONF/user_roles` files placed on each node in the cluster.
The TransportChangePasswordAction impl is designed to update the password for users in the `native` realm specifically, and a failure on cloud to change the password for `elastic` using the Change Password API fails with the error that the user does not exist.
The solution here leverages `fileUserPasswdStore.userExists` to do a low cost check on whether the request username belongs to the `file` realm and will exit early with an informative error message if that is the case.
* Update docs/changelog/127621.yaml
* re-do the ticket in Continuation-passing style. Previous unanswered questions around CI/CD are resolved.
* link issue
* accidental commit, reverting
* Update docs/changelog/127621.yaml
* try to check for membership in all non-native realms
* [CI] Auto commit changes from spotless
* improve checks to fix failing tests
* improve
* improve
* improve logic with GroupedActionListener
* return early
* extra spaces
* revert
* Update x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/user/TransportChangePasswordAction.java
Co-authored-by: Slobodan Adamović <slobodanadamovic@users.noreply.github.com>
* Update x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authc/esnative/ClientReservedRealm.java
Co-authored-by: Slobodan Adamović <slobodanadamovic@users.noreply.github.com>
* PR feedback
* fix imports
* fix test
* add test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Slobodan Adamović <slobodanadamovic@users.noreply.github.com>
During cluster upgrade, the anomaly detection jobs must be reassigned from one ML node to another. During this reassignment, the jobs transition through several states, including "opening" and "opened". If, during this transition, the master node becomes temporarily unavailable, e.g., due to reassignment, the new job state is not successfully committed to the cluster state. Therefore, once the new master became available, the cluster state was inconsistent: some anomaly detection jobs were opened, but their state got stuck as "opening".
This PR introduces a retryable action for updating the job state to ensure that the job state is successfully updated and the cluster state remains consistent during the upgrade.
Fixes#126148
* propgating retrievers to inner retrievers
* test feature taken care of
* Small changes in concurrent multipart upload interfaces (#128977)
Small changes in BlobContainer interface and wrapper.
Relates ES-11815
* Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054)
The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index.
The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers.
The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers.
This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage.
Closes#128541
* [Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
* ESQL: Check for errors while loading blocks (#129016)
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
* Make `PhaseCacheManagementTests` project-aware (#129047)
The functionality in `PhaseCacheManagement` was already project-aware,
but these tests were still using deprecated methods.
* Vector test tools (#128934)
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
* ES|QL: refactor generative tests (#129028)
* Add a test of LOOKUP JOIN against a time series index (#129007)
Add a spec test of `LOOKUP JOIN` against a time series index.
* Make ILM `ClusterStateWaitStep` project-aware (#129042)
This is part of an iterative process to make ILM project-aware.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078
* Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076)
The `ClusterState` parameter of the `asyncPredicate` is not used
anywhere.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098
* Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867
* Threadpool merge executor is aware of available disk space (#127613)
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
* Add option to include or exclude vectors from _source retrieval (#128735)
This PR introduces a new include_vectors option to the _source retrieval context.
When set to false, vectors are excluded from the returned _source.
This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely.
By default, vectors remain included unless explicitly excluded.
* Remove direct minScore propagation to inner retrievers
* cleaned up skip
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149
* Add transport version for ML inference Mistral chat completion (#129033)
* Add transport version for ML inference Mistral chat completion
* Add changelog for Mistral Chat Completion version fix
* Revert "Add changelog for Mistral Chat Completion version fix"
This reverts commit 7a57416bdc.
* Correct index path validation (#129144)
All we care about is if reindex is true or false. We shouldn't worry
about force merge. Because if reindex is true, we will create the
directory, if its false, we won't.
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148
* Implemented completion task for Google VertexAI (#128694)
* Google Vertex AI completion model, response entity and tests
* Fixed GoogleVertexAiServiceTest for Service configuration
* Changelog
* Removed downcasting and using `moveToFirstToken`
* Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses
* Added unit tests
* PR feedback
* Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion
* Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel
* Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel
* Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest
---------
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* ES|QL - kNN function initial support (#127322)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* [Inference API] Add "rerank" task type to "elastic" provider (#126022)
* Rename target destination for microbenchmarks (#128878)
* Include direct memory and non-heap memory in ML memory calculations (take #2) (#128742)
* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Throw better exception for unsupported aggregations over shape fields (#129139)
* Update Test Framework To Handle Query Rewrites That Rely on Non-Null Searchers (#129160)
* Update ReproduceInfoPrinter to correctly print a reproduction line for Lucene & build candidate upgrade tests (#129044)
* Increment inference stats counter for shard bulk inference calls (#129140)
This change updates the inference stats counter to include chunked inference calls performed by the shard bulk inference filter on all semantic text fields.
It ensures that usage of inference on semantic text fields is properly recorded in the stats.
* Synthetic source: avoid storing multi fields of type text and match_only_text by default. (#129126)
Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field.
Without this change, ES would store field otherwise twice in a multi-field configuration.
For example:
```
...
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
...
```
In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field.
This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field.
* Adding `scheduled_report_id` field to kibana reporting template (#127827)
* Adding scheduled_report_id field to kibana reporting template
* Incrementing stack template registry version
* ES|QL: Add FORK generative tests (#129135)
* ES|QL Completion command syntax change (#129189)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* ES|QL Completion command syntax change (#129189)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* ES|QL Completion command syntax change (#129189)
* Add Cluster Feature for L2 Norm (#129181)
* propgating retrievers to inner retrievers
* test feature taken care of
* Small changes in concurrent multipart upload interfaces (#128977)
Small changes in BlobContainer interface and wrapper.
Relates ES-11815
* Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054)
The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index.
The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers.
The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers.
This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage.
Closes#128541
* [Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
* ESQL: Check for errors while loading blocks (#129016)
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
* Make `PhaseCacheManagementTests` project-aware (#129047)
The functionality in `PhaseCacheManagement` was already project-aware,
but these tests were still using deprecated methods.
* Vector test tools (#128934)
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
* ES|QL: refactor generative tests (#129028)
* Add a test of LOOKUP JOIN against a time series index (#129007)
Add a spec test of `LOOKUP JOIN` against a time series index.
* Make ILM `ClusterStateWaitStep` project-aware (#129042)
This is part of an iterative process to make ILM project-aware.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078
* Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076)
The `ClusterState` parameter of the `asyncPredicate` is not used
anywhere.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098
* Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867
* Threadpool merge executor is aware of available disk space (#127613)
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
* Add option to include or exclude vectors from _source retrieval (#128735)
This PR introduces a new include_vectors option to the _source retrieval context.
When set to false, vectors are excluded from the returned _source.
This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely.
By default, vectors remain included unless explicitly excluded.
* Remove direct minScore propagation to inner retrievers
* cleaned up skip
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149
* Add transport version for ML inference Mistral chat completion (#129033)
* Add transport version for ML inference Mistral chat completion
* Add changelog for Mistral Chat Completion version fix
* Revert "Add changelog for Mistral Chat Completion version fix"
This reverts commit 7a57416bdc.
* Correct index path validation (#129144)
All we care about is if reindex is true or false. We shouldn't worry
about force merge. Because if reindex is true, we will create the
directory, if its false, we won't.
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148
* Implemented completion task for Google VertexAI (#128694)
* Google Vertex AI completion model, response entity and tests
* Fixed GoogleVertexAiServiceTest for Service configuration
* Changelog
* Removed downcasting and using `moveToFirstToken`
* Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses
* Added unit tests
* PR feedback
* Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion
* Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel
* Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel
* Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest
---------
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* Added cluster feature to yaml
* Node feature added
* Duplicate line - result of merge removed
* Update docs/changelog/129181.yaml
* Update 129181.yaml
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Luigi Dell'Aquila <luigi.dellaquila@gmail.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Albert Zaharovits <email+github@zalbert.me>
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Leonardo Hoet <55866308+leo-hoet@users.noreply.github.com>
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* Fix DRA dependenciesInfo task dependency resolution (#129209)
* IVF Hierarchical KMeans Flush & Merge (#128675)
added hierarchical kmeans as a clustering algorithm to better partitionin the space when running ivf on flush and merge
* Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {lookup-join.EnrichLookupStatsBug ASYNC} #129228
* Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {lookup-join.EnrichLookupStatsBug SYNC} #129229
* [ES|QL] Specify population in StdDev docs (#129225)
There are 2 types of Standard Deviation: population and
sample, this commit clarifies that the existing is population.
* Unmute IngestGeoIpClientYamlTestSuiteIT (#129178)
* Fix an NPE in the ES|QL completion command. (#129235)
* ESQL: fix bwc test by adding min required version (#129204)
Closes#129093Closes#129094Closes#129095Closes#129102Closes#129103
* ESQL: Fix test by add excluding capability (#129202)
Closes#129078Closes#129082
* Fix vault field name (#129184)
* Remove all usages of Metadata customs removal methods (#129043)
This removes all non-test usage of
```
Metadata.Builder.removeProjectCustom(String)
Metadata.Builder.removeProjectCustomIf(BiPredicate)
```
And replaces it with appropriate calls to the equivalent method on
`ProjectMetadata.Builder`
In most cases this _does not_ make the code project aware, but does
reduce the number of deprecated methods in use.
* Replace tuple with record (#128976)
* improve support for bytecode patching signed jars (#128613)
* improve support for bytecode patching signed jars
* Update docs/changelog/128613.yaml
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
* rename ES|QL sample capability (#129193)
* ESQL: Mute GenerativeForkIT for some LOOKUP JOIN tests (#129248)
* ESQL: Extend `RENAME` syntax to allow a `new = old` syntax (#129212)
This extends RENAME's grammar to allow a new syntax: `| RENAME new_name
= old_name` This is supported along the existing `... old_name AS
new_name` syntax.
Closes#129208
* [DOCS] Adds preview tag to the CHANGE_POINT ES|QL command in the command list. (#129247)
* ESQL: Skip unused STATS groups by adding a Top N BlockHash implementation (#127148)
- Add a new `LongTopNBlockHash` implementation taking care of skipping unused values.
- Add a `TopNUniqueSet` to take care of storing the top N values (without nulls).
- Add a `TopNMultivalueDedupeLong` class helping with it (An adaptation of the existing `MultivalueDedupeLong`).
- Add some tests to `HashAggregationOperator`. It wasn't changed much, but helps a bit with the E2E.
- Add MicroBenchmarks for TopN groupings, to ensure we're actually improving things with this.
* Add "Searchable Snapshots" to changelog validation schema (#129180)
We created a new ":Distributed Indexing/Searchable Snapshots" label recently on Github, so I think it makes sense to also have a "Searchable Snapshots" label in the changelog. It also makes sense since there is automatic changelog generation based on the pull request label.
* ESQL: Fix FieldAttribute name usage in InferNonNullAggConstraint (#128910)
* Fix InferNonNullAggConstraint with union types
* Begin fixing LucenePushdownPredicates with union types
* Introduce a dedicated wrapper record FieldName to be used where field names are really required.
The fixes consist of using FieldAttribute.fieldName() instead of .name() or .field().name(). .name() can be some temporary string unrelated to the actual name of the Lucene index field, whereas .field().name() doesn't know about parent fields; .fieldName() gives the full field name (from the root of the document).
The biggest offender of such misuse is SearchStats; make this always require a FieldName, not a String - and make FieldAttribute#fieldName handily return an instance of FieldName so users of SearchStats don't accidentally use the return value of FieldAttribute#name.
* Remove usages of `Metadata.Builder#indexGraveyard` (#129041)
And replace it with appropriate calls to the equivalent method on
`ProjectMetadata.Builder`.
In most cases this _does not_ make the code project aware, but does
reduce the number of deprecated methods in use.
Concerns both the getter and the setter.
* Mute org.elasticsearch.compute.data.sort.LongTopNSetTests testCrankyBreaker #129257
* Enable Shard-Level Search-load rate metric (#128660)
Introduces a new search load metric to the stats infrastructure, measured and tracked on a per-shard basis. The metric represents the Exponentially Weighted Moving Rate (EWMR) of search operations, calculated using the "took" time from each completed search phase.
* [ESQL] Fix typo in search-functions.md (#129260)
^^
* ESQL: Log partial failures (#129164)
Now that ESQL has `allow_partial_results` we can reply with a `200` even
though some nodes failed to run ESQL. This could happen because the node
is restarting. Or because of a bug. Or a disconnect. All kinds of
things. This logs those partial failures so an operator can look at them
and get a sense of why they are happening.
* Update Gradle wrapper to 8.14.2 (#129179)
* Fix ivf nodestats impl for getOffHeapByteSize (#129259)
This fixes a silly bug where we didn't override `OffHeapStats` for IVF.
* feat: enable date_detection for all apm data streams (#128913)
* feat: enable date_detection for all apm data streams
* Update resources.yaml
* Create 128913.yml
---------
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
* [BC Upgrage] Fix incorrect version parsing in tests (#129243)
This PR introduces several fixes to various IT tests, related to the use and misuse of the version identifier for the start cluster:
wherever we can, we replace of versions in test code with features
where we can't, we make sure we use the actual stack version (the one provided by -Dtests.bwc.main.version and not the bogus "0.0.0" version string)
when requesting the cluster version we make sure we do use the "unresolved" version identifier (the value of the tests.old_cluster_version system property e.g. 0.0.0 ) so we resolve the right distribution
These changes enabled the tests to be used in BC upgrade tests (and potentially in serverless upgrade tests too, where they would have also failed)
Relates to ES-12010
Precedes #128614, #128823 and #128983
* [Build] Build maven aggregation zip as part of DRA build (#129175)
* [Build] Build maven aggregation zip as part of DRA build
* Update path for aggregation zip
* Throttle indexing when disk IO throttling is disabled (#129245)
The threadpool-based merge scheduler triggers indexing throttling if merges are still getting enqueued faster than they're executed, while they are also disk IO unthrottled. This PR fixes the case where indexing throttling was incorrectly NOT triggered when disk IO throttling was disabled via the index settings.
* Register match_phrase as a function not a snapshot function (#129255)
* Register match_phrase as a function not a snapshot function
* Update usage
* [Gradle] Spotless plugin update (#115750)
- provides better configuration cache support
- requires some rework due to changed defaults
* Adding support to exclude semantic_text subfields (#127664)
* Adding support to exclude semantic_text subfields
* Update docs/changelog/127664.yaml
* Updating changelog file
* remove duplicate test from yaml file
* Adding support to exclude semantic_text subfields from mapper builders
* Adding support for generic field types
* refactoring to use builder and setting exclude value from semantic_text mapper
* update in semantic_text mapper and fetcher to incorporate the support functionality
* Fix code style issue
* adding node feature for yaml tests
* Adding more restrictive checks on yaml tests and few refactoring
* Returns metadata fields from metadata mappers
* returns all source fields for fieldcaps
* gather all fields and iterate to process for fieldcaps api
* revert back all changes from MappedFieldtype and subclasses
* revert back exclude logic from semantic_text mapper
* fix lint issues
* fix lint issues
* Adding runtime fields into fieldCaps
* Fix linting issue
* removing unused functions that used in previous implementation
* fix multifield tests failure
* getting alias fields for field caps
* adding support for query time runtime fields
* [CI] Auto commit changes from spotless
* Fix empty mapping fieldCaps call
* Address passthrough behavior for mappers
* Fix SearchAsYoutype mapper failures
* rename abstract method to have more meaningful name
* Rename mapper function to match its functionality
* Adding filtering for infernece subfields
* revert back previous implementation changes
* Adding yaml test for field caps not filtering multi-field
* Fixing yaml test
* Adding comment why .infernece filter is added
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Revert "[Gradle] Spotless plugin update (#115750)"
This reverts commit 6370d600b0.
* Switch IVF Writer to ES Logger (#129224)
update to use ES logger instead of infostream and fixing native access warnings
* Add heap usage estimate to ClusterInfo (#128723)
Co-authored-by: ywangd <yang.wang@elastic.co>
Co-authored-by: rjernst <ryan@elastic.co>
Relates: ES-11445
* Revert "Use IndexOrDocValuesQuery in NumberFieldType#termQuery implementations (#128293)" (#129206)
This reverts commit de7c91c1d9.
* Delegated authorization using Microsoft Graph (SDK) (#128396)
* Delegated authorization using Microsoft Graph (SDK)
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
Co-authored-by: Johannes Fredén <109296772+jfreden@users.noreply.github.com>
* Add `none` chunking strategy to disable automatic chunking for inference endpoints (#129150)
This introduces a `none` chunking strategy that disables automatic chunking when using an inference endpoint.
It enables users to provide pre-chunked input directly to a `semantic_text` field without any additional splitting.
The chunking strategy can be configured either on the inference endpoint or directly in the `semantic_text` field definition.
**Example:**
```json
PUT test-index
{
"mappings": {
"properties": {
"my_semantic_field": {
"type": "semantic_text",
"chunking_settings": {
"strategy": "none" <1>
}
}
}
}
}
```
<1> Disables automatic chunking on `my_semantic_field`.
```json
PUT test-index/_doc/1
{
"my_semantic_field": ["my first chunk", "my second chunk", ...] <1>
...
}
```
<1> Pre-chunked input provided as an array of strings.
Each array element represents a single chunk that will be sent directly to the inference service without further processing.
* Fix broken bwc logic in text field mapper introduced by #129126 (#129308)
A missing condition in the bwc logic caused a text field to be a stored, while before #129126, this wasn't the case.
* [ESQL] Fix SpatialDocValuesExtraction rule replacing TimeSeries agg node (#129273)
`TimeSeriesAggregateExec` (TS) node inherits from `AggregateExec` (STATS). The `SpatialDocValuesExtraction` rule was replacing all `AggregateExec` instances with another `AggregateExec`, whether the same class or not.
* Make `TransportMoveToStepAction` project-aware (#129252)
Future work is necessary to make the YAML tests pass in MP mode.
* [DOCS] Adds term vectors API examples (#129328)
* [DOCS] Adds term vectors API examples.
* Update docs/reference/elasticsearch/rest-apis/term-vectors-examples.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* [DOCS] Addresses feedback.
* [DOCS] Fixes link.
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* [ESQL] Fix TopNSetTestCase test and unmute it (#129327)
Closes https://github.com/elastic/elasticsearch/issues/129257
* ESQL: Change queries ID to be the same as the async (#127472)
This PR changes the list and query API for ESQL, such that the ID now follows the same format as async query IDs. This is saved as part of the task status. For async queries, this is easy, but for sync queries, this is slightly more complicated, since when creating them, we don't have access to a node ID. So instead, the status itself is just the doc ID portion of the async execution ID, which is used for salting, since this part needs to be consistent, so that when we list the queries, we can compute the async execution ID correctly.
Also, I've removed the individual ID, node, and data node tags, as mentioned in the ticket.
In addition, I've changed the accept and content-type to be JSON for lists.
Resolves#127187
* Adjust unpromotable shard refresh request validation to allow RefreshResult.NO_REFRESH (#129176)
When a primary shard uses the read-only engine, it always returns a
RefreshResult.NO_REFRESH for refreshes. Since #93600 we added an extra
roundtrip to hook unpromotable shard refresh logic. This hook is always
executed, even if there are no unpromotable shards, but the
UnpromotableShardRefreshRequest would fail if the primary shard returns
a RefreshResult.NO_REFRESH result.
Fix to be backported to several versions as it's annoying.
Closes#129036
* Add a Multi-Project Search Rest Test (#128657)
This commit adds a Rest IT specifically for search in MultiProject.
Everything was already working as expected, but we were a bit light on
explicit testing for search, which as _the_ core capability of
Elasticsearch is worth testing thoroughly and clearly.
* Modified LinearRetriever to include minScore
* cleaned up
* Made the same changes we did in textSimilarity
* Fixed a minor error
* cleaned up
* Minscore is working :)
* chore: empty commit to trigger PR update
* Update docs/changelog/129359.yaml
* Update 10_linear_retriever.yml
* [CI] Auto commit changes from spotless
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Luigi Dell'Aquila <luigi.dellaquila@gmail.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Albert Zaharovits <email+github@zalbert.me>
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Leonardo Hoet <55866308+leo-hoet@users.noreply.github.com>
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Jan Kuipers <148754765+jan-elastic@users.noreply.github.com>
Co-authored-by: Tim Grein <tim.grein@elastic.co>
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Ignacio Vera <ignacio.vera@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Ying Mao <ying.mao@elastic.co>
Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com>
Co-authored-by: Aurélien FOUCRET <aurelien.foucret@gmail.com>
Co-authored-by: John Wagster <john.wagster@elastic.co>
Co-authored-by: Larisa Motova <larisa.motova@elastic.co>
Co-authored-by: Sam Xiao <sam.xiao@elastic.co>
Co-authored-by: Richard Dennehy <richard.dennehy@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: Iván Cea Fontenla <ivancea96@outlook.com>
Co-authored-by: Dimitris Rempapis <dimitris.rempapis@elastic.co>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: kruskall <99559985+kruskall@users.noreply.github.com>
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
Co-authored-by: Samiul Monir <150824886+Samiul-TheSoccerFan@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Nick Tindall <nick.tindall@elastic.co>
Co-authored-by: ywangd <yang.wang@elastic.co>
Co-authored-by: rjernst <ryan@elastic.co>
Co-authored-by: Johannes Fredén <109296772+jfreden@users.noreply.github.com>
Co-authored-by: Gal Lalouche <gal.lalouche@elastic.co>
Co-authored-by: Tim Vernum <tim@adjective.org>
* propgating retrievers to inner retrievers
* test feature taken care of
* Small changes in concurrent multipart upload interfaces (#128977)
Small changes in BlobContainer interface and wrapper.
Relates ES-11815
* Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054)
The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index.
The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers.
The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers.
This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage.
Closes#128541
* [Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
* ESQL: Check for errors while loading blocks (#129016)
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
* Make `PhaseCacheManagementTests` project-aware (#129047)
The functionality in `PhaseCacheManagement` was already project-aware,
but these tests were still using deprecated methods.
* Vector test tools (#128934)
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
* ES|QL: refactor generative tests (#129028)
* Add a test of LOOKUP JOIN against a time series index (#129007)
Add a spec test of `LOOKUP JOIN` against a time series index.
* Make ILM `ClusterStateWaitStep` project-aware (#129042)
This is part of an iterative process to make ILM project-aware.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078
* Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076)
The `ClusterState` parameter of the `asyncPredicate` is not used
anywhere.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098
* Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867
* Threadpool merge executor is aware of available disk space (#127613)
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
* Add option to include or exclude vectors from _source retrieval (#128735)
This PR introduces a new include_vectors option to the _source retrieval context.
When set to false, vectors are excluded from the returned _source.
This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely.
By default, vectors remain included unless explicitly excluded.
* Remove direct minScore propagation to inner retrievers
* cleaned up skip
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149
* Add transport version for ML inference Mistral chat completion (#129033)
* Add transport version for ML inference Mistral chat completion
* Add changelog for Mistral Chat Completion version fix
* Revert "Add changelog for Mistral Chat Completion version fix"
This reverts commit 7a57416bdc.
* Correct index path validation (#129144)
All we care about is if reindex is true or false. We shouldn't worry
about force merge. Because if reindex is true, we will create the
directory, if its false, we won't.
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148
* Implemented completion task for Google VertexAI (#128694)
* Google Vertex AI completion model, response entity and tests
* Fixed GoogleVertexAiServiceTest for Service configuration
* Changelog
* Removed downcasting and using `moveToFirstToken`
* Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses
* Added unit tests
* PR feedback
* Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion
* Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel
* Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel
* Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest
---------
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* Fixing minscore filtering in the text similarity reranker
* ES|QL - kNN function initial support (#127322)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* [Inference API] Add "rerank" task type to "elastic" provider (#126022)
* Rename target destination for microbenchmarks (#128878)
* Include direct memory and non-heap memory in ML memory calculations (take #2) (#128742)
* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Throw better exception for unsupported aggregations over shape fields (#129139)
* Update Test Framework To Handle Query Rewrites That Rely on Non-Null Searchers (#129160)
* Update ReproduceInfoPrinter to correctly print a reproduction line for Lucene & build candidate upgrade tests (#129044)
* Increment inference stats counter for shard bulk inference calls (#129140)
This change updates the inference stats counter to include chunked inference calls performed by the shard bulk inference filter on all semantic text fields.
It ensures that usage of inference on semantic text fields is properly recorded in the stats.
* Synthetic source: avoid storing multi fields of type text and match_only_text by default. (#129126)
Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field.
Without this change, ES would store field otherwise twice in a multi-field configuration.
For example:
```
...
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
...
```
In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field.
This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field.
* Adding `scheduled_report_id` field to kibana reporting template (#127827)
* Adding scheduled_report_id field to kibana reporting template
* Incrementing stack template registry version
* ES|QL: Add FORK generative tests (#129135)
* ES|QL Completion command syntax change (#129189)
* propagated minscore to rankdsocsretrieverbuilder
* Modified the file to include minscore and the test case to verify it
* Revert "Use IndexOrDocValuesQuery in NumberFieldType#termQuery implementations (#128293)" (#129206)
This reverts commit de7c91c1d9.
* Fixed the rankdocsretriever builder
* Update docs/changelog/129223.yaml
* Update 129223.yaml
* trying to introduce cluster featureS
* included cluster features in the test
* Fixed the merge issue
* [CI] Auto commit changes from spotless
* Removed local variable from RankDocsRetrieverBuilder
* Update RankDocsRetrieverBuilder.java
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Luigi Dell'Aquila <luigi.dellaquila@gmail.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Albert Zaharovits <email+github@zalbert.me>
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Leonardo Hoet <55866308+leo-hoet@users.noreply.github.com>
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Jan Kuipers <148754765+jan-elastic@users.noreply.github.com>
Co-authored-by: Tim Grein <tim.grein@elastic.co>
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Ignacio Vera <ignacio.vera@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Ying Mao <ying.mao@elastic.co>
Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com>
Co-authored-by: Aurélien FOUCRET <aurelien.foucret@gmail.com>
