This PR addresses issues around aggregations cancellation, mentioned in https://github.com/elastic/elasticsearch/issues/108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree.
Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to.
---------
Conflicts:
server/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java
test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* [8.x] ESQL: use field_caps native nested fields filtering (#117201) (#117375) (#121645)
* Just filter the nested fields natively with field_caps support
(cherry picked from commit 73381dbeb1)
* Add import
We need to explicitly add the incubating vector API module to the third
party audit task on Java 24.
(cherry picked from commit 0c667ecd2a)
# Conflicts:
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/precommit/ThirdPartyAuditTask.java
Under very unfortunate conditions tests that check xContent objects
roundtrip parsing (like i.e. SearchHitsTests#testFromXContent)
can fail when we happen to randomly pick YAML xContent type and create
random (realistic)Unicode character sequences that may contain the
character U+0085 (133) from the Latin1 code page. That specific character
doesn't get parsed back to its original form for YAML xContent, which can
lead to rare but hard to diagnose test failures.
This change adds logic to AbstractXContentTestCase#test() which lies at
the core of most of our xContent roundtrip tests that disallows test
instances containing that particular character when using YAML xContent
type.
Closes#97716
Add missing apm-server tail sampling monitoring metrics to stack monitoring mapping. They were missed in #110568.
(cherry picked from commit f3f5135f06)
# Conflicts:
# x-pack/plugin/monitoring/src/main/java/org/elasticsearch/xpack/monitoring/MonitoringTemplateRegistry.java
The upper bound of randomVersionBetween is inclusive; therefore, for
testing the fallback version of the request, we need to use the version
preceding 8.16.0 rather than 8.16.0 itself.
Closes#117937
If the `MasterService` needs to log a create-snapshot task description
then it will call `CreateSnapshotTask#toString`, which today calls
`RepositoryData#toString` which is not overridden so ends up calling
`RepositoryData#hashCode`. This can be extraordinarily expensive in a
large repository. Worse, if there's masses of create-snapshot tasks to
execute then it'll do this repeatedly, because each one only ends up
yielding a short hex string so we don't reach the description length
limit very easily.
With this commit we provide a more efficient implementation of
`CreateSnapshotTask#toString` and also override
`RepositoryData#toString` to protect against some other caller running
into the same issue.
* [Gradle] Make rolling upgrade tests configuration cache compatible (#119577)
With this, all rolling upgrade tests that involve a
`nextNodeToNextVersion` update are gradle configuration cache
compatible.
Simplify API around test cluster registry and cc compatible usage of
test cluster in TestClusterAware tasks.
(cherry picked from commit 7b6bdfa323)
# Conflicts:
# qa/ccs-rolling-upgrade-remote-cluster/build.gradle
# x-pack/plugin/sql/qa/jdbc/security/build.gradle
# x-pack/plugin/sql/qa/server/security/build.gradle
* Fix backport merge issue
Adding `security_solution-*-*` in list of index nae to avoid the pattern collisions.
(cherry picked from commit 0638d3977a)
Co-authored-by: Smriti <152067238+smriti0321@users.noreply.github.com>
* Improve memory aspects of enrich cache (#120256)
This commit reduces the occupied heap space of the enrich cache and
corrects inaccuracies in tracking the occupied heap space (for cache
size limitation purposes).
---------
Co-authored-by: Joe Gallo <joegallo@gmail.com>
* Fix compilation
---------
Co-authored-by: Joe Gallo <joegallo@gmail.com>
* Test ML model server (#120270)
* Fix model downloading for very small models.
* Test MlModelServer
* Tiny ELSER
* unmute TextEmbeddingCrudIT and DefaultEndPointsIT
* update ELSER
* Improve MlModelServer
* tiny E5
* more logging
* improved E5 model
* tiny reranker
* scan for ports
* [CI] Auto commit changes from spotless
* Serve default models when optimized model is requested
* @ClassRule
* polish code
* Respect dynamic setting ML model repo
* fix metadata for optimized models
* improve logging
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* backport HttpHeaderParser
* Fix stripping platform
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
