* ESQL: Speed up VALUES for many buckets (#123073)
Speeds up the VALUES agg when collecting from many buckets.
Specifically, this speeds up the algorithm used to `finish` the
aggregation. Most specifically, this makes the algorithm more tollerant
to large numbers of groups being collected. The old algorithm was
`O(n^2)` with the number of groups. The new one is `O(n)`
```
(groups)
1 219.683 ± 1.069 -> 223.477 ± 1.990 ms/op
1000 426.323 ± 75.963 -> 463.670 ± 7.275 ms/op
100000 36690.871 ± 4656.350 -> 7800.332 ± 2775.869 ms/op
200000 89422.113 ± 2972.606 -> 21920.288 ± 3427.962 ms/op
400000 timed out at 10 minutes -> 40051.524 ± 2011.706 ms/op
```
The `1` group version was not changed at all. That's just noise in the
measurement. The small bump in the `1000` case is almost certainly worth
it and real. The huge drop in the `100000` case is quite real.
* Fix
* Compile
* Fix failing test(s) in `TimeSeriesDataStreamsIT` (#123378)
When these tests were run around midnight, the use of
`DataStream#getDefaultBackingIndexName` could result in a potential
mismatch in the generated index name and the one that the cluster
actually created. Instead, we need to obtain the backing index and
extract the desired index name from there.
Fixes#123086
Relates #123376
(cherry picked from commit f0f0eeb077)
# Conflicts:
# x-pack/plugin/ilm/qa/multi-node/src/javaRestTest/java/org/elasticsearch/xpack/ilm/TimeSeriesDataStreamsIT.java
* Fix compilation
Rather than checking the license (updating the usage map) on every
single shard, just do it once at the start of a computation that needs
to forecast write loads.
Backport of #123346 to 8.x
Closes#123247
* Bump json-smart and oauth2-oidc-sdk (#122737)
* Bump json-smart and oauth2-oidc-sdk
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
(cherry picked from commit e16664573e)
# Conflicts:
# gradle/verification-metadata.xml
* fixup! Add back verification data for test dep
If metrics that have the same timestamp and dimensions aren't grouped into the same document, ES will consider them to be a duplicate.
The _metric_names_hash field will be set by the OTel ES exporter.
As it's mapped as a time_series_dimensions, it creates a different _tsid for documents with different sets of metrics.
The tradeoff is that if the composition of the metrics grouping changes over time, a different _tsid will be created.
That has an impact on the rate aggregation for counters.
The nebula info broker plugin takes the information for the manifest from the java project settings rather then from
the compile task configuration. Instead of setting the compiler task configuration explicitly we now set the project
configuration accordingly. Also tweaked the javaTestCompile tasks to keep compiling with general minimum runtime version as we did before
(cherry picked from commit 6e6e42f5d4)
The aggs timeout test waits for the agg to return and then double checks
that the agg is stopped using the tasks API. We're seeing some failures
where the tasks API reports that the agg is still running. I can't
reproduce them because computers. This adds two things:
1. Logs the hot_threads so we can see if the query is indeed still
running.
2. Retries the _tasks API for a minute. If it goes away soon after the
_search returns that's *fine*. If it sticks around for more than a
few seconds then the cancel isn't working. We wait for a minute
because CI can't be trusted to do anything quickly.
Closes#121993
This PR addresses issues around aggregations cancellation, mentioned in https://github.com/elastic/elasticsearch/issues/108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree.
Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to.
---------
Conflicts:
server/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java
test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* [8.x] ESQL: use field_caps native nested fields filtering (#117201) (#117375) (#121645)
* Just filter the nested fields natively with field_caps support
(cherry picked from commit 73381dbeb1)
* Add import
Add missing apm-server tail sampling monitoring metrics to stack monitoring mapping. They were missed in #110568.
(cherry picked from commit f3f5135f06)
# Conflicts:
# x-pack/plugin/monitoring/src/main/java/org/elasticsearch/xpack/monitoring/MonitoringTemplateRegistry.java
The upper bound of randomVersionBetween is inclusive; therefore, for
testing the fallback version of the request, we need to use the version
preceding 8.16.0 rather than 8.16.0 itself.
Closes#117937
* [Gradle] Make rolling upgrade tests configuration cache compatible (#119577)
With this, all rolling upgrade tests that involve a
`nextNodeToNextVersion` update are gradle configuration cache
compatible.
Simplify API around test cluster registry and cc compatible usage of
test cluster in TestClusterAware tasks.
(cherry picked from commit 7b6bdfa323)
# Conflicts:
# qa/ccs-rolling-upgrade-remote-cluster/build.gradle
# x-pack/plugin/sql/qa/jdbc/security/build.gradle
# x-pack/plugin/sql/qa/server/security/build.gradle
* Fix backport merge issue
* Improve memory aspects of enrich cache (#120256)
This commit reduces the occupied heap space of the enrich cache and
corrects inaccuracies in tracking the occupied heap space (for cache
size limitation purposes).
---------
Co-authored-by: Joe Gallo <joegallo@gmail.com>
* Fix compilation
---------
Co-authored-by: Joe Gallo <joegallo@gmail.com>
* Test ML model server (#120270)
* Fix model downloading for very small models.
* Test MlModelServer
* Tiny ELSER
* unmute TextEmbeddingCrudIT and DefaultEndPointsIT
* update ELSER
* Improve MlModelServer
* tiny E5
* more logging
* improved E5 model
* tiny reranker
* scan for ports
* [CI] Auto commit changes from spotless
* Serve default models when optimized model is requested
* @ClassRule
* polish code
* Respect dynamic setting ML model repo
* fix metadata for optimized models
* improve logging
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* backport HttpHeaderParser
* Fix stripping platform
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Fixes two bugs in _resolve/cluster.
First, the code that detects older clusters versions and does a fallback to the _resolve/index
endpoint was using an outdated string match for error detection. That has been adjusted.
Second, upon security exceptions, the _resolve/cluster endpoint was marking the clusters as connected: true,
under the assumption that all security exceptions related to cross cluster calls and remote index access were
coming from the remote cluster, but that is not always the case. Some cross-cluster security violations can
be detected on the local querying cluster after issuing the remoteClient.execute call but before the transport
layer actually sends the request remotely. So we now mark the connected status as false for all ElasticsearchSecurityException cases. End user docs have been updated with this information.
This removes a redundant thread creation when triggering a rolling
restart as the method is already async and drops the check for cluster
health as that might hit a node that's being shut down (the master node
in particular).
* Refactor change point detection (#114289)
* Move change detection code to separate class
* Uniformize ChangeDetector and SkipeAndDipDetector
* Separate ChangeDetectorTests and ChangePointAggregatorTests.
* Public entrypoint for change point detection
* Move p-value computation to ChangeDetector
* Move main entrypoint to a separate file
* Fix change point detection for uncertain non-stationary distributions. (#119578)
* Fix change point detection for uncertain non-stationary distributions.
* Replace -1 by ChangeType.NO_CHANGE_POINT
This PR updates `bc-fips` and `bctls-fips` dependencies to the latest
minor versions.
(cherry picked from commit 6ea3e01958)
Co-authored-by: Slobodan Adamović <slobodanadamovic@users.noreply.github.com>
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
(cherry picked from commit ba61f8c7f7)
# Conflicts:
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/distribution/DockerCloudElasticsearchDistributionType.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/distribution/DockerUbiElasticsearchDistributionType.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/Fixture.java
# plugins/repository-hdfs/hadoop-client-api/build.gradle
# server/src/main/java/org/elasticsearch/inference/ChunkingOptions.java
# x-pack/plugin/kql/build.gradle
# x-pack/plugin/migrate/build.gradle
# x-pack/plugin/security/qa/security-basic/build.gradle
This adds infrastructure to make the legacy test cluster plugin and
the legacy test cluster based test plugins generally configuration cache compatible.
(cherry picked from commit 8c20ac5884)
# Conflicts:
# qa/mixed-cluster/build.gradle
* Reapply "[Build] Do not invalidate configuration cache when branch is switched (#118894)" (#119300) (#119325)
* Reapply "[Build] Do not invalidate configuration cache when branch is switched (#118894)" (#119300)
The original PR (#118894) has broken serverless.
* Fix gitinfo plugin for serverless usage
* Update buildscan git revision reference
(cherry picked from commit 5278159987)
# Conflicts:
# build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/PublishPlugin.java
* Fix merge conflict
SmbTestContainer base image upgraded from Ubuntu 16.04 to 24.04 to avoid
hanging Python module compilation when installing samba package.
Installing SMB had to be moved from container building to starting because
SYS_ADMIN capability is required.
(cherry picked from commit a0f64d2c9d)
# Conflicts:
# .buildkite/pipelines/pull-request/packaging-tests-unix.yml
Co-authored-by: Mariusz Józala <377355+jozala@users.noreply.github.com>
Restore a field originally populated by Fleet final ingest pipeline
in all APM documents.
---------
Signed-off-by: inge4pres <francesco.gualazzi@elastic.co>
(cherry picked from commit 78385f3701)
