* Cancel expired async search task when a remote returns its results (#126583)
A while ago we enabled using ccs_minimize_roundtrips in async search.
This makes it possible for users of async search to send a single search
request per remote cluster, and minimize the impact of network latency.
With non minimized roundtrips, we have pretty recurring cancellation checks:
as part of the execution, we detect that a task expired whenever each shard comes
back with its results.
In a scenario where the coord node does not hold data, or only remote data is
targeted by an async search, we have much less chance of detecting cancellation
if roundtrips are minimized. The local coordinator would do nothing other than
waiting for the minimized results from each remote cluster.
One scenario where we can check for cancellation is when each cluster comes
back with its full set of results. This commit adds such check, plus some testing
for async search cancellation with minimized roundtrips.
* compile error
Reindex task may not have completed by time status is called, so remove assertion that relies on this. Following assertions are set in the action which starts task, so will be correct whether or not task has completed.
If metrics that have the same timestamp and dimensions aren't grouped into the same document, ES will consider them to be a duplicate.
The _metric_names_hash field will be set by the OTel ES exporter.
As it's mapped as a time_series_dimensions, it creates a different _tsid for documents with different sets of metrics.
The tradeoff is that if the composition of the metrics grouping changes over time, a different _tsid will be created.
That has an impact on the rate aggregation for counters.
(cherry picked from commit 5e8865deac)
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
* Set `keyUsage` for generated HTTP certificates and self-signed CA (#126376)
The `elasticsearch-certutil http` command, and security auto-configuration,
generate the HTTP certificate and CA without setting the `keyUsage` extension.
This PR fixes this by setting (by default):
- `keyCertSign` and `cRLSign` for self-signed CAs
- `digitalSignature` and `keyEncipherment` for HTTP certificates and CSRs
These defaults can be overridden when running `elasticsearch-certutil http`
command. The user will be prompted to change them as they wish.
For `elasticsearch-certutil ca`, the default value can be overridden by passing
the `--keysage` option, e.g.
```
elasticsearch-certutil ca --keyusage "digitalSignature,keyCertSign,cRLSign" -pem
```
Fixes#117769
(cherry picked from commit 284121ad9f)
# Conflicts:
# docs/reference/elasticsearch/command-line-tools/certutil.md
* fix compilation error
* [CI] Auto commit changes from spotless
* fix failing test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Log stack traces on data nodes before they are cleared for transport (#125732)
We recently cleared stack traces on data nodes before transport back to the coordinating node
when error_trace=false to reduce unnecessary data transfer and memory on the coordinating
node (#118266). However, all logging of exceptions happens on the coordinating node, so stack
traces disappeared from any logs. This change logs stack traces directly on the data node when
error_trace=false.
(cherry picked from commit 9f6eb1d4e3)
* Revert changes to Layout.java
The change in 80125a4bac is a quick fix
and allows breaking an invariant of Layout. Revert that.
* Simplify ReplaceMissingFieldWithNull
When encountering projections, it tries to do the job of field
extraction for missing fields by injecting an Eval that creates a
literal null with the same name id as the field attribute for the
missing field. This is wrong:
1. We only insert an Eval in case that a Project relies on the missing
attribute. There could be other plan nodes that rely on the missing
attribute.
2. Even for Projects, we only insert an Eval in case we squarely project
for the field - in case of aliases (e.g. from RENAME), we do nothing.
3. In case of multiple Projects that use this attribute, we create
multiple attributes with the original field attribute's id, causing
a wrong Layout. This triggered
https://github.com/elastic/elasticsearch/issues/121754.
* Revive logic for EsRelation instead of Project
* Update LocalLogicalPlanOptimizerTests
* Update test expectations
* Do not prune attributes from EsRelation
This can lead to empty output, which leads to the EsRelation being
replaced by a LocalRelation with 0 rows.
* Add tests + capability
* Add comments
* [CI] Auto commit changes from spotless
* Update docs/changelog/125764.yaml
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Add integration tests that verify ST_EXTENT_AGG needs a license (#125584) (#125712)
* Fix java version problem on backport
* Use PLATINUM as license level for spatial features
In #120505 we introduced a capabilities check in the yaml test `deprecation/10_basic/Test Deprecations` but we forgot to add them in the `RestDeprecationInfoAction`.
In this PR we add the capabilities which will enable the test and we make the test resilient to the warning that occurs when the `.security-7` index is present.
Load field caps from store if they haven't been initialised through a refresh yet.
Keep the plain reads to not mess with performance characteristics too much on the good path but protect against confusing races when loading field infos now (that probably should have been ordered stores in the first place but this was safe due to other locks/volatiles on the refresh path).
Closes#125483
Follow up to #125345. If the query contained both a nanos and a millis comparison, we were formatting the dates incorrectly for the lucene push down. This PR adds a test and a fix for that case.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Fixes#125439
We were incorrectly formatting nanosecond dates when building lucene queries. We had missed this in our testing because none of the CSV tests were running against Lucene. This happened because the date nanos test data includes multivalue fields. Our warning behavior for multivalue fields is inconsistent between queries run in Lucene and queries run in pure ES|QL without pushdown. Our warning tests, however, require that the specified warnings be present in all execution paths. When we first built the date nanos CSV tests, we worked around this by always using an MV function to unpack the multivalue fields. But we forgot that using an MV function prevents the entire query from being pushed down to Lucene, and thus that path wasn't being tested.
In this PR, I've duplicated many of the tests to have a version that doesn't use the MV function, and uses warningRegex instead of warning. The regex version does not fail if the header is absent, so it's safe to use in both modes. Rewriting the tests this way revealed several situations in which this bug can manifest, all of which are fixed in this PR. I cannot be confidant that there aren't more paths that can trigger this bug and aren't covered by these tests, but I haven't found any yet.
I've left some trace level logging that I found helpful while debugging this.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Cohere embeddings are expected to be normalized to unit vectors, but due to floating point precision issues,
our check ({@link DenseVectorFieldMapper#isNotUnitVector(float)}) often fails.
This change fixes this bug by setting the default similarity for newly created Cohere inference endpoint to cosine.
Closes#122878
