The DirectEncoder currently returns the incorrect value for the
positionCount() method, which should be the number of positions ready in
the current batch. We need to keep track of whether a position is loaded
via encodeNextBatch() and consumed via the read() method. However, we
can always return 1 for positionCount(), indicating that one position is
already loaded. Our tests failed to catch this because mv_ordering
wasn't enabled when generating test blocks, effectively disabling the
DirectEncoders.
Closes#108268
This PR moves the doPrivileged wrapper closer to the actual deletion
request to ensure the necesary security context is established at all
times. Also added a new repository setting to configure max size for s3
deleteObjects request.
Fixes: #108049
(cherry picked from commit bcf4297e89)
# Conflicts:
# docs/reference/snapshot-restore/repository-s3.asciidoc
# modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3BlobStore.java
This will check and fail if certain functions would generate a result
exceeding a certain fixed byte size.
This prevents an operation/query to fail the entire VM.
Sometimes, CombineProjections does not correctly update an aggregation's groupings when combining with a preceding projection.
Fix this by resolving any aliases used in the groupings and de-duplicating them.
---------
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
(cherry picked from commit adaa4763f3)
# Conflicts:
# x-pack/plugin/esql/qa/testFixtures/src/main/resources/stats.csv-spec
# x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java
this test intends to test the bulkProcessor2's request consumer (see DeprecationIndexingComponent#getBulkProcessor) scheduling requests before the startup is completed (flush is enabled). To verify this behaviour the flush has to happen before the templates are loaded. To test this reliably the flush interval in the test should be as small as possible (not hardcoded 5s as of now)
This commit introduces a setting (not meant to be exposed/documented) to allow for the flush interval to be configured. It also adds additional trace logging to help with troubleshooting.
relates #104716
For the _query endpoint, add a parameter for the ESQL language version to the JSON payload.
For now, it is optional and is only validated with no further action.
A lot of places in the code use a `DataStream` constructor that sets the
`rolloverOnWrite` flag to `false`. For some places, this was
intentional, but for others, this was erroneous (and for most tests, it
didn't matter much).
This PR fixes the erroneous spots and avoids similar unintentional
behavior in the future by removing the constructor in question
altogether. Most use cases just want to copy the flag over and if you
_do_ want to set the flag to false, it makes more sense to do so
explicitly yourself rather than letting the constructor do it for you.
An additional small bonus is that we have one less constructor for the
`DataStream` class :).
Follow up of
[this](https://github.com/elastic/elasticsearch/pull/107035#discussion_r1549299287)
discussion.
Recursive aliases (eval x = 1, x1 = x) were not taken into account when
combining projections causing the target field to be lost (and only the
immediate intermediate named expression to be used instead which became
invalid).
Fix#107083
(cherry picked from commit a9388e16ad)
Fix a bug where PruneColumns would sometimes completely replace an Aggregate, producing the wrong number of rows as a result.
(cherry picked from commit edc9e6787a and updated the test skips)
If as part of the persistent task assignment the source downsample index no longer exists, then the persistent task framework will continuously try to find an assignment and fail with IndexNotFoundException (which gets logged as a warning on elected master node).
This fixes a bug in resolving the shard routing, so that if the index no longer exists any node is returned and the persistent task can fail gracefully at a later stage.
The original fix via #98769 didn't get this part right.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
The Query API Key Information endpoint supports aggs since #104895.
But some lang clients actually use the aggregations term in requests, as the preferred synonym.
This PR adds support for the aggregations request term as a synonym for the existing aggs term.
Backport of #107054
The typed_keys request parameter is the canonical parameter,
that's also used in the regular index _search enpoint, in order to
return the types of aggregations in the response.
This is required by typed language clients of the _security/_query/api_key
endpoint that are using aggregations.
Backport of #106873
All nodes on the mixed cluster need to be at least on version
8.10 since PR #97557 introduced execution of downsampling tasks
using the persisten task framework which is incompatible with
how execution was coordinated before.
Missing a check on the transport version results in unreadable cluster state
if it includes a serialized instance of DownsampleShardTaskParams.
serie indices.
Reading an optional array requires reading a boolean first which is required to
know if an array of values exists in serialized form. From 8.13 on we try to
read such a boolean which is not there because older versions don't write any
boolean nor any string array. Here we include the check on versions for backward
compatibility skipping reading any boolean or array whatsoever whenever not possible.
Customers using downsampling might have cluster states including such serielized
objects and would be unable to upgrade to version 8.13. They will be able to
upgrade to any version including this fix.
This fix has a side effect #106880
Co-authored-by: Salvatore Campagna <93581129+salvatore-campagna@users.noreply.github.com>
When there are no remote or local clusters for a given source index, we
call the listener's `onFailure` method with a `CheckpointException`.
A running transform will fail and retry, eventually moving into an
unhealthy and failed state. Any call to the stats API will note the
checkpoint failure and return.
This fixes a timeout issue calling the Transform stats API and prevents
the Transform from being stuck in indexing.
Fix#106790Fix#104533
Back when we introduced queries against runtime fields, Elasticsearch did not support
inter-segment concurrency yet. At the time, it was fine to assume that segments will be
searched sequentially. AbstractStringScriptFieldAutomatonQuery used to have a BytesRefBuilder
instance shared across the segments, which gets re-initialized when each segment starts its work.
This is no longer possible with inter-segment concurrency.
Closes#105911
We have instances where BWC tests configure old ES version nodes with
the integTest distribution. This isn't a valid configuration, and while
we in reality resolve the default distribution artifact, we have other
configuration logic that behaves differently based on whether the
integTest distro was _requested_. Specifically, what to set ES_JAVA_HOME
to. This bug resulted in us attempting to run old nodes using the
current bundled JDK version, which may be incompatible with that older
version of Elasticsearch.
Closes#104858
If we proceed without waiting for pages, we might cancel the main
request before starting the data-node request. As a result, the exchange
sinks on data-nodes won't be removed until the inactive_timeout elapses,
which is longer than the assertBusy timeout.
Closes#106443
I investigated a heap attack test failure and found that an ESQL request
was stuck. This occurred in the following:
1. The ExchangeSource on the coordinator was blocked on reading because
there were no available pages.
2. Meanwhile, the ExchangeSink on the data node had pages ready for
fetching.
3. When an exchange request tried to fetch pages, it failed due to a
CircuitBreakingException. Despite the failure, no cancellation was
triggered because the status of the ExchangeSource on the coordinator
remained unchanged. To fix this issue, this PR introduces two changes:
Resumes the ExchangeSourceOperator and Driver on the coordinator,
eventually allowing the coordinator to trigger cancellation of the
request when failing to fetch pages.
Ensures that an exchange sink on the data nodes fails when a data node
request is cancelled. This callback was inadvertently omitted when
introducing the node-level reduction in Run empty reduction node level
on data nodes #106204.
I plan to spend some time to harden the exchange and compute service.
Closes#106262
The tests that assert sorting on spatial types causes consistent error messages, also were flaky for the non-error message cases under rare circumstances where the results were returned in different order. We now sort those on a sortable field for deterministic behaviour.
Some index requests target shard IDs specifically, which may not match the indices that the request targets as given by `IndicesRequest#indices()`, which requires a different interception strategy in order to make sure those requests are handled correctly in all cases and that any malformed messages are caught early to aid in troubleshooting.
This PR adds and interface allowing requests to report the shard IDs they target as well as the index names, and adjusts the interception of those requests as appropriate to handle those shard IDs in the cases where they are relevant.
* Fix error on sorting unsortable geo_point and cartesian_point
Without a LIMIT the correct error worked, but with LIMIT it did not. This fix mimics the same error with LIMIT and adds tests for all three scenarios:
* Without limit
* With Limit
* From row with limit
* Update docs/changelog/106351.yaml
* Add tests for geo_shape and cartesian_shape also
* Updated changelog
* Separate point and shape error messages
* Move error to later so we get it only if geo field is actually used in sort.
* Implemented planner check in Verifier instead
This is a much better solution.
* Revert previous solution
* Also check non-field attributes so the same error is provided for ROW
* Changed "can't" to "cannot"
* Add unit tests for verifier error
* Added sort limitations to documentation
* Added unit tests for spatial fields in VerifierTests
* Don't run the new yaml tests on older versions
These tests mostly test the validation errors which were changed only in 8.14.0, so should not be tested in earlier versions.
* Simplify check based on code review, skip duplicate forEachDown
* Add regression tests that test ACS and entity id mismatch, causing
us to go into the initCause branch
* Fix up exception creation: initCause it not
allowed because ElasticsearchException
initialises the cause to `null` already if
it isn't passed as a contructor param.
Signed-off-by: lloydmeta <lloydmeta@gmail.com>
`look_ahead_time` is set to 1 minute, the `assertBusy` loop needs to
wait for longer than that to get a readonly backing index.
Note that this is only relevant when the `UpdateTimeSeriesRangeService`
kicks in to bump the end time of the head index. This is rare (it runs
every 10 minutes) but can happen.
Fixes#101428
