* Fix privileges for system index migration WRITE block (#121327)
This PR removes a potential cause of data loss when migrating system indices. It does this by changing the way we set a "write-block" on the system index to migrate - now using a dedicated transport request rather than a settings update. Furthermore, we no longer delete the write-block prior to deleting the index, as this was another source of potential data loss. Additionally, we now remove the block if the migration fails.
* Update release notes
* Delete docs/changelog/122214.yaml
This PR addresses issues around aggregations cancellation, mentioned in https://github.com/elastic/elasticsearch/issues/108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree.
Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
The node environment has many paths. The accessors for these currently
use a "file" suffix, but they are always directories. This commit
renames the accessors to make it clear these paths are directories.
If a custom analyzer provided in _analyze API can not be built, return
400 instead of the current 500. This most probably means that the user's
provided analyzer specifications are wrong.
Closes#121443
This commit adds compatibility tests that target ES revisions that align with specific Lucene versions. In this case, we are intending to upgrade from Lucene 10.0 to 10.1. Since no on-prem Elasticsearch release exists with 10.0, we need another method to ensure compatibility with Lucene 10.0 indicies.
The work here is a bit hacky since all our compatibility testing infrastructure is centered around versions and we're now effectively doing compatibility tests between two different revisions of Elasticsearch that both report the same version. Ideally this specific testing would be replaced by unit tests, rather that reusing our full cluster restart tests for this purpose.
We'll also want to bump the commit referenced in the CI pipelines here to align with the last commit using Lucene 10.0.
This PR removes a potential cause of data loss when migrating system indices. It does this by changing the way we set a "write-block" on the system index to migrate - now using a dedicated transport request rather than a settings update. Furthermore, we no longer delete the write-block prior to deleting the index, as this was another source of potential data loss. Additionally, we now remove the block if the migration fails.
The message_id field may be unmapped if documents were indexed into some
indices but not all. This change specifies the unmapped type for
message_id, allowing it to be sorted in such cases.
Closes#120796
Making the `es.queryable_built_in_roles_enabled` feature flag enabled by default.
This feature makes the built-in roles automatically indexed in `.security` index and available
for querying via Query Role API. The consequence of this is that `.security` index is now
created eagerly (if it's not existing) on cluster formation.
In order to keep the scope of this PR small, the feature is disabled for some of the tests,
because they are either non-trivial to adjust or the gain is not worthy the effort to do it now.
The tests will be adjusted in a follow-up PR and later the flag will be removed completely.
Relates to #117581
This call has the side effect that if you are iterating a number of hits calling this method, you will be increasing the
memory usage by a non trivial number which in most of cases is unwanted. Therefore this commit removes this caching
all together and add an assertion so the method is call once during the lifetime of the object.
In this PR we explore how we can include the failure store indices while
disallowing the user to use selectors in the `indices` field of the
create and restore snapshot requests.
**Security concerns**
The create and restore snapshot require cluster privileges only, so we
do not have to worry about how to handle index level security (see
[Create Snapshot
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/create-snapshot-api.html#create-snapshot-api-prereqs)
& [Restore Snapshot
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html#restore-snapshot-api-prereqs)).
**Challenges**
While there are other APIs that do not support selectors but they do
include the failure indices in their response such as `GET
_data_stream/<target>`, `GET _data_stream/<target>/stats`, etc; the
snapshot APIs are a little bit different. The reason for that is that
the data stream APIs first collect only the relevant data streams and
then they select the backing indices and/or the failure indices. On the
other hand, the snapshot API that works with both indices and data
streams cannot take such a shortcut and needs to use the
`concreteIndices` method.
We propose, to add a flag that when the selectors are not "allowed" can
determine if we need to include the failure store or not. In the past we
had something similar called the default selector, but it was more
flexible and it was used a fallback always.
Our goal now is to only specify the behaviour we want when the selectors
are not supported. This new flag allowed also to simplify the concrete
index resolution in `GET _data_stream/<target>/stats`
Relates to https://github.com/elastic/elasticsearch/issues/119545
Re-enables system index migration tests in main. Previously, these tests were only running on the last minor version, leaving the system indices migration infrastructure largely untested for most of the time.
Instrument write access to System properties by means of the `WriteSystemPropertiesEntitlement`.
`System.setProperties(Properties)` is always denied.
Part of #ES-10359
* Exhaustive testParseFractionalNumber
* Refactor: encapsulate ByteSizeUnit constructor
* Refactor: store size in bytes
* Support up to 2 decimals in parsed ByteSizeValue
* Fix test for rounding up with no warnings
* ByteSizeUnit transport changes
* Update docs/changelog/120142.yaml
* Changelog details and impact
* Fix change log breaking.area
* Address PR comments
Support for `ST_EXTENT_AGG` was added in https://github.com/elastic/elasticsearch/pull/118829, and then partially optimized in https://github.com/elastic/elasticsearch/pull/118829. This optimization worked only for cartesian_shape fields, and worked by extracting the Extent from the doc-values and re-encoding it as a WKB `BBOX` geometry. This does not work for geo_shape, where we need to retain all 6 integers stored in the doc-values, in order to perform the datelline choice only at reduce time during the final phase of the aggregation.
Since both geo_shape and cartesian_shape perform the aggregations using integers, and the original Extent values in the doc-values are integers, this PR expands the previous optimization by:
* Saving all Extent values into a multi-valued field in an IntBlock for both cartesian_shape and geo_shape
* Simplifying the logic around merging intermediate states for all cases (geo/cartesian and grouped and non-grouped aggs)
* Widening test cases for testing more combinations of aggregations and types, and fixing a few bugs found
* Enhancing cartesian extent to convert from 6 ints to 4 ints at block loading time (for efficiency)
* Fixing bugs in both cartesian and geo extents for generating intermediate state with missing groups (flaky tests in serverless)
* Moved the int order to always match Rectangle for 4-int and Extent for 6-int cases (improved internal consistency)
Since the PR already changed the meaning of the invalid/infinite values of the intermediate state integers, it was already not compatible with the previous cluster versions. We disabled mixed-cluster testing to prevent errors as a result of that. This leaves us the opportunity to make further changes that are mixed-cluster incompatible, hence the decision to perform this consistency update now.
This test will fail to see the rejection if the GET retries on a
different shard copy, so with this commit we make sure there are no
spare shard copies on which to retry.
Closes#113939
This PR upgrades the minio docker image from
RELEASE.2021-03-01T04-20-55Z which is 3+ years old to the latest
RELEASE.2024-12-18T13-15-44Z.
Relates: #118548
