Remove `-da:org.elasticsearch.index.mapper.DocumentMapper` and `-da:org.elasticsearch.index.mapper.MapperService` from mixed cluster/bwc cluster setups. Given that #122606 is now backported to the 8.18 branch.
This commit adds compatibility tests that target ES revisions that align with specific Lucene versions. In this case, we are intending to upgrade from Lucene 10.0 to 10.1. Since no on-prem Elasticsearch release exists with 10.0, we need another method to ensure compatibility with Lucene 10.0 indicies.
The work here is a bit hacky since all our compatibility testing infrastructure is centered around versions and we're now effectively doing compatibility tests between two different revisions of Elasticsearch that both report the same version. Ideally this specific testing would be replaced by unit tests, rather that reusing our full cluster restart tests for this purpose.
We'll also want to bump the commit referenced in the CI pipelines here to align with the last commit using Lucene 10.0.
Enable logsdb by default if logsdb.prior_logs_usage has not been set to true.
Meaning that if no data streams were created matching with the logs-- pattern in 8.x, then logsdb will be enabled by default for data streams matching with logs-*-* pattern.
Also removes LogsPatternUsageService as with version 9.0 and beyond, this component is no longer necessary.
Followup from #120708Closes#106489
Follow on from #116929. It is valid to do a full restart from any v8 cluster into a v9 cluster, so we need to maintain test conditions for those versions. Rather than historical versions, we can now use version features instead.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
As title says, the archived setting needs to be moved out to not
interfere with other test methods in the same test class.
Resolves: #111798Resolves: #111777Resolves: #111774Resolves: #111799
Support for replicating closed indices was added in #39506 (7.1.0), we can expect the the cluster always supports replication of closed indices in 8.0/9.0
The most relevant ES changes that upgrading to Lucene 10 requires are:
- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area
Co-authored-by: Christoph Büscher <christophbuescher@posteo.de>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com>
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
In order to better track the work needed to prepare this branch for the
major version upgrade, this commit adds a mandatory `owner` field to all
`UpdateForV9` (and `UpdateForV10`) annotations.
The only usage of `MappedFieldType#extractTerm` comes from `SpanTermQueryBuilder`
which attempts to extract a single term from a generic Query obtained from calling
`MappedFieldType#termQuery`. We can move this logic directly within its only caller,
and instead of using instanceof checks, we can rely on the query visitor API.
This additionally allows us to remove one of the leftover usages of TermInSetQuery#getTermData
which is deprecated in Lucene
This setting had been removed in the past, it was reintroudced for bw comp with 7.x with #109341. It can now be removed from main as it no longer supports indices created with 7.x
Test LogsDB backward compatibility with a rolling upgrade and full cluster restart.
We try to start indexing logs using a `standard` index, then we switch to a `LogsDB` index.
We also improve the existing test which switches between the two index modes, `standard`
and `logs`.
Currently these tests run against any old cluster older than 8.0.0, but
the fix that allowed `index.mapper.dynamic` to exist is only available
in 7.17.22.
Adjust these tests to only run if old cluster is after version 7.17.21
and before 8.0.0
Currently when upgrading a 7.x cluster to 8.x with
`index.mapper.dynamic` index setting defined the following happens:
- In case of a full cluster restart upgrade, then the index setting gets archived and after the upgrade the cluster is in a green health.
- In case of a rolling cluster restart upgrade, then shards of indices with the index setting fail to allocate as nodes start with 8.x version. The result is that the cluster has a red health and the index setting isn't archived. Closing and opening the index should archive the index setting and allocate the shards.
The change is about ensuring the same behavior happens when upgrading a
cluster from 7.x to 8.x with indices that have the
`index.mapper.dynamic` index setting defined. By re-defining the
`index.mapper.dynamic `index setting with
`IndexSettingDeprecatedInV7AndRemovedInV8` property, the index is
allowed to exist in 7.x indices, but can't be defined in new indices
after the upgrade. This way we don't have to rely on index archiving and
upgrading via full cluster restart or rolling restart will yield the
same outcome.
Based on the test in #109301. Relates to #109160 and #96075
To simplify the migration away from version based skip checks in YAML specs,
this PR adds a synthetic version feature `gte_vX.Y.Z` for any version at or before 8.14.0.
New test specs for 8.14 or later are expected to use respective new cluster features,
or a test-only feature supplied via ESRestTestCase#createAdditionalFeatureSpecifications
if sufficient.
Tests that send REST requests with bodies must today build up a separate
`String` containing the body contents as JSON. This is kinda ugly, and
also means we do not cover the other supported body formats in these
tests. This commit introduces a utility to allow construction of REST
requests with `XContent` bodies directly, and generalizes things to
choose randomly between JSON and other supported body formats.
* Introducing TestFeatureService to ESRestTestCase
- Added RestTestLegacyFeatures to encompass legacy (historical) features that have been removed from production code but are still needed by REST tests
- Encapsulated Mark's getHistoricalFeatures method inside a FeatureProvider (ESRestTestCaseHistoricalFeatures)
- ESRestTestCaseHistoricalFeatures is not yet used, as we need to figure out how to deal with old cluster tests
Checks for soft delete support (and deprecation of non-soft delete), peer recovery retention leases and replication of closed indices are using IndexVersion in production code. Moving these checks from Version to IndexVersion in rest tests too.
When the test was trying to test recovering translog ops,
since we flush on close/shutdown, it failed because it never
recovered any translog ops.
The code for translog recovery is irrelevant due to that and
this PR proposes to remove it.
Alternatively, we could simulate killing nodes forcibly before
upgrading, but (a) that seems out of the ordinary for upgrades,
and (b) in trying that, it did not consistently pass the test
because sometimes the flush on close still happened.
Fixes#52031
This is one of the few uses of Version#minimumCompatibilityVersion that isn't covered by other planned work. Since the minimum compatibility version for 7.14.0 is always going to be 6.8.0, we can just hardcode the value.
Re-applies the changes from #99572 to move some bwc tests to a junit-based build infrastructure. Some tests that did not handle the move well have been kept in rolling-upgrade-legacy using the old gradle-based infrastructure
