This change adds migration testing for the ".async-search" system index
to the full cluster upgrade tests that perform updates from versions N-2
to N via N-1.
The test creates a system index by using async_search on a cluster with
version N-2, then calls the "_migrate" API in version N-1 and finally
checks that on the upgraded cluster in N we are still able to retrieve
async search results from previous versions and can still write to the
system index. This is necessary to ensure we don't end up with a
write-only async search system index when migrating to version 9.
When adding support for upgrading closed indices in
N-2 version, I randomized the Translog.Durability setting
of the closed index with the aim to test the 2 phases
closing process.
This caused at least 1 test failure on Windows with the
index being closed and the cluster upgraded before the
synchronization of the translog had a chance to be
executed. I think this cause the engine to be reset on
the replica that is promoted as a primary, causing the
loss of the operations that were not yet persisted.
Closes#121257
Allow shard recovery for shards in version N-2 that have been verified before being closed, but not verified as read-only, in 7.x or 8.x. Reopening such closed indices automatically adds an index.blocks.write.
Requires #120595 for the 8.x changes.
Closes ES-10320
Closes#121170Closes#121171
Ensure that a `write` block cannot be removed on a read-only compatible index in version N-2, while allowing to change a `read_only` block into a `write` block if needed as well as closing/reopening such indices.
Requires #120647 to be merged on `8.x`.
Relates ES-10320
This change contains follows ups now the Add Index Block API change is
merged.
The index setting `index.verified_read_only`can now be `PrivateIndex`
and not `Dynamic` anymore. Regular indices in version N-2 can recover if
they have the `index.block.read_only` too. And finally, upgrade tests
can use the Add Index Block API instead of manually flushing and adding
blocks explicitly.
This change requires #120537 for `8.x` (tests will fail until it is
merged).
Relates #119743 Relates #120522
This change allows indices created in version 7.x to be opened
as read-only indices in a 9.0 cluster. It requires the corresponding
change in 8.18 (#120105). To be opened as read-only, indices must
be regular, searchable snapshot or legacy indices created in a N-2
version. They must also have the index.blocks.write setting and
regular indices require the additional index.verified_read_only.
Relates #120105
Relates ES-10320
The tests added in #119468 do not work when executed in the
`release-test` pipeline: at some point a 9.0 non-snapshot node tries to
join the 8.18.0-SNAPSHOT cluster which is impossible due to a missing
`esql.metrics_syntax` feature which is only declared in snapshot builds.
Additionaly, the tests rely on unrelease code in 8.x so they won't work
until 8.18.0 is released.
I don't know how we deal with such issues today, but I'd like to unmute
the tests so that they are executed on CI while being disabled in
`release-test`.
Releates #119550
It's stupid to pick a low value for `index.mapping.total_fields.limit`
when at the same time you also add random runtime fields => pick a value
large enough to avoid failing future random mapping updates.
Closes#119646Closes#119632Closes#119631
This change adds rolling upgrade BWC tests for searchable
snapshots in N-2 version, so that they are tested in a mixed
versions cluster.
The tests do not use replicas because it may or may not be
assigned depending on which node the primary is assigned
on (ie, if the cluster has 1 new node and 2 old node and the
primary shard is assigned to the new node, then a replica
cannot be assigned until at least another node is upgraded).
It also renames the existing full cluster restart BWC tests and
mutualizes some common code in AbstractIndexCompatibilityTestCase.
Relates ES-10432
This change (along with the required change #118923 for 8.18)
relaxes the index compatibility version checks to allow archive
and searchable snapshot indices in version N-2 to exist on a 9.x
cluster.
Relates ES-10274
This change introduces a new QA project to test Lucene
support for reading indices created in version N-2.
The test suite is inspired from the various full-cluster
restart suites we already have. It creates a cluster in
version N-2 (today 7.17.25), then upgrades the cluster
to N-1 (today 8.18.0) and finally upgrades the cluster
to the current version (today 9.0), allowing to execute
test methods after every upgrade.
The test suite has two variants: one for searchable
snapshots and one for snapshot restore. The suites
demonstrates that Elasticsearch does not allow
reading indices written in version N-2 but we hope
to make this feasible. Also, the tests can be used for
investigation and debug with the command
`./gradlew ":qa:lucene-index-compatibility:check" --debug-jvm-server`
Relates ES-10274
