The RunningSnapshotIT upgrade test adds shutdown markers to all nodes
and removes them once all nodes are upgraded. If an index gets created
in a mixed cluster, for example by ILM or deprecation messages, the
index cannot be allocated because all nodes are shutting down. Since the
cluster ready check between node upgrades expects a yellow cluster, the
unassigned index prevents the ready check to succeed and eventually
timeout. This PR fixes it by removing shutdown marker for the 1st
upgrade node to allow it hosting new indices.
Resolves: #129644Resolves: #129645Resolves: #129646
This PR introduces several fixes to various IT tests, related to the use and misuse of the version identifier for the start cluster:
wherever we can, we replace of versions in test code with features
where we can't, we make sure we use the actual stack version (the one provided by -Dtests.bwc.main.version and not the bogus "0.0.0" version string)
when requesting the cluster version we make sure we do use the "unresolved" version identifier (the value of the tests.old_cluster_version system property e.g. 0.0.0 ) so we resolve the right distribution
These changes enabled the tests to be used in BC upgrade tests (and potentially in serverless upgrade tests too, where they would have also failed)
Relates to ES-12010
Precedes #128614, #128823 and #128983
This change enables security in a number of tsdb and logsdb integration tests. A number of java/yaml rest tests in logsdb module, additionally logsdb and tsdb rolling upgrade tests.
A recent bug (#128050) wouldn't have happened if logsdb rolling upgrade tests ran with security enabled.
Before this PR sorting on integer, short and byte fields types used
SortField.Type.LONG. This made sort optimization impossible for these
field types.
This PR uses SortField.Type.INT for integer, short and byte fields. This
enables sort optimization.
There are several caveats with changing sort type that are addressed: -
Before mixed sort on integer and long fields was automatically
supported, as both field types used SortField.TYPE.LONG. Now when
merging results from different shards, we need to convert sort to LONG
and results to long values. - Similar for collapsing when there is mixed
INT and LONG sort types. - Index sorting. Similarly, before for index
sorting on integer field, SortField.Type.LONG was used. This sort type
is stored in the index writer config on disk and can't be modified. Now
when providing sortField() for index sorting, we need to account for
index version: for older indices return sort with SortField.Type.LONG
and for new indices return SortField.Type.INT.
---
There is only 1 change that may be considered not backwards compatible:
Before if an integer field was [missing a
value](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/sort-search-results#_missing_values)
, it sort values will return Long.MAX_VALUE in a search response. With
this integer, it sort valeu will return Integer.MAX_VALUE. But I think
this change is ok, as in our documentation, we don't provide information
what value will be returned, we just say it will be sorted last.
---
Also closes#127965 (as same type validation in added for collapse
queries)
* [main] Move system indices migration to migrate plugin
It seems the best way to fix#122949 is to use existing data stream reindex API. However, this API is located in the migrate x-pack plugin. This commit moves the system indices migration logic (REST handlers, transport actions, and task) to the migrate plugin.
Port of #123551
* [CI] Auto commit changes from spotless
* Fix compilation
* Fix tests
* Fix test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This PR adds project-id to both SnapshotsInProgress and Snapshot so that
they are aware of projects and ready to handle snapshots from multiple
projects.
Relates: ES-10224
Remove `-da:org.elasticsearch.index.mapper.DocumentMapper` and `-da:org.elasticsearch.index.mapper.MapperService` from mixed cluster/bwc cluster setups. Given that #122606 is now backported to the 8.18 branch.
* Use min node version to guard injecting settings in logs provider
* Update docs/changelog/123005.yaml
* no random in cluster init
* Cover basic license in logsdb rolling upgrade tests
I have ran this many times locally, and it never failed. Maybe there is
something "magical" in CI.
Added some additional info in the assertion logging.
In this PR we improve the deprecation warning about configuring source
in the mapping.
- We reduce the size of the warning message so it looks better in kibana.
- We keep the original message in the details.
- We use an alias help url, so we can associate it with the guide when it's created.
This refactoring was motivated by the following issues with the current
state of the code:
- The `TransformDeprecationChecker` is listed as plugin checker, but later we remove is from the `plugin_settings` and add it to the `cluster_settings`. This made me consider that the checker might be dealing with transform deprecation warnings but if they are listed under the `cliuster_settings`, it fits better to be part of `ClusterDeprecationChecker`.
- The `DeprecationInfo` is a data class, but it has a method `from` which constructs an `DeprecationInfo.Response` instance. However, this is not a simple factory class but it actually runs all the checks and it also tries to assert that it is not executed on a transport thread. Considering this, I thought it might fit better to the `TransportDeprecationInfoAction`, this way all the logic is in one place and all the checkers are wired and used in the same class.
- Constructing the node settings deprecation issues requires to merge the deprecation warnings of the individual nodes. We considered bringing together the execution of the remote request and the construction of the response in a new class called `NodeDeprecationChecker` that resembles the patterns of the other Checker classes.
- Reinstated the `PLUGIN_CHECKERS` even if we have only one check, so other developers can easier add their plugin checks.
- Finally, we noticed that the way we synthesise the remote requests is difficult to read and maintain because each call is nested under the previous one. We propose in this PR a different pattern that uses the `RefCountingListener` to combine the different remote calls and store their results in a container class named `PrecomputedData`
- **Bonus**: Removed the `LegacyIndexTemplateDeprecationChecker.java` which was not used.
With the introduction of our new backing algorithm and making rescoring
easier with the `rescore_vector` API, let's mark bbq as GA.
Additionally, this commit adds rolling upgrade tests to ensure
stability.
Enable logsdb by default if logsdb.prior_logs_usage has not been set to true.
Meaning that if no data streams were created matching with the logs-- pattern in 8.x, then logsdb will be enabled by default for data streams matching with logs-*-* pattern.
Also removes LogsPatternUsageService as with version 9.0 and beyond, this component is no longer necessary.
Followup from #120708Closes#106489
This change contains follows ups now the Add Index Block API change is
merged.
The index setting `index.verified_read_only`can now be `PrivateIndex`
and not `Dynamic` anymore. Regular indices in version N-2 can recover if
they have the `index.block.read_only` too. And finally, upgrade tests
can use the Add Index Block API instead of manually flushing and adding
blocks explicitly.
This change requires #120537 for `8.x` (tests will fail until it is
merged).
Relates #119743 Relates #120522
When marking read-only now flush and mark index as verified guaranteeing
that we can upgrade safely to next version with N-1 indices (becoming N-2).
Use this in the deprecation check.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
to fix following test bugs:
* Deal with trail already started error.
* Ensure k8s.pod.name field is mapped as keyword
Closes#115755, #115756, #115757, #115758, #115758
The main difference between other rolling upgrade tests is that these tests index more data while performing the rolling upgrade and no rollover is performed during rolling upgrade. For example this makes it more likely for merging to happen, which could uncover bwc bugs.
Note that currently both test suites start trial license so that synthetic source gets used.
It was deprecated in #104209 (8.13) and shouldn't be set or returned in 9.0
The Desired Nodes API is an internal API, and users shouldn't depend on its backward compatibility.
Multiple @Before methods in junit are run in random order. This commit
cosolidates the @Before methods of ParameterizedRollingUpgradeTestCase
since the code has interdependencies.
closes#114330
This PR exposes operator-defined, cluster-state role mappings in the
[Get role mappings
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-get-role-mapping.html).
Cluster-state role mappings are returned with a reserved suffix
`-read-only-operator-mapping`, to disambiguate with native role mappings
stored in the security index. CS role mappings are also marked with a
`_read_only` metadata flag. It's possible to query a CS role mapping
using its name both with and without the suffix.
CS role mappings can be viewed via the API, but cannot be modified. To
clarify this, the PUT and DELETE role mapping endpoints return header
warnings if native role mappings that name-clash with CS role mappings
are created, modified, or deleted.
The PR also prevents the creation or role mappings with names ending in
`-read-only-operator-mapping` to ensure that CS role mappings and native
role mappings can always be fully disambiguated.
Finally, the PR changes how CS role mappings are persisted in
cluster-state. CS role mappings are written (and read from disk) in the
`XContent` format. This format omits the role mapping's name. This means
that if CS role mappings are ever recovered from disk (e.g., during a
master-node restart), their names are erased. To address this, this PR
changes CS role mapping serialization to persist the name of a mapping
in a reserved metadata field, and recover it from metadata during
serialization. This allows us to persist the name without BWC-breaks in
role mapping `XContent` format. It also allows us to ensure that role
mappings are re-written to cluster state in the new, name-preserving
format the first time operator file settings are processed.
Depends on: https://github.com/elastic/elasticsearch/pull/114295
Relates: ES-9628
