* [Deprecation API] Adjust details in the SourceFieldMapper deprecation warning (#122041)
In this PR we improve the deprecation warning about configuring source
in the mapping.
- We reduce the size of the warning message so it looks better in kibana.
- We keep the original message in the details.
- We use an alias help url, so we can associate it with the guide when it's created.
* Remove bwc code
This refactoring was motivated by the following issues with the current
state of the code:
- The `TransformDeprecationChecker` is listed as plugin checker, but later we remove is from the `plugin_settings` and add it to the `cluster_settings`. This made me consider that the checker might be dealing with transform deprecation warnings but if they are listed under the `cliuster_settings`, it fits better to be part of `ClusterDeprecationChecker`.
- The `DeprecationInfo` is a data class, but it has a method `from` which constructs an `DeprecationInfo.Response` instance. However, this is not a simple factory class but it actually runs all the checks and it also tries to assert that it is not executed on a transport thread. Considering this, I thought it might fit better to the `TransportDeprecationInfoAction`, this way all the logic is in one place and all the checkers are wired and used in the same class.
- Constructing the node settings deprecation issues requires to merge the deprecation warnings of the individual nodes. We considered bringing together the execution of the remote request and the construction of the response in a new class called `NodeDeprecationChecker` that resembles the patterns of the other Checker classes.
- Reinstated the `PLUGIN_CHECKERS` even if we have only one check, so other developers can easier add their plugin checks.
- Finally, we noticed that the way we synthesise the remote requests is difficult to read and maintain because each call is nested under the previous one. We propose in this PR a different pattern that uses the `RefCountingListener` to combine the different remote calls and store their results in a container class named `PrecomputedData`
- **Bonus**: Removed the `LegacyIndexTemplateDeprecationChecker.java` which was not used.
With the introduction of our new backing algorithm and making rescoring
easier with the `rescore_vector` API, let's mark bbq as GA.
Additionally, this commit adds rolling upgrade tests to ensure
stability.
Enable logsdb by default if logsdb.prior_logs_usage has not been set to true.
Meaning that if no data streams were created matching with the logs-- pattern in 8.x, then logsdb will be enabled by default for data streams matching with logs-*-* pattern.
Also removes LogsPatternUsageService as with version 9.0 and beyond, this component is no longer necessary.
Followup from #120708Closes#106489
This change contains follows ups now the Add Index Block API change is
merged.
The index setting `index.verified_read_only`can now be `PrivateIndex`
and not `Dynamic` anymore. Regular indices in version N-2 can recover if
they have the `index.block.read_only` too. And finally, upgrade tests
can use the Add Index Block API instead of manually flushing and adding
blocks explicitly.
This change requires #120537 for `8.x` (tests will fail until it is
merged).
Relates #119743 Relates #120522
When marking read-only now flush and mark index as verified guaranteeing
that we can upgrade safely to next version with N-1 indices (becoming N-2).
Use this in the deprecation check.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
to fix following test bugs:
* Deal with trail already started error.
* Ensure k8s.pod.name field is mapped as keyword
Closes#115755, #115756, #115757, #115758, #115758
The main difference between other rolling upgrade tests is that these tests index more data while performing the rolling upgrade and no rollover is performed during rolling upgrade. For example this makes it more likely for merging to happen, which could uncover bwc bugs.
Note that currently both test suites start trial license so that synthetic source gets used.
It was deprecated in #104209 (8.13) and shouldn't be set or returned in 9.0
The Desired Nodes API is an internal API, and users shouldn't depend on its backward compatibility.
Multiple @Before methods in junit are run in random order. This commit
cosolidates the @Before methods of ParameterizedRollingUpgradeTestCase
since the code has interdependencies.
closes#114330
This PR exposes operator-defined, cluster-state role mappings in the
[Get role mappings
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-get-role-mapping.html).
Cluster-state role mappings are returned with a reserved suffix
`-read-only-operator-mapping`, to disambiguate with native role mappings
stored in the security index. CS role mappings are also marked with a
`_read_only` metadata flag. It's possible to query a CS role mapping
using its name both with and without the suffix.
CS role mappings can be viewed via the API, but cannot be modified. To
clarify this, the PUT and DELETE role mapping endpoints return header
warnings if native role mappings that name-clash with CS role mappings
are created, modified, or deleted.
The PR also prevents the creation or role mappings with names ending in
`-read-only-operator-mapping` to ensure that CS role mappings and native
role mappings can always be fully disambiguated.
Finally, the PR changes how CS role mappings are persisted in
cluster-state. CS role mappings are written (and read from disk) in the
`XContent` format. This format omits the role mapping's name. This means
that if CS role mappings are ever recovered from disk (e.g., during a
master-node restart), their names are erased. To address this, this PR
changes CS role mapping serialization to persist the name of a mapping
in a reserved metadata field, and recover it from metadata during
serialization. This allows us to persist the name without BWC-breaks in
role mapping `XContent` format. It also allows us to ensure that role
mappings are re-written to cluster state in the new, name-preserving
format the first time operator file settings are processed.
Depends on: https://github.com/elastic/elasticsearch/pull/114295
Relates: ES-9628
Changes `FileSettingsService` to reprocess file settings on every
restart or master node change, even if versions match between file and
cluster-state metadata. If the file version is lower than the metadata
version, processing is still skipped to avoid applying stale settings.
This makes it easier for consumers of file settings to change their
behavior w.r.t. file settings contents. For instance, an update of how
role mappings are stored will automatically apply on the next restart,
without the need to manually increment the file settings version to
force reprocessing.
Relates: ES-9628
Including the cluster state in responses to the `POST _cluster/state`
API was deprecated in #90399 (v8.6.0) requiring callers to pass
`?metric=none` to avoid the deprecation warning. This commit adjusts the
behaviour as promised in v9 so that this API never returns the cluster
state, and deprecates the `?metric` parameter itself.
Closes#88978
In order to better track the work needed to prepare this branch for the
major version upgrade, this commit adds a mandatory `owner` field to all
`UpdateForV9` (and `UpdateForV10`) annotations.
This setting had been removed in the past, it was reintroudced for bw comp with 7.x with #109341. It can now be removed from main as it no longer supports indices created with 7.x
Test LogsDB backward compatibility with a rolling upgrade and full cluster restart.
We try to start indexing logs using a `standard` index, then we switch to a `LogsDB` index.
We also improve the existing test which switches between the two index modes, `standard`
and `logs`.
This adds a new quantization mechanism for HNSW and flat indices. Here
we add `int4` quantization via the `int4_hnsw` and `int4_flat` index
types. This quantization methodology further reduces the memory required
for fast HNSW, meaning that the memory required is 8x smaller than with
regular float32 values.
8x reduction means that 1M 1024 dimension vectors goes from requiring
3.8GB to 477MB.
Recall continues to stay steady, there is some reduction that is
recoverable via slightly oversampling and reranking. For example over
500k CohereV3 vectors, only 5 extra vectors are required to be gathered
to achieve over 0.98 recall in a brute-force scenario.
Currently these tests run against any old cluster older than 8.0.0, but
the fix that allowed `index.mapper.dynamic` to exist is only available
in 7.17.22.
Adjust these tests to only run if old cluster is after version 7.17.21
and before 8.0.0
Currently when upgrading a 7.x cluster to 8.x with
`index.mapper.dynamic` index setting defined the following happens:
- In case of a full cluster restart upgrade, then the index setting gets archived and after the upgrade the cluster is in a green health.
- In case of a rolling cluster restart upgrade, then shards of indices with the index setting fail to allocate as nodes start with 8.x version. The result is that the cluster has a red health and the index setting isn't archived. Closing and opening the index should archive the index setting and allocate the shards.
The change is about ensuring the same behavior happens when upgrading a
cluster from 7.x to 8.x with indices that have the
`index.mapper.dynamic` index setting defined. By re-defining the
`index.mapper.dynamic `index setting with
`IndexSettingDeprecatedInV7AndRemovedInV8` property, the index is
allowed to exist in 7.x indices, but can't be defined in new indices
after the upgrade. This way we don't have to rely on index archiving and
upgrading via full cluster restart or rolling restart will yield the
same outcome.
Based on the test in #109301. Relates to #109160 and #96075
