This removes the method
getDataStreamNames(
IndexNameExpressionResolver,
ClusterState,
String[],
IndicesOptions
)
from DataStreamsActionUtil and replaces all callers with the project
scoped version.
This necessitated the introduction of
`AcknowledgedTransportMasterNodeProjectAction` and some changes to
`TransportMasterNodeReadProjectAction` (and subclasses)
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
As a drive-by, this removes another usage of the trappy default master
node timeout.
Features added before 8.18 can be removed, starting with 9.0. But first they need to be marked as assumed, so existing code knows they could be removed in later builds.
This PR introduces a new syntactical feature to index expression resolution: The selector.
Selectors, denoted with a :: followed by a recognized suffix will allow users to specify which component of
an index abstraction they would like to operate on within an API call. In this case, an index abstraction is a
concrete index, data stream, or alias; Any abstraction that can be resolved to a set of indices/shards. We
define a component of an index abstraction to be some searchable unit of the index abstraction.
This exposes new OTel node and index based metrics for indexing failures due to version conflicts.
In addition, the /_cat/shards, /_cat/indices and /_cat/nodes APIs also expose the same metric, under the newly added column iifvc.
Relates: #107601
As a precursor to converting the reserved state handlers to work on
`ProjectMetadata`, convert `MetadataIndexTemplateService` to work on
`ProjectMetadata`, with temporary adaptor methods.
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
Since 8.0, the default behavior for closing indices has been to wait for the active shards based on the index setting. The special value index-setting allows users to use this in 7.x as well. We'd like to make true our promise in the deprecation warning that this parameter should no longer be used, since it will become unsupported in 9.0.0.
This setting enables or disables the failure store for data streams
based on matching the data stream name against a list of patterns. It
acts as a default, and is overridden if the failure store is
explicitly enabled or disabled either in a component template or using
the data stream options API.
(See the PR for explanations of some of the changes here.)
In this PR we reconciliate the failure indices of a data stream just
like we do for the backing indices. The only difference is that a data
stream can have an empty list of failure indices, while it cannot have
an empty list of backing indices.
An easy way to create a situation where certain backing or failure
indices are not included in a snapshot is via using exclusions in the
multi-target expression of the snapshot. For example:
```
PUT /_snapshot/my_repository/my-snapshot?wait_for_completion=true
{
"indices": "my-ds*", "-.fs-my-ds-000001"
}
```
This PR makes open/close indices and add index blocks work by replaces a
large percentage of usages for the deprecated methods with the
corresponding project-aware versions. The ones that are no longer used
are removed. Some deprecated methods still remain after this PR. They
require even more cascading changes in areas that have not been touched,
such as snapshot/restore, watcher, security index settings etc. I prefer
to leave them out since this PR is already big enough. The main changes
are in MetadataIndexStateService class where both routing table and
metadata need to be specific to the right project.
In this PR we move the failure store configuration from the
`data_stream` in a composable index template to the `template` section
under the `data_stream_options`. The reason for this change is that we
want to be able to compose the failure store configuration via component
templates.
**Previous composable index template**
```
{
"index_patterns": [....],
"template": {
......
},
"data_stream": {
"failure_store": true
}
}
```
**New composable index template**
```
{
"index_patterns": [....],
"template": {
"data_stream_options": {
"failure_store": {
"enabled": true
}
},
......
},
"data_stream": {}
}
```
**Composition logic**
| Component Template A | Composable Index Template | Resolved data
stream options |
|-------------------------|-----------------------------|-------------------------------|
| `{"failure_store": {"enabled": true}}` | - |`{"failure_store":
{"enabled": true}}`| | `{"failure_store": {"enabled": true}}` |
`{"failure_store": {"enabled": false}}` |`{"failure_store": {"enabled":
false}}`| | - | `{"failure_store": {"enabled": true}}`
|`{"failure_store": {"enabled": true}}`| | `{"failure_store": null}` |
`{"failure_store": {"enabled": true}}` |`{"failure_store": {"enabled":
true}}`| | `{"failure_store": {"enabled": true}}` | `{"failure_store":
null}` |`{}`|
More context and discussions can be found in the comments of
https://github.com/elastic/elasticsearch/pull/113863.
We can parallelize starting the clusters and a few other things
to effectively speed up these tests by 2x which comes out to about a minute
of execution time saved for all of those in :server:internalClusterTests
on my workstation.
Here, we only need to extract the minimum and maximum values of the
timestamp field; therefore, using a stats searcher should suffice. This
is important for frozen indices.
Historical features are now trivially true on v9 - so we can remove the features, and the check.
Historical features do not affect cluster state, so this has no compatibility restrictions.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
When a document is rejected because of indexing pressure, it should not be redirected to the failure store.
The failure store is not meant to be a dead letter queue - it’s a best effort storage location for documents that cannot be ingested because there is some kind of fault in their shape or content, this way a user can fix them.
In the case of indexing pressure there is nothing wrong with the document itself. In this PR we fix the redirection to the failure store and we add an integration test to test the interaction of the failure store and incremental bulk's short circuit failure feature.
Closes ES-9577.
Co-authored-by: gmarouli <mary.gouseti@elastic.co>
