This commit allows the cluster health API to function in a multi-project
setting. Note: the API is only able to operate on a single project per
request. In the future, we might want to consider changing the API to
allow for multi-project responses, but that's a discussion for later.
This is part of an ongoing effort to get the cleanup part of integration
tests to work in a multi-project setting.
**Introduction**
> In order to make adoption of failure stores simpler for all users, we
are introducing a new syntactical feature to index expression
resolution: The selector. > > Selectors, denoted with a :: followed by a
recognized suffix will allow users to specify which component of an
index abstraction they would like to operate on within an API call. In
this case, an index abstraction is a concrete index, data stream, or
alias; Any abstraction that can be resolved to a set of indices/shards.
We define a component of an index abstraction to be some searchable unit
of the index abstraction. > > To start, we will support two components:
data and failures. Concrete indices are their own data components, while
the data component for index aliases are all of the indices contained
therein. For data streams, the data component corresponds to their
backing indices. Data stream aliases mirror this, treating all backing
indices of the data streams they correspond to as their data component.
> > The failure component is only supported by data streams and data
stream aliases. The failure component of these abstractions refer to the
data streams' failure stores. Indices and index aliases do not have a
failure component.
For more details and examples see
https://github.com/elastic/elasticsearch/pull/113144. All this work has
been cherry picked from there.
**Purpose of this PR**
This PR is introducing the `::*` as another selector option and not as a
combination of `::data` and `::failure`. The reason for this change is
that we need to differentiate between:
- `my-index::*` which should resolve to `my-index::data` only and not to `my-index::failures` and
- a user explicitly requesting `my-index::data, my-index::failures` which should result potentially to an error.
In this PR we add a test and we fix the issues we encountered when we
enabled the failure store for TSDS and logsdb.
**Logsdb** Logsdb worked out of the box, so we just added the test that
indexes with a bulk request a couple of documents and tests how they are
ingested.
**TSDS** Here it was a bit trickier. We encountered the following
issues:
- TSDS requires a timestamp to determine the write index of the data stream meaning the failure happens earlier than we have anticipated so far. We added a special exception to detect this case and we treat it accordingly.
- The template of a TSDS data stream sets certain settings that we do not want to have in the failure store index. We added an allowlist that gets applied before we add the necessary index settings.
Furthermore, we added a test case to capture this.
The most relevant ES changes that upgrading to Lucene 10 requires are:
- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area
Co-authored-by: Christoph Büscher <christophbuescher@posteo.de>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com>
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
**Introduction**
> In order to make adoption of failure stores simpler for all users, we
are introducing a new syntactical feature to index expression
resolution: The selector. > > Selectors, denoted with a :: followed by a
recognized suffix will allow users to specify which component of an
index abstraction they would like to operate on within an API call. In
this case, an index abstraction is a concrete index, data stream, or
alias; Any abstraction that can be resolved to a set of indices/shards.
We define a component of an index abstraction to be some searchable unit
of the index abstraction. > > To start, we will support two components:
data and failures. Concrete indices are their own data components, while
the data component for index aliases are all of the indices contained
therein. For data streams, the data component corresponds to their
backing indices. Data stream aliases mirror this, treating all backing
indices of the data streams they correspond to as their data component.
> > The failure component is only supported by data streams and data
stream aliases. The failure component of these abstractions refer to the
data streams' failure stores. Indices and index aliases do not have a
failure component.
For more details and examples see
https://github.com/elastic/elasticsearch/pull/113144. All this work has
been cherry picked from there.
**Purpose of this PR**
This PR is replacing the `FailureStoreOptions` with the
`SelectorOptions`, there shouldn't be any perceivable change to the user
since we kept the query parameter "failure_store" for now. It will be
removed in the next PR which will introduce the parsing of the
expressions.
_The current PR is just a refactoring and does not and should not change
any existing behaviour._
* Add data stream template validation
to snapshot restore
* Add data stream template validation
to data stream promotion endpoint
* Add new assertion for response headers
Add a new assertion to synchronously execute a request and check the
response contains a specific warning header
* Test for warning header on snapshot restore
When missing templates
* Test for promotion warnings
* Add documentation for the potential error states
* PR changes
* Spotless reformatting
* Add logic to look in snapshot global metadata
This checks if the snapshot contains a matching template for the DS
* Comment on test cleanup to explain it was copied
* Removed cluster service field
With logsdb another index mode is available, the isTimeSeries parameter is limiting. Instead, we should just push down the index mode from template to index settings provider.
Follow up from #113451
Relates to #113583
In this PR we introduce two endpoint PUT and GET to manage the data
stream options and consequently the failure store configuration on the
data stream level. This means that we can manage the failure store of
existing data streams.
The APIs look like:
```
# Enable/disable
PUT _data_stream/my-data-stream/_options
{
"failure_store": {
"enabled": true
}
}
# Remove existing configuration
DELETE _data_stream/my-data-stream/_options
# Retrieve
GET _data_stream/my-data-stream/_options
{
"failure_store": {
"enabled": true
}
}
```
Future work:
- Document the new APIs
- Convert `DataStreamOptionsIT.java` to a yaml test.
**Introduction**
> In order to make adoption of failure stores simpler for all users, we
are introducing a new syntactical feature to index expression
resolution: The selector. > > Selectors, denoted with a :: followed by a
recognized suffix will allow users to specify which component of an
index abstraction they would like to operate on within an API call. In
this case, an index abstraction is a concrete index, data stream, or
alias; Any abstraction that can be resolved to a set of indices/shards.
We define a component of an index abstraction to be some searchable unit
of the index abstraction. > > To start, we will support two components:
data and failures. Concrete indices are their own data components, while
the data component for index aliases are all of the indices contained
therein. For data streams, the data component corresponds to their
backing indices. Data stream aliases mirror this, treating all backing
indices of the data streams they correspond to as their data component.
> > The failure component is only supported by data streams and data
stream aliases. The failure component of these abstractions refer to the
data streams' failure stores. Indices and index aliases do not have a
failure component.
For more details and examples see
https://github.com/elastic/elasticsearch/pull/113144. All this work has
been cherry picked from there.
**Purpose of this PR**
This PR is replacing the the indices options boolean constructor with
the builders. The goal is to give me and the reviewer a very narrow
scope change when we can ensure we did not make any mistakes during the
conversion. Also it will reduce a bit the change list in
https://github.com/elastic/elasticsearch/pull/113144/files.
This removes the possibility for a plugin to provide factory retention settings. Factory retention settings have been deprecated and completely replaced by #111972.
Note: this feature is not in use. If someone wants to set global retention they can use the cluster settings as defined in #111972.
REST APIs which declare their supported parameters must consume exactly
those parameters: consuming an unsupported parameter means that requests
including that parameter will be rejected, whereas failing to consume a
supported parameter means that this parameter has no effect and should
be removed.
This commit adds an assertion to verify that we are consuming the
correct parameters.
Closes#113854
REST APIs which declare their supported parameters must consume exactly
those parameters: consuming an unsupported parameter means that requests
including that parameter will be rejected, whereas failing to consume a
supported parameter means that this parameter has no effect and should
be removed.
This commit adds an assertion to verify that we are consuming the
correct parameters.
Closes#113854
A few of today's REST handler implementations compute a new set of
supported parameters on each request. This is needlessly inefficient
since the set never changes. This commit fixes those implementations and
adds assertions to verify that we are returning the exact same instance
each time.
In LogsDB we would like to use a default value of `8191` for the index-level setting
`index.mapping.ignore_above`. The value for `ignore_above` is the _character count_,
but Lucene counts bytes. Here we set the limit to `32766 / 4 = 8191` since UTF-8
characters may occupy at most 4 bytes.
* Make xpack.otel_data.registry.enabled default to true
* Update build.gradle
* Disable otel-data plugin for tests where it's not relevant
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Today we keep track of a data stream's index mode. However if index mode
is logsdb, then we don't set it at data stream level. We only set it is
index mode is time series.
This change keeps track of logsdb index mode at data stream creation and
rollover time.
Followup changes: * Update the get data stream api to return that a data
stream is in logsdb mode. (today we do return additional information in
case of time series data stream) * Update the
IndexSettingProvider#getAdditionalIndexSettings(...) parameter list by
including `indexMode` instead of `isTimeSeries`.
Relates to #113583
