Transport actions have associated request and response classes. However,
the base type restrictions are not necessary to duplicate when creating
a map of transport actions. Relatedly, the ActionHandler class doesn't
actually need strongly typed action type and classes since they are lost
when shoved into the node client map. This commit removes these type
restrictions and generic parameters.
This test had a copy paste mistake. When the cluster has only one data
node the replicas cannot be assigned so we end up with a force merge
error. In the case of the failure store this was not asserted correctly.
On the other hand, this test only checked for the existence of an error
and it was not ensuring that the current error is not the rollover error
that should have recovered. We make this test a bit more explicit.
Fixes: https://github.com/elastic/elasticsearch/issues/126252
**Issue** The data stream lifecycle does not register correctly rollover
errors for failure store.
**Observed bahaviour** When data stream lifecycle encounters a rollover
error it records it unless it sees that the current write index of this
data stream doesn't match the source index of the request. However, the
write index check does not use the failure write index but the write
backing index, so the failure gets ignored
**Desired behaviour** When data stream lifecycle encounters a rollover
error it will check the relevant write index before it determines if it
should be recorded or not.
In this PR we introduce the data stream API in the `es-rest-api` using
the feature flag feature. This enabled us to use the `yamlRestTests`
tests instead of the `javaRestTests`.
Now that all actions that DLM depends on are project-aware, we can make DLM itself project-aware.
There still exists only one instance of `DataStreamLifecycleService`, it just loops over all the projects - which matches the approach we've taken for similar scenarios thus far.
This tracks the highest value seen for the recent write load metric
any time the stats for a shard was computed, exposes this value
alongside the recent value, and persists it in index metadata
alongside it too.
The new test in `IndexShardTests` is designed to more thoroughly test
the recent write load metric previously added, as well as to test the
peak metric being added here.
ES-10037 #comment Added peak load metric in https://github.com/elastic/elasticsearch/pull/125521
* Specify index component when retrieving lifecycle
* Add getters for the failure lifecycle
* Conceptually introduce the failure store lifecycle (even for now it's the same)
These tests had the potential to fail when two consecutive GET data
streams requests would hit two different nodes, where one node already
had the cluster state that contained the new backing index and the other
node didn't yet.
Caused by #122852Fixes#124882Fixes#124885
These tests had the potential to fail when two consecutive GET data
streams requests would hit two different nodes, where one node already
had the cluster state that contained the new backing index and the other
node didn't yet.
Caused by #122852Fixes#124846Fixes#124950Fixes#124999
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
This PR updates the different methods in TestProjectResolvers so that
their names are more accurate and behaviours to be more as expected.
For example, In MP-1749, we differentiate between single-project and
single-project only resolvers. The later should not support multi-project.
This is part of the work to make DLM project-aware.
These two features were pretty tightly coupled, so I saved some effort
by combining them in one PR.
This uses the recently-added `ExponentiallyWeightedMovingRate` class
to calculate a write load which favours more recent load and include
this alongside the existing unweighted all-time write load in
`IndexingStats.Stats`.
As of this change, the new load metric is not used anywhere, although
it can be retrieved with the index stats or node stats APIs.
This makes using usesDefaultDistribution in our test setup for explicit by requiring a reason why it's needed.
This is helpful as part of revisiting the need for all those usages in our code base.
This field is only used (by security) for requests, having it in responses is redundant.
Also, we have a couple of responses that are singletons/quasi-enums where setting the value
needlessly might introduce some strange contention even though it's a plain store.
This isn't just a cosmetic change. It makes it clear at compile time that each response instance
is exclusively defined by the bytes that it is read from. This makes it easier to reason about the
validity of suggested optimizations like https://github.com/elastic/elasticsearch/pull/120010
This PR adds a new MetadataDeleteDataStreamService that allows us to delete system data streams prior to a restore operation. This fixes a bug where system data streams were previously un-restorable.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
Makes the execution and use of enrich policies project-aware.
Note: this does not make the enrich cache project-aware. That is to be
handled in a follow-up PR.
This commit improves memory accounting by incorporating document
expansion during shard bulk execution. Additionally, it introduces a new
limit on the maximum document size, which defaults to 5% of the
available heap.
This limit can be configured using the new setting:
indexing_pressure.memory.max_operation_size
These changes help prevent excessive memory consumption and
improve indexing stability.
Closes ES-10777
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
This PR extends the work done in #121751 by enabling a sparse doc values index for the @timestamp field in LogsDB.
Similar to the previous PR, the setting index.mapping.use_doc_values_skipper will override the index mapping parameter when all of the following conditions are met:
* The index mode is LogsDB.
* The field name is @timestamp.
* Index sorting is configured on @timestamp (regardless of whether it is a primary sort field or not).
* Doc values are enabled.
This ensures that only one index structure is defined on the @timestamp field:
* If the conditions above are met, the inverted index is replaced with a sparse doc values index.
* This prevents both the inverted index and sparse doc values index from being enabled together, reducing unnecessary storage overhead.
This change aligns with our goal of optimizing LogsDB for storage efficiency while possibly maintaining reasonable query latency performance. It will enable us to run benchmarks and evaluate the impact of sparse indexing on the @timestamp field as well.
The node environment has many paths. The accessors for these currently
use a "file" suffix, but they are always directories. This commit
renames the accessors to make it clear these paths are directories.
Updates the on failure handler to accept the additional context required to report to the client whether
or not the document could have been rerouted if failure store was enabled.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
The message_id field may be unmapped if documents were indexed into some
indices but not all. This change specifies the unmapped type for
message_id, allowing it to be sorted in such cases.
Closes#120796
