mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-28 17:34:17 -04:00
ad90d1f0f6
In this PR we introduce cluster settings to manage the global data stream retention. We introduce two settings `data_streams.lifecycle.retention.max` & `data_streams.lifecycle.retention.default` that configure the respective retentions. The settings are loaded and monitored by the `DataStreamGlobalRetentionSettings`. The validation has also moved there. We preserved the `DataStreamGlobalRetention` record to reduce the impact of this change. The purpose of this method is to be simply a wrapper record that groups the retention settings together. Temporarily, the `DataStreamGlobalRetentionSettings` is using the DataStreamFactoryRetention which is marked as deprecated for migration purposes.
93 lines
5.2 KiB
Text
93 lines
5.2 KiB
Text
[role="xpack"]
|
|
[[data-stream-lifecycle-settings]]
|
|
=== Data stream lifecycle settings in {es}
|
|
[subs="attributes"]
|
|
++++
|
|
<titleabbrev>Data stream lifecycle settings</titleabbrev>
|
|
++++
|
|
|
|
These are the settings available for configuring <<data-stream-lifecycle, data stream lifecycle>>.
|
|
|
|
==== Cluster level settings
|
|
|
|
[[data-streams-lifecycle-retention-max]]
|
|
`data_streams.lifecycle.retention.max`::
|
|
(<<dynamic-cluster-setting,Dynamic>>, <<time-units, time unit value>>)
|
|
The maximum retention period that will apply to all user data streams managed by the data stream lifecycle. The max retention will also
|
|
override the retention of a data stream whose configured retention exceeds the max retention. It should be greater than `10s`.
|
|
|
|
[[data-streams-lifecycle-retention-default]]
|
|
`data_streams.lifecycle.retention.default`::
|
|
(<<dynamic-cluster-setting,Dynamic>>, <<time-units, time unit value>>)
|
|
The retention period that will apply to all user data streams managed by the data stream lifecycle that do not have retention configured.
|
|
It should be greater than `10s` and less or equals than <<data-streams-lifecycle-retention-max, `data_streams.lifecycle.retention.max`>>.
|
|
|
|
[[data-streams-lifecycle-poll-interval]]
|
|
`data_streams.lifecycle.poll_interval`::
|
|
(<<dynamic-cluster-setting,Dynamic>>, <<time-units, time unit value>>)
|
|
How often {es} checks what is the next action for all data streams with a built-in lifecycle. Defaults to `5m`.
|
|
|
|
[[cluster-lifecycle-default-rollover]]
|
|
`cluster.lifecycle.default.rollover`::
|
|
(<<dynamic-cluster-setting,Dynamic>>, string)
|
|
This property accepts a key value pair formatted string and configures the conditions that would trigger a data stream
|
|
to <<index-rollover,rollover>> when it has `lifecycle` configured. This property is an implementation detail and subject to
|
|
change. Currently, it defaults to `max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000`,
|
|
this means that your data stream will rollover if any of the following conditions are met:
|
|
|
|
* Either any primary shard reaches the size of 50GB,
|
|
* or any primary shard contains 200.000.000 documents
|
|
* or the index reaches a certain age which depends on the retention time of your data stream,
|
|
* **and** has at least one document.
|
|
|
|
[[data-streams-lifecycle-target-merge-factor]]
|
|
`data_streams.lifecycle.target.merge.policy.merge_factor`::
|
|
(<<dynamic-cluster-setting,Dynamic>>, integer)
|
|
Data stream lifecycle implements <<data-streams-lifecycle-how-it-works, tail merging>> by
|
|
updating the lucene merge policy factor for the target backing index. The merge factor
|
|
is both the number of segments that should be merged together, and the maximum number
|
|
of segments that we expect to find on a given tier.
|
|
This setting controls what value does <<data-stream-lifecycle, Data stream lifecycle>>
|
|
configures on the target index. It defaults to `16`.
|
|
The value will be visible under the `index.merge.policy.merge_factor` index setting
|
|
on the target index.
|
|
|
|
[[data-streams-lifecycle-target-floor-segment]]
|
|
`data_streams.lifecycle.target.merge.policy.floor_segment`::
|
|
(<<dynamic-cluster-setting,Dynamic>>)
|
|
Data stream lifecycle implements <<data-streams-lifecycle-how-it-works, tail merging>> by
|
|
updating the lucene merge policy floor segment for the target backing index. This floor
|
|
segment size is a way to prevent indices from having a long tail of very small segments.
|
|
This setting controls what value does <<data-stream-lifecycle, data stream lifecycle>>
|
|
configures on the target index. It defaults to `100MB`.
|
|
|
|
[[data-streams-lifecycle-signalling-error-retry-interval]]
|
|
`data_streams.lifecycle.signalling.error_retry_interval`::
|
|
(<<dynamic-cluster-setting,Dynamic>>, integer)
|
|
Represents the number of retries data stream lifecycle has to perform for an index
|
|
in an error step in order to signal that the index is not progressing (i.e. it's
|
|
stuck in an error step).
|
|
The current signalling mechanism is a log statement at the `error` level however,
|
|
the signalling mechanism can be extended in the future.
|
|
Defaults to 10 retries.
|
|
|
|
|
|
==== Index level settings
|
|
The following index-level settings are typically configured on the backing indices of a data stream.
|
|
|
|
[[index-lifecycle-prefer-ilm]]
|
|
`index.lifecycle.prefer_ilm`::
|
|
(<<indices-update-settings,Dynamic>>, boolean)
|
|
This setting determines which feature is managing the backing index of a data stream if, and only if, the backing index
|
|
has an <<index-lifecycle-management,{ilm}>> ({ilm-init}) policy and the data stream has also a built-in lifecycle. When
|
|
`true` this index is managed by {ilm-init}, when `false` the backing index is managed by the data stream lifecycle.
|
|
Defaults to `true`.
|
|
|
|
[[index-data-stream-lifecycle-origination-date]]
|
|
`index.lifecycle.origination_date`::
|
|
(<<indices-update-settings,Dynamic>>, long)
|
|
If specified, this is the timestamp used to calculate the backing index generation age after this backing index has been
|
|
<<index-rollover,rolled over>>. The generation age is used to determine data retention, consequently, you can use this
|
|
setting if you create a backing index that contains older data and want to ensure that the retention period or
|
|
other parts of the lifecycle will be applied based on the data's original timestamp and not the timestamp they got
|
|
indexed. Specified as a Unix epoch value in milliseconds.
|