mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-29 01:44:36 -04:00
e71ea6e6d7
In this PR we enable all new data streams to be managed by the data
stream lifecycle by default. This is implemented by adding an empty
`lifecycle: {}` upon new data stream creation.
Opting out is represented by a the `enabled` flag:
```
{
"lifecycle": {
"enabled": false
}
}
```
This change has the following implications on when is an index managed
and by which feature:
| Parent data stream lifecycle| ILM| `prefer_ilm`|Managed by|
|----------------------------|----|----------------|-| | default | yes|
true| ILM| | default | yes| false| data stream lifecycle| |default |
no|true/false|data stream lifecycle| |opt-out or
missing|yes|true/false|ILM| |opt-out or missing|no|true/false|unmanaged|
Data streams that have been created before the data stream lifecycle is
enabled will not have the default lifecycle.
Next steps: - We need to document this when the feature will be GA
(https://github.com/elastic/elasticsearch/issues/97973).
138 lines
5.1 KiB
Text
138 lines
5.1 KiB
Text
[role="xpack"]
|
|
[[tutorial-manage-existing-data-stream]]
|
|
=== Tutorial: Update existing data stream
|
|
|
|
preview::[]
|
|
|
|
To update the lifecycle of an existing data stream you do the following actions:
|
|
|
|
. <<set-lifecycle>>
|
|
. <<delete-lifecycle>>
|
|
|
|
[discrete]
|
|
[[set-lifecycle]]
|
|
==== Set a data stream's lifecycle
|
|
|
|
To add or to change the retention period of your data stream you can use the <<data-streams-put-lifecycle, PUT lifecycle API>>.
|
|
|
|
* You can set infinite retention period, meaning that your data should never be deleted. For example:
|
|
+
|
|
[source,console]
|
|
----
|
|
PUT _data_stream/my-data-stream/_lifecycle
|
|
{ } <1>
|
|
----
|
|
// TEST[setup:my_data_stream]
|
|
<1> An empty payload means that your data stream is still managed but the data will never be deleted. Managing a time
|
|
series data stream such as logs or metrics enables {es} to better store your data even if you do not use a retention period.
|
|
|
|
* Or you can set the retention period of your choice. For example:
|
|
+
|
|
[source,console]
|
|
----
|
|
PUT _data_stream/my-data-stream/_lifecycle
|
|
{
|
|
"data_retention": "30d" <1>
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
<1> The retention period of this data stream is set to 30 days. This means that {es} is allowed to delete data that is
|
|
older than 30 days at its own discretion.
|
|
|
|
The changes in the lifecycle are applied on all backing indices of the data stream. You can see the effect of the change
|
|
via the <<data-streams-explain-lifecycle, explain API>>:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET .ds-my-data-stream-*/_lifecycle/explain
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
The response will look like:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"indices": {
|
|
".ds-my-data-stream-2023.04.19-000002": {
|
|
"index": ".ds-my-data-stream-2023.04.19-000002", <1>
|
|
"managed_by_lifecycle": true, <2>
|
|
"index_creation_date_millis": 1681919221417,
|
|
"time_since_index_creation": "6.85s", <3>
|
|
"lifecycle": {
|
|
"enabled": true,
|
|
"data_retention": "30d" <4>
|
|
}
|
|
},
|
|
".ds-my-data-stream-2023.04.17-000001": {
|
|
"index": ".ds-my-data-stream-2023.04.17-000001", <5>
|
|
"managed_by_lifecycle": true, <6>
|
|
"index_creation_date_millis": 1681745209501,
|
|
"time_since_index_creation": "48d", <7>
|
|
"rollover_date_millis": 1681919221419,
|
|
"time_since_rollover": "6.84s", <8>
|
|
"generation_time": "6.84s", <9>
|
|
"lifecycle": {
|
|
"enabled": true,
|
|
"data_retention": "30d" <10>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
// TESTRESPONSE[skip:the result is for illustrating purposes only]
|
|
<1> The name of the backing index.
|
|
<2> This index is managed by a data stream lifecycle.
|
|
<3> The time that has passed since this index has been created.
|
|
<4> The data retention for this index is at least 30 days, as it was recently updated.
|
|
<5> The name of the backing index.
|
|
<6> This index is managed by the built-in data stream lifecycle.
|
|
<7> The time that has passed since this index has been created.
|
|
<8> The time that has passed since this index was <<index-rollover,rolled over>>.
|
|
<9> The time that will be used to determine when it's safe to delete this index and all its data.
|
|
<10> The data retention for this index as well is at least 30 days, as it was recently updated.
|
|
|
|
[discrete]
|
|
[[delete-lifecycle]]
|
|
==== Remove lifecycle for a data stream
|
|
|
|
To remove the lifecycle of a data stream you can use the <<data-streams-delete-lifecycle-request,delete lifecycle API>>. As consequence,
|
|
the maintenance operations that were applied by the lifecycle will no longer be applied to the data stream and all its
|
|
backing indices. For example:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE _data_stream/my-data-stream/_lifecycle
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
You can then use the <<data-streams-explain-lifecycle, explain API>> again to see that the indices are no longer managed.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET .ds-my-data-stream-*/_lifecycle/explain
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
// TEST[teardown:data_stream_cleanup]
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"indices": {
|
|
".ds-my-data-stream-2023.04.19-000002": {
|
|
"index": ".ds-my-data-stream-2023.04.19-000002", <1>
|
|
"managed_by_lifecycle": false <2>
|
|
},
|
|
".ds-my-data-stream-2023.04.17-000001": {
|
|
"index": ".ds-my-data-stream-2023.04.19-000001", <3>
|
|
"managed_by_lifecycle": false <4>
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[skip:the result is for illustrating purposes only]
|
|
<1> The name of the backing index.
|
|
<2> Indication that the index is not managed by the data stream lifecycle.
|
|
<3> The name of another backing index.
|
|
<4> Indication that the index is not managed by the data stream lifecycle.
|