mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
d59ee6cb45
This change also updated the deprecation warning on all rollup pages from Rollups will be removed in a future version. Use <<downsampling,downsampling>> instead. to Rollups will be removed in a future version. Please <<rollup-migrating-to-downsampling,migrate>> to <<downsampling,downsampling>> instead..
120 lines
4.2 KiB
Text
120 lines
4.2 KiB
Text
[role="xpack"]
|
|
[[rollup-migrating-to-downsampling]]
|
|
=== Migrating from {rollup-cap} to downsampling
|
|
++++
|
|
<titleabbrev>Migrating to downsampling</titleabbrev>
|
|
++++
|
|
|
|
Rollup and downsampling are two different features that allow historical metrics to be rolled up.
|
|
From a high level rollup is more flexible compared to downsampling, but downsampling is a more robust and
|
|
easier feature to downsample metrics.
|
|
|
|
The following aspects of downsampling are easier or more robust:
|
|
|
|
* No need to schedule jobs. Downsampling is integrated with Index Lifecycle Management (ILM) and Data Stream Lifecycle (DSL).
|
|
* No separate search API. Downsampled indices can be accessed via the search api and es|ql.
|
|
* No separate rollup configuration. Downsampling uses the time series dimension and metric configuration from the mapping.
|
|
|
|
It isn't possible to migrate all rollup usages to downsampling. The main requirement
|
|
is that the data should be stored in Elasticsearch as <<tsds,time series data stream (TSDS)>>.
|
|
Rollup usages that basically roll the data up by time and all dimensions can migrate to downsampling.
|
|
|
|
An example rollup usage that can be migrated to downsampling:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _rollup/job/sensor
|
|
{
|
|
"index_pattern": "sensor-*",
|
|
"rollup_index": "sensor_rollup",
|
|
"cron": "0 0 * * * *", <1>
|
|
"page_size": 1000,
|
|
"groups": { <2>
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"fixed_interval": "60m" <3>
|
|
},
|
|
"terms": {
|
|
"fields": [ "node" ]
|
|
}
|
|
},
|
|
"metrics": [
|
|
{
|
|
"field": "temperature",
|
|
"metrics": [ "min", "max", "sum" ] <4>
|
|
},
|
|
{
|
|
"field": "voltage",
|
|
"metrics": [ "avg" ] <4>
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:sensor_index]
|
|
|
|
The equivalent <<tsds,time series data stream (TSDS)>> setup that uses downsampling via DSL:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _index_template/sensor-template
|
|
{
|
|
"index_patterns": ["sensor-*"],
|
|
"data_stream": { },
|
|
"template": {
|
|
"lifecycle": {
|
|
"downsampling": [
|
|
{
|
|
"after": "1d", <1>
|
|
"fixed_interval": "1h" <3>
|
|
}
|
|
]
|
|
},
|
|
"settings": {
|
|
"index.mode": "time_series"
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"node": {
|
|
"type": "keyword",
|
|
"time_series_dimension": true <2>
|
|
},
|
|
"temperature": {
|
|
"type": "half_float",
|
|
"time_series_metric": "gauge" <4>
|
|
},
|
|
"voltage": {
|
|
"type": "half_float",
|
|
"time_series_metric": "gauge" <4>
|
|
},
|
|
"@timestamp": { <2>
|
|
"type": "date"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
DELETE _index_template/sensor-template
|
|
----
|
|
// TEST[continued]
|
|
////
|
|
|
|
The downsample configuration is included in the above template for a <<tsds,time series data stream (TSDS)>>.
|
|
Only the `downsampling` part is necessary to enable downsampling, which indicates when to downsample to what fixed interval.
|
|
|
|
<1> In the rollup job, the `cron` field determines when the rollup documents. In the index template,
|
|
the `after` field determines when downsampling will rollup documents (note that this the time after a rollover has been performed).
|
|
<2> In the rollup job, the `groups` field determines all dimensions of the group documents are rolled up to. In the index template,
|
|
the fields with `time_series_dimension` set `true` and the `@timestamp` field determine the group.
|
|
<3> In the rollup job, the `fixed_interval` field determines how timestamps are aggregated as part of the grouping.
|
|
In the index template, the `fixed_interval` field has the same purpose. Note that downsampling does not support calendar intervals.
|
|
<4> In the rollup job, the `metrics` field define the metrics and how to store these metrics. In the index template,
|
|
all fields with a `time_series_metric` are metric fields. If a field has `gauge` as `time_series_metric` attribute
|
|
value, then min, max, sum and value counts are stored for this field in the downsampled index. If a field has
|
|
`counter` as `time_series_metric` attribute value, then only the last value stored for this field in the downsampled
|
|
index.
|