mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
164 lines
5 KiB
Text
164 lines
5 KiB
Text
[role="xpack"]
|
|
[[indices-downsample-data-stream]]
|
|
=== Downsample index API
|
|
++++
|
|
<titleabbrev>Downsample</titleabbrev>
|
|
++++
|
|
|
|
.New API reference
|
|
[sidebar]
|
|
--
|
|
For the most up-to-date API details, refer to {api-es}/group/endpoint-data-stream[Data stream APIs].
|
|
--
|
|
|
|
Aggregates a time series (TSDS) index and stores
|
|
pre-computed statistical summaries (`min`, `max`, `sum`, `value_count` and
|
|
`avg`) for each metric field grouped by a configured time interval. For example,
|
|
a TSDS index that contains metrics sampled every 10 seconds can be downsampled
|
|
to an hourly index. All documents within an hour interval are summarized and
|
|
stored as a single document in the downsample index.
|
|
|
|
// tag::downsample-example[]
|
|
////
|
|
[source,console]
|
|
----
|
|
PUT /my-time-series-index
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"mode": "time_series",
|
|
"time_series": {
|
|
"start_time": "2022-06-10T00:00:00Z",
|
|
"end_time": "2022-06-30T23:59:59Z"
|
|
},
|
|
"routing_path": [
|
|
"test.namespace"
|
|
],
|
|
"number_of_replicas": 0,
|
|
"number_of_shards": 2
|
|
}
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"@timestamp": {
|
|
"type": "date"
|
|
},
|
|
"metric": {
|
|
"type": "long",
|
|
"time_series_metric": "gauge"
|
|
},
|
|
"dimension": {
|
|
"type": "keyword",
|
|
"time_series_dimension": true
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
PUT /my-time-series-index/_block/write
|
|
|
|
----
|
|
// TEST
|
|
////
|
|
|
|
[source,console]
|
|
----
|
|
POST /my-time-series-index/_downsample/my-downsampled-time-series-index
|
|
{
|
|
"fixed_interval": "1d"
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
DELETE /my-time-series-index*
|
|
DELETE _data_stream/*
|
|
DELETE _index_template/*
|
|
----
|
|
// TEST[continued]
|
|
////
|
|
// end::downsample-example[]
|
|
|
|
[[downsample-api-request]]
|
|
==== {api-request-title}
|
|
|
|
`POST /<source-index>/_downsample/<output-downsampled-index>`
|
|
|
|
[[downsample-api-prereqs]]
|
|
==== {api-prereq-title}
|
|
|
|
* Only indices in a <<tsds,time series data stream>> are supported.
|
|
|
|
* If the {es} {security-features} are enabled, you must have the `all`
|
|
or `manage` <<privileges-list-indices,index privilege>> for the data stream.
|
|
|
|
* Neither <<field-and-document-access-control,field nor document level security>> can be defined on the source index.
|
|
|
|
* The source index must be read only (`index.blocks.write: true`).
|
|
|
|
[[downsample-api-path-params]]
|
|
==== {api-path-parms-title}
|
|
|
|
`<source-index>`::
|
|
(Optional, string) Name of the time series index to downsample.
|
|
|
|
`<output-downsampled_index>`::
|
|
+
|
|
--
|
|
(Required, string) Name of the index to create.
|
|
|
|
include::{es-ref-dir}/indices/create-index.asciidoc[tag=index-name-reqs]
|
|
--
|
|
|
|
[role="child_attributes"]
|
|
[[downsample-api-query-parms]]
|
|
==== {api-query-parms-title}
|
|
|
|
`fixed_interval`:: (Required, <<time-units,time units>>) The interval at which
|
|
to aggregate the original time series index. For example, `60m` produces a
|
|
document for each 60 minute (hourly) interval. This follows standard time
|
|
formatting syntax as used elsewhere in {es}.
|
|
+
|
|
NOTE: Smaller, more granular intervals take up proportionally more space.
|
|
|
|
[[downsample-api-process]]
|
|
==== The downsampling process
|
|
|
|
The downsampling operation traverses the source TSDS index and performs the
|
|
following steps:
|
|
|
|
. Creates a new document for each value of the `_tsid` field and each
|
|
`@timestamp` value, rounded to the `fixed_interval` defined in the downsample
|
|
configuration.
|
|
. For each new document, copies all <<time-series-dimension,time
|
|
series dimensions>> from the source index to the target index. Dimensions in a
|
|
TSDS are constant, so this is done only once per bucket.
|
|
. For each <<time-series-metric,time series metric>> field, computes aggregations
|
|
for all documents in the bucket. Depending on the metric type of each metric
|
|
field a different set of pre-aggregated results is stored:
|
|
|
|
** `gauge`: The `min`, `max`, `sum`, and `value_count` are stored; `value_count`
|
|
is stored as type `aggregate_metric_double`.
|
|
** `counter`: The `last_value` is stored.
|
|
. For all other fields, the most recent value is copied to the target index.
|
|
|
|
[[downsample-api-mappings]]
|
|
==== Source and target index field mappings
|
|
|
|
Fields in the target, downsampled index are created based on fields in the
|
|
original source index, as follows:
|
|
|
|
. All fields mapped with the `time-series-dimension` parameter are created in
|
|
the target downsample index with the same mapping as in the source index.
|
|
. All fields mapped with the `time_series_metric` parameter are created
|
|
in the target downsample index with the same mapping as in the source
|
|
index. An exception is that for fields mapped as `time_series_metric: gauge`
|
|
the field type is changed to `aggregate_metric_double`.
|
|
. All other fields that are neither dimensions nor metrics (that is, label
|
|
fields), are created in the target downsample index with the same mapping
|
|
that they had in the source index.
|
|
|
|
Check the <<downsampling,Downsampling>> documentation for an overview and
|
|
examples of running downsampling manually and as part of an ILM policy.
|