mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-19 04:45:07 -04:00
[TSDB] Metric fields in the field caps API (#88695)
To assist the user in configuring the visualizations correctly while leveraging TSDB functionality, information about TSDB configuration should be exposed via the field caps API per field. Especially for metrics fields, it must be clear which fields are metrics and if they belong to only time-series indexes or mixed time-series and non-time-series indexes. To further distinguish metric fields when they belong to any of the following indices: - Standard (non-time-series) indexes - Time series indexes - Downsampled time series indexes This PR modifies the field caps API so that the mapping parameters time_series_dimension and time_series_dimension are presented only when they are set on fields of time-series indexes. Those parameters are completely ignored when they are set on standard (non-time-series) indexes. This PR revisits some of the conventions adopted by #78790
This commit is contained in:
Christos Soulios
GitHub
parent
188f8872c6
commit
b81f4187ab
11 changed files with 474 additions and 79 deletions
5
docs/changelog/88695.yaml
Normal file
5
docs/changelog/88695.yaml
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
pr: 88695
|
||||
summary: "[TSDB] Metric fields in the field caps API"
|
||||
area: TSDB
|
||||
type: enhancement
|
||||
issues: []
|
||||
|
|
@ -137,11 +137,13 @@ field types are all described as the `keyword` type family.
|
|||
|
||||
`time_series_dimension`::
|
||||
preview:[]
|
||||
Whether this field is used as a time series dimension.
|
||||
Whether this field is used as a time series dimension on all indices.
|
||||
For non-time-series indices this field is not present.
|
||||
|
||||
`time_series_metric`::
|
||||
preview:[]
|
||||
Contains metric type if this fields is used as a time series metrics, absent if the field is not used as metric.
|
||||
Contains the metric type if the field is used as a time series metric on all indices, absent if the field is
|
||||
not used as a metric. For non-time-series indices this field is not included.
|
||||
|
||||
`indices`::
|
||||
The list of indices where this field has the same type family, or null if all indices
|
||||
|
|
@ -157,12 +159,12 @@ field types are all described as the `keyword` type family.
|
|||
|
||||
`non_dimension_indices`::
|
||||
experimental:[]
|
||||
If this list is present in response then some indices have the field marked as a dimension and other indices, the
|
||||
If this list is present in the response, some indices have the field marked as a dimension and other indices, the
|
||||
ones in this list, do not.
|
||||
|
||||
`metric_conflicts_indices`::
|
||||
experimental:[]
|
||||
The list of indices where this field is present if these indices don't have the same `time_series_metric` value for
|
||||
The list of indices where this field is present, if these indices don't have the same `time_series_metric` value for
|
||||
this field.
|
||||
|
||||
`meta`::
|
||||
|
|
|
|||
|
|
@ -12,6 +12,11 @@ setup:
|
|||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mode: time_series
|
||||
routing_path: [ metricset, k8s.pod.uid ]
|
||||
time_series:
|
||||
start_time: 2021-04-28T00:00:00Z
|
||||
end_time: 2021-04-29T00:00:00Z
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
|
|
@ -59,6 +64,11 @@ setup:
|
|||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mode: time_series
|
||||
routing_path: [ k8s.pod.uid ]
|
||||
time_series:
|
||||
start_time: 2021-04-28T00:00:00Z
|
||||
end_time: 2021-04-29T00:00:00Z
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
|
|
|
|||
|
|
@ -183,8 +183,8 @@ can't shadow metrics:
|
|||
# Test that _tsid field is not added if an index is not a time-series index
|
||||
no _tsid in standard indices:
|
||||
- skip:
|
||||
version: " - 8.0.99"
|
||||
reason: _tsid support introduced in 8.1.0
|
||||
version: " - 8.4.99"
|
||||
reason: time series params only on time series indices introduced in 8.5.0
|
||||
|
||||
- do:
|
||||
indices.create:
|
||||
|
|
@ -209,11 +209,11 @@ no _tsid in standard indices:
|
|||
|
||||
- match: {fields.metricset.keyword.searchable: true}
|
||||
- match: {fields.metricset.keyword.aggregatable: true}
|
||||
- match: {fields.metricset.keyword.time_series_dimension: true}
|
||||
- is_false: fields.metricset.keyword.indices
|
||||
- is_false: fields.metricset.keyword.non_searchable_indices
|
||||
- is_false: fields.metricset.keyword.non_aggregatable_indices
|
||||
- is_false: fields._tsid # _tsid metadata field must not exist in non-time-series indices
|
||||
- is_false: fields.metricset.keyword.time_series_dimension # time_series_dimension param is ignored in non-time-series indices
|
||||
|
||||
---
|
||||
no nested dimensions:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,231 @@
|
|||
setup:
|
||||
- skip:
|
||||
version: " - 8.4.99"
|
||||
reason: metric params only on time series indexes introduced in 8.5.0
|
||||
|
||||
- do:
|
||||
indices.create:
|
||||
index: test_time_series
|
||||
body:
|
||||
settings:
|
||||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mode: time_series
|
||||
routing_path: [ metricset, k8s.pod.uid ]
|
||||
time_series:
|
||||
start_time: 2021-04-28T00:00:00Z
|
||||
end_time: 2021-04-29T00:00:00Z
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
type: date
|
||||
metricset:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
k8s:
|
||||
properties:
|
||||
pod:
|
||||
properties:
|
||||
availability_zone:
|
||||
type: short
|
||||
time_series_dimension: true
|
||||
uid:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
name:
|
||||
type: keyword
|
||||
ip:
|
||||
type: ip
|
||||
time_series_dimension: true
|
||||
network:
|
||||
properties:
|
||||
tx:
|
||||
type: long
|
||||
time_series_metric: counter
|
||||
rx:
|
||||
type: long
|
||||
time_series_metric: gauge
|
||||
|
||||
- do:
|
||||
indices.create:
|
||||
index: test_non_time_series
|
||||
body:
|
||||
settings:
|
||||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
type: date
|
||||
metricset:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
k8s:
|
||||
properties:
|
||||
pod:
|
||||
properties:
|
||||
availability_zone:
|
||||
type: short
|
||||
time_series_dimension: true
|
||||
uid:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
name:
|
||||
type: keyword
|
||||
ip:
|
||||
type: ip
|
||||
time_series_dimension: true
|
||||
network:
|
||||
properties:
|
||||
tx:
|
||||
type: long
|
||||
time_series_metric: counter
|
||||
rx:
|
||||
type: long
|
||||
time_series_metric: gauge
|
||||
|
||||
---
|
||||
field caps on time_series indices:
|
||||
- skip:
|
||||
version: " - 8.3.99"
|
||||
reason: metric params only on time series indexes introduced in 8.4.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: test_time_series
|
||||
fields: [ k8s.pod.uid, k8s.pod.network.rx, k8s.pod.network.tx, k8s.pod.ip, metricset, _tsid ]
|
||||
|
||||
- match: { fields.k8s\.pod\.uid.keyword.type: keyword }
|
||||
- match: { fields.k8s\.pod\.uid.keyword.searchable: true }
|
||||
- match: { fields.k8s\.pod\.uid.keyword.aggregatable: true }
|
||||
- match: { fields.k8s\.pod\.uid.keyword.time_series_dimension: true }
|
||||
- is_false: fields.k8s\.pod\.uid.keyword.indices
|
||||
- is_false: fields.k8s\.pod\.uid.keyword.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.uid.keyword.non_aggregatable_indices
|
||||
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.type: long }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.searchable: true }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.aggregatable: true }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.time_series_metric: gauge }
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
|
||||
|
||||
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.type: long }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.searchable: true }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.aggregatable: true }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.time_series_metric: counter }
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.non_aggregatable_indices
|
||||
|
||||
- match: { fields.k8s\.pod\.ip.ip.type: ip }
|
||||
- match: { fields.k8s\.pod\.ip.ip.searchable: true }
|
||||
- match: { fields.k8s\.pod\.ip.ip.aggregatable: true }
|
||||
- is_false: fields.k8s\.pod\.ip.ip.indices
|
||||
- is_false: fields.k8s\.pod\.ip.ip.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.ip.ip.non_aggregatable_indices
|
||||
|
||||
- match: { fields.metricset.keyword.type: keyword }
|
||||
- match: { fields.metricset.keyword.searchable: true }
|
||||
- match: { fields.metricset.keyword.aggregatable: true }
|
||||
- match: { fields.metricset.keyword.time_series_dimension: true }
|
||||
- is_false: fields.metricset.keyword.non_dimension_indices
|
||||
- is_false: fields.metricset.keyword.indices
|
||||
- is_false: fields.metricset.keyword.non_searchable_indices
|
||||
- is_false: fields.metricset.keyword.non_aggregatable_indices
|
||||
|
||||
- match: { fields._tsid._tsid.metadata_field: true }
|
||||
- match: { fields._tsid._tsid.searchable: false }
|
||||
- match: { fields._tsid._tsid.aggregatable: true }
|
||||
- is_false: fields._tsid._tsid.indices
|
||||
- is_false: fields._tsid._tsid.non_searchable_indices
|
||||
- is_false: fields._tsid._tsid.non_aggregatable_indices
|
||||
|
||||
|
||||
---
|
||||
field caps on standard indices:
|
||||
- skip:
|
||||
version: " - 8.3.99"
|
||||
reason: metric params only on time series indexes introduced in 8.4.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: test_non_time_series
|
||||
fields: [ _tsid, metricset, k8s.pod.network.rx, k8s.pod.network.tx, k8s.pod.network.rx ]
|
||||
|
||||
- match: { fields.metricset.keyword.type: keyword }
|
||||
- match: { fields.metricset.keyword.searchable: true }
|
||||
- match: { fields.metricset.keyword.aggregatable: true }
|
||||
- is_false: fields.metricset.keyword.time_series_dimension
|
||||
- is_false: fields.metricset.keyword.non_dimension_indices
|
||||
- is_false: fields.metricset.keyword.indices
|
||||
- is_false: fields.metricset.keyword.non_searchable_indices
|
||||
- is_false: fields.metricset.keyword.non_aggregatable_indices
|
||||
|
||||
- is_false: fields._tsid # _tsid metadata field must not exist in non-time-series indices
|
||||
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.type: long }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.searchable: true }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.aggregatable: true }
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.time_series_metric
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.metric_conflicts_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.gauge
|
||||
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.type: long }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.searchable: true }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.aggregatable: true }
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.time_series_metric
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.non_aggregatable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.counter
|
||||
|
||||
|
||||
---
|
||||
field caps on mixed indices:
|
||||
- skip:
|
||||
version: " - 8.3.99"
|
||||
reason: metric params only on time series indexes introduced in 8.4.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: test_*
|
||||
fields: [ metricset, k8s.pod.availability_zone, k8s.pod.network.tx, k8s.pod.network.rx ]
|
||||
|
||||
- match: { fields.metricset.keyword.type: keyword }
|
||||
- match: { fields.metricset.keyword.searchable: true }
|
||||
- match: { fields.metricset.keyword.aggregatable: true }
|
||||
- is_false: fields.metricset.keyword.time_series_dimension
|
||||
- match: { fields.metricset.keyword.non_dimension_indices: [ "test_non_time_series" ] }
|
||||
- is_false: fields.metricset.keyword.indices
|
||||
- is_false: fields.metricset.keyword.non_searchable_indices
|
||||
- is_false: fields.metricset.keyword.non_aggregatable_indices
|
||||
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.type: long }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.searchable: true }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.aggregatable: true }
|
||||
- match: { fields.k8s\.pod\.network\.rx.long.metric_conflicts_indices: [ "test_non_time_series", "test_time_series" ] }
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.time_series_metric
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.indices
|
||||
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.type: long }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.searchable: true }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.aggregatable: true }
|
||||
- match: { fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices: [ "test_non_time_series", "test_time_series" ] }
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.time_series_metric
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.non_aggregatable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.tx.long.indices
|
||||
|
||||
|
|
@ -304,45 +304,6 @@ aggregate a tag:
|
|||
term:
|
||||
_tsid: wont't work
|
||||
|
||||
---
|
||||
field capabilities:
|
||||
- skip:
|
||||
version: " - 8.1.99"
|
||||
reason: tsdb indexing changed in 8.2.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: test
|
||||
fields: [k8s.pod.uid, k8s.pod.network.rx, k8s.pod.ip, metricset, _tsid]
|
||||
|
||||
- match: {fields.k8s\.pod\.uid.keyword.searchable: true}
|
||||
- match: {fields.k8s\.pod\.uid.keyword.aggregatable: true}
|
||||
- match: {fields.k8s\.pod\.uid.keyword.time_series_dimension: true}
|
||||
- is_false: fields.k8s\.pod\.uid.keyword.indices
|
||||
- is_false: fields.k8s\.pod\.uid.keyword.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.uid.keyword.non_aggregatable_indices
|
||||
- match: {fields.k8s\.pod\.network\.rx.long.searchable: true}
|
||||
- match: {fields.k8s\.pod\.network\.rx.long.aggregatable: true}
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
|
||||
- match: {fields.k8s\.pod\.ip.ip.searchable: true}
|
||||
- match: {fields.k8s\.pod\.ip.ip.aggregatable: true}
|
||||
- is_false: fields.k8s\.pod\.ip.ip.indices
|
||||
- is_false: fields.k8s\.pod\.ip.ip.non_searchable_indices
|
||||
- is_false: fields.k8s\.pod\.ip.ip.non_aggregatable_indices
|
||||
- match: {fields.metricset.keyword.searchable: true}
|
||||
- match: {fields.metricset.keyword.aggregatable: true}
|
||||
- match: {fields.metricset.keyword.time_series_dimension: true}
|
||||
- is_false: fields.metricset.keyword.indices
|
||||
- is_false: fields.metricset.keyword.non_searchable_indices
|
||||
- is_false: fields.metricset.keyword.non_aggregatable_indices
|
||||
- match: {fields._tsid._tsid.metadata_field: true}
|
||||
- match: {fields._tsid._tsid.searchable: false}
|
||||
- match: {fields._tsid._tsid.aggregatable: true}
|
||||
- is_false: fields._tsid._tsid.indices
|
||||
- is_false: fields._tsid._tsid.non_searchable_indices
|
||||
- is_false: fields._tsid._tsid.non_aggregatable_indices
|
||||
|
||||
---
|
||||
sort by tsid:
|
||||
|
|
|
|||
|
|
@ -24,7 +24,9 @@ import org.elasticsearch.common.breaker.CircuitBreaker;
|
|||
import org.elasticsearch.common.breaker.CircuitBreakingException;
|
||||
import org.elasticsearch.common.io.stream.StreamInput;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.index.IndexMode;
|
||||
import org.elasticsearch.index.IndexService;
|
||||
import org.elasticsearch.index.IndexSettings;
|
||||
import org.elasticsearch.index.mapper.DocumentParserContext;
|
||||
import org.elasticsearch.index.mapper.KeywordFieldMapper;
|
||||
import org.elasticsearch.index.mapper.MetadataFieldMapper;
|
||||
|
|
@ -119,7 +121,14 @@ public class FieldCapabilitiesIT extends ESIntegTestCase {
|
|||
.endObject()
|
||||
.endObject()
|
||||
.endObject();
|
||||
assertAcked(prepareCreate("old_index").setMapping(oldIndexMapping));
|
||||
|
||||
Settings settings = Settings.builder()
|
||||
.put(IndexSettings.MODE.getKey(), IndexMode.TIME_SERIES)
|
||||
.putList(IndexMetadata.INDEX_ROUTING_PATH.getKey(), List.of("some_dimension"))
|
||||
.put(IndexSettings.TIME_SERIES_START_TIME.getKey(), "2006-01-08T23:40:53.384Z")
|
||||
.put(IndexSettings.TIME_SERIES_END_TIME.getKey(), "2106-01-08T23:40:53.384Z")
|
||||
.build();
|
||||
assertAcked(prepareCreate("old_index").setSettings(settings).setMapping(oldIndexMapping));
|
||||
|
||||
XContentBuilder newIndexMapping = XContentFactory.jsonBuilder()
|
||||
.startObject()
|
||||
|
|
|
|||
|
|
@ -12,12 +12,14 @@ import org.elasticsearch.action.ActionListener;
|
|||
import org.elasticsearch.action.index.IndexRequestBuilder;
|
||||
import org.elasticsearch.action.support.IndicesOptions;
|
||||
import org.elasticsearch.action.support.ListenableActionFuture;
|
||||
import org.elasticsearch.cluster.metadata.IndexMetadata;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.time.DateFormatter;
|
||||
import org.elasticsearch.common.util.Maps;
|
||||
import org.elasticsearch.core.CheckedConsumer;
|
||||
import org.elasticsearch.core.TimeValue;
|
||||
import org.elasticsearch.core.Tuple;
|
||||
import org.elasticsearch.index.IndexMode;
|
||||
import org.elasticsearch.index.IndexSettings;
|
||||
import org.elasticsearch.index.mapper.DateFieldMapper;
|
||||
import org.elasticsearch.search.aggregations.MultiBucketConsumerService;
|
||||
|
|
@ -29,6 +31,7 @@ import org.elasticsearch.xcontent.json.JsonXContent;
|
|||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
|
@ -80,7 +83,9 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
dimensionMapping.accept(mapping);
|
||||
mapping.field("time_series_dimension", true);
|
||||
mapping.endObject();
|
||||
});
|
||||
// Add a keyword dimension as a routing parameter
|
||||
mapping.startObject("k").field("type", "keyword").field("time_series_dimension", true).endObject();
|
||||
}, List.of("k"));
|
||||
String[] dates = new String[] {
|
||||
"2021-01-01T00:10:00.000Z",
|
||||
"2021-01-01T00:11:00.000Z",
|
||||
|
|
@ -88,36 +93,37 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
"2021-01-01T00:20:00.000Z", };
|
||||
indexRandom(
|
||||
true,
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[0], "dim", d1, "v", 1, "m", 1)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[1], "dim", d1, "v", 2, "m", 2)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[2], "dim", d1, "v", 3, "m", 3)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[3], "dim", d1, "v", 4, "m", 4)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[1], "dim", d2, "v", 5, "m", 6))
|
||||
false,
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[0], "k", "k", "dim", d1, "v", 1, "m", 1)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[1], "k", "k", "dim", d1, "v", 2, "m", 2)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[2], "k", "k", "dim", d1, "v", 3, "m", 3)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[3], "k", "k", "dim", d1, "v", 4, "m", 4)),
|
||||
client().prepareIndex("tsdb").setSource(Map.of("@timestamp", dates[1], "k", "k", "dim", d2, "v", 5, "m", 6))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
latest(between(1, MultiBucketConsumerService.DEFAULT_MAX_BUCKETS), TimeValue.timeValueMinutes(5), dates[0]),
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1)), List.of(Map.entry(dates[0], 1.0)))
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[0], 1.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
latest(between(1, MultiBucketConsumerService.DEFAULT_MAX_BUCKETS), TimeValue.timeValueMinutes(10), dates[2]),
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1)), List.of(Map.entry(dates[2], 3.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2)), List.of(Map.entry(dates[2], 5.0)))
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[2], 3.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[2], 5.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
latest(between(1, MultiBucketConsumerService.DEFAULT_MAX_BUCKETS), TimeValue.timeValueMinutes(15), dates[3]),
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1)), List.of(Map.entry(dates[3], 4.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2)), List.of(Map.entry(dates[3], 5.0)))
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[3], 4.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[3], 5.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
range(between(1, MAX_RESULT_WINDOW), TimeValue.timeValueMinutes(15), TimeValue.timeValueMinutes(10), null, dates[3]),
|
||||
matchesMap().entry(
|
||||
Tuple.tuple("v", Map.of("dim", d1)),
|
||||
Tuple.tuple("v", Map.of("dim", d1, "k", "k")),
|
||||
List.of(Map.entry(dates[1], 2.0), Map.entry(dates[2], 3.0), Map.entry(dates[3], 4.0))
|
||||
).entry(Tuple.tuple("v", Map.of("dim", d2)), List.of(Map.entry(dates[1], 5.0)))
|
||||
).entry(Tuple.tuple("v", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[1], 5.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
|
|
@ -129,7 +135,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
dates[3]
|
||||
),
|
||||
matchesMap().entry(
|
||||
Tuple.tuple("v", Map.of("dim", d1)),
|
||||
Tuple.tuple("v", Map.of("dim", d1, "k", "k")),
|
||||
List.of(
|
||||
Map.entry("2021-01-01T00:11:00.000Z", 2.0),
|
||||
Map.entry("2021-01-01T00:12:00.000Z", 2.0),
|
||||
|
|
@ -144,7 +150,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
)
|
||||
)
|
||||
.entry(
|
||||
Tuple.tuple("v", Map.of("dim", d2)),
|
||||
Tuple.tuple("v", Map.of("dim", d2, "k", "k")),
|
||||
List.of(
|
||||
Map.entry("2021-01-01T00:11:00.000Z", 5.0),
|
||||
Map.entry("2021-01-01T00:12:00.000Z", 5.0),
|
||||
|
|
@ -168,8 +174,8 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
TimeValue.timeValueMinutes(15),
|
||||
FORMATTER.parse(dates[3]).getLong(INSTANT_SECONDS) * 1000
|
||||
),
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1)), List.of(Map.entry(dates[3], 4.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2)), List.of(Map.entry(dates[3], 5.0)))
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[3], 4.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[3], 5.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
|
|
@ -180,7 +186,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
TimeValue.timeValueMinutes(15),
|
||||
FORMATTER.parse(dates[3]).getLong(INSTANT_SECONDS) * 1000
|
||||
),
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d1)), List.of(Map.entry(dates[3], 4.0)))
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[3], 4.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
|
|
@ -191,7 +197,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
TimeValue.timeValueMinutes(15),
|
||||
FORMATTER.parse(dates[3]).getLong(INSTANT_SECONDS) * 1000
|
||||
),
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d2)), List.of(Map.entry(dates[3], 6.0)))
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[3], 6.0)))
|
||||
);
|
||||
|
||||
if ("a".equals(d1)) {
|
||||
|
|
@ -204,7 +210,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
TimeValue.timeValueMinutes(15),
|
||||
FORMATTER.parse(dates[3]).getLong(INSTANT_SECONDS) * 1000
|
||||
),
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d2)), List.of(Map.entry(dates[3], 6.0)))
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[3], 6.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
|
|
@ -215,7 +221,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
TimeValue.timeValueMinutes(15),
|
||||
FORMATTER.parse(dates[3]).getLong(INSTANT_SECONDS) * 1000
|
||||
),
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d1)), List.of(Map.entry(dates[3], 4.0)))
|
||||
matchesMap().entry(Tuple.tuple("m", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[3], 4.0)))
|
||||
);
|
||||
|
||||
assertMap(
|
||||
|
|
@ -226,8 +232,8 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
TimeValue.timeValueMinutes(15),
|
||||
FORMATTER.parse(dates[3]).getLong(INSTANT_SECONDS) * 1000
|
||||
),
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1)), List.of(Map.entry(dates[3], 4.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2)), List.of(Map.entry(dates[3], 5.0)))
|
||||
matchesMap().entry(Tuple.tuple("v", Map.of("dim", d1, "k", "k")), List.of(Map.entry(dates[3], 4.0)))
|
||||
.entry(Tuple.tuple("v", Map.of("dim", d2, "k", "k")), List.of(Map.entry(dates[3], 5.0)))
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -289,7 +295,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
expectedLatest = expectedLatest.entry(dimensions, List.of(Map.entry(max, value)));
|
||||
expectedValues = expectedValues.entry(dimensions, expectedValuesForTimeSeries);
|
||||
}
|
||||
indexRandom(true, docs);
|
||||
indexRandom(true, false, docs);
|
||||
assertMap(latest(iterationSize, TimeValue.timeValueMillis(maxMillis - minMillis), maxMillis), expectedLatest);
|
||||
assertMap(
|
||||
range(
|
||||
|
|
@ -328,7 +334,7 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
expectedValues.add(Map.entry(timestamp, v));
|
||||
docs.add(client().prepareIndex("tsdb").setSource(Map.of("@timestamp", timestamp, "dim", "dim", "v", v)));
|
||||
}
|
||||
indexRandom(true, docs);
|
||||
indexRandom(true, false, docs);
|
||||
assertMap(
|
||||
range(
|
||||
iterationBuckets,
|
||||
|
|
@ -356,10 +362,11 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
for (String d : keywordDimensions) {
|
||||
mapping.startObject(d).field("type", "keyword").field("time_series_dimension", true).endObject();
|
||||
}
|
||||
});
|
||||
}, Arrays.asList(keywordDimensions));
|
||||
}
|
||||
|
||||
private void createTsdbIndex(CheckedConsumer<XContentBuilder, IOException> dimensionMapping) throws IOException {
|
||||
private void createTsdbIndex(CheckedConsumer<XContentBuilder, IOException> dimensionMapping, List<String> routingDims)
|
||||
throws IOException {
|
||||
XContentBuilder mapping = JsonXContent.contentBuilder();
|
||||
mapping.startObject().startObject("properties");
|
||||
mapping.startObject("@timestamp").field("type", "date").endObject();
|
||||
|
|
@ -367,7 +374,14 @@ public class TimeSeriesMetricsIT extends ESIntegTestCase {
|
|||
mapping.startObject("m").field("type", "double").field("time_series_metric", "gauge").endObject();
|
||||
dimensionMapping.accept(mapping);
|
||||
mapping.endObject().endObject();
|
||||
client().admin().indices().prepareCreate("tsdb").setMapping(mapping).get();
|
||||
|
||||
Settings settings = Settings.builder()
|
||||
.put(IndexSettings.MODE.getKey(), IndexMode.TIME_SERIES)
|
||||
.putList(IndexMetadata.INDEX_ROUTING_PATH.getKey(), routingDims)
|
||||
.put(IndexSettings.TIME_SERIES_START_TIME.getKey(), "2000-01-08T23:40:53.384Z")
|
||||
.put(IndexSettings.TIME_SERIES_END_TIME.getKey(), "2106-01-08T23:40:53.384Z")
|
||||
.build();
|
||||
client().admin().indices().prepareCreate("tsdb").setSettings(settings).setMapping(mapping).get();
|
||||
}
|
||||
|
||||
private Map<Tuple<String, Map<String, Object>>, List<Map.Entry<String, Double>>> latest(
|
||||
|
|
|
|||
|
|
@ -110,18 +110,23 @@ class FieldCapabilitiesFetcher {
|
|||
boolean includeParentObjects = checkIncludeParents(filters);
|
||||
|
||||
Predicate<MappedFieldType> filter = buildFilter(indexFieldfilter, filters, types, context);
|
||||
boolean isTimeSeriesIndex = context.getIndexSettings().getTimestampBounds() != null;
|
||||
Map<String, IndexFieldCapabilities> responseMap = new HashMap<>();
|
||||
for (String field : fieldNames) {
|
||||
MappedFieldType ft = context.getFieldType(field);
|
||||
if (filter.test(ft)) {
|
||||
IndexFieldCapabilities fieldCap = new IndexFieldCapabilities(
|
||||
field,
|
||||
ft.familyTypeName(),
|
||||
// This is a nasty hack so that we expose aggregate_metric_double field,
|
||||
// when the index is a time series index and the field is marked as metric.
|
||||
// This code should be reverted once PR https://github.com/elastic/elasticsearch/pull/87849
|
||||
// is merged.
|
||||
isTimeSeriesIndex && ft.getMetricType() != null ? ft.typeName() : ft.familyTypeName(),
|
||||
context.isMetadataField(field),
|
||||
ft.isSearchable(),
|
||||
ft.isAggregatable(),
|
||||
ft.isDimension(),
|
||||
ft.getMetricType(),
|
||||
isTimeSeriesIndex ? ft.isDimension() : false,
|
||||
isTimeSeriesIndex ? ft.getMetricType() : null,
|
||||
ft.meta()
|
||||
);
|
||||
responseMap.put(field, fieldCap);
|
||||
|
|
|
|||
|
|
@ -52,7 +52,6 @@ public class IndexFieldCapabilities implements Writeable {
|
|||
TimeSeriesParams.MetricType metricType,
|
||||
Map<String, String> meta
|
||||
) {
|
||||
|
||||
this.name = name;
|
||||
this.type = type;
|
||||
this.isMetadatafield = isMetadatafield;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,159 @@
|
|||
setup:
|
||||
- skip:
|
||||
version: " - 8.4.99"
|
||||
reason: metric params only on time series indices introduced in 8.5.0
|
||||
|
||||
- do:
|
||||
indices.create:
|
||||
index: test_rollup
|
||||
body:
|
||||
settings:
|
||||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mode: time_series
|
||||
routing_path: [ metricset, k8s.pod.uid ]
|
||||
time_series:
|
||||
start_time: 2021-04-28T00:00:00Z
|
||||
end_time: 2021-04-29T00:00:00Z
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
type: date
|
||||
metricset:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
metric:
|
||||
type: aggregate_metric_double
|
||||
metrics: [ min, max, sum, value_count ]
|
||||
default_metric: max
|
||||
time_series_metric: gauge
|
||||
|
||||
- do:
|
||||
indices.create:
|
||||
index: test_time_series
|
||||
body:
|
||||
settings:
|
||||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mode: time_series
|
||||
routing_path: [ metricset, k8s.pod.uid ]
|
||||
time_series:
|
||||
start_time: 2021-04-28T00:00:00Z
|
||||
end_time: 2021-04-29T00:00:00Z
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
type: date
|
||||
metricset:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
metric:
|
||||
type: double
|
||||
time_series_metric: gauge
|
||||
|
||||
- do:
|
||||
indices.create:
|
||||
index: test_non_time_series
|
||||
body:
|
||||
settings:
|
||||
index:
|
||||
number_of_replicas: 0
|
||||
number_of_shards: 2
|
||||
mappings:
|
||||
properties:
|
||||
"@timestamp":
|
||||
type: date
|
||||
metricset:
|
||||
type: keyword
|
||||
time_series_dimension: true
|
||||
metric:
|
||||
type: double
|
||||
time_series_metric: gauge
|
||||
|
||||
|
||||
---
|
||||
# Test field_caps on a rollup index
|
||||
field caps on rollup indices:
|
||||
- skip:
|
||||
version: " - 8.4.99"
|
||||
reason: metric params only on time series indices introduced in 8.5.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: test_rollup
|
||||
fields: [ metric ]
|
||||
|
||||
- match: { fields.metric.aggregate_metric_double.type: aggregate_metric_double }
|
||||
- match: { fields.metric.aggregate_metric_double.searchable: true }
|
||||
- match: { fields.metric.aggregate_metric_double.aggregatable: true }
|
||||
- match: { fields.metric.aggregate_metric_double.time_series_metric: gauge }
|
||||
- is_false: fields.metric.aggregate_metric_double.indices
|
||||
- is_false: fields.metric.aggregate_metric_double.non_searchable_indices
|
||||
- is_false: fields.metric.aggregate_metric_double.non_aggregatable_indices
|
||||
- is_false: fields.metric.aggregate_metric_double.metric_conflicts_indices
|
||||
- is_false: fields.metric.double
|
||||
|
||||
|
||||
---
|
||||
# Test field_caps on time-series index (mix of raw and rollup indices)
|
||||
field caps on time series indices:
|
||||
- skip:
|
||||
version: " - 8.4.99"
|
||||
reason: metric params only on time series indices introduced in 8.5.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: [ test_time_series, test_rollup ]
|
||||
fields: [ metric ]
|
||||
|
||||
- match: { fields.metric.double.type: double }
|
||||
- match: { fields.metric.double.searchable: true }
|
||||
- match: { fields.metric.double.aggregatable: true }
|
||||
- match: { fields.metric.double.time_series_metric: gauge }
|
||||
- match: { fields.metric.double.indices: [ "test_time_series" ] }
|
||||
- is_false: fields.metric.double.non_searchable_indices
|
||||
- is_false: fields.metric.double.non_aggregatable_indices
|
||||
- is_false: fields.metric.double.metric_conflicts_indices
|
||||
|
||||
- match: { fields.metric.aggregate_metric_double.type: aggregate_metric_double }
|
||||
- match: { fields.metric.aggregate_metric_double.searchable: true }
|
||||
- match: { fields.metric.aggregate_metric_double.aggregatable: true }
|
||||
- match: { fields.metric.aggregate_metric_double.time_series_metric: gauge }
|
||||
- match: { fields.metric.aggregate_metric_double.indices: [ "test_rollup" ] }
|
||||
|
||||
- is_false: fields.metric.aggregate_metric_double.non_searchable_indices
|
||||
- is_false: fields.metric.aggregate_metric_double.non_aggregatable_indices
|
||||
- is_false: fields.metric.aggregate_metric_double.metric_conflicts_indices
|
||||
|
||||
---
|
||||
# Test field_caps on mixed standard and time-series (mix of raw and rollup) indices
|
||||
field caps on all indices:
|
||||
- skip:
|
||||
version: " - 8.4.99"
|
||||
reason: metric params only on time series indices introduced in 8.5.0
|
||||
|
||||
- do:
|
||||
field_caps:
|
||||
index: [ test_time_series, test_rollup, test_non_time_series ]
|
||||
fields: [ metric ]
|
||||
|
||||
- match: { fields.metric.double.type: double }
|
||||
- match: { fields.metric.double.searchable: true }
|
||||
- match: { fields.metric.double.aggregatable: true }
|
||||
- match: { fields.metric.double.indices: [ "test_non_time_series", "test_time_series" ] }
|
||||
- match: { fields.metric.double.metric_conflicts_indices: [ "test_non_time_series", "test_time_series" ] }
|
||||
- is_false: fields.metric.double.non_searchable_indices
|
||||
- is_false: fields.metric.double.non_aggregatable_indices
|
||||
- is_false: fields.metric.double.time_series_metric
|
||||
|
||||
- match: { fields.metric.aggregate_metric_double.type: aggregate_metric_double }
|
||||
- match: { fields.metric.aggregate_metric_double.searchable: true }
|
||||
- match: { fields.metric.aggregate_metric_double.aggregatable: true }
|
||||
- match: { fields.metric.aggregate_metric_double.time_series_metric: gauge }
|
||||
- match: { fields.metric.aggregate_metric_double.indices: [ "test_rollup" ] }
|
||||
|
||||
- is_false: fields.metric.aggregate_metric_double.non_searchable_indices
|
||||
- is_false: fields.metric.aggregate_metric_double.non_aggregatable_indices
|
||||
- is_false: fields.metric.aggregate_metric_double.metric_conflicts_indices
|
||||
Loading…
Add table
Reference in a new issue