If metrics that have the same timestamp and dimensions aren't grouped into the same document, ES will consider them to be a duplicate.
The _metric_names_hash field will be set by the OTel ES exporter.
As it's mapped as a time_series_dimensions, it creates a different _tsid for documents with different sets of metrics.
The tradeoff is that if the composition of the metrics grouping changes over time, a different _tsid will be created.
That has an impact on the rate aggregation for counters.
The default field limit of 1000 seems undersized, especially for the default data stream.
Each distinct attribute/dimension name and metric name count towards that limit
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
Historical features are now trivially true on v9 - so we can remove the features, and the check.
Historical features do not affect cluster state, so this has no compatibility restrictions.
Here we check for the existence of a `host.name` field in index sort settings
when the index mode is `logsdb` and decide to inject the field in the mapping
depending on whether it exists or not. By default `host.name` is required for
sorting in LogsDB. This reduces the chances for errors at mapping or template
composition time as a result of injecting the `host.name` field only if strictly
required. A user who wants to override index sort settings without including
a `host.name` field would be able to do so without finding an additional
`host.name` field in the mappings (injected automatically). If users override the
sort settings and a `host.name` field is not included we don't need
to inject such field since sorting does not require it anymore.
As a result of this change we have the following:
* the user does not provide any index sorting configuration: we are responsible for injecting the default sort fields and their mapping (for `logsdb`)
* the user explicitly provides non-empty index sorting configuration: the user is also responsible for providing correct mappings and we do not modify index sorting or mappings
Note also that all sort settings `index.sort.*` are `final` which means doing this
check once, when mappings are merged at template composition time, is enough.
* Hide aggregated metrics generated for the APM UI
* Update 30_aggregated_metrics_tests.yml
* Review feedback
- Introduced templates for 1 minute aggregations
- Moved dynamic templates `ecs_ip` and `all_strings_to_keywords` into a dedicated file and now I pull the file in instead of repeating them
- Introduced `metrics-[x]m.otel@custom`
- Added tests with terms aggregation that assert by default 1 bucket (only 1m) with metricsset.interval, and with allowing hidden indices it's 3 buckets (1m, 10m, 60m)
* Update 30_aggregated_metrics_tests.yml
Simplify - no need to separate tests for the 3bucket queries.
* Rename metrics-otel-fixed@mappings to ecs-tsdb@mappings
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add tests for exceptions
* Adapt aliases and move into a separate file
* Clarify description in alias files
* Handling stacktrace
* Move error aliases to logs-otel@mappings.yaml
* Update resources.yaml
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
<!-- Thank you for your interest in and contributing to Elasticsearch!
There are a few simple things to check before submitting your pull
request that can help with the review process. You should delete these
items from your submission, but they are here to help bring them to your
attention. --> Fix error `failed to parse: class java.lang.String cannot
be cast to class java.util.List (java.lang.String and java.util.List are
in module java.base of loader 'bootstrap')`
* Make xpack.otel_data.registry.enabled default to true
* Update build.gradle
* Disable otel-data plugin for tests where it's not relevant
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Closes https://github.com/elastic/elasticsearch/issues/110387
Having this in now affords us not having to introduce version checks in
the ES exporter later. We can simply use the same serialization logic
for metric attributes as we do for other signals. This also enables us
to properly map `*.ip` fields to the ip field type as ip fields
containing a list of IPs are not converted to a comma-separated list.
* Add YamlTemplateRegistry and OtelIndexTemplateRegistry with resource YAML files
* Fix traces-otel template
* Adding first yml tests
* Base APMIndexTemplateRegistry on YamlTemplateRegistry
* Update OTelPlugin.java
* Update APMIndexTemplateRegistry.java
* Update YamlIngestPipelineConfig.java
* Adding traces tests
* Update x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
* Add mapper-version
* Fix code-style
* Rename `status.status_code` to `status.code`
* Update otel@mappings.yaml
Revert back to date due to missing support in ES|QL for date_nanos
* Move dynamic_templates to metrics@mappings in core
* Run gradlew :x-pack:plugin:core:spotlessApply
* Update x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
* Update 20_metic_tests.yml
Workaround for TSDB timestamp issue: we push a custom template with higher priority and set time_series.start_time.
* Update CODEOWNERS
Adding obs-ds-intake-services as owner of the new otel-data plugin.
Since we had some changes, also updating the owner of apm-data to the same team.
* Change dynamic: strict to false
* Skip "Reject invalid top level field" test
* Update 20_metic_tests.yml
* Add boolean as dimension test (skipping it for now)
* Add booleans_to_keywords and enable corresponding test
* Remove processor.event top level mapping
Reason: for metrics and logs we can rely on the name of the datastream. For spans vs. transactions there are other fields we can use.
* Remove booleans_to_keywords
Because booleans are supported now as dimension on TSDB
* Add alias service.language.name -> telemetry.sdk.language
* cleanup
* Update README.md
* Update README.md
* Update docs/changelog/111091.yaml
* Move traces@settings and traces@mappings to core
* Update traces-otel@mappings.yaml
* Review feedback
* Adapt `match` style in tests
* Update docs/changelog/111091.yaml
* Apply suggestions from code review
Co-authored-by: Vishal Raj <vishal.raj@elastic.co>
* Update x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
* Changing trace_flags to long
Related discussion: https://github.com/elastic/elasticsearch/pull/111091#discussion_r1706698491
* Remove trace_flags
see: https://github.com/elastic/opentelemetry-dev/pull/368#pullrequestreview-2229633970
* Apply suggestions from code review
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
* Review feedback
* Add store_array_source for span links
* Define constant `data_stream.type` in `template.yaml`s
* Create package-info.java
* Move ecs-tsdb@mappings to index template
Add test to verify that @custom template can add dynamic templates with a higher precedence
* Update metrics@mappings.json
Remove summary_gauge and summary_counter since they are covered by summary_metrics
* Move clusterService.getClusterSettings().addSettingsUpdateConsumer to registry
* Fix code-style
* Update x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
* Enable logsdb
* Update traces@settings.json
No lifecycle needed for OTel at this point
---------
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
Co-authored-by: Vishal Raj <vishal.raj@elastic.co>
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Felix Barnsteiner <felix.barnsteiner@elastic.co>
