Reverts https://github.com/elastic/elasticsearch/pull/112465
After a week and a half trying the team CODEOWNERS, there are some
problems with not being able to (easily?) identify the direct, personal
review requests from team requests in notifications.
Unless it's solved, and given that these CODEOWNERS are more like an
"auto-documentation" tool right now, we are better reverting it.
* Add YamlTemplateRegistry and OtelIndexTemplateRegistry with resource YAML files
* Fix traces-otel template
* Adding first yml tests
* Base APMIndexTemplateRegistry on YamlTemplateRegistry
* Update OTelPlugin.java
* Update APMIndexTemplateRegistry.java
* Update YamlIngestPipelineConfig.java
* Adding traces tests
* Update x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
* Add mapper-version
* Fix code-style
* Rename `status.status_code` to `status.code`
* Update otel@mappings.yaml
Revert back to date due to missing support in ES|QL for date_nanos
* Move dynamic_templates to metrics@mappings in core
* Run gradlew :x-pack:plugin:core:spotlessApply
* Update x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
* Update 20_metic_tests.yml
Workaround for TSDB timestamp issue: we push a custom template with higher priority and set time_series.start_time.
* Update CODEOWNERS
Adding obs-ds-intake-services as owner of the new otel-data plugin.
Since we had some changes, also updating the owner of apm-data to the same team.
* Change dynamic: strict to false
* Skip "Reject invalid top level field" test
* Update 20_metic_tests.yml
* Add boolean as dimension test (skipping it for now)
* Add booleans_to_keywords and enable corresponding test
* Remove processor.event top level mapping
Reason: for metrics and logs we can rely on the name of the datastream. For spans vs. transactions there are other fields we can use.
* Remove booleans_to_keywords
Because booleans are supported now as dimension on TSDB
* Add alias service.language.name -> telemetry.sdk.language
* cleanup
* Update README.md
* Update README.md
* Update docs/changelog/111091.yaml
* Move traces@settings and traces@mappings to core
* Update traces-otel@mappings.yaml
* Review feedback
* Adapt `match` style in tests
* Update docs/changelog/111091.yaml
* Apply suggestions from code review
Co-authored-by: Vishal Raj <vishal.raj@elastic.co>
* Update x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
* Changing trace_flags to long
Related discussion: https://github.com/elastic/elasticsearch/pull/111091#discussion_r1706698491
* Remove trace_flags
see: https://github.com/elastic/opentelemetry-dev/pull/368#pullrequestreview-2229633970
* Apply suggestions from code review
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
* Review feedback
* Add store_array_source for span links
* Define constant `data_stream.type` in `template.yaml`s
* Create package-info.java
* Move ecs-tsdb@mappings to index template
Add test to verify that @custom template can add dynamic templates with a higher precedence
* Update metrics@mappings.json
Remove summary_gauge and summary_counter since they are covered by summary_metrics
* Move clusterService.getClusterSettings().addSettingsUpdateConsumer to registry
* Fix code-style
* Update x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
* Enable logsdb
* Update traces@settings.json
No lifecycle needed for OTel at this point
---------
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
Co-authored-by: Vishal Raj <vishal.raj@elastic.co>
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Felix Barnsteiner <felix.barnsteiner@elastic.co>
* [logstash bridge]: stableapi bootstrap
Adds a new `logstash-bridge` project in `/libs` that exports api-stable
wrappers for the elasticsearch-internal types that Logstash's Elastic
Integration Filter relies on to provide ingest pipeline execution inside
of Logstash.
These bridge classes prevent Elasticsearch-internal refactorings from
breaking the Logstash-owned project.
* Update docs/changelog/108171.yaml
* rename StableAPI -> StableBridgeAPI
This PR makes the ES security code-owners of privilege-related source
files. This covers named cluster and index privileges. We may extend
code-ownership to other security-related files, but this feels like a
reasonable start.
This commit updates the Gradle wrapper validation action to the latest
release and pins it to that version by hash per our company best
practices for GitHub Workflows & Actions.
Cross-cutting refactoring often pings the apm-server team
and we're not really the best placed for reviewing these
changes. The apm-data plugin is structured so the apm-server
team can work on the templates and ingest pipelines without
touching any Java code, so reduce the scope to just the
assets and YAML REST tests.
* x-pack/plugin/apm: introduce x-pack-apm plugin
* Dependency fix and tests
* Restore addition to ESRestTestCase
* Replace IngestPipelineConfig instantiation
* Update DataStreamUpgradeRestIT to expect logs-apm.error
* Adding rollover functionality
* Extend basic rollover funtionality tests
* Start adding integration test
* Hide rollup data streams
* Apm ingest fixes
- Map transaction.duration.us independently of span.duration.us,
fix ingest pipeline. We might consider aliasing later.
- Set event.ingested
- Set processor.event, using constant_keyword where possible
* Fix error.grouping_name script
* Only set event.ingested for traces-apm.sampled
* Enabling APMRolloverIT
* Spotless...
* Assertion change
* Wait a bit before assertBusy
* Adjust template and pipeline names to convention
See https://github.com/elastic/elasticsearch/issues/96267#issuecomment-1662129512
- Index templates have a "@template" suffix
- Component templates are split into settings and mappings,
and have a @settings and @mappings suffix respectively
- Ingest pipelines generally have a @pipeline suffix,
except for one special case where we use @default-pipeline
We no longer have a component template for every data stream, rather we
have reusable component templates where it makes sense. For example, we
now have metrics-apm@settings and metrics-apm@mappings, which are
included by all APM metrics data streams.
We now set both default_pipeline and final_pipeline in the index
templates, preventing users from overriding them completely. The
default pipeline is always apm@default-pipeline, which performs
some rerouting of legacy data, and invokes user-defined @custom
pipelines based on the data_stream.type and data_stream.dataset
fields. The final pipeline just performs built-in processing.
* No need for manual cluster change events anymore
* Fix test
* Lower template priority to 110
Certain index templates installed by Fleet are given
a priority of 150, so lower the priority to avoid
conflicts.
See a18c68f7ac/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts (L58)
* Add logging
* Remove explicit timeout for rollovers
* Add _meta to component templates too
* Increase template priorities to 140
This goes closer to the Fleet-installed template priorities,
allowing for more builtin templates to be introduced without
affecting APM.
* Test multiple index template upgrades with rollovers
* Enfore correct component template versions
* Rename to x-pack-apm-data, default to disabled
Because there's already an "apm" plugin,
we haven't yet solved all issues with rollovers,
and the templates need some more love.
* Fix checkstyle
* Fix more checkstyle
* Fix even more linting issues
* Adjust test to disabled by default
* remove todo
* Fixing test
* Refactor leftovers
* Fix constant keyword mapping conflict
* Fix after merge
* Comment fixes
* Rename ApmIngestPipelineConfig to YamlIngestPipelineConfig
* Always return a registry
* spotless, what else
* Add rollover integration test to core
* Remove APMRolloverIT
* Refactor in renamed components
---------
Co-authored-by: eyalkoren <41850454+eyalkoren@users.noreply.github.com>
The ES security team currently manually tags the Kibana security team
for reviews on changes to the Kibana system role definition. This PR
aims to automate via codeowners.
This PR updates the pull request template to refer to main instead
of master as well as it updates some URLs.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Currently we use GitHub issue templates containing markdown for each
template. This was an improvement over the single issue template before
that, and there are several bits of information that we ask for, but
filling out the template is cumbersome because it is completely
markdown, and users often leave bits of markdown comments meant for them
to read in the resulting issue.
This commit converts the existing issue templates to use the new github
issue template forms. These are configured through yml, and present an
html form to the user that is customized for each issue type.
The Elasticsearch project has a security policy set on it, which causes
a "Report a security vulnerability" section in the new issue template
with a link to the security policy. This commit removes the older link
to sending an email to the security list, since it is confusing whether
a user should look at the policy (which references the security email)
or directly click the fake "Open" button.
