diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 201559a5d..e58ea8531 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -55,6 +55,10 @@ include::static/advanced-pipeline.asciidoc[] :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/life-of-an-event.asciidoc include::static/life-of-an-event.asciidoc[] +// Elastic Common Schema (ECS) +:editurl!: +include::static/ecs-compatibility.asciidoc[] + // Processing details :edit_url!: diff --git a/docs/static/ecs-compatibility.asciidoc b/docs/static/ecs-compatibility.asciidoc new file mode 100644 index 000000000..d41da363c --- /dev/null +++ b/docs/static/ecs-compatibility.asciidoc @@ -0,0 +1,96 @@ +[[ecs-ls]] +=== ECS in Logstash + +// LS8 will ship with ECS v8, but until ECS v8 is ready we rely on ECS v1 as an approximation. +:ls8-ecs-major-version: v1 + +The {ecs-ref}/index.html[Elastic Common Schema (ECS)] is an open source specification, developed with support from the Elastic user community. +ECS defines a common set of fields to be used for storing event data, such as logs and metrics, in {es}. +With ECS, users can normalize event data to better analyze, visualize, and correlate the data represented in their events. + +[[ecs-compatibility]] +==== ECS compatibility + +Many plugins implement an ECS-compatibility mode, which causes them to produce and manipulate events in a manner that is compatible with the Elastic Common Schema (ECS). + +Any plugin that supports this mode will also have an `ecs_compatibility` option, which allows you to configure which mode the individual plugin instance should operate in. +If left unspecified for an individual plugin, the pipeline's `pipeline.ecs_compatibility` setting will be observed. +This allows you to configure plugins to use ECS -- or to lock in your existing non-ECS behavior -- in advance of your {ls} 8.0 upgrade where ECS will be enabled by default. + +ECS Compatibility modes do not prevent you from explicitly configuring a plugin in a manner that conflicts with ECS. +Instead, they ensure that _implicit_ configuration avoids conflicts. + +NOTE: Until {ls} 8.0 and the final 7.x are released, any value for `pipeline.ecs_compatibility` other than `disabled` are considered BETA and unsupported. + As we continue to release plugins with ECS Compatibility modes, having this flag set will cause even patch-level upgrades to _automatically_ consume breaking changes in the upgraded plugins, changing the shape of data the plugin produces. + +ifeval::["{ls8-ecs-major-version}"!="v8"] +NOTE: ECS `v8` will be the default in the GA release of {ls} 8.0.0, and will be available at or before the final minor release of {ls} 7. + We expect the scope of breaking changes in ECS 8 to be limited, but progress toward the definition of ECS v8 can be tracked https://github.com/elastic/ecs/issues/839[here]. +endif::[] + +[[ecs-configuration]] +===== Configuring ECS + +ECS will be on by default in {ls} 8, but you can begin using it now by configuring individual plugins with `ecs_compatibility`. +You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn't change when you perform your next major upgrade. + +====== Specific plugin instance + +Use a plugin's `ecs_compatibility` option to override the default value on the plugin instance. + +For example, if you want a specific instance of the GeoIP Filter to behave without ECS compatibility, you can adjust its definition in your pipeline without affecting any other plugin instances. + +[source,text] +----- +filter { + geoip { + source => "[host][ip]" + ecs_compatibility => disabled + } +} +----- + +Alternatively, if you had a UDP input with a CEF codec, and wanted both to use an ECS mode while still running {ls} 7, you can adjust their definitions to specify the major version of ECS to use. + +[source,text,subs="attributes"] +----- +input { + udp { + port => 1234 + ecs_compatibility => {ls8-ecs-major-version} + codec => cef { + ecs_compatibility => {ls8-ecs-major-version} + } + } +} +----- + +[[ecs-configuration-pipeline]] +====== All plugins in a given pipeline + +If you wish to provide a specific default value for `ecs_compatibility` to _all_ plugins in a pipeline, you can do so with the `pipeline.ecs_compatibility` setting in your pipeline definition in `config/pipelines.yml` or Central Management. +This setting will be used unless overridden by a specific plugin instance. +If unspecified for an individual pipeline, the global value will be used. + +[source,yaml,subs="attributes"] +----- +- pipeline.id: my-legacy-pipeline + path.config: "/etc/path/to/legacy-pipeline.config" + pipeline.ecs_compatibility: disabled +- pipeline.id: my-ecs-pipeline + path.config: "/etc/path/to/ecs-pipeline.config" + pipeline.ecs_compatibility: {ls8-ecs-major-version} +----- + +NOTE: Until the final minor release of {ls} 7 that coincides with the General Availability of {ls} 8.0.0, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported because it will produce undesireable consequences when performing upgrades. + As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause the affected plugins to silently and automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces. + +[[ecs-configuration-all]] +====== All plugins in all pipelines + +Similarly, you can set the default value for the whole {ls} process by setting the `pipeline.ecs_compatibility` value in `config/logstash.yml`. + +[source,yaml] +----- +pipeline.ecs_compatibility: disabled +-----