mirror of
https://github.com/elastic/logstash.git
synced 2025-06-28 17:53:28 -04:00
95 lines
4.4 KiB
Text
95 lines
4.4 KiB
Text
[[ea-integrations]]
|
|
== Using {ls} with Elastic {integrations}
|
|
|
|
You can take advantage of the extensive, built-in capabilities of Elastic {integrations}--such as managing data collection, transformation, and visualization--and then use {ls} for additional data processing and output options.
|
|
{ls} can further expand capabilities for use cases where you need additional processing, or if you need your data delivered to multiple destinations.
|
|
|
|
[discrete]
|
|
[[integrations-value]]
|
|
=== Elastic {integrations}: ingesting to visualizing
|
|
|
|
https://docs.elastic.co/integrations[Elastic {integrations}] provide quick, end-to-end solutions for:
|
|
|
|
* ingesting data from a variety of data sources,
|
|
* ensuring compliance with the {ecs-ref}/index.html[Elastic Common Schema (ECS)],
|
|
* getting the data into the {stack}, and
|
|
* visualizing it with purpose-built dashboards.
|
|
|
|
{integrations} are available for https://docs.elastic.co/integrations/all_integrations[popular services and platforms], such as Nginx, AWS, and MongoDB, as well as many generic input types like log files.
|
|
Each integration includes pre-packaged assets to help reduce the time between ingest and insights.
|
|
|
|
To see available integrations, go to the {kib} home page, and click **Add {integrations}**.
|
|
You can use the query bar to search for integrations you may want to use.
|
|
When you find an integration for your data source, the UI walks you through adding and configuring it.
|
|
|
|
[discrete]
|
|
[[integrations-and-ls]]
|
|
=== Extend {integrations} with {ls}
|
|
|
|
Logstash can run the ingest pipeline component of your Elastic integration when you use the Logstash `filter-elastic_integration` plugin in your {ls} pipeline.
|
|
|
|
Adding the `filter-elastic_integration` plugin as the _first_ filter plugin keeps the pipeline's behavior as close as possible to the behavior you'd expect if the bytes were processed by the integration in {es}.
|
|
The more you modify an event before calling the `elastic_integration` filter, the higher the risk that the modifications will have meaningful effect in how the event is transformed.
|
|
|
|
.How to
|
|
|
|
****
|
|
Create a {ls} pipeline that uses the <<plugins-inputs-elastic_agent,elastic_agent input>> plugin, and the <<plugins-filters-elastic_integration,elastic_integration filter>> plugin as the _first_ filter in your {ls} pipeline.
|
|
You can add more filters for additional processing, but they must come after the `logstash-filter-elastic_integration` plugin in your configuration.
|
|
Add one or more output plugins to complete your pipeline.
|
|
****
|
|
|
|
**Sample pipeline configuration**
|
|
|
|
[source,ruby]
|
|
-----
|
|
input {
|
|
elastic_agent {
|
|
port => 5044
|
|
}
|
|
}
|
|
|
|
filter {
|
|
elastic_integration{ <1>
|
|
cloud_id => "<cloud id>"
|
|
cloud_auth => "<your_cloud-auth"
|
|
}
|
|
|
|
translate { <2>
|
|
source => "[http][host]"
|
|
target => "[@metadata][tenant]"
|
|
dictionary_path => "/etc/conf.d/logstash/tenants.yml"
|
|
}
|
|
}
|
|
|
|
output { <3>
|
|
if [@metadata][tenant] == "tenant01" {
|
|
elasticsearch {
|
|
cloud_id => "<cloud id>"
|
|
api_key => "<api key>"
|
|
}
|
|
} else if [@metadata][tenant] == "tenant02" {
|
|
elasticsearch {
|
|
cloud_id => "<cloud id>"
|
|
api_key => "<api key>"
|
|
}
|
|
}
|
|
}
|
|
-----
|
|
|
|
<1> Use `filter-elastic_integration` as the first filter in your pipeline
|
|
<2> You can use additional filters as long as they follow `filter-elastic_integration`
|
|
<3> Sample config to output data to multiple destinations
|
|
|
|
[discrete]
|
|
[[es-tips]]
|
|
==== Using `filter-elastic_integration` with `output-elasticsearch`
|
|
|
|
Elastic {integrations} are designed to work with {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams[data streams] and {logstash-ref}/plugins-outputs-elasticsearch.html#_compatibility_with_the_elastic_common_schema_ecs[ECS-compatible] output.
|
|
Be sure that these features are enabled in the {logstash-ref}/plugins-outputs-elasticsearch.html[`output-elasticsearch`] plugin.
|
|
|
|
* Set {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream[`data-stream`] to `true`. +
|
|
(Check out {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams[Data streams] for additional data streams settings.)
|
|
* Set {logstash-ref}/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ecs_compatibility[`ecs-compatibility`] to `v1` or `v8`.
|
|
|
|
Check out the {logstash-ref}/plugins-outputs-elasticsearch.html[`output-elasticsearch` plugin] docs for additional settings.
|