mirror of
https://github.com/elastic/logstash.git
synced 2025-06-28 09:46:03 -04:00
177 lines
6.4 KiB
Text
177 lines
6.4 KiB
Text
[[filebeat-modules]]
|
|
|
|
== Working with {filebeat} Modules
|
|
|
|
{filebeat} comes packaged with pre-built
|
|
{filebeat-ref}/filebeat-modules.html[modules] that contain the configurations
|
|
needed to collect, parse, enrich, and visualize data from various log file
|
|
formats. Each {filebeat} module consists of one or more filesets that contain
|
|
ingest node pipelines, {es} templates, {filebeat} input configurations, and
|
|
{kib} dashboards.
|
|
|
|
You can use {filebeat} modules with {ls}, but you need to do some extra setup.
|
|
The simplest approach is to <<use-ingest-pipelines,set up and use the ingest
|
|
pipelines>> provided by {filebeat}. If the ingest pipelines don't meet your
|
|
requirements, you can
|
|
<<logstash-config-for-filebeat-modules,create {ls} configurations>> to use
|
|
instead of the ingest pipelines.
|
|
|
|
Either approach allows you to use the configurations, index templates, and
|
|
dashboards available with {filebeat} modules, as long as you maintain the
|
|
field structure expected by the index and dashboards.
|
|
|
|
[[use-ingest-pipelines]]
|
|
=== Use ingest pipelines for parsing
|
|
|
|
When you use {filebeat} modules with {ls}, you can use the ingest pipelines
|
|
provided by {filebeat} to parse the data. You need to load the pipelines
|
|
into {es} and configure {ls} to use them.
|
|
|
|
*To load the ingest pipelines:*
|
|
|
|
On the system where {filebeat} is installed, run the `setup` command with the
|
|
`--pipelines` option specified to load ingest pipelines for specific modules.
|
|
For example, the following command loads ingest pipelines for the system and
|
|
nginx modules:
|
|
|
|
[source,shell]
|
|
-----
|
|
filebeat setup --pipelines --modules nginx,system
|
|
-----
|
|
|
|
A connection to {es} is required for this setup step because {filebeat} needs to
|
|
load the ingest pipelines into {es}. If necessary, you can temporarily disable
|
|
your configured output and enable the {es} output before running the command.
|
|
|
|
*To configure {ls} to use the pipelines:*
|
|
|
|
On the system where {ls} is installed, create a {ls} pipeline configuration
|
|
that reads from a {ls} input, such as {beats} or Kafka, and sends events to an
|
|
{es} output. Set the `pipeline` option in the {es} output to
|
|
`%{[@metadata][pipeline]}` to use the ingest pipelines that you loaded
|
|
previously.
|
|
|
|
Here's an example configuration that reads data from the Beats input and uses
|
|
{filebeat} ingest pipelines to parse data collected by modules:
|
|
|
|
[source,yaml]
|
|
-----
|
|
input {
|
|
beats {
|
|
port => 5044
|
|
}
|
|
}
|
|
|
|
output {
|
|
if [@metadata][pipeline] {
|
|
elasticsearch {
|
|
hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
|
|
manage_template => false
|
|
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
|
pipeline => "%{[@metadata][pipeline]}" <1>
|
|
user => "elastic"
|
|
password => "secret"
|
|
}
|
|
} else {
|
|
elasticsearch {
|
|
hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
|
|
manage_template => false
|
|
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
|
user => "elastic"
|
|
password => "secret"
|
|
}
|
|
}
|
|
}
|
|
-----
|
|
<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting
|
|
configures {ls} to select the correct ingest pipeline based on metadata
|
|
passed in the event.
|
|
|
|
See the {filebeat} {filebeat-ref}/filebeat-modules-overview.html[Modules]
|
|
documentation for more information about setting up and running modules.
|
|
|
|
For a full example, see <<use-filebeat-modules-kafka>>.
|
|
|
|
[[logstash-config-for-filebeat-modules]]
|
|
=== Use {ls} pipelines for parsing
|
|
|
|
The examples in this section show how to build {ls} pipeline configurations that
|
|
replace the ingest pipelines provided with {filebeat} modules. The pipelines
|
|
take the data collected by {filebeat} modules, parse it into fields expected by
|
|
the {filebeat} index, and send the fields to {es} so that you can visualize the
|
|
data in the pre-built dashboards provided by {filebeat}.
|
|
|
|
This approach is more time consuming than using the existing ingest pipelines to
|
|
parse the data, but it gives you more control over how the data is processed.
|
|
By writing your own pipeline configurations, you can do additional processing,
|
|
such as dropping fields, after the fields are extracted, or you can move your
|
|
load from {es} ingest nodes to {ls} nodes.
|
|
|
|
Before deciding to replaced the ingest pipelines with {ls} configurations,
|
|
read <<use-ingest-pipelines>>.
|
|
|
|
Here are some examples that show how to implement {ls} configurations to replace
|
|
ingest pipelines:
|
|
|
|
* <<parsing-apache2>>
|
|
* <<parsing-mysql>>
|
|
* <<parsing-nginx>>
|
|
* <<parsing-system>>
|
|
|
|
TIP: {ls} provides an <<ingest-converter,ingest pipeline conversion tool>>
|
|
to help you migrate ingest pipeline definitions to {ls} configs. The tool does
|
|
not currently support all the processors that are available for ingest node, but
|
|
it's a good starting point.
|
|
|
|
[[parsing-apache2]]
|
|
==== Apache 2 Logs
|
|
|
|
The {ls} pipeline configuration in this example shows how to ship and parse
|
|
access and error logs collected by the
|
|
{filebeat-ref}/filebeat-module-apache.html[`apache` {filebeat} module].
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/apache2/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
|
|
[[parsing-mysql]]
|
|
==== MySQL Logs
|
|
|
|
The {ls} pipeline configuration in this example shows how to ship and parse
|
|
error and slowlog logs collected by the
|
|
{filebeat-ref}/filebeat-module-mysql.html[`mysql` {filebeat} module].
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/mysql/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
|
|
[[parsing-nginx]]
|
|
==== Nginx Logs
|
|
|
|
The {ls} pipeline configuration in this example shows how to ship and parse
|
|
access and error logs collected by the
|
|
{filebeat-ref}/filebeat-module-nginx.html[`nginx` {filebeat} module].
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/nginx/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
|
|
[[parsing-system]]
|
|
==== System Logs
|
|
|
|
The {ls} pipeline configuration in this example shows how to ship and parse
|
|
system logs collected by the
|
|
{filebeat-ref}/filebeat-module-system.html[`system` {filebeat} module].
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/system/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
include::fb-ls-kafka-example.asciidoc[]
|