[[filebeat-modules]]

== Working with {filebeat} Modules

{filebeat} comes packaged with pre-built
{filebeat-ref}/filebeat-modules.html[modules] that contain the configurations
needed to collect, parse, enrich, and visualize data from various log file
formats. Each {filebeat} module consists of one or more filesets that contain
ingest node pipelines, {es} templates, {filebeat} input configurations, and
{kib} dashboards.

You can use {filebeat} modules with {ls}, but you need to do some extra setup.
The simplest approach is to <<use-ingest-pipelines,set up and use the ingest
pipelines>> provided by {filebeat}. If the ingest pipelines don't meet your
requirements, you can
<<logstash-config-for-filebeat-modules,create {ls} configurations>> to use
instead of the ingest pipelines.

Either approach allows you to use the configurations, index templates, and
dashboards available with {filebeat} modules, as long as you maintain the
field structure expected by the index and dashboards.

[[use-ingest-pipelines]]
=== Use ingest pipelines for parsing

When you use {filebeat} modules with {ls}, you can use the ingest pipelines
provided by {filebeat} to parse the data. You need to load the pipelines
into {es} and configure {ls} to use them.

*To load the ingest pipelines:*

On the system where {filebeat} is installed, run the `setup` command with the
`--pipelines` option specified to load ingest pipelines for specific modules.
For example, the following command loads ingest pipelines for the system and
nginx modules:

[source,shell]
-----
filebeat setup --pipelines --modules nginx,system
-----

A connection to {es} is required for this setup step because {filebeat} needs to
load the ingest pipelines into {es}. If necessary, you can temporarily disable
your configured output and enable the {es} output before running the command.

*To configure {ls} to use the pipelines:*

On the system where {ls} is installed, create a {ls} pipeline configuration
that reads from a {ls} input, such as {beats} or Kafka, and sends events to an
{es} output. Set the `pipeline` option in the {es} output to
`%{[@metadata][pipeline]}` to use the ingest pipelines that you loaded
previously.

Here's an example configuration that reads data from the Beats input and uses
{filebeat} ingest pipelines to parse data collected by modules:

[source,yaml]
-----
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" <1>
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "secret"
    }
  }
}
-----
<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting
configures {ls} to select the correct ingest pipeline based on metadata
passed in the event.

See the {filebeat} {filebeat-ref}/filebeat-modules-overview.html[Modules]
documentation for more information about setting up and running modules.

For a full example, see <<use-filebeat-modules-kafka>>.

[[logstash-config-for-filebeat-modules]]
=== Use {ls} pipelines for parsing

The examples in this section show how to build {ls} pipeline configurations that
replace the ingest pipelines provided with {filebeat} modules. The pipelines
take the data collected by {filebeat} modules, parse it into fields expected by
the {filebeat} index, and send the fields to {es} so that you can visualize the
data in the pre-built dashboards provided by {filebeat}.

This approach is more time consuming than using the existing ingest pipelines to
parse the data, but it gives you more control over how the data is processed.
By writing your own pipeline configurations, you can do additional processing,
such as dropping fields, after the fields are extracted, or you can move your
load from {es} ingest nodes to {ls} nodes.

Before deciding to replaced the ingest pipelines with {ls} configurations,
read <<use-ingest-pipelines>>.

Here are some examples that show how to implement {ls} configurations to replace
ingest pipelines:

* <<parsing-apache2>>
* <<parsing-mysql>>
* <<parsing-nginx>>
* <<parsing-system>>

TIP: {ls} provides an <<ingest-converter,ingest pipeline conversion tool>>
to help you migrate ingest pipeline definitions to {ls} configs. The tool does
not currently support all the processors that are available for ingest node, but
it's a good starting point.

[[parsing-apache2]]
==== Apache 2 Logs

The {ls} pipeline configuration in this example shows how to ship and parse
access and error logs collected by the
{filebeat-ref}/filebeat-module-apache.html[`apache` {filebeat} module].

[source,json]
----------------------------------------------------------------------------
include::filebeat_modules/apache2/pipeline.conf[]
----------------------------------------------------------------------------


[[parsing-mysql]]
==== MySQL Logs

The {ls} pipeline configuration in this example shows how to ship and parse
error and slowlog logs collected by the
{filebeat-ref}/filebeat-module-mysql.html[`mysql` {filebeat} module].

[source,json]
----------------------------------------------------------------------------
include::filebeat_modules/mysql/pipeline.conf[]
----------------------------------------------------------------------------


[[parsing-nginx]]
==== Nginx Logs

The {ls} pipeline configuration in this example shows how to ship and parse
access and error logs collected by the
{filebeat-ref}/filebeat-module-nginx.html[`nginx` {filebeat} module].

[source,json]
----------------------------------------------------------------------------
include::filebeat_modules/nginx/pipeline.conf[]
----------------------------------------------------------------------------


[[parsing-system]]
==== System Logs

The {ls} pipeline configuration in this example shows how to ship and parse
system logs collected by the
{filebeat-ref}/filebeat-module-system.html[`system` {filebeat} module].

[source,json]
----------------------------------------------------------------------------
include::filebeat_modules/system/pipeline.conf[]
----------------------------------------------------------------------------

include::fb-ls-kafka-example.asciidoc[]
