Improve docs about using Filebeat modules with Logstash (#10438)

* Improve docs about using Filebeat modules with Logstash * Add fixes from review Fixes #10482
2025-04-24 22:57:16 -04:00 · 2019-02-15 12:39:28 -08:00 · 2019-02-15 12:39:28 -08:00 · ca6ad64c7f
commit ca6ad64c7f
parent 1c88792d49
2 changed files with 255 additions and 112 deletions
--- a/docs/static/fb-ls-kafka-example.asciidoc
+++ b/docs/static/fb-ls-kafka-example.asciidoc
@ -0,0 +1,151 @@
+[[use-filebeat-modules-kafka]]
+=== Example: Set up {filebeat} modules to work with Kafka and {ls}
+
+This section shows how to set up {filebeat}
+{filebeat-ref}/filebeat-modules-overview.html[modules] to work with {ls} when
+you are using Kafka in between {filebeat} and {ls} in your publishing pipeline.
+The main goal of this example is to show how to load ingest pipelines from
+{filebeat} and use them with {ls}.
+
+The examples in this section show simple configurations with topic names hard
+coded. For a full list of configuration options, see documentation about
+configuring the <<plugins-inputs-kafka,Kafka input plugin>>. Also see
+{filebeat-ref}/kafka-output.html[Configure the Kafka output] in the _{filebeat}
+Reference_.
+
+==== Set up and run {filebeat}
+
+. If you haven't already set up the {filebeat} index template and sample {kib}
+dashboards, run the {filebeat} `setup` command to do that now: 
+
+[source,shell]
+----------------------------------------------------------------------
+filebeat -e setup
+----------------------------------------------------------------------
+
+The `-e` flag is optional and sends output to standard error instead of syslog.
+
+A connection to {es} and {kib} is required for this one-time setup
+step because {filebeat} needs to create the index template in {es} and
+load the sample dashboards into {kib}. For more information about configuring
+the connection to {es}, see the Filebeat modules
+{filebeat-ref}/filebeat-modules-quickstart.html[quick start].
+
+After the template and dashboards are loaded, you'll see the message `INFO
+{kib} dashboards successfully loaded. Loaded dashboards`.
+
+. Run the `modules enable` command to enable the modules that you want to run.
+For example:
+
+[source,shell]
+----------------------------------------------------------------------
+filebeat modules enable system
+----------------------------------------------------------------------
+
+You can further configure the module by editing the config file under the
+{filebeat} `modules.d` directory. For example, if the log files are not in the
+location expected by the module, you can set the `var.paths` option.
+
+. Run the `setup` command with the `--pipelines` and `--modules` options
+specified to load ingest pipelines for the modules you've enabled. This step
+also requires a connection to {es}. If you want use a {ls} pipeline instead of
+ingest node to parse the data, skip this step.
+
+[source,shell]
+----------------------------------------------------------------------
+filebeat setup --pipelines --modules system
+----------------------------------------------------------------------
+
+. Configure {filebeat} to send log lines to Kafka. To do this, in the
+filebeat.yml+ config file, disable the {es} output by commenting it out, and
+enable the Kafka output. For example:
+
+[source,yaml]
+-----
+#output.elasticsearch:
+  #hosts: ["localhost:9200"]
+output.kafka:
+  hosts: ["kafka:9092"]
+  topic: "filebeat"
+  codec.json:
+    pretty: false
+-----
+
+. Start {filebeat}. For example:
+
+[source,shell]
+----------------------------------------------------------------------
+filebeat -e
+----------------------------------------------------------------------
+
+{filebeat} will attempt to send messages to {ls} and continue until {ls} is
+available to receive them.
+
+NOTE: Depending on how you've installed {filebeat}, you might see errors
+related to file ownership or permissions when you try to run {filebeat} modules.
+See {beats-ref}/config-file-permissions.html[Config File Ownership and Permissions]
+in the _Beats Platform Reference_ if you encounter errors related to file
+ownership or permissions.
+
+
+==== Create and start the {ls} pipeline
+
+. On the system where {ls} is installed, create a {ls} pipeline configuration
+that reads from a Kafka input and sends events to an {es} output:
+
+--
+[source,yaml]
+-----
+input {
+  kafka {
+    bootstrap_servers => "myhost:9092"
+    topics => ["filebeat"]
+    codec => json
+  }
+}
+
+output {
+  if [@metadata][pipeline] {
+    elasticsearch {
+      hosts => "https://myEShost:9200"
+      manage_template => false
+      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
+      pipeline => "%{[@metadata][pipeline]}" <1>
+      user => "elastic"
+      password => "secret"
+    }
+  } else {
+    elasticsearch {
+      hosts => "https://myEShost:9200"
+      manage_template => false
+      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
+      user => "elastic"
+      password => "secret"
+    }
+  }
+}
+-----
+<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting
+configures {ls} to select the correct ingest pipeline based on metadata
+passed in the event.
+
+If you want use a {ls} pipeline instead of ingest node to parse the data, see
+the `filter` and `output` settings in the examples under
+<<logstash-config-for-filebeat-modules>>.
+--
+
+. Start {ls}, passing in the pipeline configuration file you just defined. For
+example:
+
+[source,shell]
+----------------------------------------------------------------------
+bin/logstash -f mypipeline.conf
+----------------------------------------------------------------------
+
+{ls} should start a pipeline and begin receiving events from the Kafka input.
+
+==== Visualize the data
+
+To visualize the data in {kib}, launch the {kib} web interface by pointing your
+browser to port 5601. For example, http://127.0.0.1:5601[http://127.0.0.1:5601].
+Click *Dashboards* then view the {filebeat} dashboards.
--- a/docs/static/filebeat-modules.asciidoc
+++ b/docs/static/filebeat-modules.asciidoc
@ -1,143 +1,134 @@
 [[filebeat-modules]]

-== Working with Filebeat Modules
+== Working with {filebeat} Modules

-Filebeat comes packaged with pre-built  {filebeat-ref}/filebeat-modules.html[modules]
-that contain the configurations needed to collect, parse, enrich, and visualize
-data from various log file formats. Each Filebeat module consists of one or more
-filesets that contain ingest node pipelines, Elasticsearch templates, Filebeat
-prospector configurations, and Kibana dashboards.
+{filebeat} comes packaged with pre-built
+{filebeat-ref}/filebeat-modules.html[modules] that contain the configurations
+needed to collect, parse, enrich, and visualize data from various log file
+formats. Each {filebeat} module consists of one or more filesets that contain
+ingest node pipelines, {es} templates, {filebeat} input configurations, and
+{kib} dashboards.

-Filebeat modules are a great way to get started, but you might find that ingest
-pipelines don't offer the processing power that you require. If that's the case,
-you'll need to use Logstash.
+You can use {filebeat} modules with {ls}, but you need to do some extra setup.
+The simplest approach is to <<use-ingest-pipelines,set up and use the ingest
+pipelines>> provided by {filebeat}. If the ingest pipelines don't meet your
+requirements, you can
+<<logstash-config-for-filebeat-modules,create {ls} configurations>> to use
+instead of the ingest pipelines.

-[float]
-[[graduating-to-Logstash]]
-=== Using Logstash instead of Ingest Node
+Either approach allows you to use the configurations, index templates, and
+dashboards available with {filebeat} modules, as long as you maintain the
+field structure expected by the index and dashboards.

-Logstash provides an <<ingest-converter,ingest pipeline conversion tool>>
-to help you migrate ingest pipeline definitions to Logstash configs. However,
-the tool does not currently support all the processors that are available for
-ingest node.
+[[use-ingest-pipelines]]
+=== Use ingest pipelines for parsing

-You can follow the steps in this section to build and run Logstash
-configurations that parse the data collected by Filebeat modules. Then you'll be
-able to use the same dashboards available with Filebeat to visualize your data
-in Kibana.
+When you use {filebeat} modules with {ls}, you can use the ingest pipelines
+provided by {filebeat} to parse the data. You need to load the pipelines
+into {es} and configure {ls} to use them.

-[float]
-==== Create and start the Logstash pipeline
+*To load the ingest pipelines:*

-. Create a Logstash pipeline configuration that reads from the Beats input and
-parses the events.
-+
-See <<logstash-config-for-filebeat-modules>> for detailed examples.
+On the system where {filebeat} is installed, run the `setup` command with the
+`--pipelines` option specified to load ingest pipelines for specific modules.
+For example, the following command loads ingest pipelines for the system and
+nginx modules:

-. Start Logstash, passing in the pipeline configuration file that parses the
-log. For example:
-+
 [source,shell]
----------------------------------------------------------------------
-bin/logstash -f mypipeline.conf
----------------------------------------------------------------------
-+
-You'll see the following message when Logstash is running and listening for
-input from Beats: 
-+
-[source,shell]
----------------------------------------------------------------------
-[2017-10-13T00:01:15,413][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"127.0.0.1:5044"}
-[2017-10-13T00:01:15,443][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"main"}
----------------------------------------------------------------------
+-----
+filebeat setup --pipelines --modules nginx,system
+-----

+A connection to {es} is required for this setup step because {filebeat} needs to
+load the ingest pipelines into {es}. If necessary, you can temporarily disable
+your configured output and enable the {es} output before running the command.

-The Logstash pipeline is now ready to receive events from Filebeat. Next, you
-set up and run Filebeat.
+*To configure {ls} to use the pipelines:*

-[float]
-==== Set up and run Filebeat
+On the system where {ls} is installed, create a {ls} pipeline configuration
+that reads from a {ls} input, such as {beats} or Kafka, and sends events to an
+{es} output. Set the `pipeline` option in the {es} output to
+`%{[@metadata][pipeline]}` to use the ingest pipelines that you loaded
+previously.

-. If you haven't already set up the Filebeat index template and sample Kibana
-dashboards, run the Filebeat `setup` command to do that now: 
-+
-[source,shell]
----------------------------------------------------------------------
-./filebeat -e setup
----------------------------------------------------------------------
-+
-The `-e` flag is optional and sends output to standard error instead of syslog.
-+
-A connection to Elasticsearch and Kibana is required for this one-time setup
-step because Filebeat needs to create the index template in Elasticsearch and
-load the sample dashboards into Kibana. 
-+
-After the template and dashboards are loaded, you'll see the message `INFO
-Kibana dashboards successfully loaded. Loaded dashboards`.
+Here's an example configuration that reads data from the Beats input and uses
+{filebeat} ingest pipelines to parse data collected by modules:

-. Configure Filebeat to send log lines to Logstash. To do this, in the
-+filebeat.yml+ config file, disable the Elasticsearch output, and enable the
-Logstash output. For example:
-+
 [source,yaml]
 -----
-#output.elasticsearch:
-  #hosts: ["localhost:9200"]
-output.logstash:
-  hosts: ["localhost:5044"]
+input {
+  beats {
+    port => 5044
+  }
+}
+
+output {
+  if [@metadata][pipeline] {
+    elasticsearch {
+      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
+      manage_template => false
+      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
+      pipeline => "%{[@metadata][pipeline]}" <1>
+      user => "elastic"
+      password => "secret"
+    }
+  } else {
+    elasticsearch {
+      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
+      manage_template => false
+      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
+      user => "elastic"
+      password => "secret"
+    }
+  }
+}
 -----
+<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting
+configures {ls} to select the correct ingest pipeline based on metadata
+passed in the event.

-. Run the `modules enable` command to enable the modules that you want to run.
-For example:
-+
-[source,shell]
----------------------------------------------------------------------
-./filebeat modules enable nginx
----------------------------------------------------------------------
-+
-You can further configure the module by editing the config file under the
-Filebeat `modules.d` directory. For example, if the log files are not in the
-location expected by the module, you can set the `var.paths` option.
+See the {filebeat} {filebeat-ref}/filebeat-modules-overview.html[Modules]
+documentation for more information about setting up and running modules.

-. Start Filebeat. For example, to start Filebeat in the foreground, use:
-+
-[source,shell]
----------------------------------------------------------------------
-./filebeat -e
----------------------------------------------------------------------
-+
-NOTE: Depending on how you've installed Filebeat, you might see errors
-related to file ownership or permissions when you try to run Filebeat modules.
-See {beats-ref}/config-file-permissions.html[Config File Ownership and Permissions]
-in the _Beats Platform Reference_ if you encounter errors related to file
-ownership or permissions.
-+
-See {filebeat-ref}/filebeat-starting.html[Starting Filebeat] for more info.
-
-[float]
-==== Visualize the data
-
-To visualize the data in Kibana, launch the Kibana web interface by pointing
-your browser to port 5601. For example,
-http://127.0.0.1:5601[http://127.0.0.1:5601]. 
+For a full example, see <<use-filebeat-modules-kafka>>.

 [[logstash-config-for-filebeat-modules]]
-=== Configuration Examples
+=== Use {ls} pipelines for parsing

-The examples in this section show you how to build Logstash pipelines that parse
-data sent collected by Filebeat modules:
+The examples in this section show how to build {ls} pipeline configurations that
+replace the ingest pipelines provided with {filebeat} modules. The pipelines
+take the data collected by {filebeat} modules, parse it into fields expected by
+the {filebeat} index, and send the fields to {es} so that you can visualize the
+data in the pre-built dashboards provided by {filebeat}.
+
+This approach is more time consuming than using the existing ingest pipelines to
+parse the data, but it gives you more control over how the data is processed.
+By writing your own pipeline configurations, you can do additional processing,
+such as dropping fields, after the fields are extracted, or you can move your
+load from {es} ingest nodes to {ls} nodes.
+
+Before deciding to replaced the ingest pipelines with {ls} configurations,
+read <<use-ingest-pipelines>>.
+
+Here are some examples that show how to implement {ls} configurations to replace
+ingest pipelines:

 * <<parsing-apache2>>
 * <<parsing-mysql>>
 * <<parsing-nginx>>
 * <<parsing-system>>

+TIP: {ls} provides an <<ingest-converter,ingest pipeline conversion tool>>
+to help you migrate ingest pipeline definitions to {ls} configs. The tool does
+not currently support all the processors that are available for ingest node, but
+it's a good starting point.
+
 [[parsing-apache2]]
 ==== Apache 2 Logs

-The Logstash pipeline configuration in this example shows how to ship and parse
+The {ls} pipeline configuration in this example shows how to ship and parse
 access and error logs collected by the
-{filebeat-ref}/filebeat-module-apache.html[`apache` Filebeat module].
+{filebeat-ref}/filebeat-module-apache.html[`apache` {filebeat} module].

 [source,json]
 ----------------------------------------------------------------------------
@ -148,9 +139,9 @@ include::filebeat_modules/apache2/pipeline.conf[]
 [[parsing-mysql]]
 ==== MySQL Logs

-The Logstash pipeline configuration in this example shows how to ship and parse
+The {ls} pipeline configuration in this example shows how to ship and parse
 error and slowlog logs collected by the
-{filebeat-ref}/filebeat-module-mysql.html[`mysql` Filebeat module].
+{filebeat-ref}/filebeat-module-mysql.html[`mysql` {filebeat} module].

 [source,json]
 ----------------------------------------------------------------------------
@ -161,9 +152,9 @@ include::filebeat_modules/mysql/pipeline.conf[]
 [[parsing-nginx]]
 ==== Nginx Logs

-The Logstash pipeline configuration in this example shows how to ship and parse
+The {ls} pipeline configuration in this example shows how to ship and parse
 access and error logs collected by the
-{filebeat-ref}/filebeat-module-nginx.html[`nginx` Filebeat module].
+{filebeat-ref}/filebeat-module-nginx.html[`nginx` {filebeat} module].

 [source,json]
 ----------------------------------------------------------------------------
@ -174,12 +165,13 @@ include::filebeat_modules/nginx/pipeline.conf[]
 [[parsing-system]]
 ==== System Logs

-The Logstash pipeline configuration in this example shows how to ship and parse
+The {ls} pipeline configuration in this example shows how to ship and parse
 system logs collected by the
-{filebeat-ref}/filebeat-module-system.html[`system` Filebeat module].
+{filebeat-ref}/filebeat-module-system.html[`system` {filebeat} module].

 [source,json]
 ----------------------------------------------------------------------------
 include::filebeat_modules/system/pipeline.conf[]
 ----------------------------------------------------------------------------

+include::fb-ls-kafka-example.asciidoc[]