mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
Improve docs about using Filebeat modules with Logstash (#10438)
* Improve docs about using Filebeat modules with Logstash * Add fixes from review Fixes #10482
This commit is contained in:
DeDe Morton
parent
1c88792d49
commit
ca6ad64c7f
2 changed files with 255 additions and 112 deletions
151
docs/static/fb-ls-kafka-example.asciidoc
vendored
Normal file
151
docs/static/fb-ls-kafka-example.asciidoc
vendored
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
[[use-filebeat-modules-kafka]]
|
||||
=== Example: Set up {filebeat} modules to work with Kafka and {ls}
|
||||
|
||||
This section shows how to set up {filebeat}
|
||||
{filebeat-ref}/filebeat-modules-overview.html[modules] to work with {ls} when
|
||||
you are using Kafka in between {filebeat} and {ls} in your publishing pipeline.
|
||||
The main goal of this example is to show how to load ingest pipelines from
|
||||
{filebeat} and use them with {ls}.
|
||||
|
||||
The examples in this section show simple configurations with topic names hard
|
||||
coded. For a full list of configuration options, see documentation about
|
||||
configuring the <<plugins-inputs-kafka,Kafka input plugin>>. Also see
|
||||
{filebeat-ref}/kafka-output.html[Configure the Kafka output] in the _{filebeat}
|
||||
Reference_.
|
||||
|
||||
==== Set up and run {filebeat}
|
||||
|
||||
. If you haven't already set up the {filebeat} index template and sample {kib}
|
||||
dashboards, run the {filebeat} `setup` command to do that now:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
filebeat -e setup
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
The `-e` flag is optional and sends output to standard error instead of syslog.
|
||||
+
|
||||
A connection to {es} and {kib} is required for this one-time setup
|
||||
step because {filebeat} needs to create the index template in {es} and
|
||||
load the sample dashboards into {kib}. For more information about configuring
|
||||
the connection to {es}, see the Filebeat modules
|
||||
{filebeat-ref}/filebeat-modules-quickstart.html[quick start].
|
||||
+
|
||||
After the template and dashboards are loaded, you'll see the message `INFO
|
||||
{kib} dashboards successfully loaded. Loaded dashboards`.
|
||||
|
||||
. Run the `modules enable` command to enable the modules that you want to run.
|
||||
For example:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
filebeat modules enable system
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
You can further configure the module by editing the config file under the
|
||||
{filebeat} `modules.d` directory. For example, if the log files are not in the
|
||||
location expected by the module, you can set the `var.paths` option.
|
||||
|
||||
. Run the `setup` command with the `--pipelines` and `--modules` options
|
||||
specified to load ingest pipelines for the modules you've enabled. This step
|
||||
also requires a connection to {es}. If you want use a {ls} pipeline instead of
|
||||
ingest node to parse the data, skip this step.
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
filebeat setup --pipelines --modules system
|
||||
----------------------------------------------------------------------
|
||||
|
||||
. Configure {filebeat} to send log lines to Kafka. To do this, in the
|
||||
+filebeat.yml+ config file, disable the {es} output by commenting it out, and
|
||||
enable the Kafka output. For example:
|
||||
+
|
||||
[source,yaml]
|
||||
-----
|
||||
#output.elasticsearch:
|
||||
#hosts: ["localhost:9200"]
|
||||
output.kafka:
|
||||
hosts: ["kafka:9092"]
|
||||
topic: "filebeat"
|
||||
codec.json:
|
||||
pretty: false
|
||||
-----
|
||||
|
||||
. Start {filebeat}. For example:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
filebeat -e
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
{filebeat} will attempt to send messages to {ls} and continue until {ls} is
|
||||
available to receive them.
|
||||
+
|
||||
NOTE: Depending on how you've installed {filebeat}, you might see errors
|
||||
related to file ownership or permissions when you try to run {filebeat} modules.
|
||||
See {beats-ref}/config-file-permissions.html[Config File Ownership and Permissions]
|
||||
in the _Beats Platform Reference_ if you encounter errors related to file
|
||||
ownership or permissions.
|
||||
|
||||
|
||||
==== Create and start the {ls} pipeline
|
||||
|
||||
. On the system where {ls} is installed, create a {ls} pipeline configuration
|
||||
that reads from a Kafka input and sends events to an {es} output:
|
||||
+
|
||||
--
|
||||
[source,yaml]
|
||||
-----
|
||||
input {
|
||||
kafka {
|
||||
bootstrap_servers => "myhost:9092"
|
||||
topics => ["filebeat"]
|
||||
codec => json
|
||||
}
|
||||
}
|
||||
|
||||
output {
|
||||
if [@metadata][pipeline] {
|
||||
elasticsearch {
|
||||
hosts => "https://myEShost:9200"
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||||
pipeline => "%{[@metadata][pipeline]}" <1>
|
||||
user => "elastic"
|
||||
password => "secret"
|
||||
}
|
||||
} else {
|
||||
elasticsearch {
|
||||
hosts => "https://myEShost:9200"
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||||
user => "elastic"
|
||||
password => "secret"
|
||||
}
|
||||
}
|
||||
}
|
||||
-----
|
||||
<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting
|
||||
configures {ls} to select the correct ingest pipeline based on metadata
|
||||
passed in the event.
|
||||
|
||||
If you want use a {ls} pipeline instead of ingest node to parse the data, see
|
||||
the `filter` and `output` settings in the examples under
|
||||
<<logstash-config-for-filebeat-modules>>.
|
||||
--
|
||||
|
||||
. Start {ls}, passing in the pipeline configuration file you just defined. For
|
||||
example:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
bin/logstash -f mypipeline.conf
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
{ls} should start a pipeline and begin receiving events from the Kafka input.
|
||||
|
||||
==== Visualize the data
|
||||
|
||||
To visualize the data in {kib}, launch the {kib} web interface by pointing your
|
||||
browser to port 5601. For example, http://127.0.0.1:5601[http://127.0.0.1:5601].
|
||||
Click *Dashboards* then view the {filebeat} dashboards.
|
||||
216
docs/static/filebeat-modules.asciidoc
vendored
216
docs/static/filebeat-modules.asciidoc
vendored
|
|
@ -1,143 +1,134 @@
|
|||
[[filebeat-modules]]
|
||||
|
||||
== Working with Filebeat Modules
|
||||
== Working with {filebeat} Modules
|
||||
|
||||
Filebeat comes packaged with pre-built {filebeat-ref}/filebeat-modules.html[modules]
|
||||
that contain the configurations needed to collect, parse, enrich, and visualize
|
||||
data from various log file formats. Each Filebeat module consists of one or more
|
||||
filesets that contain ingest node pipelines, Elasticsearch templates, Filebeat
|
||||
prospector configurations, and Kibana dashboards.
|
||||
{filebeat} comes packaged with pre-built
|
||||
{filebeat-ref}/filebeat-modules.html[modules] that contain the configurations
|
||||
needed to collect, parse, enrich, and visualize data from various log file
|
||||
formats. Each {filebeat} module consists of one or more filesets that contain
|
||||
ingest node pipelines, {es} templates, {filebeat} input configurations, and
|
||||
{kib} dashboards.
|
||||
|
||||
Filebeat modules are a great way to get started, but you might find that ingest
|
||||
pipelines don't offer the processing power that you require. If that's the case,
|
||||
you'll need to use Logstash.
|
||||
You can use {filebeat} modules with {ls}, but you need to do some extra setup.
|
||||
The simplest approach is to <<use-ingest-pipelines,set up and use the ingest
|
||||
pipelines>> provided by {filebeat}. If the ingest pipelines don't meet your
|
||||
requirements, you can
|
||||
<<logstash-config-for-filebeat-modules,create {ls} configurations>> to use
|
||||
instead of the ingest pipelines.
|
||||
|
||||
[float]
|
||||
[[graduating-to-Logstash]]
|
||||
=== Using Logstash instead of Ingest Node
|
||||
Either approach allows you to use the configurations, index templates, and
|
||||
dashboards available with {filebeat} modules, as long as you maintain the
|
||||
field structure expected by the index and dashboards.
|
||||
|
||||
Logstash provides an <<ingest-converter,ingest pipeline conversion tool>>
|
||||
to help you migrate ingest pipeline definitions to Logstash configs. However,
|
||||
the tool does not currently support all the processors that are available for
|
||||
ingest node.
|
||||
[[use-ingest-pipelines]]
|
||||
=== Use ingest pipelines for parsing
|
||||
|
||||
You can follow the steps in this section to build and run Logstash
|
||||
configurations that parse the data collected by Filebeat modules. Then you'll be
|
||||
able to use the same dashboards available with Filebeat to visualize your data
|
||||
in Kibana.
|
||||
When you use {filebeat} modules with {ls}, you can use the ingest pipelines
|
||||
provided by {filebeat} to parse the data. You need to load the pipelines
|
||||
into {es} and configure {ls} to use them.
|
||||
|
||||
[float]
|
||||
==== Create and start the Logstash pipeline
|
||||
*To load the ingest pipelines:*
|
||||
|
||||
. Create a Logstash pipeline configuration that reads from the Beats input and
|
||||
parses the events.
|
||||
+
|
||||
See <<logstash-config-for-filebeat-modules>> for detailed examples.
|
||||
On the system where {filebeat} is installed, run the `setup` command with the
|
||||
`--pipelines` option specified to load ingest pipelines for specific modules.
|
||||
For example, the following command loads ingest pipelines for the system and
|
||||
nginx modules:
|
||||
|
||||
. Start Logstash, passing in the pipeline configuration file that parses the
|
||||
log. For example:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
bin/logstash -f mypipeline.conf
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
You'll see the following message when Logstash is running and listening for
|
||||
input from Beats:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
[2017-10-13T00:01:15,413][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"127.0.0.1:5044"}
|
||||
[2017-10-13T00:01:15,443][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
|
||||
----------------------------------------------------------------------
|
||||
-----
|
||||
filebeat setup --pipelines --modules nginx,system
|
||||
-----
|
||||
|
||||
A connection to {es} is required for this setup step because {filebeat} needs to
|
||||
load the ingest pipelines into {es}. If necessary, you can temporarily disable
|
||||
your configured output and enable the {es} output before running the command.
|
||||
|
||||
The Logstash pipeline is now ready to receive events from Filebeat. Next, you
|
||||
set up and run Filebeat.
|
||||
*To configure {ls} to use the pipelines:*
|
||||
|
||||
[float]
|
||||
==== Set up and run Filebeat
|
||||
On the system where {ls} is installed, create a {ls} pipeline configuration
|
||||
that reads from a {ls} input, such as {beats} or Kafka, and sends events to an
|
||||
{es} output. Set the `pipeline` option in the {es} output to
|
||||
`%{[@metadata][pipeline]}` to use the ingest pipelines that you loaded
|
||||
previously.
|
||||
|
||||
. If you haven't already set up the Filebeat index template and sample Kibana
|
||||
dashboards, run the Filebeat `setup` command to do that now:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
./filebeat -e setup
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
The `-e` flag is optional and sends output to standard error instead of syslog.
|
||||
+
|
||||
A connection to Elasticsearch and Kibana is required for this one-time setup
|
||||
step because Filebeat needs to create the index template in Elasticsearch and
|
||||
load the sample dashboards into Kibana.
|
||||
+
|
||||
After the template and dashboards are loaded, you'll see the message `INFO
|
||||
Kibana dashboards successfully loaded. Loaded dashboards`.
|
||||
Here's an example configuration that reads data from the Beats input and uses
|
||||
{filebeat} ingest pipelines to parse data collected by modules:
|
||||
|
||||
. Configure Filebeat to send log lines to Logstash. To do this, in the
|
||||
+filebeat.yml+ config file, disable the Elasticsearch output, and enable the
|
||||
Logstash output. For example:
|
||||
+
|
||||
[source,yaml]
|
||||
-----
|
||||
#output.elasticsearch:
|
||||
#hosts: ["localhost:9200"]
|
||||
output.logstash:
|
||||
hosts: ["localhost:5044"]
|
||||
input {
|
||||
beats {
|
||||
port => 5044
|
||||
}
|
||||
}
|
||||
|
||||
output {
|
||||
if [@metadata][pipeline] {
|
||||
elasticsearch {
|
||||
hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||||
pipeline => "%{[@metadata][pipeline]}" <1>
|
||||
user => "elastic"
|
||||
password => "secret"
|
||||
}
|
||||
} else {
|
||||
elasticsearch {
|
||||
hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||||
user => "elastic"
|
||||
password => "secret"
|
||||
}
|
||||
}
|
||||
}
|
||||
-----
|
||||
<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting
|
||||
configures {ls} to select the correct ingest pipeline based on metadata
|
||||
passed in the event.
|
||||
|
||||
. Run the `modules enable` command to enable the modules that you want to run.
|
||||
For example:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
./filebeat modules enable nginx
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
You can further configure the module by editing the config file under the
|
||||
Filebeat `modules.d` directory. For example, if the log files are not in the
|
||||
location expected by the module, you can set the `var.paths` option.
|
||||
See the {filebeat} {filebeat-ref}/filebeat-modules-overview.html[Modules]
|
||||
documentation for more information about setting up and running modules.
|
||||
|
||||
. Start Filebeat. For example, to start Filebeat in the foreground, use:
|
||||
+
|
||||
[source,shell]
|
||||
----------------------------------------------------------------------
|
||||
./filebeat -e
|
||||
----------------------------------------------------------------------
|
||||
+
|
||||
NOTE: Depending on how you've installed Filebeat, you might see errors
|
||||
related to file ownership or permissions when you try to run Filebeat modules.
|
||||
See {beats-ref}/config-file-permissions.html[Config File Ownership and Permissions]
|
||||
in the _Beats Platform Reference_ if you encounter errors related to file
|
||||
ownership or permissions.
|
||||
+
|
||||
See {filebeat-ref}/filebeat-starting.html[Starting Filebeat] for more info.
|
||||
|
||||
[float]
|
||||
==== Visualize the data
|
||||
|
||||
To visualize the data in Kibana, launch the Kibana web interface by pointing
|
||||
your browser to port 5601. For example,
|
||||
http://127.0.0.1:5601[http://127.0.0.1:5601].
|
||||
For a full example, see <<use-filebeat-modules-kafka>>.
|
||||
|
||||
[[logstash-config-for-filebeat-modules]]
|
||||
=== Configuration Examples
|
||||
=== Use {ls} pipelines for parsing
|
||||
|
||||
The examples in this section show you how to build Logstash pipelines that parse
|
||||
data sent collected by Filebeat modules:
|
||||
The examples in this section show how to build {ls} pipeline configurations that
|
||||
replace the ingest pipelines provided with {filebeat} modules. The pipelines
|
||||
take the data collected by {filebeat} modules, parse it into fields expected by
|
||||
the {filebeat} index, and send the fields to {es} so that you can visualize the
|
||||
data in the pre-built dashboards provided by {filebeat}.
|
||||
|
||||
This approach is more time consuming than using the existing ingest pipelines to
|
||||
parse the data, but it gives you more control over how the data is processed.
|
||||
By writing your own pipeline configurations, you can do additional processing,
|
||||
such as dropping fields, after the fields are extracted, or you can move your
|
||||
load from {es} ingest nodes to {ls} nodes.
|
||||
|
||||
Before deciding to replaced the ingest pipelines with {ls} configurations,
|
||||
read <<use-ingest-pipelines>>.
|
||||
|
||||
Here are some examples that show how to implement {ls} configurations to replace
|
||||
ingest pipelines:
|
||||
|
||||
* <<parsing-apache2>>
|
||||
* <<parsing-mysql>>
|
||||
* <<parsing-nginx>>
|
||||
* <<parsing-system>>
|
||||
|
||||
TIP: {ls} provides an <<ingest-converter,ingest pipeline conversion tool>>
|
||||
to help you migrate ingest pipeline definitions to {ls} configs. The tool does
|
||||
not currently support all the processors that are available for ingest node, but
|
||||
it's a good starting point.
|
||||
|
||||
[[parsing-apache2]]
|
||||
==== Apache 2 Logs
|
||||
|
||||
The Logstash pipeline configuration in this example shows how to ship and parse
|
||||
The {ls} pipeline configuration in this example shows how to ship and parse
|
||||
access and error logs collected by the
|
||||
{filebeat-ref}/filebeat-module-apache.html[`apache` Filebeat module].
|
||||
{filebeat-ref}/filebeat-module-apache.html[`apache` {filebeat} module].
|
||||
|
||||
[source,json]
|
||||
----------------------------------------------------------------------------
|
||||
|
|
@ -148,9 +139,9 @@ include::filebeat_modules/apache2/pipeline.conf[]
|
|||
[[parsing-mysql]]
|
||||
==== MySQL Logs
|
||||
|
||||
The Logstash pipeline configuration in this example shows how to ship and parse
|
||||
The {ls} pipeline configuration in this example shows how to ship and parse
|
||||
error and slowlog logs collected by the
|
||||
{filebeat-ref}/filebeat-module-mysql.html[`mysql` Filebeat module].
|
||||
{filebeat-ref}/filebeat-module-mysql.html[`mysql` {filebeat} module].
|
||||
|
||||
[source,json]
|
||||
----------------------------------------------------------------------------
|
||||
|
|
@ -161,9 +152,9 @@ include::filebeat_modules/mysql/pipeline.conf[]
|
|||
[[parsing-nginx]]
|
||||
==== Nginx Logs
|
||||
|
||||
The Logstash pipeline configuration in this example shows how to ship and parse
|
||||
The {ls} pipeline configuration in this example shows how to ship and parse
|
||||
access and error logs collected by the
|
||||
{filebeat-ref}/filebeat-module-nginx.html[`nginx` Filebeat module].
|
||||
{filebeat-ref}/filebeat-module-nginx.html[`nginx` {filebeat} module].
|
||||
|
||||
[source,json]
|
||||
----------------------------------------------------------------------------
|
||||
|
|
@ -174,12 +165,13 @@ include::filebeat_modules/nginx/pipeline.conf[]
|
|||
[[parsing-system]]
|
||||
==== System Logs
|
||||
|
||||
The Logstash pipeline configuration in this example shows how to ship and parse
|
||||
The {ls} pipeline configuration in this example shows how to ship and parse
|
||||
system logs collected by the
|
||||
{filebeat-ref}/filebeat-module-system.html[`system` Filebeat module].
|
||||
{filebeat-ref}/filebeat-module-system.html[`system` {filebeat} module].
|
||||
|
||||
[source,json]
|
||||
----------------------------------------------------------------------------
|
||||
include::filebeat_modules/system/pipeline.conf[]
|
||||
----------------------------------------------------------------------------
|
||||
|
||||
include::fb-ls-kafka-example.asciidoc[]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue