mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
279 lines
9.4 KiB
Text
279 lines
9.4 KiB
Text
[[filebeat-modules]]
|
|
|
|
== Working with Filebeat Modules
|
|
|
|
Starting with version 5.3, Filebeat comes packaged with pre-built
|
|
{filebeat}filebeat-modules.html[modules] that contain the configurations needed
|
|
to collect, parse, enrich, and visualize data from various log file formats.
|
|
Each Filebeat module consists of one or more filesets that contain ingest node
|
|
pipelines, Elasticsearch templates, Filebeat prospector configurations, and
|
|
Kibana dashboards.
|
|
|
|
Filebeat modules are a great way to get started, but you might find that ingest
|
|
pipelines don't offer the processing power that you require. If that's the case,
|
|
you'll need to use Logstash.
|
|
|
|
[float]
|
|
[[graduating-to-Logstash]]
|
|
=== Graduating to Logstash
|
|
|
|
You may need to graduate to using Logstash instead of ingest pipelines if you
|
|
want to:
|
|
|
|
* Use multiple outputs. Ingest pipelines were designed to only support
|
|
Elasticsearch as an output, but you may want to use more than one output. For
|
|
example, you may want to archive your incoming data to S3 as well as indexing
|
|
it in Elasticsearch.
|
|
* Use the <<persistent-queues,persistent queue>> feature to handle spikes when
|
|
ingesting data (from Beats and other sources).
|
|
* Take advantage of the richer transformation capabilities in Logstash, such as
|
|
external lookups.
|
|
|
|
Currently, we don't provide an automatic migration path from ingest pipelines
|
|
to Logstash pipelines (but that's coming). For now, you can follow the steps in
|
|
this section to configure Filebeat and build Logstash pipeline configurations
|
|
that are equivalent to the ingest node pipelines available with the Filebeat
|
|
modules. Then you'll be able to use the same dashboards available with Filebeat
|
|
to visualize your data in Kibana.
|
|
|
|
Follow the steps in this section to build and run Logstash configurations that
|
|
provide capabilities similar to Filebeat modules.
|
|
|
|
. Load the Filebeat index pattern and sample Kibana dashboards. To do this, you
|
|
need to run the Filebeat module with the Elasticsearch output enabled and
|
|
specify the `-setup` flag.
|
|
+
|
|
For example, to load the sample dashboards for Nginx, run:
|
|
+
|
|
[source,shell]
|
|
----------------------------------------------------------------------
|
|
./filebeat -e -modules=nginx -setup -E "output.elasticsearch.hosts=["http://localhost:9200"]"
|
|
----------------------------------------------------------------------
|
|
+
|
|
A connection to Elasticsearch is required for this one-time setup step because
|
|
Filebeat needs to create the index pattern and load the sample dashboards into the
|
|
Kibana index.
|
|
+
|
|
After the template and dashboards are loaded, you'll see the message
|
|
`INFO Elasticsearch template with name 'filebeat' loaded`. You can shut
|
|
down Filebeat.
|
|
|
|
. Configure Filebeat to send log lines to Logstash.
|
|
+
|
|
See <<logstash-config-for-filebeat-modules>> for detailed examples.
|
|
|
|
. Create a Logstash pipeline configuration that reads from the Beats input and
|
|
parses the log events.
|
|
+
|
|
See <<logstash-config-for-filebeat-modules>> for detailed examples.
|
|
|
|
. Start Filebeat. For example, to start Filebeat in the foreground, use:
|
|
+
|
|
[source,shell]
|
|
----------------------------------------------------------------------
|
|
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
|
----------------------------------------------------------------------
|
|
+
|
|
NOTE: Depending on how you've installed Filebeat, you might see errors
|
|
related to file ownership or permissions when you try to run Filebeat modules.
|
|
See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
|
|
in the _Beats Platform Reference_ if you encounter errors related to file
|
|
ownership or permissions.
|
|
+
|
|
See {filebeat}/filebeat-starting.html[Starting Filebeat] for more info.
|
|
|
|
. Start Logstash, passing in the pipeline configuration file that parses the
|
|
log. For example:
|
|
+
|
|
[source,shell]
|
|
----------------------------------------------------------------------
|
|
bin/logstash -f mypipeline.conf
|
|
----------------------------------------------------------------------
|
|
+
|
|
You'll see the following message when Logstash is running and listening for
|
|
input from Beats:
|
|
+
|
|
[source,shell]
|
|
----------------------------------------------------------------------
|
|
[2017-03-17T16:31:40,319][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"127.0.0.1:5044"}
|
|
[2017-03-17T16:31:40,350][INFO ][logstash.pipeline ] Pipeline main started
|
|
----------------------------------------------------------------------
|
|
|
|
. To visualize the data in Kibana, launch the Kibana web interface by pointing
|
|
your browser to port 5601. For example,
|
|
http://127.0.0.1:5601[http://127.0.0.1:5601].
|
|
|
|
[[logstash-config-for-filebeat-modules]]
|
|
=== Configuration Examples
|
|
|
|
The examples in this section show you how to configure Filebeat and build
|
|
Logstash pipelines that parse:
|
|
|
|
* <<parsing-apache2>>
|
|
* <<parsing-mysql>>
|
|
* <<parsing-nginx>>
|
|
* <<parsing-system>>
|
|
|
|
Of course, the paths that you specify in the Filebeat config depend on the location
|
|
of the logs you are harvesting. The examples show common default locations.
|
|
|
|
[[parsing-apache2]]
|
|
==== Apache 2 Logs
|
|
|
|
Here are some configuration examples for shipping and parsing Apache 2 access and
|
|
error logs.
|
|
|
|
===== Apache 2 Access Logs
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/apache2/access/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/apache2/access/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
===== Apache 2 Error Logs
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/apache2/error/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/apache2/error/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
[[parsing-mysql]]
|
|
==== MySQL Logs
|
|
|
|
Here are some configuration examples for shipping and parsing MySQL error and
|
|
slowlog logs.
|
|
|
|
===== MySQL Error Logs
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/mysql/error/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/mysql/error/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
===== MySQL Slowlog
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/mysql/slowlog/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/mysql/slowlog/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
[[parsing-nginx]]
|
|
==== Nginx Logs
|
|
|
|
Here are some configuration examples for shipping and parsing Nginx access and
|
|
error logs.
|
|
|
|
===== Nginx Access Logs
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/nginx/access/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/nginx/access/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
|
|
===== Nginx Error Logs
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/nginx/error/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/nginx/error/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
[[parsing-system]]
|
|
==== System Logs
|
|
|
|
Here are some configuration examples for shipping and parsing system
|
|
logs.
|
|
|
|
===== System Authorization Logs
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/system/auth/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/system/auth/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|
|
|
|
===== Syslog
|
|
|
|
Example Filebeat config:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
include::filebeat_modules/system/syslog/filebeat.yml[]
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
Example Logstash pipeline config:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------------
|
|
include::filebeat_modules/system/syslog/pipeline.conf[]
|
|
----------------------------------------------------------------------------
|