Merge pull request #6832 from dedemorton/fbm_docs_issue#6542

Add docs about configuring LS for use with Filebeat modules

docs/index.asciidoc

@@ -17,6 +17,7 @@ release-state can be: released | prerelease | unreleased
 :ref:                   https://www.elastic.co/guide/en/elasticsearch/reference/current/
 :xpack:                https://www.elastic.co/guide/en/x-pack/current
 :logstash:              https://www.elastic.co/guide/en/logstash/current/
+:libbeat:               https://www.elastic.co/guide/en/beats/libbeat/current/
 :filebeat:              https://www.elastic.co/guide/en/beats/filebeat/current/
 :lsissue:               https://github.com/elastic/logstash/issues/
 :security:              X-Pack Security
@@ -100,6 +101,10 @@ include::static/managing-multiline-events.asciidoc[]
 
 include::static/glob-support.asciidoc[]
 
+// Working with Filebeat Modules
+
+include::static/filebeat-modules.asciidoc[]
+
 // Deploying & Scaling
 
 include::static/deploying.asciidoc[]

docs/static/filebeat-modules.asciidoc

@@ -0,0 +1,279 @@
+[[filebeat-modules]]
+
+== Working with Filebeat Modules
+
+Starting with version 5.3, Filebeat comes packaged with pre-built 
+{filebeat}filebeat-modules.html[modules] that contain the configurations needed
+to collect, parse, enrich, and visualize data from various log file formats.
+Each Filebeat module consists of one or more filesets that contain ingest node
+pipelines, Elasticsearch templates, Filebeat prospector configurations, and
+Kibana dashboards.
+
+Filebeat modules are a great way to get started, but you might find that ingest
+pipelines don't offer the processing power that you require. If that's the case,
+you'll need to use Logstash.
+
+[float]
+[[graduating-to-Logstash]]
+=== Graduating to Logstash
+
+You may need to graduate to using Logstash instead of ingest pipelines if you
+want to:
+
+* Use multiple outputs. Ingest pipelines were designed to only support
+Elasticsearch as an output, but you may want to use more than one output. For
+example, you may want to archive your incoming data to S3 as well as indexing
+it in Elasticsearch.
+* Use the <<persistent-queues,persistent queue>> feature to handle spikes when
+ingesting data (from Beats and other sources).
+* Take advantage of the richer transformation capabilities in Logstash, such as
+external lookups.
+
+Currently, we don't provide an automatic migration path from ingest pipelines
+to Logstash pipelines (but that's coming). For now, you can follow the steps in
+this section to configure Filebeat and build Logstash pipeline configurations
+that are equivalent to the ingest node pipelines available with the Filebeat
+modules. Then you'll be able to use the same dashboards available with Filebeat
+to visualize your data in Kibana.
+
+Follow the steps in this section to build and run Logstash configurations that
+provide capabilities similar to Filebeat modules.
+
+. Load the Filebeat index pattern and sample Kibana dashboards. To do this, you
+need to run the Filebeat module with the Elasticsearch output enabled and
+specify the `-setup` flag. 
++
+For example, to load the sample dashboards for Nginx, run:
++
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -e -modules=nginx -setup -E "output.elasticsearch.hosts=["http://localhost:9200"]"
+----------------------------------------------------------------------
++
+A connection to Elasticsearch is required for this one-time setup step because
+Filebeat needs to create the index pattern and load the sample dashboards into the
+Kibana index. 
++
+After the template and dashboards are loaded, you'll see the message
+`INFO Elasticsearch template with name 'filebeat' loaded`. You can shut
+down Filebeat.
+
+. Configure Filebeat to send log lines to Logstash.
++
+See <<logstash-config-for-filebeat-modules>> for detailed examples.
+
+. Create a Logstash pipeline configuration that reads from the Beats input and
+parses the log events.
++
+See <<logstash-config-for-filebeat-modules>> for detailed examples.
+
+. Start Filebeat. For example, to start Filebeat in the foreground, use:
++
+[source,shell]
+----------------------------------------------------------------------
+sudo ./filebeat -e -c filebeat.yml -d "publish"
+----------------------------------------------------------------------
++
+NOTE: Depending on how you've installed Filebeat, you might see errors
+related to file ownership or permissions when you try to run Filebeat modules.
+See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions]
+in the _Beats Platform Reference_ if you encounter errors related to file
+ownership or permissions.
++
+See {filebeat}/filebeat-starting.html[Starting Filebeat] for more info.
+
+. Start Logstash, passing in the pipeline configuration file that parses the
+log. For example:
++
+[source,shell]
+----------------------------------------------------------------------
+bin/logstash -f mypipeline.conf
+----------------------------------------------------------------------
++
+You'll see the following message when Logstash is running and listening for
+input from Beats: 
++
+[source,shell]
+----------------------------------------------------------------------
+[2017-03-17T16:31:40,319][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"127.0.0.1:5044"}
+[2017-03-17T16:31:40,350][INFO ][logstash.pipeline        ] Pipeline main started
+----------------------------------------------------------------------
+
+. To visualize the data in Kibana, launch the Kibana web interface by pointing
+your browser to port 5601. For example,
+http://127.0.0.1:5601[http://127.0.0.1:5601]. 
+
+[[logstash-config-for-filebeat-modules]]
+=== Configuration Examples
+
+The examples in this section show you how to configure Filebeat and build
+Logstash pipelines that parse:
+
+* <<parsing-apache2>>
+* <<parsing-mysql>>
+* <<parsing-nginx>>
+* <<parsing-system>>
+
+Of course, the paths that you specify in the Filebeat config depend on the location
+of the logs you are harvesting. The examples show common default locations.
+
+[[parsing-apache2]]
+==== Apache 2 Logs
+
+Here are some configuration examples for shipping and parsing Apache 2 access and
+error logs.
+
+===== Apache 2 Access Logs
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/apache2/access/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/apache2/access/pipeline.conf[]
+----------------------------------------------------------------------------
+
+===== Apache 2 Error Logs
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/apache2/error/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/apache2/error/pipeline.conf[]
+----------------------------------------------------------------------------
+
+[[parsing-mysql]]
+==== MySQL Logs
+
+Here are some configuration examples for shipping and parsing MySQL error and
+slowlog logs.
+
+===== MySQL Error Logs
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/mysql/error/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/mysql/error/pipeline.conf[]
+----------------------------------------------------------------------------
+
+===== MySQL Slowlog
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/mysql/slowlog/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/mysql/slowlog/pipeline.conf[]
+----------------------------------------------------------------------------
+
+[[parsing-nginx]]
+==== Nginx Logs
+
+Here are some configuration examples for shipping and parsing Nginx access and
+error logs.
+
+===== Nginx Access Logs
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/nginx/access/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/nginx/access/pipeline.conf[]
+----------------------------------------------------------------------------
+
+
+===== Nginx Error Logs
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/nginx/error/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/nginx/error/pipeline.conf[]
+----------------------------------------------------------------------------
+
+[[parsing-system]]
+==== System Logs
+
+Here are some configuration examples for shipping and parsing system
+logs.
+
+===== System Authorization Logs
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/system/auth/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/system/auth/pipeline.conf[]
+----------------------------------------------------------------------------
+
+===== Syslog
+
+Example Filebeat config:
+
+[source,yml]
+----------------------------------------------------------------------
+include::filebeat_modules/system/syslog/filebeat.yml[]
+----------------------------------------------------------------------
+
+
+Example Logstash pipeline config:
+
+[source,json]
+----------------------------------------------------------------------------
+include::filebeat_modules/system/syslog/pipeline.conf[]
+----------------------------------------------------------------------------

docs/static/filebeat_modules/apache2/access/filebeat.yml

@@ -0,0 +1,8 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/apache2/access.log* 
+    - /var/log/apache2/other_vhosts_access.log*
+  exclude_files: [".gz$"]
+output.logstash:
+  hosts: ["localhost:5044"]

docs/static/filebeat_modules/apache2/error/filebeat.yml

@@ -0,0 +1,8 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/apache2/error.log*
+  exclude_files: [".gz$"]
+output.logstash:
+  hosts: ["localhost:5044"]
+  

docs/static/filebeat_modules/mysql/error/filebeat.yml

@@ -0,0 +1,8 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/mysql/error.log*
+    - /var/log/mysqld.log*
+  exclude_files: [".gz$"]
+output.logstash:
+  hosts: ["localhost:5044"]

docs/static/filebeat_modules/mysql/slowlog/filebeat.yml

@@ -0,0 +1,12 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/mysql/mysql-slow.log*
+    - /var/lib/mysql/hostname-slow.log
+  exclude_files: [".gz$"]
+  multiline:
+    pattern: "^# User@Host: "
+    negate: true
+    match: after
+output.logstash:
+  hosts: ["localhost:5044"]

docs/static/filebeat_modules/nginx/access/filebeat.yml

@@ -0,0 +1,7 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/nginx/access.log*
+  exclude_files: [".gz$"]
+output.logstash:
+  hosts: ["localhost:5044"]

docs/static/filebeat_modules/nginx/error/filebeat.yml

@@ -0,0 +1,7 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/nginx/error.log*
+  exclude_files: [".gz$"]
+output.logstash:
+  hosts: ["localhost:5044"]

docs/static/filebeat_modules/system/auth/filebeat.yml

@@ -0,0 +1,11 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/auth.log*
+    - /var/log/secure*
+  exclude_files: [".gz$"]
+  multiline:
+    pattern: "^\\s"
+    match: after
+output.logstash:
+  hosts: ["localhost:5044"]

docs/static/filebeat_modules/system/syslog/filebeat.yml

@@ -0,0 +1,11 @@
+filebeat.prospectors:
+- input_type: log
+  paths:
+    - /var/log/messages*
+    - /var/log/syslog*
+  exclude_files: [".gz$"]
+  multiline:
+    pattern: "^\\s"
+    match: after
+output.logstash:
+  hosts: ["localhost:5044"]