diff --git a/docs/static/fb-ls-kafka-example.asciidoc b/docs/static/fb-ls-kafka-example.asciidoc new file mode 100644 index 000000000..86b428f9b --- /dev/null +++ b/docs/static/fb-ls-kafka-example.asciidoc @@ -0,0 +1,151 @@ +[[use-filebeat-modules-kafka]] +=== Example: Set up {filebeat} modules to work with Kafka and {ls} + +This section shows how to set up {filebeat} +{filebeat-ref}/filebeat-modules-overview.html[modules] to work with {ls} when +you are using Kafka in between {filebeat} and {ls} in your publishing pipeline. +The main goal of this example is to show how to load ingest pipelines from +{filebeat} and use them with {ls}. + +The examples in this section show simple configurations with topic names hard +coded. For a full list of configuration options, see documentation about +configuring the <>. Also see +{filebeat-ref}/kafka-output.html[Configure the Kafka output] in the _{filebeat} +Reference_. + +==== Set up and run {filebeat} + +. If you haven't already set up the {filebeat} index template and sample {kib} +dashboards, run the {filebeat} `setup` command to do that now: ++ +[source,shell] +---------------------------------------------------------------------- +filebeat -e setup +---------------------------------------------------------------------- ++ +The `-e` flag is optional and sends output to standard error instead of syslog. ++ +A connection to {es} and {kib} is required for this one-time setup +step because {filebeat} needs to create the index template in {es} and +load the sample dashboards into {kib}. For more information about configuring +the connection to {es}, see the Filebeat modules +{filebeat-ref}/filebeat-modules-quickstart.html[quick start]. ++ +After the template and dashboards are loaded, you'll see the message `INFO +{kib} dashboards successfully loaded. Loaded dashboards`. + +. Run the `modules enable` command to enable the modules that you want to run. +For example: ++ +[source,shell] +---------------------------------------------------------------------- +filebeat modules enable system +---------------------------------------------------------------------- ++ +You can further configure the module by editing the config file under the +{filebeat} `modules.d` directory. For example, if the log files are not in the +location expected by the module, you can set the `var.paths` option. + +. Run the `setup` command with the `--pipelines` and `--modules` options +specified to load ingest pipelines for the modules you've enabled. This step +also requires a connection to {es}. If you want use a {ls} pipeline instead of +ingest node to parse the data, skip this step. ++ +[source,shell] +---------------------------------------------------------------------- +filebeat setup --pipelines --modules system +---------------------------------------------------------------------- + +. Configure {filebeat} to send log lines to Kafka. To do this, in the ++filebeat.yml+ config file, disable the {es} output by commenting it out, and +enable the Kafka output. For example: ++ +[source,yaml] +----- +#output.elasticsearch: + #hosts: ["localhost:9200"] +output.kafka: + hosts: ["kafka:9092"] + topic: "filebeat" + codec.json: + pretty: false +----- + +. Start {filebeat}. For example: ++ +[source,shell] +---------------------------------------------------------------------- +filebeat -e +---------------------------------------------------------------------- ++ +{filebeat} will attempt to send messages to {ls} and continue until {ls} is +available to receive them. ++ +NOTE: Depending on how you've installed {filebeat}, you might see errors +related to file ownership or permissions when you try to run {filebeat} modules. +See {beats-ref}/config-file-permissions.html[Config File Ownership and Permissions] +in the _Beats Platform Reference_ if you encounter errors related to file +ownership or permissions. + + +==== Create and start the {ls} pipeline + +. On the system where {ls} is installed, create a {ls} pipeline configuration +that reads from a Kafka input and sends events to an {es} output: ++ +-- +[source,yaml] +----- +input { + kafka { + bootstrap_servers => "myhost:9092" + topics => ["filebeat"] + codec => json + } +} + +output { + if [@metadata][pipeline] { + elasticsearch { + hosts => "https://myEShost:9200" + manage_template => false + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + pipeline => "%{[@metadata][pipeline]}" <1> + user => "elastic" + password => "secret" + } + } else { + elasticsearch { + hosts => "https://myEShost:9200" + manage_template => false + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + user => "elastic" + password => "secret" + } + } +} +----- +<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting +configures {ls} to select the correct ingest pipeline based on metadata +passed in the event. + +If you want use a {ls} pipeline instead of ingest node to parse the data, see +the `filter` and `output` settings in the examples under +<>. +-- + +. Start {ls}, passing in the pipeline configuration file you just defined. For +example: ++ +[source,shell] +---------------------------------------------------------------------- +bin/logstash -f mypipeline.conf +---------------------------------------------------------------------- ++ +{ls} should start a pipeline and begin receiving events from the Kafka input. + +==== Visualize the data + +To visualize the data in {kib}, launch the {kib} web interface by pointing your +browser to port 5601. For example, http://127.0.0.1:5601[http://127.0.0.1:5601]. +Click *Dashboards* then view the {filebeat} dashboards. \ No newline at end of file diff --git a/docs/static/filebeat-modules.asciidoc b/docs/static/filebeat-modules.asciidoc index 1b601f685..dd43a974d 100644 --- a/docs/static/filebeat-modules.asciidoc +++ b/docs/static/filebeat-modules.asciidoc @@ -1,143 +1,134 @@ [[filebeat-modules]] -== Working with Filebeat Modules +== Working with {filebeat} Modules -Filebeat comes packaged with pre-built {filebeat-ref}/filebeat-modules.html[modules] -that contain the configurations needed to collect, parse, enrich, and visualize -data from various log file formats. Each Filebeat module consists of one or more -filesets that contain ingest node pipelines, Elasticsearch templates, Filebeat -prospector configurations, and Kibana dashboards. +{filebeat} comes packaged with pre-built +{filebeat-ref}/filebeat-modules.html[modules] that contain the configurations +needed to collect, parse, enrich, and visualize data from various log file +formats. Each {filebeat} module consists of one or more filesets that contain +ingest node pipelines, {es} templates, {filebeat} input configurations, and +{kib} dashboards. -Filebeat modules are a great way to get started, but you might find that ingest -pipelines don't offer the processing power that you require. If that's the case, -you'll need to use Logstash. +You can use {filebeat} modules with {ls}, but you need to do some extra setup. +The simplest approach is to <> provided by {filebeat}. If the ingest pipelines don't meet your +requirements, you can +<> to use +instead of the ingest pipelines. -[float] -[[graduating-to-Logstash]] -=== Using Logstash instead of Ingest Node +Either approach allows you to use the configurations, index templates, and +dashboards available with {filebeat} modules, as long as you maintain the +field structure expected by the index and dashboards. -Logstash provides an <> -to help you migrate ingest pipeline definitions to Logstash configs. However, -the tool does not currently support all the processors that are available for -ingest node. +[[use-ingest-pipelines]] +=== Use ingest pipelines for parsing -You can follow the steps in this section to build and run Logstash -configurations that parse the data collected by Filebeat modules. Then you'll be -able to use the same dashboards available with Filebeat to visualize your data -in Kibana. +When you use {filebeat} modules with {ls}, you can use the ingest pipelines +provided by {filebeat} to parse the data. You need to load the pipelines +into {es} and configure {ls} to use them. -[float] -==== Create and start the Logstash pipeline +*To load the ingest pipelines:* -. Create a Logstash pipeline configuration that reads from the Beats input and -parses the events. -+ -See <> for detailed examples. +On the system where {filebeat} is installed, run the `setup` command with the +`--pipelines` option specified to load ingest pipelines for specific modules. +For example, the following command loads ingest pipelines for the system and +nginx modules: -. Start Logstash, passing in the pipeline configuration file that parses the -log. For example: -+ [source,shell] ----------------------------------------------------------------------- -bin/logstash -f mypipeline.conf ----------------------------------------------------------------------- -+ -You'll see the following message when Logstash is running and listening for -input from Beats: -+ -[source,shell] ----------------------------------------------------------------------- -[2017-10-13T00:01:15,413][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"127.0.0.1:5044"} -[2017-10-13T00:01:15,443][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"} ----------------------------------------------------------------------- +----- +filebeat setup --pipelines --modules nginx,system +----- +A connection to {es} is required for this setup step because {filebeat} needs to +load the ingest pipelines into {es}. If necessary, you can temporarily disable +your configured output and enable the {es} output before running the command. -The Logstash pipeline is now ready to receive events from Filebeat. Next, you -set up and run Filebeat. +*To configure {ls} to use the pipelines:* -[float] -==== Set up and run Filebeat +On the system where {ls} is installed, create a {ls} pipeline configuration +that reads from a {ls} input, such as {beats} or Kafka, and sends events to an +{es} output. Set the `pipeline` option in the {es} output to +`%{[@metadata][pipeline]}` to use the ingest pipelines that you loaded +previously. -. If you haven't already set up the Filebeat index template and sample Kibana -dashboards, run the Filebeat `setup` command to do that now: -+ -[source,shell] ----------------------------------------------------------------------- -./filebeat -e setup ----------------------------------------------------------------------- -+ -The `-e` flag is optional and sends output to standard error instead of syslog. -+ -A connection to Elasticsearch and Kibana is required for this one-time setup -step because Filebeat needs to create the index template in Elasticsearch and -load the sample dashboards into Kibana. -+ -After the template and dashboards are loaded, you'll see the message `INFO -Kibana dashboards successfully loaded. Loaded dashboards`. +Here's an example configuration that reads data from the Beats input and uses +{filebeat} ingest pipelines to parse data collected by modules: -. Configure Filebeat to send log lines to Logstash. To do this, in the -+filebeat.yml+ config file, disable the Elasticsearch output, and enable the -Logstash output. For example: -+ [source,yaml] ----- -#output.elasticsearch: - #hosts: ["localhost:9200"] -output.logstash: - hosts: ["localhost:5044"] +input { + beats { + port => 5044 + } +} + +output { + if [@metadata][pipeline] { + elasticsearch { + hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243" + manage_template => false + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + pipeline => "%{[@metadata][pipeline]}" <1> + user => "elastic" + password => "secret" + } + } else { + elasticsearch { + hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243" + manage_template => false + index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" + user => "elastic" + password => "secret" + } + } +} ----- +<1> Set the `pipeline` option to `%{[@metadata][pipeline]}`. This setting +configures {ls} to select the correct ingest pipeline based on metadata +passed in the event. -. Run the `modules enable` command to enable the modules that you want to run. -For example: -+ -[source,shell] ----------------------------------------------------------------------- -./filebeat modules enable nginx ----------------------------------------------------------------------- -+ -You can further configure the module by editing the config file under the -Filebeat `modules.d` directory. For example, if the log files are not in the -location expected by the module, you can set the `var.paths` option. +See the {filebeat} {filebeat-ref}/filebeat-modules-overview.html[Modules] +documentation for more information about setting up and running modules. -. Start Filebeat. For example, to start Filebeat in the foreground, use: -+ -[source,shell] ----------------------------------------------------------------------- -./filebeat -e ----------------------------------------------------------------------- -+ -NOTE: Depending on how you've installed Filebeat, you might see errors -related to file ownership or permissions when you try to run Filebeat modules. -See {beats-ref}/config-file-permissions.html[Config File Ownership and Permissions] -in the _Beats Platform Reference_ if you encounter errors related to file -ownership or permissions. -+ -See {filebeat-ref}/filebeat-starting.html[Starting Filebeat] for more info. - -[float] -==== Visualize the data - -To visualize the data in Kibana, launch the Kibana web interface by pointing -your browser to port 5601. For example, -http://127.0.0.1:5601[http://127.0.0.1:5601]. +For a full example, see <>. [[logstash-config-for-filebeat-modules]] -=== Configuration Examples +=== Use {ls} pipelines for parsing -The examples in this section show you how to build Logstash pipelines that parse -data sent collected by Filebeat modules: +The examples in this section show how to build {ls} pipeline configurations that +replace the ingest pipelines provided with {filebeat} modules. The pipelines +take the data collected by {filebeat} modules, parse it into fields expected by +the {filebeat} index, and send the fields to {es} so that you can visualize the +data in the pre-built dashboards provided by {filebeat}. + +This approach is more time consuming than using the existing ingest pipelines to +parse the data, but it gives you more control over how the data is processed. +By writing your own pipeline configurations, you can do additional processing, +such as dropping fields, after the fields are extracted, or you can move your +load from {es} ingest nodes to {ls} nodes. + +Before deciding to replaced the ingest pipelines with {ls} configurations, +read <>. + +Here are some examples that show how to implement {ls} configurations to replace +ingest pipelines: * <> * <> * <> * <> +TIP: {ls} provides an <> +to help you migrate ingest pipeline definitions to {ls} configs. The tool does +not currently support all the processors that are available for ingest node, but +it's a good starting point. + [[parsing-apache2]] ==== Apache 2 Logs -The Logstash pipeline configuration in this example shows how to ship and parse +The {ls} pipeline configuration in this example shows how to ship and parse access and error logs collected by the -{filebeat-ref}/filebeat-module-apache.html[`apache` Filebeat module]. +{filebeat-ref}/filebeat-module-apache.html[`apache` {filebeat} module]. [source,json] ---------------------------------------------------------------------------- @@ -148,9 +139,9 @@ include::filebeat_modules/apache2/pipeline.conf[] [[parsing-mysql]] ==== MySQL Logs -The Logstash pipeline configuration in this example shows how to ship and parse +The {ls} pipeline configuration in this example shows how to ship and parse error and slowlog logs collected by the -{filebeat-ref}/filebeat-module-mysql.html[`mysql` Filebeat module]. +{filebeat-ref}/filebeat-module-mysql.html[`mysql` {filebeat} module]. [source,json] ---------------------------------------------------------------------------- @@ -161,9 +152,9 @@ include::filebeat_modules/mysql/pipeline.conf[] [[parsing-nginx]] ==== Nginx Logs -The Logstash pipeline configuration in this example shows how to ship and parse +The {ls} pipeline configuration in this example shows how to ship and parse access and error logs collected by the -{filebeat-ref}/filebeat-module-nginx.html[`nginx` Filebeat module]. +{filebeat-ref}/filebeat-module-nginx.html[`nginx` {filebeat} module]. [source,json] ---------------------------------------------------------------------------- @@ -174,12 +165,13 @@ include::filebeat_modules/nginx/pipeline.conf[] [[parsing-system]] ==== System Logs -The Logstash pipeline configuration in this example shows how to ship and parse +The {ls} pipeline configuration in this example shows how to ship and parse system logs collected by the -{filebeat-ref}/filebeat-module-system.html[`system` Filebeat module]. +{filebeat-ref}/filebeat-module-system.html[`system` {filebeat} module]. [source,json] ---------------------------------------------------------------------------- include::filebeat_modules/system/pipeline.conf[] ---------------------------------------------------------------------------- +include::fb-ls-kafka-example.asciidoc[]