mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 06:37:19 -04:00
DeDe Morton
parent
7132c68cf4
commit
683bbd72ed
1 changed files with 389 additions and 141 deletions
530
docs/static/advanced-pipeline.asciidoc
vendored
530
docs/static/advanced-pipeline.asciidoc
vendored
|
|
@ -4,14 +4,72 @@
|
|||
In <<first-event>>, you created a basic Logstash pipeline to test your Logstash setup. In the real world, a Logstash
|
||||
pipeline is a bit more complex: it typically has one or more input, filter, and output plugins.
|
||||
|
||||
In this section, you create a Logstash pipeline that takes Apache web logs as input, parses those
|
||||
In this section, you create a Logstash pipeline that uses Filebeat to take Apache web logs as input, parses those
|
||||
logs to create specific, named fields from the logs, and writes the parsed data to an Elasticsearch cluster. Rather than
|
||||
defining the pipeline configuration at the command line, you'll define the pipeline in a config file.
|
||||
|
||||
The following text represents the skeleton of a configuration pipeline:
|
||||
To get started, go https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz[here] to
|
||||
download the sample data set used in this example. Unpack the file.
|
||||
|
||||
|
||||
[[configuring-filebeat]]
|
||||
==== Configuring Filebeat to Send Log Lines to Logstash
|
||||
|
||||
Before you create the Logstash pipeline, you'll configure Filebeat to send log lines to Logstash. https://github.com/elastic/beats/tree/master/filebeat[Filebeat] is a lightweight, resource-friendly tool that
|
||||
collects logs from files on the server and forwards these logs to your Logstash instance for processing. Filebeat is
|
||||
designed for reliability and low latency. Filebeat uses the computing resources of the machine hosting the source data,
|
||||
and the {logstash}plugins-inputs-beats.html[`Beats input`] plugin minimizes the
|
||||
resource demands on the Logstash instance.
|
||||
|
||||
NOTE: In a typical use case, Filebeat runs on a separate machine from the machine running your
|
||||
Logstash instance. For the purposes of this tutorial, Logstash and Filebeat are running on the
|
||||
same machine.
|
||||
|
||||
The default Logstash installation includes the {logstash}plugins-inputs-beats.html[`Beats input`] plugin. To install
|
||||
Filebeat on your data source machine, download the appropriate package from the Filebeat https://www.elastic.co/downloads/beats/filebeat[product page]. You can also refer to
|
||||
{filebeat}filebeat-getting-started.html[Getting Started with Filebeat] in the Beats documentation for additional
|
||||
installation instructions.
|
||||
|
||||
After installing Filebeat, you need to configure it. Open the `filebeat.yml` file located in your Filebeat installation
|
||||
directory, and replace the contents with the following lines. Make sure `paths` points to the example Apache log file,
|
||||
`logstash-tutorial.log`, that you downloaded earlier:
|
||||
|
||||
[source,yaml]
|
||||
--------------------------------------------------------------------------------
|
||||
filebeat.prospectors:
|
||||
- input_type: log
|
||||
paths:
|
||||
- /path/to/file/logstash-tutorial.log <1>
|
||||
output.logstash:
|
||||
hosts: ["localhost:5043"]
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
<1> Absolute path to the file or files that Filebeat processes.
|
||||
|
||||
Save your changes.
|
||||
|
||||
To keep the configuration simple, you won't specify TLS/SSL settings as you would in a real world
|
||||
scenario.
|
||||
|
||||
At the data source machine, run Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Filebeat will attempt to connect on port 5403. Until Logstash starts with an active Beats plugin, there
|
||||
won’t be any answer on that port, so any messages you see regarding failure to connect on that port are normal for now.
|
||||
|
||||
==== Configuring Logstash for Filebeat Input
|
||||
|
||||
Next, you create a Logstash configuration pipeline that uses the Beats input plugin to receive
|
||||
events from Beats.
|
||||
|
||||
The following text represents the skeleton of a configuration pipeline:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
# The # character at the beginning of a line indicates a comment. Use
|
||||
# comments to describe your configuration.
|
||||
input {
|
||||
|
|
@ -28,44 +86,98 @@ output {
|
|||
This skeleton is non-functional, because the input and output sections don’t have any valid options defined.
|
||||
|
||||
To get started, copy and paste the skeleton configuration pipeline into a file named `first-pipeline.conf` in your home
|
||||
Logstash directory. Then go https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz[here] to
|
||||
download the sample data set used in this example. Unpack the file.
|
||||
Logstash directory.
|
||||
|
||||
[float]
|
||||
[[configuring-file-input]]
|
||||
==== Configuring Logstash for File Input
|
||||
Next, configure your Logstash instance to use the Beats input plugin by adding the following lines to the `input` section
|
||||
of the `first-pipeline.conf` file:
|
||||
|
||||
NOTE: This example uses the file input plugin for convenience. To tail files in the real world, you'll use
|
||||
Filebeat to ship log events to Logstash. You learn how to <<configuring-lsf,configure the Filebeat input plugin>> later
|
||||
when you build a more sophisticated pipeline.
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
To begin your Logstash pipeline, configure the Logstash instance to read from a file by using the
|
||||
{logstash}plugins-inputs-file.html[`file`] input plugin.
|
||||
You'll configure Logstash to write to Elasticsearch later. For now, you can add the following line
|
||||
to the `output` section so that the output is printed to stdout when you run Logstash:
|
||||
|
||||
Edit the `first-pipeline.conf` file and replace the entire `input` section with the following text:
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
stdout { codec => rubydebug }
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
When you're done, the contents of `first-pipeline.conf` should look like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
file {
|
||||
path => "/path/to/file/*.log"
|
||||
start_position => beginning <1>
|
||||
ignore_older => 0 <2>
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
# The filter part of this file is commented out to indicate that it is
|
||||
# optional.
|
||||
# filter {
|
||||
#
|
||||
# }
|
||||
output {
|
||||
stdout { codec => rubydebug }
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
<1> The default behavior of the file input plugin is to monitor a file for new information, in a manner similar to the
|
||||
UNIX `tail -f` command. To change this default behavior and process the entire file, we need to specify the position
|
||||
where Logstash starts processing the file.
|
||||
<2> The default behavior of the file input plugin is to ignore files whose last modification is greater than 86400s. To change this default behavior and process the tutorial file (which is probably much older than a day), we need to configure Logstash so that it does not ignore old files.
|
||||
To verify your configuration, run the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f first-pipeline.conf --config.test_and_exit
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `--config.test_and_exit` option parses your configuration file and reports any errors.
|
||||
|
||||
If the configuration file passes the configuration test, start Logstash with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f first-pipeline.conf --config.reload.automatic
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `--config.reload.automatic` option enables automatic config reloading so that you don't have to stop and restart Logstash
|
||||
every time you modify the configuration file.
|
||||
|
||||
If your pipeline is working correctly, you should see a series of events like the following written to the console:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"@timestamp" => 2016-10-11T20:54:06.733Z,
|
||||
"offset" => 325,
|
||||
"@version" => "1",
|
||||
"beat" => {
|
||||
"hostname" => "My-MacBook-Pro.local",
|
||||
"name" => "My-MacBook-Pro.local"
|
||||
},
|
||||
"input_type" => "log",
|
||||
"host" => "My-MacBook-Pro.local",
|
||||
"source" => "/path/to/file/logstash-tutorial.log",
|
||||
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"type" => "log",
|
||||
"tags" => [
|
||||
[0] "beats_input_codec_plain_applied"
|
||||
]
|
||||
}
|
||||
...
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Replace `/path/to/file` with the absolute path to the location of `logstash-tutorial.log` in your file system.
|
||||
|
||||
[float]
|
||||
[[configuring-grok-filter]]
|
||||
==== Parsing Web Logs with the Grok Filter Plugin
|
||||
|
||||
Now you have a working pipeline that reads log lines from Filebeat. However you'll notice that the format of the log messages
|
||||
is not ideal. You want to parse the log messages to create specific, named fields from the logs.
|
||||
To do this, you'll use the `grok` filter plugin.
|
||||
|
||||
The {logstash}plugins-filters-grok.html[`grok`] filter plugin is one of several plugins that are available by default in
|
||||
Logstash. For details on how to manage Logstash plugins, see the <<working-with-plugins,reference documentation>> for
|
||||
the plugin manager.
|
||||
|
|
@ -110,25 +222,82 @@ filter {
|
|||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
After processing the log file with the grok pattern, the sample line will have the following JSON representation:
|
||||
When you're done, the contents of `first-pipeline.conf` should look like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
}
|
||||
}
|
||||
output {
|
||||
stdout { codec => rubydebug }
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Save your changes. Because you've enabled automatic config reloading, you don't have to restart Logstash to
|
||||
pick up your changes. However, you do need to force Filebeat to read the log file from scratch. To do this,
|
||||
go to the terminal window where Filebeat is running and press Ctrl+C to shut down Filebeat. Then delete the
|
||||
Filebeat registry file. For example, run:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo rm data/registry
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Since Filebeat stores the state of each file it harvests in the registry, deleting the registry file forces
|
||||
Filebeat to read all the files it's harvesting from scratch.
|
||||
|
||||
Next, restart Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
After processing the log file with the grok pattern, the events will have the following JSON representation:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"clientip" : "83.149.9.216",
|
||||
"ident" : ,
|
||||
"auth" : ,
|
||||
"timestamp" : "04/Jan/2015:05:13:42 +0000",
|
||||
"verb" : "GET",
|
||||
"request" : "/presentations/logstash-monitorama-2013/images/kibana-search.png",
|
||||
"httpversion" : "HTTP/1.1",
|
||||
"response" : "200",
|
||||
"bytes" : "203023",
|
||||
"referrer" : "http://semicomplete.com/presentations/logstash-monitorama-2013/",
|
||||
"agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
|
||||
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
|
||||
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"offset" => 325,
|
||||
"auth" => "-",
|
||||
"ident" => "-",
|
||||
"input_type" => "log",
|
||||
"verb" => "GET",
|
||||
"source" => "/path/to/file/logstash-tutorial.log",
|
||||
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"type" => "log",
|
||||
"tags" => [
|
||||
[0] "beats_input_codec_plain_applied"
|
||||
],
|
||||
"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
|
||||
"@timestamp" => 2016-10-11T21:04:36.167Z,
|
||||
"response" => "200",
|
||||
"bytes" => "203023",
|
||||
"clientip" => "83.149.9.216",
|
||||
"@version" => "1",
|
||||
"beat" => {
|
||||
"hostname" => "My-MacBook-Pro.local",
|
||||
"name" => "My-MacBook-Pro.local"
|
||||
},
|
||||
"host" => "My-MacBook-Pro.local",
|
||||
"httpversion" => "1.1",
|
||||
"timestamp" => "04/Jan/2015:05:13:42 +0000"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Notice that the event includes the original message, but the log message is also broken down into specific fields.
|
||||
|
||||
|
||||
[float]
|
||||
[[configuring-geoip-plugin]]
|
||||
==== Enhancing Your Data with the Geoip Filter Plugin
|
||||
|
|
@ -150,11 +319,17 @@ of the `first-pipeline.conf` file:
|
|||
The `geoip` plugin configuration requires you to specify the name of the source field that contains the IP address to look up. In this example, the `clientip` field contains the IP address.
|
||||
|
||||
Since filters are evaluated in sequence, make sure that the `geoip` section is after the `grok` section of
|
||||
the configuration file and that both the `grok` and `geoip` sections are nested within the `filter` section
|
||||
like this:
|
||||
the configuration file and that both the `grok` and `geoip` sections are nested within the `filter` section.
|
||||
|
||||
When you're done, the contents of `first-pipeline.conf` should look like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
|
|
@ -162,7 +337,47 @@ like this:
|
|||
geoip {
|
||||
source => "clientip"
|
||||
}
|
||||
}
|
||||
}
|
||||
output {
|
||||
stdout { codec => rubydebug }
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Save your changes. To force Filebeat to read the log file from scratch, as you did earlier, shut down Filebeat (press Ctrl+C),
|
||||
delete the registry file, and then restart Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Notice that the event now contains geographic location information:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
|
||||
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"geoip" => {
|
||||
"timezone" => "Europe/Moscow",
|
||||
"ip" => "83.149.9.216",
|
||||
"latitude" => 55.7522,
|
||||
"continent_code" => "EU",
|
||||
"city_name" => "Moscow",
|
||||
"country_code2" => "RU",
|
||||
"country_name" => "Russia",
|
||||
"dma_code" => nil,
|
||||
"country_code3" => "RU",
|
||||
"region_name" => "Moscow",
|
||||
"location" => [
|
||||
[0] 37.6156,
|
||||
[1] 55.7522
|
||||
],
|
||||
"postal_code" => "101194",
|
||||
"longitude" => 37.6156,
|
||||
"region_code" => "MOW"
|
||||
},
|
||||
...
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
|
@ -187,23 +402,17 @@ With this configuration, Logstash uses http protocol to connect to Elasticsearch
|
|||
Logstash and Elasticsearch are running on the same instance. You can specify a remote Elasticsearch instance by using
|
||||
the `hosts` configuration to specify something like `hosts => [ "es-machine:9092" ]`.
|
||||
|
||||
[float]
|
||||
[[testing-initial-pipeline]]
|
||||
===== Testing Your Initial Pipeline
|
||||
|
||||
At this point, your `first-pipeline.conf` file has input, filter, and output sections properly configured, and looks
|
||||
something like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
file {
|
||||
path => "/Users/myusername/tutorialdata/*.log"
|
||||
start_position => beginning
|
||||
ignore_older => 0
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
}
|
||||
|
|
@ -218,36 +427,39 @@ output {
|
|||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
To verify your configuration, run the following command:
|
||||
Save your changes. To force Filebeat to read the log file from scratch, as you did earlier, shut down Filebeat (press Ctrl+C),
|
||||
delete the registry file, and then restart Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f first-pipeline.conf --config.test_and_exit
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `--config.test_and_exit` option parses your configuration file and reports any errors. When the configuration file
|
||||
passes the configuration test, start Logstash with the following command:
|
||||
[float]
|
||||
[[testing-initial-pipeline]]
|
||||
===== Testing Your Pipeline
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f first-pipeline.conf
|
||||
--------------------------------------------------------------------------------
|
||||
Now that the Logstash pipeline is configured to index the data into an
|
||||
Elasticsearch cluster, you can query Elasticsearch.
|
||||
|
||||
Try a test query to Elasticsearch based on the fields created by the `grok` filter plugin:
|
||||
Try a test query to Elasticsearch based on the fields created by the `grok` filter plugin.
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=response=200'
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format.
|
||||
NOTE: The date used in the index name is based on UTC, not the timezone where Logstash is running.
|
||||
If the query returns `index_not_found_exception`, make sure that `logstash-$DATE` reflects the actual
|
||||
name of the index. To see a list of available indexes, use this query: `curl 'localhost:9200/_cat/indices?v'`.
|
||||
|
||||
We get multiple hits back. For example:
|
||||
You should get multiple hits back. For example:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"took" : 4,
|
||||
"took" : 21,
|
||||
"timed_out" : false,
|
||||
"_shards" : {
|
||||
"total" : 5,
|
||||
|
|
@ -256,64 +468,80 @@ We get multiple hits back. For example:
|
|||
},
|
||||
"hits" : {
|
||||
"total" : 98,
|
||||
"max_score" : 4.833623,
|
||||
"hits" : [ {
|
||||
"_index" : "logstash-2016.05.27",
|
||||
"_type" : "logs",
|
||||
"_id" : "AVT0nBiGe_tzyi1erg7-",
|
||||
"_score" : 4.833623,
|
||||
"_source" : {
|
||||
"request" : "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
|
||||
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"geoip" : {
|
||||
"timezone" : "Europe/Moscow",
|
||||
"ip" : "83.149.9.216",
|
||||
"latitude" : 55.7522,
|
||||
"continent_code" : "EU",
|
||||
"city_name" : "Moscow",
|
||||
"country_code2" : "RU",
|
||||
"country_name" : "Russia",
|
||||
"dma_code" : null,
|
||||
"country_code3" : "RU",
|
||||
"region_name" : "Moscow",
|
||||
"location" : [ 37.6156, 55.7522 ],
|
||||
"postal_code" : "101194",
|
||||
"longitude" : 37.6156,
|
||||
"region_code" : "MOW"
|
||||
},
|
||||
"auth" : "-",
|
||||
"ident" : "-",
|
||||
"verb" : "GET",
|
||||
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] \"GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1\" 200 52878 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
|
||||
"@timestamp" : "2016-05-27T23:45:50.828Z",
|
||||
"response" : "200",
|
||||
"bytes" : "52878",
|
||||
"clientip" : "83.149.9.216",
|
||||
"@version" : "1",
|
||||
"host" : "myexamplehost",
|
||||
"httpversion" : "1.1",
|
||||
"timestamp" : "04/Jan/2015:05:13:45 +0000"
|
||||
"max_score" : 3.745223,
|
||||
"hits" : [
|
||||
{
|
||||
"_index" : "logstash-2016.10.11",
|
||||
"_type" : "log",
|
||||
"_id" : "AVe14gMiYMkU36o_eVsA",
|
||||
"_score" : 3.745223,
|
||||
"_source" : {
|
||||
"request" : "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
|
||||
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"geoip" : {
|
||||
"timezone" : "Europe/Moscow",
|
||||
"ip" : "83.149.9.216",
|
||||
"latitude" : 55.7522,
|
||||
"continent_code" : "EU",
|
||||
"city_name" : "Moscow",
|
||||
"country_code2" : "RU",
|
||||
"country_name" : "Russia",
|
||||
"dma_code" : null,
|
||||
"country_code3" : "RU",
|
||||
"region_name" : "Moscow",
|
||||
"location" : [
|
||||
37.6156,
|
||||
55.7522
|
||||
],
|
||||
"postal_code" : "101194",
|
||||
"longitude" : 37.6156,
|
||||
"region_code" : "MOW"
|
||||
},
|
||||
"offset" : 2932,
|
||||
"auth" : "-",
|
||||
"ident" : "-",
|
||||
"input_type" : "log",
|
||||
"verb" : "GET",
|
||||
"source" : "/path/to/file/logstash-tutorial.log",
|
||||
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] \"GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1\" 200 52878 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"type" : "log",
|
||||
"tags" : [
|
||||
"beats_input_codec_plain_applied"
|
||||
],
|
||||
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
|
||||
"@timestamp" : "2016-10-11T22:34:25.317Z",
|
||||
"response" : "200",
|
||||
"bytes" : "52878",
|
||||
"clientip" : "83.149.9.216",
|
||||
"@version" : "1",
|
||||
"beat" : {
|
||||
"hostname" : "My-MacBook-Pro.local",
|
||||
"name" : "My-MacBook-Pro.local"
|
||||
},
|
||||
"host" : "My-MacBook-Pro.local",
|
||||
"httpversion" : "1.1",
|
||||
"timestamp" : "04/Jan/2015:05:13:45 +0000"
|
||||
}
|
||||
}
|
||||
},
|
||||
...
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Try another search for the geographic information derived from the IP address:
|
||||
Try another search for the geographic information derived from the IP address.
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=geoip.city_name=Buffalo'
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format.
|
||||
|
||||
A few log entries come from Buffalo, so the query produces the following response:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"took" : 2,
|
||||
"took" : 3,
|
||||
"timed_out" : false,
|
||||
"_shards" : {
|
||||
"total" : 5,
|
||||
|
|
@ -322,49 +550,69 @@ A few log entries come from Buffalo, so the query produces the following respons
|
|||
},
|
||||
"hits" : {
|
||||
"total" : 3,
|
||||
"max_score" : 1.0520113,
|
||||
"hits" : [ {
|
||||
"_index" : "logstash-2016.05.27",
|
||||
"_type" : "logs",
|
||||
"_id" : "AVT0nBiHe_tzyi1erg9T",
|
||||
"_score" : 1.0520113,
|
||||
"_source" : {
|
||||
"request" : "/blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29",
|
||||
"agent" : "\"Tiny Tiny RSS/1.11 (http://tt-rss.org/)\"",
|
||||
"geoip" : {
|
||||
"timezone" : "America/New_York",
|
||||
"ip" : "198.46.149.143",
|
||||
"latitude" : 42.9864,
|
||||
"continent_code" : "NA",
|
||||
"city_name" : "Buffalo",
|
||||
"country_code2" : "US",
|
||||
"country_name" : "United States",
|
||||
"dma_code" : 514,
|
||||
"country_code3" : "US",
|
||||
"region_name" : "New York",
|
||||
"location" : [ -78.7279, 42.9864 ],
|
||||
"postal_code" : "14221",
|
||||
"longitude" : -78.7279,
|
||||
"region_code" : "NY"
|
||||
},
|
||||
"auth" : "-",
|
||||
"ident" : "-",
|
||||
"verb" : "GET",
|
||||
"message" : "198.46.149.143 - - [04/Jan/2015:05:29:13 +0000] \"GET /blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1\" 200 10756 \"-\" \"Tiny Tiny RSS/1.11 (http://tt-rss.org/)\"",
|
||||
"referrer" : "\"-\"",
|
||||
"@timestamp" : "2016-05-27T23:45:50.836Z",
|
||||
"response" : "200",
|
||||
"bytes" : "10756",
|
||||
"clientip" : "198.46.149.143",
|
||||
"@version" : "1",
|
||||
"host" : "myexamplehost",
|
||||
"httpversion" : "1.1",
|
||||
"timestamp" : "04/Jan/2015:05:29:13 +0000"
|
||||
}
|
||||
},
|
||||
...
|
||||
"max_score" : 2.6390574,
|
||||
"hits" : [
|
||||
{
|
||||
"_index" : "logstash-2016.10.11",
|
||||
"_type" : "log",
|
||||
"_id" : "AVe14gMjYMkU36o_eVtO",
|
||||
"_score" : 2.6390574,
|
||||
"_source" : {
|
||||
"request" : "/?flav=rss20",
|
||||
"agent" : "\"-\"",
|
||||
"geoip" : {
|
||||
"timezone" : "America/New_York",
|
||||
"ip" : "108.174.55.234",
|
||||
"latitude" : 42.9864,
|
||||
"continent_code" : "NA",
|
||||
"city_name" : "Buffalo",
|
||||
"country_code2" : "US",
|
||||
"country_name" : "United States",
|
||||
"dma_code" : 514,
|
||||
"country_code3" : "US",
|
||||
"region_name" : "New York",
|
||||
"location" : [
|
||||
-78.7279,
|
||||
42.9864
|
||||
],
|
||||
"postal_code" : "14221",
|
||||
"longitude" : -78.7279,
|
||||
"region_code" : "NY"
|
||||
},
|
||||
"offset" : 21471,
|
||||
"auth" : "-",
|
||||
"ident" : "-",
|
||||
"input_type" : "log",
|
||||
"verb" : "GET",
|
||||
"source" : "/path/to/file/logstash-tutorial.log",
|
||||
"message" : "108.174.55.234 - - [04/Jan/2015:05:27:45 +0000] \"GET /?flav=rss20 HTTP/1.1\" 200 29941 \"-\" \"-\"",
|
||||
"type" : "log",
|
||||
"tags" : [
|
||||
"beats_input_codec_plain_applied"
|
||||
],
|
||||
"referrer" : "\"-\"",
|
||||
"@timestamp" : "2016-10-11T22:34:25.318Z",
|
||||
"response" : "200",
|
||||
"bytes" : "29941",
|
||||
"clientip" : "108.174.55.234",
|
||||
"@version" : "1",
|
||||
"beat" : {
|
||||
"hostname" : "My-MacBook-Pro.local",
|
||||
"name" : "My-MacBook-Pro.local"
|
||||
},
|
||||
"host" : "My-MacBook-Pro.local",
|
||||
"httpversion" : "1.1",
|
||||
"timestamp" : "04/Jan/2015:05:27:45 +0000"
|
||||
}
|
||||
},
|
||||
...
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
You've successfully created a pipeline that uses Filebeat to take Apache web logs as input, parses those logs to
|
||||
create specific, named fields from the logs, and writes the parsed data to an Elasticsearch cluster. Next, you
|
||||
learn how to create a pipeline that uses multiple input and output plugins.
|
||||
|
||||
[[multiple-input-output-plugins]]
|
||||
=== Stitching Together Multiple Input and Output Plugins
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue