mirror of
https://github.com/elastic/logstash.git
synced 2025-06-28 09:46:03 -04:00
307 lines
8.8 KiB
Text
307 lines
8.8 KiB
Text
[[configuration]]
|
|
== Configuring Logstash
|
|
|
|
To configure Logstash, you create a config file that specifies which plugins you want to use and settings for each plugin.
|
|
You can reference event fields in a configuration and use conditionals to process events when they meet certain
|
|
criteria. When you run logstash, you use the `-f` to specify your config file.
|
|
|
|
Let's step through creating a simple config file and using it to run Logstash. Create a file named "logstash-simple.conf" and save it in the same directory as Logstash.
|
|
|
|
[source,ruby]
|
|
----------------------------------
|
|
input { stdin { } }
|
|
output {
|
|
elasticsearch { hosts => ["localhost:9200"] }
|
|
stdout { codec => rubydebug }
|
|
}
|
|
----------------------------------
|
|
|
|
Then, run logstash and specify the configuration file with the `-f` flag.
|
|
|
|
[source,ruby]
|
|
----------------------------------
|
|
bin/logstash -f logstash-simple.conf
|
|
----------------------------------
|
|
|
|
Et voilà! Logstash reads the specified configuration file and outputs to both Elasticsearch and stdout. Note that if you see a message in stdout that reads "Elasticsearch Unreachable" that you will need to make sure Elasticsearch is installed and up and reachable on port 9200. Before we
|
|
move on to some <<config-examples,more complex examples>>, let's take a closer look at what's in a config file.
|
|
|
|
[[configuration-file-structure]]
|
|
=== Structure of a config file
|
|
|
|
A Logstash config file has a separate section for each type of plugin you want to add to the event processing pipeline. For example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
# This is a comment. You should use comments to describe
|
|
# parts of your configuration.
|
|
input {
|
|
...
|
|
}
|
|
|
|
filter {
|
|
...
|
|
}
|
|
|
|
output {
|
|
...
|
|
}
|
|
----------------------------------
|
|
|
|
Each section contains the configuration options for one or more plugins. If you specify
|
|
multiple filters, they are applied in the order of their appearance in the configuration file.
|
|
|
|
|
|
[discrete]
|
|
[[plugin_configuration]]
|
|
=== Plugin configuration
|
|
|
|
The configuration of a plugin consists of the plugin name followed
|
|
by a block of settings for that plugin. For example, this input section configures two file inputs:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
input {
|
|
file {
|
|
path => "/var/log/messages"
|
|
type => "syslog"
|
|
}
|
|
|
|
file {
|
|
path => "/var/log/apache/access.log"
|
|
type => "apache"
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
In this example, two settings are configured for each of the file inputs: 'path' and 'type'.
|
|
|
|
The settings you can configure vary according to the plugin type. For information about each plugin, see <<input-plugins,Input Plugins>>, <<output-plugins, Output Plugins>>, <<filter-plugins,Filter Plugins>>, and <<codec-plugins,Codec Plugins>>.
|
|
|
|
[discrete]
|
|
[[plugin-value-types]]
|
|
=== Value types
|
|
|
|
A plugin can require that the value for a setting be a
|
|
certain type, such as boolean, list, or hash. The following value
|
|
types are supported.
|
|
|
|
[[array]]
|
|
==== Array
|
|
|
|
This type is now mostly deprecated in favor of using a standard type like `string` with the plugin defining the `:list => true` property for better type checking. It is still needed to handle lists of hashes or mixed types where type checking is not desired.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
users => [ {id => 1, name => bob}, {id => 2, name => jane} ]
|
|
----------------------------------
|
|
|
|
[[list]]
|
|
[discrete]
|
|
==== Lists
|
|
|
|
Not a type in and of itself, but a property types can have.
|
|
This makes it possible to type check multiple values.
|
|
Plugin authors can enable list checking by specifying `:list => true` when declaring an argument.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
path => [ "/var/log/messages", "/var/log/*.log" ]
|
|
uris => [ "http://elastic.co", "http://example.net" ]
|
|
----------------------------------
|
|
|
|
This example configures `path`, which is a `string` to be a list that contains an element for each of the three strings. It also will configure the `uris` parameter to be a list of URIs, failing if any of the URIs provided are not valid.
|
|
|
|
|
|
[[boolean]]
|
|
[discrete]
|
|
==== Boolean
|
|
|
|
A boolean must be either `true` or `false`. Note that the `true` and `false` keywords
|
|
are not enclosed in quotes.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
ssl_enable => true
|
|
----------------------------------
|
|
|
|
[[bytes]]
|
|
[discrete]
|
|
==== Bytes
|
|
|
|
A bytes field is a string field that represents a valid unit of bytes. It is a
|
|
convenient way to declare specific sizes in your plugin options. Both SI (k M G T P E Z Y)
|
|
and Binary (Ki Mi Gi Ti Pi Ei Zi Yi) units are supported. Binary units are in
|
|
base-1024 and SI units are in base-1000. This field is case-insensitive
|
|
and accepts space between the value and the unit. If no unit is specified, the integer string
|
|
represents the number of bytes.
|
|
|
|
Examples:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
my_bytes => "1113" # 1113 bytes
|
|
my_bytes => "10MiB" # 10485760 bytes
|
|
my_bytes => "100kib" # 102400 bytes
|
|
my_bytes => "180 mb" # 180000000 bytes
|
|
----------------------------------
|
|
|
|
[[codec]]
|
|
[discrete]
|
|
==== Codec
|
|
|
|
A codec is the name of Logstash codec used to represent the data. Codecs can be
|
|
used in both inputs and outputs.
|
|
|
|
Input codecs provide a convenient way to decode your data before it enters the input.
|
|
Output codecs provide a convenient way to encode your data before it leaves the output.
|
|
Using an input or output codec eliminates the need for a separate filter in your Logstash pipeline.
|
|
|
|
A list of available codecs can be found at the <<codec-plugins,Codec Plugins>> page.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
codec => "json"
|
|
----------------------------------
|
|
|
|
[[hash]]
|
|
[discrete]
|
|
==== Hash
|
|
|
|
A hash is a collection of key value pairs specified in the format `"field1" => "value1"`.
|
|
Note that multiple key value entries are separated by spaces rather than commas.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
match => {
|
|
"field1" => "value1"
|
|
"field2" => "value2"
|
|
...
|
|
}
|
|
# or as a single line. No commas between entries:
|
|
match => { "field1" => "value1" "field2" => "value2" }
|
|
----------------------------------
|
|
|
|
[[number]]
|
|
[discrete]
|
|
==== Number
|
|
|
|
Numbers must be valid numeric values (floating point or integer).
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
port => 33
|
|
----------------------------------
|
|
|
|
[[password]]
|
|
[discrete]
|
|
==== Password
|
|
|
|
A password is a string with a single value that is not logged or printed.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
my_password => "password"
|
|
----------------------------------
|
|
|
|
[[uri]]
|
|
[discrete]
|
|
==== URI
|
|
|
|
A URI can be anything from a full URL like 'http://elastic.co/' to a simple identifier
|
|
like 'foobar'. If the URI contains a password such as 'http://user:pass@example.net' the password
|
|
portion of the URI will not be logged or printed.
|
|
|
|
Example:
|
|
[source,js]
|
|
----------------------------------
|
|
my_uri => "http://foo:bar@example.net"
|
|
----------------------------------
|
|
|
|
|
|
[[path]]
|
|
[discrete]
|
|
==== Path
|
|
|
|
A path is a string that represents a valid operating system path.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
my_path => "/tmp/logstash"
|
|
----------------------------------
|
|
|
|
[[string]]
|
|
[discrete]
|
|
==== String
|
|
|
|
A string must be a single character sequence. Note that string values are
|
|
enclosed in quotes, either double or single.
|
|
|
|
===== Escape sequences
|
|
|
|
By default, escape sequences are not enabled. If you wish to use escape
|
|
sequences in quoted strings, you will need to set
|
|
`config.support_escapes: true` in your `logstash.yml`. When `true`, quoted
|
|
strings (double and single) will have this transformation:
|
|
|
|
|===========================
|
|
| Text | Result
|
|
| \r | carriage return (ASCII 13)
|
|
| \n | new line (ASCII 10)
|
|
| \t | tab (ASCII 9)
|
|
| \\ | backslash (ASCII 92)
|
|
| \" | double quote (ASCII 34)
|
|
| \' | single quote (ASCII 39)
|
|
|===========================
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
name => "Hello world"
|
|
name => 'It\'s a beautiful day'
|
|
----------------------------------
|
|
|
|
[[field-reference]]
|
|
[discrete]
|
|
==== Field reference
|
|
|
|
A Field Reference is a special <<string>> value representing the path to a field in an event, such as `@timestamp` or `[@timestamp]` to reference a top-level field, or `[client][ip]` to access a nested field.
|
|
The <<field-references-deepdive>> provides detailed information about the structure of Field References.
|
|
When provided as a configuration option, Field References need to be quoted and special characters must be escaped following the same rules as <<string>>.
|
|
|
|
[discrete]
|
|
[[comments]]
|
|
=== Comments
|
|
|
|
Comments are the same as in perl, ruby, and python. A comment starts with a '#' character, and does not need to be at the beginning of a line. For example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
# this is a comment
|
|
|
|
input { # comments can appear at the end of a line, too
|
|
# ...
|
|
}
|
|
----------------------------------
|
|
|
|
|
|
include::event-data.asciidoc[]
|
|
include::env-vars.asciidoc[]
|
|
include::pipeline-config-exps.asciidoc[]
|