mirror of
https://github.com/elastic/logstash.git
synced 2025-04-25 15:17:52 -04:00
393 lines
9.3 KiB
Text
393 lines
9.3 KiB
Text
== Logstash Config Language
|
|
[float]
|
|
=== Basic Layout
|
|
|
|
The Logstash config language aims to be simple.
|
|
|
|
There are 3 main sections: inputs, filters, outputs. Each section has configurations for each plugin available in that section.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
# This is a comment. You should use comments to describe
|
|
# parts of your configuration.
|
|
input {
|
|
...
|
|
}
|
|
|
|
filter {
|
|
...
|
|
}
|
|
|
|
output {
|
|
...
|
|
}
|
|
----------------------------------
|
|
[float]
|
|
=== Filters and Ordering
|
|
|
|
For a given event, filters are applied in the order of appearance in the configuration file.
|
|
[float]
|
|
=== Comments
|
|
|
|
Comments are the same as in perl, ruby, and python. A comment starts with a '#' character, and does not need to be at the beginning of a line. For example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
# this is a comment
|
|
|
|
input { # comments can appear at the end of a line, too
|
|
# ...
|
|
}
|
|
----------------------------------
|
|
[float]
|
|
[[plugin_configuration]]
|
|
=== Plugins
|
|
|
|
The input, filter and output sections all let you configure plugins. Plugin
|
|
configuration consists of the plugin name followed by a block of settings for
|
|
that plugin. For example, how about two file inputs:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
input {
|
|
file {
|
|
path => "/var/log/messages"
|
|
type => "syslog"
|
|
}
|
|
|
|
file {
|
|
path => "/var/log/apache/access.log"
|
|
type => "apache"
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
The above configures two file separate inputs. Both set two configuration settings each: 'path' and 'type'. Each plugin has different settings for configuring it; seek the documentation for your plugin to learn what settings are available and what they mean. For example, the [file input][fileinput] documentation will explain the meanings of the path and type settings.
|
|
|
|
[float]
|
|
=== Value Types
|
|
|
|
The documentation for a plugin may enforce a configuration field having a
|
|
certain type. Examples include boolean, string, array, number, hash,
|
|
etc.
|
|
[[boolean]]
|
|
[float]
|
|
==== Boolean
|
|
|
|
A boolean must be either `true` or `false`. Note the lack of quotes around `true` and `false`.
|
|
|
|
Examples:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
ssl_enable => true
|
|
----------------------------------
|
|
[[string]]
|
|
[float]
|
|
==== String
|
|
|
|
A string must be a single value.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
name => "Hello world"
|
|
----------------------------------
|
|
|
|
You should use quotes around string values.
|
|
[[number]]
|
|
[float]
|
|
==== Number
|
|
|
|
Numbers must be valid numerics (floating point or integer are OK).
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
port => 33
|
|
----------------------------------
|
|
[[array]]
|
|
[float]
|
|
==== Array
|
|
|
|
An array can be a single string value or multiple. If you specify the same
|
|
field multiple times, it appends to the array.
|
|
|
|
Examples:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
path => [ "/var/log/messages", "/var/log/*.log" ]
|
|
path => "/data/mysql/mysql.log"
|
|
----------------------------------
|
|
|
|
The above makes 'path' a 3-element array including all 3 strings.
|
|
[[hash]]
|
|
[float]
|
|
==== Hash
|
|
|
|
A hash is basically the same syntax as Ruby hashes.
|
|
The key and value are simply pairs, such as:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
match => {
|
|
"field1" => "value1"
|
|
"field2" => "value2"
|
|
...
|
|
}
|
|
----------------------------------
|
|
|
|
[[password]]
|
|
[float]
|
|
==== Password
|
|
|
|
A password field is basically a String field with a single value, but it will
|
|
not be logged or printed
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
my_password => "password"
|
|
----------------------------------
|
|
|
|
|
|
[[path]]
|
|
[float]
|
|
==== Path
|
|
|
|
A path field is a String field which represents a valid operating system path
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
my_path => "/tmp/logstash"
|
|
----------------------------------
|
|
|
|
[[codec]]
|
|
[float]
|
|
==== Codec
|
|
|
|
A codec is the name of Logstash codec used to represent the data. Codec can be
|
|
used in both inputs and outputs.
|
|
Input codecs are a convenient method for decoding your data before it enters the input,
|
|
without needing a separate filter in your Logstash pipeline.
|
|
Output codecs are a convenient method for encoding your data before it leaves the output,
|
|
without needing a separate filter in your Logstash pipeline.
|
|
|
|
Example:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
codec => "json"
|
|
----------------------------------
|
|
|
|
|
|
[float]
|
|
=== Field References
|
|
|
|
All events have properties. For example, an apache access log would have things
|
|
like status code (200, 404), request path ("/", "index.html"), HTTP verb (GET, POST),
|
|
client IP address, etc. Logstash calls these properties "fields."
|
|
|
|
In many cases, it is useful to be able to refer to a field by name. To do this,
|
|
you can use the Logstash field reference syntax.
|
|
|
|
By way of example, let us suppose we have this event:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
{
|
|
"agent": "Mozilla/5.0 (compatible; MSIE 9.0)",
|
|
"ip": "192.168.24.44",
|
|
"request": "/index.html"
|
|
"response": {
|
|
"status": 200,
|
|
"bytes": 52353
|
|
},
|
|
"ua": {
|
|
"os": "Windows 7"
|
|
}
|
|
}
|
|
|
|
----------------------------------
|
|
|
|
- the syntax to access fields is `[fieldname]`.
|
|
- if you are only referring to a **top-level field**, you can omit the `[]` and
|
|
simply say `fieldname`.
|
|
- in the case of **nested fields**, like the "os" field above, you need
|
|
the full path to that field: `[ua][os]`.
|
|
|
|
[float]
|
|
==== sprintf format
|
|
|
|
This syntax is also used in what Logstash calls 'sprintf format'. This format
|
|
allows you to refer to field values from within other strings. For example, the
|
|
statsd output has an 'increment' setting, to allow you to keep a count of
|
|
apache logs by status code:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
output {
|
|
statsd {
|
|
increment => "apache.%{[response][status]}"
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
You can also do time formatting in this sprintf format. Instead of specifying a field name, use the `+FORMAT` syntax where `FORMAT` is a [time format](http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html).
|
|
|
|
For example, if you want to use the file output to write to logs based on the
|
|
hour and the 'type' field:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
output {
|
|
file {
|
|
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
[float]
|
|
=== Conditionals
|
|
|
|
Sometimes you only want a filter or output to process an event under
|
|
certain conditions. For that, you'll want to use a conditional!
|
|
|
|
Conditionals in Logstash look and act the same way they do in programming
|
|
languages. You have `if`, `else if` and `else` statements. Conditionals may be
|
|
nested if you need that.
|
|
|
|
The syntax is follows:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
if EXPRESSION {
|
|
...
|
|
} else if EXPRESSION {
|
|
...
|
|
} else {
|
|
...
|
|
}
|
|
----------------------------------
|
|
|
|
What's an expression? Comparison tests, boolean logic, etc!
|
|
|
|
The following comparison operators are supported:
|
|
|
|
* equality, etc: ==, !=, <, >, <=, >=
|
|
* regexp: =~, !~
|
|
* inclusion: in, not in
|
|
|
|
The following boolean operators are supported:
|
|
|
|
* and, or, nand, xor
|
|
|
|
The following unary operators are supported:
|
|
|
|
* !
|
|
|
|
Expressions may contain expressions. Expressions may be negated with `!`.
|
|
Expressions may be grouped with parentheses `(...)`. Expressions can be long
|
|
and complex.
|
|
|
|
For example, if we want to remove the field `secret` if the field
|
|
`action` has a value of `login`:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
filter {
|
|
if [action] == "login" {
|
|
mutate { remove => "secret" }
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
The above uses the field reference syntax to get the value of the
|
|
`action` field. It is compared against the text `login` and, if equal,
|
|
allows the mutate filter to delete the field named `secret`.
|
|
|
|
How about a more complex example?
|
|
|
|
* alert nagios of any apache events with status 5xx
|
|
* record any 4xx status to elasticsearch
|
|
* record all status code hits via statsd
|
|
|
|
How about telling nagios of any http event that has a status code of 5xx?
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
output {
|
|
if [type] == "apache" {
|
|
if [status] =~ /^5\d\d/ {
|
|
nagios { ... }
|
|
} else if [status] =~ /^4\d\d/ {
|
|
elasticsearch { ... }
|
|
}
|
|
statsd { increment => "apache.%{status}" }
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
You can also do multiple expressions in a single condition:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
output {
|
|
# Send production errors to pagerduty
|
|
if [loglevel] == "ERROR" and [deployment] == "production" {
|
|
pagerduty {
|
|
...
|
|
}
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
Here are some examples for testing with the in conditional:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
filter {
|
|
if [foo] in [foobar] {
|
|
mutate { add_tag => "field in field" }
|
|
}
|
|
if [foo] in "foo" {
|
|
mutate { add_tag => "field in string" }
|
|
}
|
|
if "hello" in [greeting] {
|
|
mutate { add_tag => "string in field" }
|
|
}
|
|
if [foo] in ["hello", "world", "foo"] {
|
|
mutate { add_tag => "field in list" }
|
|
}
|
|
if [missing] in [alsomissing] {
|
|
mutate { add_tag => "shouldnotexist" }
|
|
}
|
|
if !("foo" in ["hello", "world"]) {
|
|
mutate { add_tag => "shouldexist" }
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
Or, to test if grok was successful:
|
|
|
|
[source,js]
|
|
----------------------------------
|
|
output {
|
|
if "_grokparsefailure" not in [tags] {
|
|
elasticsearch { ... }
|
|
}
|
|
}
|
|
----------------------------------
|
|
|
|
[float]
|
|
=== Further Reading
|
|
|
|
For more information, see [the plugin docs index](index)
|