mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
Doc: Add note and example for date math in conditionals (#13242)
Backports #13199 to 7.x Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
This commit is contained in:
Karen Metts
GitHub
parent
eb776ca411
commit
d3b92d58cd
1 changed files with 100 additions and 51 deletions
151
docs/static/configuration.asciidoc
vendored
151
docs/static/configuration.asciidoc
vendored
|
|
@ -27,7 +27,7 @@ Et voilà! Logstash reads the specified configuration file and outputs to both
|
|||
move on to some <<config-examples,more complex examples>>, let's take a closer look at what's in a config file.
|
||||
|
||||
[[configuration-file-structure]]
|
||||
=== Structure of a Config File
|
||||
=== Structure of a config file
|
||||
|
||||
A Logstash config file has a separate section for each type of plugin you want to add to the event processing pipeline. For example:
|
||||
|
||||
|
|
@ -52,9 +52,9 @@ Each section contains the configuration options for one or more plugins. If you
|
|||
multiple filters, they are applied in the order of their appearance in the configuration file.
|
||||
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[plugin_configuration]]
|
||||
=== Plugin Configuration
|
||||
=== Plugin configuration
|
||||
|
||||
The configuration of a plugin consists of the plugin name followed
|
||||
by a block of settings for that plugin. For example, this input section configures two file inputs:
|
||||
|
|
@ -78,9 +78,9 @@ In this example, two settings are configured for each of the file inputs: 'path'
|
|||
|
||||
The settings you can configure vary according to the plugin type. For information about each plugin, see <<input-plugins,Input Plugins>>, <<output-plugins, Output Plugins>>, <<filter-plugins,Filter Plugins>>, and <<codec-plugins,Codec Plugins>>.
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[plugin-value-types]]
|
||||
=== Value Types
|
||||
=== Value types
|
||||
|
||||
A plugin can require that the value for a setting be a
|
||||
certain type, such as boolean, list, or hash. The following value
|
||||
|
|
@ -99,7 +99,7 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[list]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Lists
|
||||
|
||||
Not a type in and of itself, but a property types can have.
|
||||
|
|
@ -118,7 +118,7 @@ This example configures `path`, which is a `string` to be a list that contains a
|
|||
|
||||
|
||||
[[boolean]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Boolean
|
||||
|
||||
A boolean must be either `true` or `false`. Note that the `true` and `false` keywords
|
||||
|
|
@ -132,7 +132,7 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[bytes]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Bytes
|
||||
|
||||
A bytes field is a string field that represents a valid unit of bytes. It is a
|
||||
|
|
@ -153,7 +153,7 @@ Examples:
|
|||
----------------------------------
|
||||
|
||||
[[codec]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Codec
|
||||
|
||||
A codec is the name of Logstash codec used to represent the data. Codecs can be
|
||||
|
|
@ -173,7 +173,7 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[hash]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Hash
|
||||
|
||||
A hash is a collection of key value pairs specified in the format `"field1" => "value1"`.
|
||||
|
|
@ -193,7 +193,7 @@ match => { "field1" => "value1" "field2" => "value2" }
|
|||
----------------------------------
|
||||
|
||||
[[number]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Number
|
||||
|
||||
Numbers must be valid numeric values (floating point or integer).
|
||||
|
|
@ -206,7 +206,7 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[password]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Password
|
||||
|
||||
A password is a string with a single value that is not logged or printed.
|
||||
|
|
@ -219,7 +219,7 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[uri]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== URI
|
||||
|
||||
A URI can be anything from a full URL like 'http://elastic.co/' to a simple identifier
|
||||
|
|
@ -234,7 +234,7 @@ Example:
|
|||
|
||||
|
||||
[[path]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== Path
|
||||
|
||||
A path is a string that represents a valid operating system path.
|
||||
|
|
@ -247,13 +247,13 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[string]]
|
||||
[float]
|
||||
[discrete]
|
||||
==== String
|
||||
|
||||
A string must be a single character sequence. Note that string values are
|
||||
enclosed in quotes, either double or single.
|
||||
|
||||
===== Escape Sequences
|
||||
===== Escape sequences
|
||||
|
||||
By default, escape sequences are not enabled. If you wish to use escape
|
||||
sequences in quoted strings, you will need to set
|
||||
|
|
@ -279,14 +279,14 @@ Example:
|
|||
----------------------------------
|
||||
|
||||
[[field-reference]]
|
||||
[float]
|
||||
==== Field Reference
|
||||
[discrete]
|
||||
==== Field reference
|
||||
|
||||
A Field Reference is a special <<string>> value representing the path to a field in an event, such as `@timestamp` or `[@timestamp]` to reference a top-level field, or `[client][ip]` to access a nested field.
|
||||
The <<field-references-deepdive>> provides detailed information about the structure of Field References.
|
||||
When provided as a configuration option, Field References need to be quoted and special characters must be escaped following the same rules as <<string>>.
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[comments]]
|
||||
=== Comments
|
||||
|
||||
|
|
@ -302,7 +302,7 @@ input { # comments can appear at the end of a line, too
|
|||
----------------------------------
|
||||
|
||||
[[event-dependent-configuration]]
|
||||
=== Accessing Event Data and Fields in the Configuration
|
||||
=== Accessing event data and fields in the configuration
|
||||
|
||||
The logstash agent is a processing pipeline with 3 stages: inputs -> filters ->
|
||||
outputs. Inputs generate events, filters modify them, outputs ship them
|
||||
|
|
@ -320,11 +320,11 @@ Because of their dependency on events and fields, the following configuration
|
|||
options will only work within filter and output blocks.
|
||||
|
||||
IMPORTANT: Field references, sprintf format and conditionals, described below,
|
||||
will not work in an input block.
|
||||
do not work in an input block.
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[logstash-config-field-references]]
|
||||
==== Field References
|
||||
==== Field references
|
||||
|
||||
It is often useful to be able to refer to a field by name. To do this,
|
||||
you can use the Logstash <<field-references-deepdive,field reference syntax>>.
|
||||
|
|
@ -359,7 +359,7 @@ field such as `request`, you can simply specify the field name.
|
|||
|
||||
For more detailed information, see <<field-references-deepdive>>.
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[sprintf]]
|
||||
==== sprintf format
|
||||
|
||||
|
|
@ -391,11 +391,11 @@ output {
|
|||
}
|
||||
----------------------------------
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[conditionals]]
|
||||
==== Conditionals
|
||||
|
||||
Sometimes you only want to filter or output an event under
|
||||
Sometimes you want to filter or output an event only under
|
||||
certain conditions. For that, you can use a conditional.
|
||||
|
||||
Conditionals in Logstash look and act the same way they do in programming
|
||||
|
|
@ -423,11 +423,11 @@ You can use the following comparison operators:
|
|||
* regexp: `=~`, `!~` (checks a pattern on the right against a string value on the left)
|
||||
* inclusion: `in`, `not in`
|
||||
|
||||
The supported boolean operators are:
|
||||
Supported boolean operators are:
|
||||
|
||||
* `and`, `or`, `nand`, `xor`
|
||||
|
||||
The supported unary operators are:
|
||||
Supported unary operators are:
|
||||
|
||||
* `!`
|
||||
|
||||
|
|
@ -510,18 +510,23 @@ doesn't exist versus a field that's simply false. The expression `if [foo]` retu
|
|||
|
||||
For more complex examples, see <<using-conditionals, Using Conditionals>>.
|
||||
|
||||
[float]
|
||||
NOTE: Sprintf date/time format in conditionals is not currently supported.
|
||||
A workaround using the `@metadata` field is available.
|
||||
See <<date-time>> for more details and an example.
|
||||
|
||||
|
||||
[discrete]
|
||||
[[metadata]]
|
||||
==== The @metadata field
|
||||
|
||||
In Logstash 1.5 and later, there is a special field called `@metadata`. The contents
|
||||
of `@metadata` will not be part of any of your events at output time, which
|
||||
In Logstash, there is a special field called `@metadata`. The contents
|
||||
of `@metadata` are not part of any of your events at output time, which
|
||||
makes it great to use for conditionals, or extending and building event fields
|
||||
with field reference and sprintf formatting.
|
||||
with field reference and `sprintf` formatting.
|
||||
|
||||
The following configuration file will yield events from STDIN. Whatever is
|
||||
typed will become the `message` field in the event. The `mutate` events in the
|
||||
filter block will add a few fields, some nested in the `@metadata` field.
|
||||
This configuration file yields events from STDIN. Whatever you type
|
||||
becomes the `message` field in the event. The `mutate` events in the
|
||||
filter block add a few fields, some nested in the `@metadata` field.
|
||||
|
||||
[source,ruby]
|
||||
----------------------------------
|
||||
|
|
@ -642,8 +647,7 @@ Pipeline main started
|
|||
That's it! No extra fields in the output, and a cleaner config file because you
|
||||
do not have to delete a "timestamp" field after conversion in the `date` filter.
|
||||
|
||||
Another use case is the CouchDB Changes input plugin (See
|
||||
https://github.com/logstash-plugins/logstash-input-couchdb_changes).
|
||||
Another use case is the https://github.com/logstash-plugins/logstash-input-couchdb_changes[CouchDB Changes input plugin].
|
||||
This plugin automatically captures CouchDB document field metadata into the
|
||||
`@metadata` field within the input plugin itself. When the events pass through
|
||||
to be indexed by Elasticsearch, the Elasticsearch output plugin allows you to
|
||||
|
|
@ -663,8 +667,53 @@ output {
|
|||
}
|
||||
----------------------------------
|
||||
|
||||
[discrete]
|
||||
[[date-time]]
|
||||
===== sprintf date/time format in conditionals
|
||||
|
||||
Sprintf date/time format in conditionals is not currently supported, but a workaround is available.
|
||||
Put the date calculation in a field so that you can use the field reference in a conditional.
|
||||
|
||||
*Example*
|
||||
|
||||
Using sprintf time format directly to add a field based on ingestion time _will not work_:
|
||||
// This counter example is formatted to be understated to help users avoid following a bad example
|
||||
|
||||
```
|
||||
----------
|
||||
# non-working example
|
||||
filter{
|
||||
if "%{+HH}:%{+mm}" < "16:30" {
|
||||
mutate {
|
||||
add_field => { "string_compare" => "%{+HH}:%{+mm} is before 16:30" }
|
||||
}
|
||||
}
|
||||
}
|
||||
----------
|
||||
```
|
||||
|
||||
This workaround gives you the intended results:
|
||||
|
||||
[source,js]
|
||||
----------------------------------
|
||||
filter {
|
||||
mutate{
|
||||
add_field => {
|
||||
"[@metadata][time]" => "%{+HH}:%{+mm}"
|
||||
}
|
||||
}
|
||||
if [@metadata][time] < "16:30" {
|
||||
mutate {
|
||||
add_field => {
|
||||
"string_compare" => "%{+HH}:%{+mm} is before 16:30"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
----------------------------------
|
||||
|
||||
[[environment-variables]]
|
||||
=== Using Environment Variables in the Configuration
|
||||
=== Using environment variables in the configuration
|
||||
|
||||
==== Overview
|
||||
|
||||
|
|
@ -682,7 +731,7 @@ environment variable is undefined.
|
|||
The following examples show you how to use environment variables to set the values of some commonly used
|
||||
configuration options.
|
||||
|
||||
===== Setting the TCP Port
|
||||
===== Setting the TCP port
|
||||
|
||||
Here's an example that uses an environment variable to set the TCP port:
|
||||
|
||||
|
|
@ -739,7 +788,7 @@ input {
|
|||
|
||||
If the environment variable is defined, Logstash uses the value specified for the variable instead of the default.
|
||||
|
||||
===== Setting the Value of a Tag
|
||||
===== Setting the value of a tag
|
||||
|
||||
Here's an example that uses an environment variable to set the value of a tag:
|
||||
|
||||
|
|
@ -770,7 +819,7 @@ filter {
|
|||
}
|
||||
----
|
||||
|
||||
===== Setting a File Path
|
||||
===== Setting a file path
|
||||
|
||||
Here's an example that uses an environment variable to set the path to a log file:
|
||||
|
||||
|
|
@ -807,16 +856,16 @@ filter {
|
|||
|
||||
|
||||
[[config-examples]]
|
||||
=== Logstash Configuration Examples
|
||||
The following examples illustrate how you can configure Logstash to filter events, process Apache logs and syslog messages, and use conditionals to control what events are processed by a filter or output.
|
||||
=== Logstash configuration examples
|
||||
These examples illustrate how you can configure Logstash to filter events, process Apache logs and syslog messages, and use conditionals to control what events are processed by a filter or output.
|
||||
|
||||
TIP: If you need help building grok patterns, try out the
|
||||
{kibana-ref}/xpack-grokdebugger.html[Grok Debugger]. The Grok Debugger is an
|
||||
{xpack} feature under the Basic License and is therefore *free to use*.
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[filter-example]]
|
||||
==== Configuring Filters
|
||||
==== Configuring filters
|
||||
Filters are an in-line processing mechanism that provide the flexibility to slice and dice your data to fit your needs. Let's take a look at some filters in action. The following configuration file sets up the `grok` and `date` filters.
|
||||
|
||||
[source,ruby]
|
||||
|
|
@ -880,8 +929,8 @@ As you can see, Logstash (with help from the `grok` filter) was able to parse th
|
|||
|
||||
The other filter used in this example is the `date` filter. This filter parses out a timestamp and uses it as the timestamp for the event (regardless of when you're ingesting the log data). You'll notice that the `@timestamp` field in this example is set to December 11, 2013, even though Logstash is ingesting the event at some point afterwards. This is handy when backfilling logs. It gives you the ability to tell Logstash "use this value as the timestamp for this event".
|
||||
|
||||
[float]
|
||||
==== Processing Apache Logs
|
||||
[discrete]
|
||||
==== Processing Apache logs
|
||||
Let's do something that's actually *useful*: process apache2 access log files! We are going to read the input from a file on the localhost, and use a <<conditionals,conditional>> to process the event according to our needs. First, create a file called something like 'logstash-apache.conf' with the following contents (you can change the log's file path to suit your needs):
|
||||
|
||||
[source,js]
|
||||
|
|
@ -946,9 +995,9 @@ When you restart Logstash, it will process both the error and access logs. Howev
|
|||
|
||||
Note that Logstash did not reprocess the events that were already seen in the access_log file. When reading from a file, Logstash saves its position and only processes new lines as they are added. Neat!
|
||||
|
||||
[float]
|
||||
[discrete]
|
||||
[[using-conditionals]]
|
||||
==== Using Conditionals
|
||||
==== Using conditionals
|
||||
You use conditionals to control what events are processed by a filter or output. For example, you could label each event according to which file it appeared in (access_log, error_log, and other random files that end with "log").
|
||||
|
||||
[source,ruby]
|
||||
|
|
@ -1009,8 +1058,8 @@ output {
|
|||
}
|
||||
----------------------------------
|
||||
|
||||
[float]
|
||||
==== Processing Syslog Messages
|
||||
[discrete]
|
||||
==== Processing Syslog messages
|
||||
Syslog is one of the most common use cases for Logstash, and one it handles exceedingly well (as long as the log lines conform roughly to RFC3164). Syslog is the de facto UNIX networked logging standard, sending messages from client machines to a local file, or to a centralized log server via rsyslog. For this example, you won't need a functioning syslog instance; we'll fake it from the command line so you can get a feel for what happens.
|
||||
|
||||
First, let's make a simple configuration file for Logstash + syslog, called 'logstash-syslog.conf'.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue