mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
135 lines
4.8 KiB
Text
135 lines
4.8 KiB
Text
[role="exclude",id="field-references-deepdive"]
|
|
== Field References Deep Dive
|
|
|
|
It is often useful to be able to refer to a field or collection of fields by name. To do this,
|
|
you can use the Logstash field reference syntax.
|
|
|
|
The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets.
|
|
|
|
_Field References_ can be expressed literally within <<conditionals,_Conditional_>> statements in your pipeline configurations,
|
|
as string arguments to your pipeline plugins, or within sprintf statements that will be used by your pipeline plugins:
|
|
|
|
[source,pipelineconf]
|
|
filter {
|
|
# +----literal----+ +----literal----+
|
|
# | | | |
|
|
if [@metadata][date] and [@metadata][time] {
|
|
mutate {
|
|
add_field {
|
|
"[@metadata][timestamp]" => "%{[@metadata][date]} %{[@metadata][time]}"
|
|
# | | | | | | | |
|
|
# +----string-argument---+ | +--field-ref----+ +--field-ref----+ |
|
|
# +-------- sprintf format string ----------+
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
[float]
|
|
[[formal-grammar]]
|
|
=== Formal Grammar
|
|
|
|
Below is the formal grammar of the Field Reference, with notes and examples.
|
|
|
|
[float]
|
|
[[formal-grammar-field-reference-literal]]
|
|
==== Field Reference Literal
|
|
|
|
A _Field Reference Literal_ is a sequence of one or more _Path Fragments_ that can be used directly in Logstash pipeline <<conditionals,conditionals>> without any additional quoting (e.g. `[request]`, `[response][status]`).
|
|
|
|
[source,antlr]
|
|
fieldReferenceLiteral
|
|
: ( pathFragment )+
|
|
;
|
|
|
|
NOTE: In Logstash 7.x and earlier, a quoted value (such as `["foo"]`) is
|
|
considered a field reference and isn't treated as a single element array. This
|
|
behavior might cause confusion in conditionals, such as `[message] in ["foo",
|
|
"bar"]` compared to `[message] in ["foo"]`. We discourage using names with
|
|
quotes, such as `"\"foo\""`, as this behavior might change in the future.
|
|
|
|
[float]
|
|
[[formal-grammar-field-reference]]
|
|
==== Field Reference (Event APIs)
|
|
|
|
The Event API's methods for manipulating the fields of an event or using the sprintf syntax are more flexible than the pipeline grammar in what they accept as a Field Reference.
|
|
Top-level fields can be referenced directly by their _Field Name_ without the square brackets, and there is some support for _Composite Field References_, simplifying use of programmatically-generated Field References.
|
|
|
|
A _Field Reference_ for use with the Event API is therefore one of:
|
|
|
|
- a single _Field Reference Literal_; OR
|
|
- a single _Field Name_ (referencing a top-level field); OR
|
|
- a single _Composite Field Reference_.
|
|
|
|
[source,antlr]
|
|
eventApiFieldReference
|
|
: fieldReferenceLiteral
|
|
| fieldName
|
|
| compositeFieldReference
|
|
;
|
|
|
|
[float]
|
|
[[formal-grammar-path-fragment]]
|
|
==== Path Fragment
|
|
|
|
A _Path Fragment_ is a _Field Name_ wrapped in square brackets (e.g., `[request]`).
|
|
|
|
[source,antlr]
|
|
pathFragment
|
|
: '[' fieldName ']'
|
|
;
|
|
|
|
[float]
|
|
[[formal-grammar-field-name]]
|
|
==== Field Name
|
|
|
|
A _Field Name_ is a sequence of characters that are _not_ square brackets (`[` or `]`).
|
|
|
|
[source,antlr]
|
|
fieldName
|
|
: ( ~( '[' | ']' ) )+
|
|
;
|
|
|
|
[float]
|
|
[[formal-grammar-event-api-composite-field-reference]]
|
|
==== Composite Field Reference
|
|
|
|
In some cases, it may be necessary to programmatically _compose_ a Field Reference from one or more Field References,
|
|
such as when manipulating fields in a plugin or while using the Ruby Filter plugin and the Event API.
|
|
|
|
[source,ruby]
|
|
fieldReference = "[path][to][deep nested field]"
|
|
compositeFieldReference = "[@metadata][#{fieldReference}][size]"
|
|
# => "[@metadata][[path][to][deep nested field]][size]"
|
|
|
|
// NOTE: table below uses "plus for passthrough" quoting to prevent double square-brackets
|
|
// from being interpreted as asciidoc anchors when converted to HTML.
|
|
[float]
|
|
===== Canonical Representations of Composite Field References
|
|
|===
|
|
| Acceptable _Composite Field Reference_ | Canonical _Field Reference_ Representation
|
|
|
|
| `+[[deep][nesting]][field]+` | `+[deep][nesting][field]+`
|
|
| `+[foo][[bar]][bingo]+` | `+[foo][bar][bingo]+`
|
|
| `+[[ok]]+` | `+[ok]+`
|
|
|===
|
|
|
|
A _Composite Field Reference_ is a sequence of one or more _Path Fragments_ or _Embedded Field References_.
|
|
|
|
[source,antlr]
|
|
compositeFieldReference
|
|
: ( pathFragment | embeddedFieldReference )+
|
|
;
|
|
|
|
_Composite Field References_ are supported by the Event API, but are _not_ supported as literals in the Pipeline Configuration.
|
|
|
|
[float]
|
|
[[formal-grammar-event-api-embedded-field-reference]]
|
|
==== Embedded Field Reference
|
|
|
|
[source,antlr]
|
|
embeddedFieldReference
|
|
: '[' fieldReference ']'
|
|
;
|
|
|
|
An _Embedded Field Reference_ is a _Field Reference_ that is itself wrapped in square brackets (`[` and `]`), and can be a component of a _Composite Field Reference_.
|