logstash/docs/static/field-reference.asciidoc

[role="exclude",id="field-references-deepdive"]
== Field References Deep Dive

It is often useful to be able to refer to a field or collection of fields by name. To do this,
you can use the Logstash field reference syntax.

The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets.
When a field name contains square brackets, they must be properly <<formal-grammar-escape-sequences, _escaped_>>.

_Field References_ can be expressed literally within <<conditionals,_Conditional_>> statements in your pipeline configurations,
as string arguments to your pipeline plugins, or within sprintf statements that will be used by your pipeline plugins:

[source,pipelineconf]
filter {
  #  +----literal----+     +----literal----+
  #  |               |     |               |
  if [@metadata][date] and [@metadata][time] {
    mutate {
      add_field {
        "[@metadata][timestamp]" => "%{[@metadata][date]} %{[@metadata][time]}"
      # |                      |    |  |               |    |               | |
      # +----string-argument---+    |  +--field-ref----+    +--field-ref----+ |
      #                             +-------- sprintf format string ----------+
      }
    }
  }
}

[float]
[[formal-grammar]]
=== Formal Grammar

Below is the formal grammar of the Field Reference, with notes and examples.

[float]
[[formal-grammar-field-reference-literal]]
==== Field Reference Literal

A _Field Reference Literal_ is a sequence of one or more _Path Fragments_ that can be used directly in Logstash pipeline <<conditionals,conditionals>> without any additional quoting (e.g. `[request]`, `[response][status]`).

[source,antlr]
fieldReferenceLiteral
  : ( pathFragment )+
  ;

NOTE: In Logstash 7.x and earlier, a quoted value (such as `["foo"]`) is
considered a field reference and isn't treated as a single element array. This
behavior might cause confusion in conditionals, such as `[message] in ["foo",
"bar"]` compared to `[message] in ["foo"]`. We discourage using names with
quotes, such as `"\"foo\""`, as this behavior might change in the future.

[float]
[[formal-grammar-field-reference]]
==== Field Reference (Event APIs)

The Event API's methods for manipulating the fields of an event or using the sprintf syntax are more flexible than the pipeline grammar in what they accept as a Field Reference.
Top-level fields can be referenced directly by their _Field Name_ without the square brackets, and there is some support for _Composite Field References_, simplifying use of programmatically-generated Field References.

A _Field Reference_ for use with the Event API is therefore one of:

 - a single _Field Reference Literal_; OR
 - a single _Field Name_ (referencing a top-level field); OR
 - a single _Composite Field Reference_.

[source,antlr]
eventApiFieldReference
  : fieldReferenceLiteral
  | fieldName
  | compositeFieldReference
  ;

[float]
[[formal-grammar-path-fragment]]
==== Path Fragment

A _Path Fragment_ is a _Field Name_ wrapped in square brackets (e.g., `[request]`).

[source,antlr]
pathFragment
  : '[' fieldName ']'
  ;

[float]
[[formal-grammar-field-name]]
==== Field Name

A _Field Name_ is a sequence of characters that are _not_ square brackets (`[` or `]`).

[source,antlr]
fieldName
  : ( ~( '[' | ']' ) )+
  ;

[float]
[[formal-grammar-event-api-composite-field-reference]]
==== Composite Field Reference

In some cases, it may be necessary to programmatically _compose_ a Field Reference from one or more Field References,
such as when manipulating fields in a plugin or while using the Ruby Filter plugin and the Event API.

[source,ruby]
    fieldReference = "[path][to][deep nested field]"
    compositeFieldReference = "[@metadata][#{fieldReference}][size]"
    # => "[@metadata][[path][to][deep nested field]][size]"

// NOTE: table below uses "plus for passthrough" quoting to prevent double square-brackets
//       from being interpreted as asciidoc anchors when converted to HTML.
[float]
===== Canonical Representations of Composite Field References
|===
| Acceptable _Composite Field Reference_ | Canonical _Field Reference_ Representation

| `+[[deep][nesting]][field]+`           | `+[deep][nesting][field]+`
| `+[foo][[bar]][bingo]+`                | `+[foo][bar][bingo]+`
| `+[[ok]]+`                             | `+[ok]+`
|===

A _Composite Field Reference_ is a sequence of one or more _Path Fragments_ or _Embedded Field References_.

[source,antlr]
compositeFieldReference
  : ( pathFragment | embeddedFieldReference )+
  ;

_Composite Field References_ are supported by the Event API, but are _not_ supported as literals in the Pipeline Configuration.

[float]
[[formal-grammar-event-api-embedded-field-reference]]
==== Embedded Field Reference

[source,antlr]
embeddedFieldReference
  : '[' fieldReference ']'
  ;

An _Embedded Field Reference_ is a _Field Reference_ that is itself wrapped in square brackets (`[` and `]`), and can be a component of a _Composite Field Reference_.

[float]
[[formal-grammar-escape-sequences]]
=== Escape Sequences

For {ls} to reference a field whose name contains a character that has special meaning in the field reference grammar, the character must be escaped.
Logstash can be globally configured to use one of two field reference escape modes:

 - `none` (default): no escape sequence processing is done. Fields containing literal square brackets cannot be referenced by the Event API.
 - `percent`: URI-style percent encoding of UTF-8 bytes. The left square bracket (`[`) is expressed as `%5B`, and the right square bracket (`]`) is expressed as `%5D`.
 - `ampersand`: HTML-style ampersand encoding (`&#` + decimal unicode codepoint + `;`). The left square bracket (`[`) is expressed as `&#91;`, and the right square bracket (`]`) is expressed as `&#93;`.