elasticsearch/docs/reference/ingest/processors/dot-expand.asciidoc

[[dot-expand-processor]]
=== Dot expander processor
++++
<titleabbrev>Dot expander</titleabbrev>
++++

Expands a field with dots into an object field. This processor allows fields
with dots in the name to be accessible by other processors in the pipeline.
Otherwise these fields can't be accessed by any processor.

[[dot-expander-options]]
.Dot Expand Options
[options="header"]
|======
| Name      | Required  | Default  | Description
| `field`   | yes       | -        | The field to expand into an object field. If set to `*`, all top-level fields will be expanded.
| `path`    | no        | -        | The field that contains the field to expand. Only required if the field to expand is part another object field, because the `field` option can only understand leaf fields.
| `override`| no        | false    | Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When `false`, the processor will merge conflicts by combining the old and the new values into an array. When `true`, the value from the expanded field will overwrite the existing value.
include::common-options.asciidoc[]
|======

[source,js]
--------------------------------------------------
{
  "dot_expander": {
    "field": "foo.bar"
  }
}
--------------------------------------------------
// NOTCONSOLE

For example the dot expand processor would turn this document:

[source,js]
--------------------------------------------------
{
  "foo.bar" : "value"
}
--------------------------------------------------
// NOTCONSOLE

into:

[source,js]
--------------------------------------------------
{
  "foo" : {
    "bar" : "value"
  }
}
--------------------------------------------------
// NOTCONSOLE

If there is already a `bar` field nested under `foo` then
this processor merges the `foo.bar` field into it. If the field is
a scalar value then it will turn that field into an array field.

For example, the following document:

[source,js]
--------------------------------------------------
{
  "foo.bar" : "value2",
  "foo" : {
    "bar" : "value1"
  }
}
--------------------------------------------------
// NOTCONSOLE

is transformed by the `dot_expander` processor into:

[source,js]
--------------------------------------------------
{
  "foo" : {
    "bar" : ["value1", "value2"]
  }
}
--------------------------------------------------
// NOTCONSOLE

Contrast that with when the `override` option is set to `true`.

[source,js]
--------------------------------------------------
{
  "dot_expander": {
    "field": "foo.bar",
    "override": true
  }
}
--------------------------------------------------
// NOTCONSOLE

In that case, the value of the expanded field overrides the value of the nested object.

[source,js]
--------------------------------------------------
{
  "foo" : {
    "bar" : "value2"
  }
}
--------------------------------------------------
// NOTCONSOLE

'''

The value of `field` can also be set to a `*` to expand all top-level dotted field names:

[source,js]
--------------------------------------------------
{
  "dot_expander": {
    "field": "*"
  }
}
--------------------------------------------------
// NOTCONSOLE

The dot expand processor would turn this document:

[source,js]
--------------------------------------------------
{
  "foo.bar" : "value",
  "baz.qux" : "value"
}
--------------------------------------------------
// NOTCONSOLE

into:

[source,js]
--------------------------------------------------
{
  "foo" : {
    "bar" : "value"
  },
  "baz" : {
    "qux" : "value"
  }
}
--------------------------------------------------
// NOTCONSOLE

'''

If the dotted field is nested within a non-dotted structure, then use the `path` option to navigate the
non-dotted structure:

[source,js]
--------------------------------------------------
{
  "dot_expander": {
    "path": "foo"
    "field": "*"
  }
}
--------------------------------------------------
// NOTCONSOLE

The dot expand processor would turn this document:

[source,js]
--------------------------------------------------
{
  "foo" : {
    "bar.one" : "value",
    "bar.two" : "value"
  }
}
--------------------------------------------------
// NOTCONSOLE

into:

[source,js]
--------------------------------------------------
{
  "foo" : {
    "bar" : {
      "one" : "value",
      "two" : "value"
    }
  }
}
--------------------------------------------------
// NOTCONSOLE

'''

If any field outside of the leaf field conflicts with a pre-existing field of the same name,
then that field needs to be renamed first.

Consider the following document:

[source,js]
--------------------------------------------------
{
  "foo": "value1",
  "foo.bar": "value2"
}
--------------------------------------------------
// NOTCONSOLE

Then the `foo` needs to be renamed first before the `dot_expander`
processor is applied. So in order for the `foo.bar` field to properly
be expanded into the `bar` field under the `foo` field the following
pipeline should be used:

[source,js]
--------------------------------------------------
{
  "processors" : [
    {
      "rename" : {
        "field" : "foo",
        "target_field" : "foo.bar"
      }
    },
    {
      "dot_expander": {
        "field": "foo.bar"
      }
    }
  ]
}
--------------------------------------------------
// NOTCONSOLE

The reason for this is that Ingest doesn't know how to automatically cast
a scalar field to an object field.