mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
5c75d004fa
Co-authored-by: debadair <debadair@elastic.co> Co-authored-by: Lisa Cawley <lcawley@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
547 lines
15 KiB
Text
547 lines
15 KiB
Text
[[dynamic-templates]]
|
|
=== Dynamic templates
|
|
|
|
Dynamic templates allow you greater control of how {es} maps your data beyond
|
|
the default <<dynamic-field-mapping,dynamic field mapping rules>>. You enable
|
|
dynamic mapping by setting the dynamic parameter to `true` or `runtime`. You
|
|
can then use dynamic templates to define custom mappings that can be applied to
|
|
dynamically added fields based on the matching condition:
|
|
|
|
* <<match-mapping-type,`match_mapping_type`>> operates on the data type that
|
|
{es} detects
|
|
* <<match-unmatch,`match` and `unmatch`>> use a pattern to match on the field
|
|
name
|
|
* <<path-match-unmatch,`path_match` and `path_unmatch`>> operate on the full
|
|
dotted path to the field
|
|
|
|
Use the `{name}` and `{dynamic_type}` <<template-variables,template variables>>
|
|
in the mapping specification as placeholders.
|
|
|
|
IMPORTANT: Dynamic field mappings are only added when a field contains a
|
|
concrete value -- not `null` or an empty array. If the
|
|
`null_value` option is used in a `dynamic_template`, it will only be applied
|
|
after the first document with a concrete value for the field has been
|
|
indexed.
|
|
|
|
Dynamic templates are specified as an array of named objects:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"dynamic_templates": [
|
|
{
|
|
"my_template_name": { <1>
|
|
... match conditions ... <2>
|
|
"mapping": { ... } <3>
|
|
}
|
|
},
|
|
...
|
|
]
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
<1> The template name can be any string value.
|
|
<2> The match conditions can include any of : `match_mapping_type`, `match`, `match_pattern`, `unmatch`, `path_match`, `path_unmatch`.
|
|
<3> The mapping that the matched field should use.
|
|
|
|
[[dynamic-templates-validation]]
|
|
==== Validating dynamic templates
|
|
If a provided mapping contains an invalid mapping snippet, a validation error
|
|
is returned. Validation occurs when applying the dynamic template at index time,
|
|
and, in most cases, when the dynamic template is updated. Providing an invalid mapping
|
|
snippet may cause the update or validation of a dynamic template to fail under certain conditions:
|
|
|
|
* If no `match_mapping_type` has been specified but the template is valid for at least one predefined mapping type,
|
|
the mapping snippet is considered valid. However, a validation error is returned at index time if a field matching
|
|
the template is indexed as a different type. For example, configuring a dynamic template with no `match_mapping_type`
|
|
is considered valid as string type, but if a field matching the dynamic template is indexed as a long, a validation
|
|
error is returned at index time. It is recommended to configure the `match_mapping_type` to the expected JSON type or
|
|
configure the desired `type` in the mapping snippet.
|
|
|
|
* If the `{name}` placeholder is used in the mapping snippet, validation is skipped when updating the dynamic
|
|
template. This is because the field name is unknown at that time. Instead, validation occurs when the template is applied
|
|
at index time.
|
|
|
|
Templates are processed in order -- the first matching template wins. When
|
|
putting new dynamic templates through the <<indices-put-mapping, update mapping>> API,
|
|
all existing templates are overwritten. This allows for dynamic templates to be
|
|
reordered or deleted after they were initially added.
|
|
|
|
[[dynamic-mapping-runtime-fields]]
|
|
==== Mapping runtime fields in a dynamic template
|
|
If you want {es} to dynamically map new fields of a certain type as runtime
|
|
fields, set `"dynamic":"runtime"` in the index mappings. These fields are not
|
|
indexed, and are loaded from `_source` at query time.
|
|
|
|
Alternatively, you can use the default dynamic mapping rules and then create
|
|
dynamic templates to map specific fields as runtime fields. You set
|
|
`"dynamic":"true"` in your index mapping, and then create a dynamic template to map
|
|
new fields of a certain type as runtime fields.
|
|
|
|
Let's say you have data where each of the fields start with `ip_`. Based on the
|
|
<<match-mapping-type,dynamic mapping rules>>, {es} maps any `string` that passes
|
|
`numeric` detection as a `float` or `long`. However, you can create a dynamic
|
|
template that maps new strings as runtime fields of type `ip`.
|
|
|
|
The following request defines a dynamic template named `strings_as_ip`. When
|
|
{es} detects new `string` fields matching the `ip*` pattern, it maps those
|
|
fields as runtime fields of type `ip`. Because `ip` fields aren't mapped
|
|
dynamically, you can use this template with either `"dynamic":"true"` or
|
|
`"dynamic":"runtime"`.
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001/
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"strings_as_ip": {
|
|
"match_mapping_type": "string",
|
|
"match": "ip*",
|
|
"runtime": {
|
|
"type": "ip"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
See <<text-only-mappings-strings,this example>> for how to use dynamic templates
|
|
to map `string` fields as either indexed fields or runtime fields.
|
|
|
|
[[match-mapping-type]]
|
|
==== `match_mapping_type`
|
|
|
|
The `match_mapping_type` is the data type detected by the JSON parser. Because
|
|
JSON doesn't distinguish a `long` from an `integer` or a `double` from
|
|
a `float`, it always chooses the wider data type such as `long` for integers
|
|
and `double` for floating-point numbers.
|
|
|
|
{es} automatically detects the following data types:
|
|
|
|
include::field-mapping.asciidoc[tag=dynamic-field-mapping-types-tag]
|
|
|
|
Use a wildcard (`*`) to match all data types.
|
|
|
|
For example, if we wanted to map all integer fields as `integer` instead of
|
|
`long`, and all `string` fields as both `text` and `keyword`, we
|
|
could use the following template:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"integers": {
|
|
"match_mapping_type": "long",
|
|
"mapping": {
|
|
"type": "integer"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"strings": {
|
|
"match_mapping_type": "string",
|
|
"mapping": {
|
|
"type": "text",
|
|
"fields": {
|
|
"raw": {
|
|
"type": "keyword",
|
|
"ignore_above": 256
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"my_integer": 5, <1>
|
|
"my_string": "Some string" <2>
|
|
}
|
|
--------------------------------------------------
|
|
|
|
<1> The `my_integer` field is mapped as an `integer`.
|
|
<2> The `my_string` field is mapped as a `text`, with a `keyword` <<multi-fields,multi field>>.
|
|
|
|
[[match-unmatch]]
|
|
==== `match` and `unmatch`
|
|
|
|
The `match` parameter uses a pattern to match on the field name, while
|
|
`unmatch` uses a pattern to exclude fields matched by `match`.
|
|
|
|
The `match_pattern` parameter adjusts the behavior of the `match` parameter
|
|
to support full Java regular expressions matching on the field name
|
|
instead of simple wildcards. For example:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"match_pattern": "regex",
|
|
"match": "^profit_\d+$"
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
The following example matches all `string` fields whose name starts with
|
|
`long_` (except for those which end with `_text`) and maps them as `long`
|
|
fields:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"longs_as_strings": {
|
|
"match_mapping_type": "string",
|
|
"match": "long_*",
|
|
"unmatch": "*_text",
|
|
"mapping": {
|
|
"type": "long"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"long_num": "5", <1>
|
|
"long_text": "foo" <2>
|
|
}
|
|
--------------------------------------------------
|
|
|
|
<1> The `long_num` field is mapped as a `long`.
|
|
<2> The `long_text` field uses the default `string` mapping.
|
|
|
|
[[path-match-unmatch]]
|
|
==== `path_match` and `path_unmatch`
|
|
|
|
The `path_match` and `path_unmatch` parameters work in the same way as `match`
|
|
and `unmatch`, but operate on the full dotted path to the field, not just the
|
|
final name, e.g. `some_object.*.some_field`.
|
|
|
|
This example copies the values of any fields in the `name` object to the
|
|
top-level `full_name` field, except for the `middle` field:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"full_name": {
|
|
"path_match": "name.*",
|
|
"path_unmatch": "*.middle",
|
|
"mapping": {
|
|
"type": "text",
|
|
"copy_to": "full_name"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"name": {
|
|
"first": "John",
|
|
"middle": "Winston",
|
|
"last": "Lennon"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
Note that the `path_match` and `path_unmatch` parameters match on object paths
|
|
in addition to leaf fields. As an example, indexing the following document will
|
|
result in an error because the `path_match` setting also matches the object
|
|
field `name.title`, which can't be mapped as text:
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001/_doc/2
|
|
{
|
|
"name": {
|
|
"first": "Paul",
|
|
"last": "McCartney",
|
|
"title": {
|
|
"value": "Sir",
|
|
"category": "order of chivalry"
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
// TEST[catch:bad_request]
|
|
|
|
[[template-variables]]
|
|
==== Template variables
|
|
|
|
The `{name}` and `{dynamic_type}` placeholders are replaced in the `mapping`
|
|
with the field name and detected dynamic type. The following example sets all
|
|
string fields to use an <<analyzer,`analyzer`>> with the same name as the
|
|
field, and disables <<doc-values,`doc_values`>> for all non-string fields:
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"named_analyzers": {
|
|
"match_mapping_type": "string",
|
|
"match": "*",
|
|
"mapping": {
|
|
"type": "text",
|
|
"analyzer": "{name}"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"no_doc_values": {
|
|
"match_mapping_type":"*",
|
|
"mapping": {
|
|
"type": "{dynamic_type}",
|
|
"doc_values": false
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"english": "Some English text", <1>
|
|
"count": 5 <2>
|
|
}
|
|
----
|
|
|
|
<1> The `english` field is mapped as a `string` field with the `english` analyzer.
|
|
<2> The `count` field is mapped as a `long` field with `doc_values` disabled.
|
|
|
|
[[template-examples]]
|
|
==== Dynamic template examples
|
|
|
|
Here are some examples of potentially useful dynamic templates:
|
|
|
|
===== Structured search
|
|
|
|
When you set `"dynamic":"true"`, {es} will map string fields as a `text` field with
|
|
a `keyword` subfield. If you are only indexing structured content and not
|
|
interested in full text search, you can make {es} map your fields
|
|
only as `keyword` fields. However, you must search on the exact same value that
|
|
was indexed to search those fields.
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"strings_as_keywords": {
|
|
"match_mapping_type": "string",
|
|
"mapping": {
|
|
"type": "keyword"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
[[text-only-mappings-strings]]
|
|
===== `text`-only mappings for strings
|
|
|
|
Contrary to the previous example, if you only care about full-text search on
|
|
string fields and don't plan on running aggregations, sorting, or exact
|
|
searches, you could tell instruct {es} to map strings as `text`:
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"strings_as_text": {
|
|
"match_mapping_type": "string",
|
|
"mapping": {
|
|
"type": "text"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
Alternatively, you can create a dynamic template to map your string fields as
|
|
`keyword` fields in the runtime section of the mapping. When {es} detects new
|
|
fields of type `string`, those fields will be created as runtime fields of
|
|
type `keyword`.
|
|
|
|
Although your `string` fields won't be indexed, their values are stored in
|
|
`_source` and can be used in search requests, aggregations, filtering, and
|
|
sorting.
|
|
|
|
For example, the following request creates a dynamic template to map `string`
|
|
fields as runtime fields of type `keyword`. Although the `runtime` definition
|
|
is blank, new `string` fields will be mapped as `keyword` runtime fields based
|
|
on the <<dynamic-field-mapping-types,dynamic mapping rules>> that {es} uses for
|
|
adding field types to the mapping. Any `string` that doesn't pass date
|
|
detection or numeric detection is automatically mapped as a `keyword`:
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"strings_as_keywords": {
|
|
"match_mapping_type": "string",
|
|
"runtime": {}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
You index a simple document:
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"english": "Some English text",
|
|
"count": 5
|
|
}
|
|
----
|
|
//TEST[continued]
|
|
|
|
When you view the mapping, you'll see that the `english` field is a runtime
|
|
field of type `keyword`:
|
|
|
|
[source,console]
|
|
----
|
|
GET my-index-000001/_mapping
|
|
----
|
|
//TEST[continued]
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"my-index-000001" : {
|
|
"mappings" : {
|
|
"dynamic_templates" : [
|
|
{
|
|
"strings_as_keywords" : {
|
|
"match_mapping_type" : "string",
|
|
"runtime" : { }
|
|
}
|
|
}
|
|
],
|
|
"runtime" : {
|
|
"english" : {
|
|
"type" : "keyword"
|
|
}
|
|
},
|
|
"properties" : {
|
|
"count" : {
|
|
"type" : "long"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----
|
|
|
|
===== Disabled norms
|
|
|
|
Norms are index-time scoring factors. If you do not care about scoring, which
|
|
would be the case for instance if you never sort documents by score, you could
|
|
disable the storage of these scoring factors in the index and save some space.
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"strings_as_keywords": {
|
|
"match_mapping_type": "string",
|
|
"mapping": {
|
|
"type": "text",
|
|
"norms": false,
|
|
"fields": {
|
|
"keyword": {
|
|
"type": "keyword",
|
|
"ignore_above": 256
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
The sub `keyword` field appears in this template to be consistent with the
|
|
default rules of dynamic mappings. Of course if you do not need them because
|
|
you don't need to perform exact search or aggregate on this field, you could
|
|
remove it as described in the previous section.
|
|
|
|
===== Time series
|
|
|
|
When doing time series analysis with Elasticsearch, it is common to have many
|
|
numeric fields that you will often aggregate on but never filter on. In such a
|
|
case, you could disable indexing on those fields to save disk space and also
|
|
maybe gain some indexing speed:
|
|
|
|
[source,console]
|
|
----
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_templates": [
|
|
{
|
|
"unindexed_longs": {
|
|
"match_mapping_type": "long",
|
|
"mapping": {
|
|
"type": "long",
|
|
"index": false
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"unindexed_doubles": {
|
|
"match_mapping_type": "double",
|
|
"mapping": {
|
|
"type": "float", <1>
|
|
"index": false
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
|
|
<1> Like the default dynamic mapping rules, doubles are mapped as floats, which
|
|
are usually accurate enough, yet require half the disk space.
|