mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 23:57:20 -04:00
2d3bcc483d
* [DOCS] Warn only one date format is added to the field date formats
When using multiple options in `dynamic_date_formats`, only one of the formats of the first document having a date matching one of the date formats provided will be used.
E.g.
```
PUT my-index-000001
{
"mappings": {
"dynamic_date_formats": [ "yyyy/MM", "MM/dd/yyyy"]
}
}
PUT my-index-000001/_doc/1
{
"create_date": "09/25/2015"
}
```
The generated mappings will be:
```
"mappings": {
"dynamic_date_formats": [
"yyyy/MM",
"MM/dd/yyyy"
],
"properties": {
"create_date": {
"type": "date",
"format": "MM/dd/yyyy"
}
}
},
```
Indexing a document with `2015/12` would lead to the `format` `"yyyy/MM"` being used for the `create_date`.
This can be misleading especially if the user is using multiple date formats on the same field.
The first document will determine the format of the `date` field being detected.
Maybe we should provide an additional example, such as:
```
PUT my-index-000001
{
"mappings": {
"dynamic_date_formats": [ "yyyy/MM||MM/dd/yyyy"]
}
}
```
My wording is not great, so feel free to amend/edit.
* Update docs/reference/mapping/dynamic/field-mapping.asciidoc
Reword and add code example
* Turned discussion of the two syntaxes into an admonition
* Fix failing tests
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
245 lines
6.4 KiB
Text
245 lines
6.4 KiB
Text
[[dynamic-field-mapping]]
|
|
=== Dynamic field mapping
|
|
|
|
When {es} detects a new field in a document, it _dynamically_ adds the field to
|
|
the type mapping by default. The <<dynamic,`dynamic`>> parameter controls this behavior.
|
|
|
|
You can explicitly instruct {es} to dynamically create fields based on incoming
|
|
documents by setting the `dynamic` parameter to `true` or `runtime`. When
|
|
dynamic field mapping is enabled, {es} uses the rules in the following table to
|
|
determine how to map data types for each field.
|
|
|
|
NOTE: The field data types in the following table are the only
|
|
<<mapping-types,field data types>> that {es} detects dynamically. You must
|
|
explicitly map all other data types.
|
|
|
|
[[dynamic-field-mapping-types]]
|
|
// tag::dynamic-field-mapping-types-tag[]
|
|
[cols="3",frame=all]
|
|
|===
|
|
h| 2+^h|{es} data type
|
|
h| JSON data type h| `"dynamic":"true"` h| `"dynamic":"runtime"`
|
|
|`null` 2*| No field added
|
|
|`true` or `false` 2*| `boolean`
|
|
|`double` | `float` | `double`
|
|
|`long` 2*| `long`
|
|
|`object` | `object` | No field added
|
|
|`array` 2*| Depends on the first non-`null` value in the array
|
|
|`string` that passes <<date-detection,date detection>> 2*| `date`
|
|
|`string` that passes <<numeric-detection,numeric detection>> | `float` or `long` | `double` or `long`
|
|
|`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword`
|
|
|===
|
|
// end::dynamic-field-mapping-types-tag[]
|
|
|
|
You can disable dynamic mapping, both at the document and at the
|
|
<<object,`object`>> level. Setting the `dynamic` parameter to
|
|
`false` ignores new fields, and `strict` rejects the document if {es}
|
|
encounters an unknown field.
|
|
|
|
TIP: Use the <<indices-put-mapping,update mapping API>> to update the `dynamic`
|
|
setting on existing fields.
|
|
|
|
You can customize dynamic field mapping rules for
|
|
<<date-detection,date detection>> and <<numeric-detection,numeric detection>>.
|
|
To define custom mappings rules that you can apply to additional dynamic
|
|
fields, use <<dynamic-templates,`dynamic_templates`>>.
|
|
|
|
[[date-detection]]
|
|
==== Date detection
|
|
|
|
If `date_detection` is enabled (default), then new string fields are checked
|
|
to see whether their contents match any of the date patterns specified in
|
|
`dynamic_date_formats`. If a match is found, a new <<date,`date`>> field is
|
|
added with the corresponding format.
|
|
|
|
The default value for `dynamic_date_formats` is:
|
|
|
|
[ <<strict-date-time,`"strict_date_optional_time"`>>,`"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"`]
|
|
|
|
For example:
|
|
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"create_date": "2015/09/02"
|
|
}
|
|
|
|
GET my-index-000001/_mapping <1>
|
|
--------------------------------------------------
|
|
|
|
<1> The `create_date` field has been added as a <<date,`date`>>
|
|
field with the <<mapping-date-format,`format`>>: +
|
|
`"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"`.
|
|
|
|
===== Disabling date detection
|
|
|
|
Dynamic date detection can be disabled by setting `date_detection` to `false`:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"date_detection": false
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1 <1>
|
|
{
|
|
"create_date": "2015/09/02"
|
|
}
|
|
--------------------------------------------------
|
|
|
|
<1> The `create_date` field has been added as a <<text,`text`>> field.
|
|
|
|
===== Customizing detected date formats
|
|
|
|
Alternatively, the `dynamic_date_formats` can be customized to support your
|
|
own <<mapping-date-format,date formats>>:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_date_formats": ["MM/dd/yyyy"]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"create_date": "09/25/2015"
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[NOTE]
|
|
====
|
|
There is a difference between configuring an array of date patterns and
|
|
configuring multiple patterns in a single string separated by `||`. When you
|
|
configure an array of date patterns, the pattern that matches the date in the
|
|
first document with an unmapped date field will determine the mapping of that
|
|
field:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_date_formats": [ "yyyy/MM", "MM/dd/yyyy"]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"create_date": "09/25/2015"
|
|
}
|
|
--------------------------------------------------
|
|
|
|
The resulting mapping will be:
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
GET my-index-000001/_mapping
|
|
----
|
|
//TEST[continued]
|
|
////
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"my-index-000001": {
|
|
"mappings": {
|
|
"dynamic_date_formats": [
|
|
"yyyy/MM",
|
|
"MM/dd/yyyy"
|
|
],
|
|
"properties": {
|
|
"create_date": {
|
|
"type": "date",
|
|
"format": "MM/dd/yyyy"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
Configuring multiple patterns in a single string separated by `||` results in a
|
|
mapping that supports any of the date formats. This enables you to index
|
|
documents that use different formats:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"dynamic_date_formats": [ "yyyy/MM||MM/dd/yyyy"]
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"create_date": "09/25/2015"
|
|
}
|
|
--------------------------------------------------
|
|
|
|
The resulting mapping will be:
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
GET my-index-000001/_mapping
|
|
----
|
|
//TEST[continued]
|
|
////
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"my-index-000001": {
|
|
"mappings": {
|
|
"dynamic_date_formats": [
|
|
"yyyy/MM||MM/dd/yyyy"
|
|
],
|
|
"properties": {
|
|
"create_date": {
|
|
"type": "date",
|
|
"format": "yyyy/MM||MM/dd/yyyy"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
====
|
|
|
|
[[numeric-detection]]
|
|
==== Numeric detection
|
|
|
|
While JSON has support for native floating point and integer data types, some
|
|
applications or languages may sometimes render numbers as strings. Usually the
|
|
correct solution is to map these fields explicitly, but numeric detection
|
|
(which is disabled by default) can be enabled to do this automatically:
|
|
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my-index-000001
|
|
{
|
|
"mappings": {
|
|
"numeric_detection": true
|
|
}
|
|
}
|
|
|
|
PUT my-index-000001/_doc/1
|
|
{
|
|
"my_float": "1.0", <1>
|
|
"my_integer": "1" <2>
|
|
}
|
|
--------------------------------------------------
|
|
|
|
<1> The `my_float` field is added as a <<number,`float`>> field.
|
|
<2> The `my_integer` field is added as a <<number,`long`>> field.
|