[DOCS] Updates to the runtime fields docs for 7.11+ (#67484)

* Moving examples to the page for retrieving runtime fields.

* Adding runtime_mappings to request body of search API.

* Updating runtime_mappings properties and adding runtime fields to search your data.

* Updating examples and hopefully fixing build failure.

* Fixing snippet formatting that was causing test failure.

* Adding page in Painless guide for runtime fields.

* Fixing typo.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit is contained in:
Adam Locke 2021-01-19 10:31:17 -05:00 committed by GitHub
parent 3e34247570
commit 744f7c67c8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 355 additions and 102 deletions

View file

@ -1,5 +1,7 @@
include::painless-walkthrough.asciidoc[]
include::painless-runtime-fields.asciidoc[]
include::painless-datetime.asciidoc[]
include::painless-method-dispatch.asciidoc[]

View file

@ -0,0 +1,89 @@
[[painless-runtime-fields]]
=== Use Painless scripts in runtime fields
A runtime field is a field that is evaluated at query time. When you define a
runtime field, you can immediately use it in search requests, aggregations,
filtering, and sorting.
When defining a runtime field, you can include a Painless script that is
evaluated at query time. This script has access to the entire context of a
document, including the original `_source` and any mapped fields plus their
values. At query time, the script runs and generates values for each scripted
field that is included in the query.
You can map a runtime field in the `runtime` section under the mapping
definition, or define runtime fields that exist only as part of a search
request. The script syntax is the same, regardless of where you define the
runtime field.
IMPORTANT: When defining a Painless script to use with runtime fields, you must
include `emit` to return calculated values.
[discrete]
[[painless-runtime-fields-mapping]]
==== Define a runtime field in the mapping
Add a `runtime` section under the {ref}/runtime-mapping-fields.html[mapping definition] to explore your data without indexing fields.
The script in the following request extracts the day of the week from the
`@timestamp` field, which is defined as a `date` type. The script calculates
the day of the week based on the value of `timestamp`, and uses `emit` to
return the calculated value.
[source,console]
----
PUT my-index/
{
"mappings": {
"runtime": {
"day_of_week": {
"type": "keyword",
"script": {
"source":
"""emit(doc['@timestamp'].value.dayOfWeekEnum
.getDisplayName(TextStyle.FULL, Locale.ROOT))"""
}
}
},
"properties": {
"timestamp": {"type": "date"}
}
}
}
----
[discrete]
[[painless-runtime-fields-query]]
==== Define a runtime field only in a search request
Use runtime fields in a search request to create a field that exists
{ref}/runtime-search-request.html[only as part of the query]. You can also {ref}/runtime-override-values.html[override field values] at query time for existing fields without
modifying the field itself.
This flexibility allows you to experiment with your data schema and fix
mistakes in your index mapping without reindexing your data.
In the following request, the values for the `day_of_week` field are calculated
dynamically, and only within the context of this search request:
[source,console]
----
GET my-index/_search
{
"runtime_mappings": {
"day_of_week": {
"type": "keyword",
"script": {
"source":
"""emit(doc['@timestamp'].value.dayOfWeekEnum
.getDisplayName(TextStyle.FULL, Locale.ROOT))"""
}
}
},
"aggs": {
"day_of_week": {
"terms": {
"field": "day_of_week"
}
}
}
}
----
//TEST[continued]

View file

@ -17,7 +17,7 @@ explicitly map all other data types.
// tag::dynamic-field-mapping-types-tag[]
[cols="3"]
|===
h| JSON data type h| `dynamic:true` h| `dynamic:runtime`
h| JSON data type h| `"dynamic":"true"` h| `"dynamic":"runtime"`
|`null` 2*| No field added
|`true` or `false` 2*| `boolean`
|`double` | `float` | `double`

View file

@ -68,12 +68,12 @@ reordered or deleted after they were initially added.
[[dynamic-mapping-runtime-fields]]
==== Mapping runtime fields in a dynamic template
If you want {es} to dynamically map new fields of a certain type as runtime
fields, set `dynamic:runtime` in the index mappings. These fields are not
fields, set `"dynamic":"runtime"` in the index mappings. These fields are not
indexed, and are loaded from `_source` at query time.
Alternatively, you can use the default dynamic mapping rules and then create
dynamic templates to map specific fields as runtime fields. You set
`dynamic:true` in your index mapping, and then create a dynamic template to map
`"dynamic":"true"` in your index mapping, and then create a dynamic template to map
new fields of a certain type as runtime fields.
Let's say you have data where each of the fields start with `ip_`. Based on the
@ -84,8 +84,8 @@ template that maps new strings as runtime fields of type `ip`.
The following request defines a dynamic template named `strings_as_ip`. When
{es} detects new `string` fields matching the `ip*` pattern, it maps those
fields as runtime fields of type `ip`. Because `ip` fields aren't mapped
dynamically, you can use this template with either `dynamic:true` or
`dynamic:runtime`.
dynamically, you can use this template with either `"dynamic":"true"` or
`"dynamic":"runtime"`.
[source,console]
----
@ -337,7 +337,7 @@ Here are some examples of potentially useful dynamic templates:
===== Structured search
When you set `dynamic:true`, {es} will map string fields as a `text` field with
When you set `"dynamic":"true"`, {es} will map string fields as a `text` field with
a `keyword` subfield. If you are only indexing structured content and not
interested in full text search, you can make {es} map your fields
only as `keyword` fields. However, you must search on the exact same value that

View file

@ -105,6 +105,7 @@ PUT my-index/
The `runtime` section can be any of these data types:
// tag::runtime-data-types[]
* `boolean`
* `date`
* `double`
@ -112,6 +113,7 @@ The `runtime` section can be any of these data types:
* `ip`
* `keyword`
* `long`
// end::runtime-data-types[]
Runtime fields with a `type` of `date` can accept the
<<mapping-date-format,`format`>> parameter exactly as the `date` field type.
@ -410,10 +412,77 @@ the values of runtime fields. Runtime fields won't display in `_source`, but
the `fields` API works for all fields, even those that were not sent as part of
the original `_source`.
[discrete]
[[runtime-define-field-dayofweek]]
==== Define a runtime field to calculate the day of week
For example, the following request adds a runtime field called `day_of_week`.
The runtime field includes a script that calculates the day of the week based
on the value of the `@timestamp` field. We'll include `"dynamic":"runtime"` in
the request so that new fields are added to the mapping as runtime fields.
[source,console]
----
PUT my-index/
{
"mappings": {
"dynamic": "runtime",
"runtime": {
"day_of_week": {
"type": "keyword",
"script": {
"source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
}
}
},
"properties": {
"timestamp": {"type": "date"}
}
}
}
----
[discrete]
[[runtime-ingest-data]]
==== Ingest some data
Let's ingest some sample data, which will result in two indexed fields:
`@timestamp` and `message`.
[source,console]
----
POST /my-index/_bulk?refresh
{ "index": {}}
{ "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
{ "index": {}}
{ "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:30:17-05:00", "message" : "40.135.0.0 - - [2020-04-30T14:30:17-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:30:53-05:00", "message" : "232.0.0.0 - - [2020-04-30T14:30:53-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:12-05:00", "message" : "26.1.0.0 - - [2020-04-30T14:31:12-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:19-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:19-05:00] \"GET /french/splash_inet.html HTTP/1.0\" 200 3781"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:27-05:00", "message" : "252.0.0.0 - - [2020-04-30T14:31:27-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_brdl.gif HTTP/1.0\" 304 0"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_arw.gif HTTP/1.0\" 304 0"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:32-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:32-05:00] \"GET /images/nav_bg_top.gif HTTP/1.0\" 200 929"}
{ "index": {}}
{ "@timestamp": "2020-04-30T14:31:43-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:43-05:00] \"GET /french/images/nav_venue_off.gif HTTP/1.0\" 304 0"}
----
//TEST[continued]
[discrete]
[[runtime-search-dayofweek]]
==== Search for the calculated day of week
The following request uses the search API to retrieve the `day_of_week` field
that <<runtime-mapping-fields,this previous request>> defined as a runtime field
in the mapping. The value for the `day_of_week` field is calculated at query
time based on the evaluation of the defined script.
that the original request defined as a runtime field in the mapping. The value
for this field is calculated dynamically at query time without reindexing
documents or indexing the `day_of_week` field. This flexibility allows you to
modify the mapping without changing any field values.
[source,console]
----
@ -428,6 +497,92 @@ GET my-index/_search
----
// TEST[continued]
The previous request returns the `day_of_week` field for all matching documents.
We can define another runtime field called `client_ip` that also operates on
the `message` field and will further refine the query:
[source,console]
----
PUT /my-index/_mapping
{
"runtime": {
"client_ip": {
"type": "ip",
"script" : {
"source" : "String m = doc[\"message\"].value; int end = m.indexOf(\" \"); emit(m.substring(0, end));"
}
}
}
}
----
//TEST[continued]
Run another query, but search for a specific IP address using the `client_ip`
runtime field:
[source,console]
----
GET my-index/_search
{
"size": 1,
"query": {
"match": {
"client_ip": "211.11.9.0"
}
},
"fields" : ["*"]
}
----
//TEST[continued]
This time, the response includes only two hits. The value for `day_of_week`
(`Sunday`) was calculated at query time using the runtime script defined in the
mapping, and the result includes only documents matching the `211.11.9.0` IP
address.
[source,console-result]
----
{
...
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "my-index",
"_id" : "oWs5KXYB-XyJbifr9mrz",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2020-06-21T15:00:01-05:00",
"message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
},
"fields" : {
"@timestamp" : [
"2020-06-21T20:00:01.000Z"
],
"client_ip" : [
"211.11.9.0"
],
"message" : [
"211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
],
"day_of_week" : [
"Sunday"
]
}
}
]
}
}
----
// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
// TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
// TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/]
[[runtime-examples]]
=== Explore your data with runtime fields
Consider a large set of log data that you want to extract fields from.
@ -612,96 +767,3 @@ and determine which fields to index.
----
// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
// TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
[[runtime-examples-calculate-value]]
==== Define a runtime field to calculate the day of week
You can add the `day_of_week` field to the mapping using the request from
<<runtime-mapping-fields,mapping a runtime field>>:
[source,console]
----
PUT /my-index/_mapping
{
"runtime": {
"day_of_week": {
"type": "keyword",
"script": {
"source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
}
}
},
"properties": {
"timestamp": {
"type": "date"
}
}
}
----
// TEST[continued]
Then, you can re-run the previous search request and also retrieve the day of
the week based on the `@timestamp` field:
[source,console]
----
GET my-index/_search
{
"size": 1,
"query": {
"match": {
"clientip": "211.11.9.0"
}
},
"fields" : ["*"]
}
----
// TEST[continued]
The value for this field is calculated dynamically at runtime without
reindexing the document or adding the `day_of_week` field. This flexibility
allows you to modify the mapping without changing any field values.
In the following response, the value for `day_of_week` (`Sunday`) was
calculated at query time using the runtime script defined in the mapping.
[source,console-result]
----
{
...
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "my-index",
"_id" : "oWs5KXYB-XyJbifr9mrz",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2020-06-21T15:00:01-05:00",
"message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
},
"fields" : {
"@timestamp" : [
"2020-06-21T20:00:01.000Z"
],
"clientip" : [
"211.11.9.0"
],
"message" : [
"211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
],
"day_of_week" : [
"Sunday"
]
}
}
]
}
}
----
// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
// TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
// TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/]

View file

@ -99,6 +99,72 @@ The API response returns the top 10 documents matching the query in the
// TESTRESPONSE[s/"took": 5/"took": "$body.took"/]
// TESTRESPONSE[s/"_id": "kxWFcnMByiguvud1Z8vC"/"_id": "$body.hits.hits.0._id"/]
[discrete]
[[run-search-runtime-fields]]
== Define fields that exist only in a query
Instead of indexing your data and then searching it, you can define
<<runtime-search-request,runtime fields>> that only exist as part of your
search query. You specify a `runtime_mappings` section in your search request
to define the runtime field, which can optionally include a Painless script.
For example, the following query defines a runtime field called `day_of_week`.
The included script calculates the day of the week based on the value of the
`@timestamp` field, and uses `emit` to return the calculated value.
The query also includes a <<search-aggregations-bucket-terms-aggregation,terms aggregation>> that operates on `day_of_week`.
[source,console]
----
GET /my-index-000001/_search
{
"runtime_mappings": {
"day_of_week": {
"type": "keyword",
"script": {
"source":
"""emit(doc['@timestamp'].value.dayOfWeekEnum
.getDisplayName(TextStyle.FULL, Locale.ROOT))"""
}
}
},
"aggs": {
"day_of_week": {
"terms": {
"field": "day_of_week"
}
}
}
}
----
// TEST[setup:my_index]
The response includes an aggregation based on the `day_of_week` runtime field.
Under `buckets` is a `key` with a value of `Sunday`. The query dynamically
calculated this value based on the script defined in the `day_of_week` runtime
field without ever indexing the field.
[source,console-result]
----
{
...
***
"aggregations" : {
"day_of_week" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Sunday",
"doc_count" : 5
}
]
}
}
}
----
// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
// TESTRESPONSE[s/\*\*\*/"hits" : $body.hits,/]
[discrete]
[[common-search-options]]
=== Common search options

View file

@ -399,6 +399,40 @@ lower `_score` are not included in the search results.
(Optional, <<query-dsl,query object>>) Defines the search definition using the
<<query-dsl,Query DSL>>.
[[search-api-body-runtime]]
`runtime_mappings`::
(Optional, object)
Defines a <<runtime-search-request,runtime field>> in the search request that
exist only as part of the query. Fields defined in the search request take
precedence over fields defined with the same name in the index mappings.
+
.Properties of `runtime_mappings` objects
[%collapsible%open]
====
`type`::
(Required, string)
<<mapping-types,Data type>> of the object, which can be any of the following:
+
include::{es-ref-dir}/mapping/runtime.asciidoc[tag=runtime-data-types]
`script`::
(Optional, string)
<<modules-scripting-using,Painless script>> that is executed at query time. The
script has access to the entire context of a document, including the original
`_source` and any mapped fields plus their values.
+
Your script must include `emit` to return calculated values. For
example:
+
[source,js]
----
"script": {
"source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
}
----
// NOTCONSOLE
====
[[request-body-search-seq-no-primary-term]]
`seq_no_primary_term`::
(Optional, Boolean) If `true`, returns sequence number and primary term of the