Adds an `experimental` annotation to the following:
* `time_series_metric` mapping parameter
* `time_series_dimension` mapping parameter
* `index.mapping.dimension_fields.limit` index setting
* `time_series_dimension` and `time_series_metric` properties in the field caps API response
# Conflicts:
# docs/reference/search/field-caps.asciidoc
Backports the following PRs:
* Add dimension mapping parameter (#74450)
Added the dimension parameter to the following field types:
keyword
ip
Numeric field types (integer, long, byte, short)
The dimension parameter is of type boolean (default: false) and is used
to mark that a field is a time series dimension field.
Relates to #74014
* Add constraints to dimension fields (#74939)
This PR adds the following constraints to dimension fields:
It must be an indexed field and must has doc values
It cannot be multi-valued
The number of dimension fields in the index mapping must not be more than 16. This should be configurable through an index property (index.mapping.dimension_fields.limit)
keyword fields cannot be more than 1024 bytes long
keyword fields must not use a normalizer
Based on the code added in PR #74450
Relates to #74660
* Expand DocumentMapperTests (#76368)
Adds a test for setting the maximum number of dimensions setting and
tests the names and types of the metadata fields in the index.
Previously we just asserted the count of metadata fields. That made it
hard to read failures.
* Fix broken test for dimension keywords (#75408)
Test was failing because it was testing 1024 bytes long keyword and assertion was failing.
Closes#75225
* Checkstyle
* Add time_series_metric parameter (#76766)
This PR adds the time_series_metric parameter to the following field types:
Numeric field types
histogram
aggregate_metric_double
* Rename `dimension` mapping parameter to `time_series_dimension` (#78012)
This PR renames dimension mapping parameter to time_series_dimension to make it consistent with time_series_metric parameter (#76766)
Relates to #74450 and #74014
* Add time series params to `unsigned_long` and `scaled_float` (#78204)
Added the time_series_metric mapping parameter to the unsigned_long and scaled_float field types
Added the time_series_dimension mapping parameter to the unsigned_long field type
Fixes#78100
Relates to #76766, #74450 and #74014
Co-authored-by: Nik Everett <nik9000@gmail.com>
Changes:
* Documents the `time_series_metric` mapping parameter for PR #76766.
* Renames the `dimension` parameter to `time_series_dimension` for PR #78012.
* Adds support for `unsigned_long` to `time_series_dimension` for PR #78204.
Adds additional information about how Elasticsearch uses polygon orientation. Elasticsearch only uses a polygon's orientation to determine if it crosses the international dateline. If so, Elasticsearch splits the polygon at the dateline.
Closes#74891
# Conflicts:
# docs/reference/mapping/types/geo-shape.asciidoc
If you specify a deprecated mapping parameter related to prefix trees, a
`geo_shape` field will use prefix tree encoding.
This adds a related admon to the top of the geoshape field docs page. This admon will
help users avoid accidentally switching from the default BKD tree encoding.
Closes#40266.
* Mention match_only_text in disk usage docs
Previously we explained how to manually disable norms, freqs, and positions. We
now have a ready-made solution in the new `match_only_text` field type.
* Fixing typo and minor grammar changes
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for #18699. This updates the routing formulas and adds xrefs for
related settings.
Closes#76072
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Changes:
* Use "geopoint" when not referring to the literal field type
* Use "geoshape" when not referring to the literal field type or query type
* Use "GeoJSON" consistently
# Conflicts:
# docs/reference/ingest/processors/enrich.asciidoc
* [DOCS] Add performance info for runtime fields
* Add script-based sorting and clarify performance
* Changing title to Incentives and reworking the intro
* Add support for range aggregations on histogram mapped fields (#74146)
This adds support for the range aggregation over `histogram` mapped fields.
Decisions made for implementation:
- Sub-aggregations are not allowed. This is to simplify implementation and follows the prior art set by the `histogram` aggregation
- Nothing fancy is done with the ranges. No filter translations as we cannot easily do a `range` filter query against histogram fields. This may be an optimization in the future.
- Ranges check the histogram value ONLY. No interpolation of values is done. If we have better statistics around the histogram this MAY be possible.
When we introduced dynamic:runtime (#65489) we decided to have it create objects dynamically under properties, as the runtime section did not (and still does not) support object fields. That proved to be a poor choice, because the runtime section is flat, supports dots in field names, and does not really need objects. Also, these end up causing unnecessary mapping conflicts.
With this commit we adapt dynamic:runtime to not dynamically create objects.
Closes#70268
* [DOCS] Add retriving from flattened fields
* Clarify sub-field syntax
* Moving sub-field retrieval to flattened field docs
* Remove full example and de-emphasize runtime fields
* Remove extraneous sample tag
* [DOCS] [7.x] Create a new page for grok content in scripting docs (#73118)
* [DOCS] Moving grok to its own scripting page
* Adding examples
* Updating cross link for grok page
* Adds same runtime field in a search request for #73262
* Clarify titles and shift navigation
* Incorporating review feedback
* Updating cross-link to Painless
* Adding doc type to response
* [DOCS] Expand information on when to use a runtime field without a script
* Reworking information based on review feedback
* Clarify case where doc_values are disabled
* A few minor changes from review feedback
We currently don't support `copy_to` for fields that take the form of objects
(e.g. `date_range` or certain kinds of `geo_point` variants). The current
problem with objects is that when DocumentParser parses anything other than
single values, it potentially advances the underlying parser past the value that
we would need to stay on for parsing the value again. While we might want to
support this in the future, for now this PR enhances the otherwise confusing
MapperParsingException with something more helpful and adds a short note in the
documentation about this restriction.
Closes#49344
This adds a new `match_only_text` field, which indexes the same data as a `text`
field that has `index_options: docs` and `norms: false` and uses the `_source`
for positional queries like `match_phrase`. Unlike `text`, this field doesn't
support scoring.
* [DOCS] Focus retrieving selected fields on fields parameter (#71506)
* Incorporating changes from reviews
* Adding clarifications from review feedback
* Slight wording revisions.
* Clarify language around format parameter and move text out of callout.
* Fix code block start tagging
As we started thinking about applying on_script_error to runtime fields, to handle script errors at search time, we would like to use the same parameter that was recently introduced for indexed fields. We decided that continue or fail gives a better indication of the behaviour compared to the current ignore or reject which is too specific to indexing documents.
This commit applies such rename.
This commit adds the ability to define an index-time geo_point field
with a script parameter, allowing you to calculate points from other
values within the indexed document.
Allow direct access to a dense_vector' values in script
through the following functions:
- getVectorValue – returns a vector's value as an array of floats
- getMagnitude – returns a vector's magnitude
Closes#51964
Backport for #71313
Currently the `fields` API fetches the root flattened field and returns it in a
structured way in the response. In addition this change makes it possible to
directly query subfields. However, requesting flattened subfields via wildcard
patterns is not possible.
Closes#70605
This commit allows you to set 'script' and 'on_script_error' parameters
on date field mappers, meaning that runtime date fields can be made indexed
simply by moving their definitions from the runtime section of the mappings
to the properties section.
We accept dates with a decimal point like `2113413.13241324` and parse
them *somehow*. But there are cases where we'll lose precision on those
dates, see #70085. This advises folks not to use that format. We'll
continue to accept those dates for backwards compatibility but you
should avoid using them.
Co-authored-by: Adrien Grand <jpountz@gmail.com>
Runtime fields are much more flexible than script_fields because you
can filter and aggregate on them so we hope folks use them! This
converts the example of using a `parent_join` field in a script to a
runtime field so folks get used to seeing them and hopefully using them.
While I was editing this I took the opportunity to replace the script
with a real-ish example. Scripts that just load the field value are nice
and short but I hope no one uses them in real life because they just add
overhead when compared to accessing the field directly. So I made the
script do something.
Relates to #69291
This commit allows you to set 'script' and 'on_script_error' parameters
on IP field mappers, meaning that runtime IP fields can be made indexed
simply by moving their definitions from the runtime section of the mappings
to the properties section.
Runtime fields are much more flexible than script_fields because you
can filter and aggregate on them so we hope folks use them! This
converts the example of using a `date_nanos` field in a script to a
runtime field so folks get used to seeing them and hopefully using them.
While I was editing this I took the opportunity to replace the script
with a real-ish example. Scripts that just load the field value are nice
and short but I hope no one uses them in real life because they just add
overhead when compared to accessing the field directly. So I made the
script do something.
Relates to #69291
Co-authored-by: Adam Locke <adam.locke@elastic.co>
