This commit adds a script parameter to long and double fields that makes
it possible to calculate a value for these fields at index time. It uses the same
script context as the equivalent runtime fields, and allows for multiple index-time
scripted fields to cross-refer while still checking for indirection loops.
* Make wildcard field use constant scoring queries for wildcard queries. Add a note about ignoring rewrite parameters on wildcard queries.
Also fixes caching issue where case sensitive and case insensitive results were cached as the same
Closes#69604
This commit updates the default format of date_nanos field
on existing and new indices to use `strict_date_optional_time_nanos` instead of
`strict_date_optional_time`.
Using `strict_date_optional_time` as the default format for date_nanos doesn't
make sense because it accepts and parses dates with nanosecond precision,
but when it formats it drops the nanoseconds.
The change should be transparent for users, these formats accept the same input.
Relates #69192Closes#67063
This field mapper only lived in its own module so it could be licensed as x-pack
basic. Now it can be moved to core, which matches its status as a core type.
Remove not completely correct statement about the size of dense_vectors
We do store a dense_vector as binary doc value with size `4*dims+4`.
But this is size before compression. As compressed size depends on
data itself, it is better to remove completely any statement
about the size.
The tip about updating a `search_analyzer` currently does not mention that most
of the time (when the current analyzer is not "default"), user need to repeat
the currently set "analyzer" parameter in the field definition. Adding this as a
short note.
You can't update the `analyzer` parameter in the PUT mappings API even if
the index is closed. This adds a TIP to call that out. And adds a TIP
for `search_quote_analyzer` which you *can* update.
* [DOCS] Adding grok support for runtime fields.
* Update response.
* Adding testresponse replacements.
* Update runtime field context and add dissect.
* Fixing backslash in the response.
* Fixing testresponse.
* Incorporating review feedback.
* Updates emit and adds cross link from ES runtime fields page.
* Reallocate runtime document
Reallocate document `runtime-fields-scriptless` from `runtime-search-request` to `runtime-mapping-fields`
* Move runtime without script section
Move runtime without script section to under the dynamic runtime mapping section
* Fix snippet formatting and remove discrete heading.
* Update test snippet.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Currently, existing runtime fields can be updated, but they cannot be removed. That allows to correct potential mistakes, but once a runtime field is added to the index mappings, it is not possible to remove it.
With this commit we introduce the ability to remove an existing runtime field by providing a null value for it through the put mapping API. If a field with such name does not exist, such specific instruction will have no effect on other existing runtime fields.
Note that the removal of runtime fields makes the recently introduced assertRefreshItNotNeeded assertion trip, because when each local node merges mappings back in, the runtime fields that were previously removed by the master node, get added back again locally. This is only a problem for the assertion that verifies that the removed refresh operation is never needed. We worked around this by tweaking the assertion to ignore runtime fields completely, for simplicity, by assertion on the serialized merged mappings and incoming mappings without the corresponding runtime section.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Adding runtime fields page for Painless context.
* Adds beta admonition to runtime fields and Painless docs.
* Fixing test errors and improving content sections.
* Adding refresh to fix test cases.
* Simplifying the ingest request to include refresh.
* Removing beta (will add in another PR) and updating examples.
* Moving examples to the page for retrieving runtime fields.
* Adding runtime_mappings to request body of search API.
* Updating runtime_mappings properties and adding runtime fields to search your data.
* Updating examples and hopefully fixing build failure.
* Fixing snippet formatting that was causing test failure.
* Adding page in Painless guide for runtime fields.
* Fixing typo.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Updating dynamic mappings for runtime fields.
* Updating example to fix test case and be more accurate.
* Changing header level for dynamic runtime.
* Clarifying language around ip fields in dynamic template.
* Changes for dynamic templates.
* Clarifying language around dynamic:true and dynamic:runtime.
* Clarifying edits and some restructuring.
* Overhauling the Mapping page.
* Incorporating changes from #66911.
* Reworking mapping page to focus on dynamic vs. explicit mapping.
* Reordering to fix test failure.
* Further clarifying mapping page.
* Reordering sections, adding headings to examples, and other clarifications.
* Incorporating review feedback.
* Adding description of for Painless script.
Added documentation for the aggregate_metric_double field that was merged in #56745
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* First steps in docs for runtime fields.
* Adding new page for runtime fields.
* Adding page for runtime fields.
* Adding more to the runtime fields topic.
* Adding parameters and retrieval options for runtime fields.
* Adding TESTSETUP for index creation.
* Incorporating review feedback.
* Incorporating reviewer feedback.
* Adding examples for runtime fields.
* Adding more context and simplifying the example.
* Changing timestamp to @timestamp throughout.
* Removing duplicate @timestamp field.
* Expanding example to hopefully fix CI builds.
* Adding skip test for result.
* Adding missing callout.
* Adding TESTRESPONSEs, which are currently broken.
* Fixing TESTRESPONSEs.
* Incorporating review feedback.
* Several clarifications, better test cases, and other changes.
* Adding missing callout in example.
* Adding substitutions to TESTRESPONSE for shorter results shown.
* Shuffling some information and adding link to script-fields.
* Fixing typo.
* Updates for API redesign -- will break builds.
* Updating examples and including info about overriding fields.
* Updating examples.
* Adding info for using runtime fields in the search request.
* Adding that queries against runtime fields are expensive.
* Incorporating feedback from reviewers.
* Minor changes from reviews.
* Adding alias for test case.
* Adding aliases to PUT example.
* Fixing test cases, for real this time.
* Updating use cases and introducing overlay throughout.
* Edits, adding 'shadowing', and explaining shadowing better.
* Streamlining tests and other changes.
* Fix formatting in example for test.
* Apply suggestions from code review
* Incorporating reviewer feedback 7 Dec
* Shifting structure of mapping page to fix cross links.
* Revisions for shadowing, overview, and other sections.
* Removing dot notation section and incorporating review changes.
* Adding updated example for shadowing.
* Streamlining shadowing example and TESTRESPONSEs.
During highlighting, we now load all values that were copied into the field
through copy_to. So there's no longer a reason to set 'store: true' to account
for fields not available in _source.
* Clarify that field data cache includes global ordinals
* Describe that the cache should be cleared once the limit is reached
* Clarify that the `_id` field does not supported aggregations anymore
* Fold the `fielddata` mapping parameter page into the `text field docs
* Improve cross-linking
Include the attempted 'match_mapping_type' into the message,
so that it is clearer that multiple validation attempts have occurred.
Dynamic template validation was recently added via #51233 and
there was some confusion over the deprecation message itself.
(in 7.x only deprecation warning will be omitted and from 8.0
an error will be returned)
Bucket aggregations compute bucket doc_count values by incrementing the doc_count by 1 for every document collected in the bucket.
When using summary fields (such as aggregate_metric_double) one field may represent more than one document. To provide this functionality we have implemented a new field mapper (named doc_count field mapper). This field is a positive integer representing the number of documents aggregated in a single summary field.
Bucket aggregations will check if a field of type doc_count exists in a document and will take this value into consideration when computing doc counts.
We standardize on some metadata entries that we plan to later leverage
in Kibana in order to provide a better out-of-the-box experience, e.g.
different visualizations make sense on gauges and counters.
