This commit updates the default format of date_nanos field
on existing and new indices to use `strict_date_optional_time_nanos` instead of
`strict_date_optional_time`.
Using `strict_date_optional_time` as the default format for date_nanos doesn't
make sense because it accepts and parses dates with nanosecond precision,
but when it formats it drops the nanoseconds.
The change should be transparent for users, these formats accept the same input.
Relates #69192Closes#67063
This field mapper only lived in its own module so it could be licensed as x-pack
basic. Now it can be moved to core, which matches its status as a core type.
The PR also removes flattened fields from xpack usage stats now that they live
in core. The usage stats were already removed in on master, but this PR was not
backported to avoid a breaking change on 7.x. We have now clarified that usage
stats updates are not considered breaking, so it is safe to backport this
change.
Remove not completely correct statement about the size of dense_vectors
We do store a dense_vector as binary doc value with size `4*dims+4`.
But this is size before compression. As compressed size depends on
data itself, it is better to remove completely any statement
about the size.
The tip about updating a `search_analyzer` currently does not mention that most
of the time (when the current analyzer is not "default"), user need to repeat
the currently set "analyzer" parameter in the field definition. Adding this as a
short note.
You can't update the `analyzer` parameter in the PUT mappings API even if
the index is closed. This adds a TIP to call that out. And adds a TIP
for `search_quote_analyzer` which you *can* update.
* [DOCS] [7.x] Adding grok support for runtime fields. (#69308)
* [DOCS] Adding grok support for runtime fields.
* Update response.
* Adding testresponse replacements.
* Update runtime field context and add dissect.
* Fixing backslash in the response.
* Fixing testresponse.
* Incorporating review feedback.
* Updates emit and adds cross link from ES runtime fields page.
* Add missing _type to response.
* Reallocate runtime document
Reallocate document `runtime-fields-scriptless` from `runtime-search-request` to `runtime-mapping-fields`
* Move runtime without script section
Move runtime without script section to under the dynamic runtime mapping section
* Fix snippet formatting and remove discrete heading.
* Update test snippet.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: S. J. Lee <occidere@naver.com>
Currently, existing runtime fields can be updated, but they cannot be removed. That allows to correct potential mistakes, but once a runtime field is added to the index mappings, it is not possible to remove it.
With this commit we introduce the ability to remove an existing runtime field by providing a null value for it through the put mapping API. If a field with such name does not exist, such specific instruction will have no effect on other existing runtime fields.
Note that the removal of runtime fields makes the recently introduced assertRefreshItNotNeeded assertion trip, because when each local node merges mappings back in, the runtime fields that were previously removed by the master node, get added back again locally. This is only a problem for the assertion that verifies that the removed refresh operation is never needed. We worked around this by tweaking the assertion to ignore runtime fields completely, for simplicity, by assertion on the serialized merged mappings and incoming mappings without the corresponding runtime section.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Adding runtime fields page for Painless context.
* Adds beta admonition to runtime fields and Painless docs.
* Fixing test errors and improving content sections.
* Adding refresh to fix test cases.
* Simplifying the ingest request to include refresh.
* Removing beta (will add in another PR) and updating examples.
* [DOCS] [7.x] Updates to the runtime fields docs for 7.11+ (#67484)
* Moving examples to the page for retrieving runtime fields.
* Adding runtime_mappings to request body of search API.
* Updating runtime_mappings properties and adding runtime fields to search your data.
* Updating examples and hopefully fixing build failure.
* Fixing snippet formatting that was causing test failure.
* Adding page in Painless guide for runtime fields.
* Fixing typo.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Fixing testresponse to include _type.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [DOCS] [7.x] Updating dynamic mappings for runtime fields (#67301)
* Updating dynamic mappings for runtime fields.
* Updating example to fix test case and be more accurate.
* Changing header level for dynamic runtime.
* Clarifying language around ip fields in dynamic template.
* Adding test warning to fix snippet test.
* Changes for dynamic templates.
* Clarifying language around dynamic:true and dynamic:runtime.
* Clarifying edits and some restructuring.
* Overhauling the Mapping page.
* Incorporating changes from #66911.
* Reworking mapping page to focus on dynamic vs. explicit mapping.
* Reordering to fix test failure.
* Further clarifying mapping page.
* Reordering sections, adding headings to examples, and other clarifications.
* Incorporating review feedback.
* Adding description of for Painless script.
* Minor revisions.
Added documentation for the aggregate_metric_double field that was merged in #56745
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* [DOCS] Add docs for runtime fields (#62653)
* First steps in docs for runtime fields.
* Adding new page for runtime fields.
* Adding page for runtime fields.
* Adding more to the runtime fields topic.
* Adding parameters and retrieval options for runtime fields.
* Adding TESTSETUP for index creation.
* Incorporating review feedback.
* Incorporating reviewer feedback.
* Adding examples for runtime fields.
* Adding more context and simplifying the example.
* Changing timestamp to @timestamp throughout.
* Removing duplicate @timestamp field.
* Expanding example to hopefully fix CI builds.
* Adding skip test for result.
* Adding missing callout.
* Adding TESTRESPONSEs, which are currently broken.
* Fixing TESTRESPONSEs.
* Incorporating review feedback.
* Several clarifications, better test cases, and other changes.
* Adding missing callout in example.
* Adding substitutions to TESTRESPONSE for shorter results shown.
* Shuffling some information and adding link to script-fields.
* Fixing typo.
* Updates for API redesign -- will break builds.
* Updating examples and including info about overriding fields.
* Updating examples.
* Adding info for using runtime fields in the search request.
* Adding that queries against runtime fields are expensive.
* Incorporating feedback from reviewers.
* Minor changes from reviews.
* Adding alias for test case.
* Adding aliases to PUT example.
* Fixing test cases, for real this time.
* Updating use cases and introducing overlay throughout.
* Edits, adding 'shadowing', and explaining shadowing better.
* Streamlining tests and other changes.
* Fix formatting in example for test.
* Apply suggestions from code review
* Incorporating reviewer feedback 7 Dec
* Shifting structure of mapping page to fix cross links.
* Revisions for shadowing, overview, and other sections.
* Removing dot notation section and incorporating review changes.
* Adding updated example for shadowing.
* Streamlining shadowing example and TESTRESPONSEs.
* Adding type field in responses for 7.x to fix test cases.
During highlighting, we now load all values that were copied into the field
through copy_to. So there's no longer a reason to set 'store: true' to account
for fields not available in _source.
* Clarify that field data cache includes global ordinals
* Describe that the cache should be cleared once the limit is reached
* Clarify that the `_id` field does not supported aggregations anymore
* Fold the `fielddata` mapping parameter page into the `text field docs
* Improve cross-linking
Backporting #60870 to 7.x branch.
Include the attempted 'match_mapping_type' into the message,
so that it is clearer that multiple validation attempts have occurred.
Dynamic template validation was recently added via #51233 and
there was some confusion over the deprecation message itself.
(in 7.x only deprecation warning will be omitted and from 8.0
an error will be returned)
Bucket aggregations compute bucket doc_count values by incrementing the doc_count by 1 for every document collected in the bucket.
When using summary fields (such as aggregate_metric_double) one field may represent more than one document. To provide this functionality we have implemented a new field mapper (named doc_count field mapper). This field is a positive integer representing the number of documents aggregated in a single summary field.
Bucket aggregations will check if a field of type doc_count exists in a document and will take this value into consideration when computing doc counts.
We standardize on some metadata entries that we plan to later leverage
in Kibana in order to provide a better out-of-the-box experience, e.g.
different visualizations make sense on gauges and counters.
As a result of this, we can remove a chunk of code from TypeParsers as well. Tests
for search/index mode analyzers have moved into their own file. This commit also
rationalises the serialization checks for parameters into a single SerializerCheck
interface that takes the values includeDefaults, isConfigured and the value
itself.
Relates to #62988
This fixes fields retrieval on unsigned_long field
1) For docvalue_fields a custom UnsignedLongLeafFieldData::getLeafValueFetcher
is implemented that correctly retrieves doc values.
2) For stored fields, an error was fixed in UnsignedLongFieldMapper
how stored values were stored. Before they were incorrectly
stored in the shifted format, now they are stored as original
values in String format.
Relates to #60050
Backport for #63119
Introduce 64-bit unsigned long field type
This field type supports
- indexing of integer values from [0, 18446744073709551615]
- precise queries (term, range)
- precise sort and terms aggregations
- other aggregations are based on conversion of long values
to double and can be imprecise for large values.
Backport for #60050Closes#32434
