Currently, existing runtime fields can be updated, but they cannot be removed. That allows to correct potential mistakes, but once a runtime field is added to the index mappings, it is not possible to remove it.
With this commit we introduce the ability to remove an existing runtime field by providing a null value for it through the put mapping API. If a field with such name does not exist, such specific instruction will have no effect on other existing runtime fields.
Note that the removal of runtime fields makes the recently introduced assertRefreshItNotNeeded assertion trip, because when each local node merges mappings back in, the runtime fields that were previously removed by the master node, get added back again locally. This is only a problem for the assertion that verifies that the removed refresh operation is never needed. We worked around this by tweaking the assertion to ignore runtime fields completely, for simplicity, by assertion on the serialized merged mappings and incoming mappings without the corresponding runtime section.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Adding runtime fields page for Painless context.
* Adds beta admonition to runtime fields and Painless docs.
* Fixing test errors and improving content sections.
* Adding refresh to fix test cases.
* Simplifying the ingest request to include refresh.
* Removing beta (will add in another PR) and updating examples.
* Moving examples to the page for retrieving runtime fields.
* Adding runtime_mappings to request body of search API.
* Updating runtime_mappings properties and adding runtime fields to search your data.
* Updating examples and hopefully fixing build failure.
* Fixing snippet formatting that was causing test failure.
* Adding page in Painless guide for runtime fields.
* Fixing typo.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Updating dynamic mappings for runtime fields.
* Updating example to fix test case and be more accurate.
* Changing header level for dynamic runtime.
* Clarifying language around ip fields in dynamic template.
* Changes for dynamic templates.
* Clarifying language around dynamic:true and dynamic:runtime.
* Clarifying edits and some restructuring.
* Overhauling the Mapping page.
* Incorporating changes from #66911.
* Reworking mapping page to focus on dynamic vs. explicit mapping.
* Reordering to fix test failure.
* Further clarifying mapping page.
* Reordering sections, adding headings to examples, and other clarifications.
* Incorporating review feedback.
* Adding description of for Painless script.
Added documentation for the aggregate_metric_double field that was merged in #56745
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* First steps in docs for runtime fields.
* Adding new page for runtime fields.
* Adding page for runtime fields.
* Adding more to the runtime fields topic.
* Adding parameters and retrieval options for runtime fields.
* Adding TESTSETUP for index creation.
* Incorporating review feedback.
* Incorporating reviewer feedback.
* Adding examples for runtime fields.
* Adding more context and simplifying the example.
* Changing timestamp to @timestamp throughout.
* Removing duplicate @timestamp field.
* Expanding example to hopefully fix CI builds.
* Adding skip test for result.
* Adding missing callout.
* Adding TESTRESPONSEs, which are currently broken.
* Fixing TESTRESPONSEs.
* Incorporating review feedback.
* Several clarifications, better test cases, and other changes.
* Adding missing callout in example.
* Adding substitutions to TESTRESPONSE for shorter results shown.
* Shuffling some information and adding link to script-fields.
* Fixing typo.
* Updates for API redesign -- will break builds.
* Updating examples and including info about overriding fields.
* Updating examples.
* Adding info for using runtime fields in the search request.
* Adding that queries against runtime fields are expensive.
* Incorporating feedback from reviewers.
* Minor changes from reviews.
* Adding alias for test case.
* Adding aliases to PUT example.
* Fixing test cases, for real this time.
* Updating use cases and introducing overlay throughout.
* Edits, adding 'shadowing', and explaining shadowing better.
* Streamlining tests and other changes.
* Fix formatting in example for test.
* Apply suggestions from code review
* Incorporating reviewer feedback 7 Dec
* Shifting structure of mapping page to fix cross links.
* Revisions for shadowing, overview, and other sections.
* Removing dot notation section and incorporating review changes.
* Adding updated example for shadowing.
* Streamlining shadowing example and TESTRESPONSEs.
During highlighting, we now load all values that were copied into the field
through copy_to. So there's no longer a reason to set 'store: true' to account
for fields not available in _source.
* Clarify that field data cache includes global ordinals
* Describe that the cache should be cleared once the limit is reached
* Clarify that the `_id` field does not supported aggregations anymore
* Fold the `fielddata` mapping parameter page into the `text field docs
* Improve cross-linking
Include the attempted 'match_mapping_type' into the message,
so that it is clearer that multiple validation attempts have occurred.
Dynamic template validation was recently added via #51233 and
there was some confusion over the deprecation message itself.
(in 7.x only deprecation warning will be omitted and from 8.0
an error will be returned)
Bucket aggregations compute bucket doc_count values by incrementing the doc_count by 1 for every document collected in the bucket.
When using summary fields (such as aggregate_metric_double) one field may represent more than one document. To provide this functionality we have implemented a new field mapper (named doc_count field mapper). This field is a positive integer representing the number of documents aggregated in a single summary field.
Bucket aggregations will check if a field of type doc_count exists in a document and will take this value into consideration when computing doc counts.
We standardize on some metadata entries that we plan to later leverage
in Kibana in order to provide a better out-of-the-box experience, e.g.
different visualizations make sense on gauges and counters.
This fixes fields retrieval on unsigned_long field
1) For docvalue_fields a custom UnsignedLongLeafFieldData::getLeafValueFetcher
is implemented that correctly retrieves doc values.
2) For stored fields, an error was fixed in UnsignedLongFieldMapper
how stored values were stored. Before they were incorrectly
stored in the shifted format, now they are stored as original
values in String format.
Relates to #60050
We don't need a special TypeFieldMapper for anything in particular; all access
to the type field can be done via a TypeFieldType that issues appropriate
deprecation warnings.
Relates to #41059
This field type supports
- indexing of integer values from [0, 18446744073709551615]
- precise queries (term, range)
- precise sort and terms aggregations
- other aggregations are based on conversion of long values
to double and can be imprecise for large values.
Closes#32434
This PR adds a new 'version' field type that allows indexing string values
representing software versions similar to the ones defined in the Semantic
Versioning definition (semver.org). The field behaves very similar to a
'keyword' field but allows efficient sorting and range queries that take into
accound the special ordering needed for version strings. For example, the main
version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't
be possible with 'keyword' fields today.
Valid version values are similar to the Semantic Versioning definition, with the
notable exception that in addition to the "main" version consiting of
major.minor.patch, we allow less or more than three numeric identifiers, i.e.
"1.2" or "1.4.6.123.12" are treated as valid too.
Relates to #48878
We now link to the top-level keyword type family page instead of its individual
subsections. This better fits the page format, where each type name is a link.
This commit makes IpFieldMapper extend ParametrizedFieldMapper. It also
updates the IpFieldMapper docs to add the ignore_malformed parameter,
which was not previously documented.
