This shrinks a runtime field definition so that it fits on the screen
without scrolling. It also converts the doc into a test so we can be
sure it continues to work.
Relates to #69291
This replaces the `script` docs for bucket aggregations with runtime
fields. We expect runtime fields to be nicer to work with because you
can also fetch them or filter on them. We expect them to be faster
because their don't need this sort of `instanceof` tree:
a92a647b9f/server/src/main/java/org/elasticsearch/search/aggregations/support/values/ScriptDoubleValues.java (L42)
Relates to #69291
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
This change exposes for each field in the _field_caps response if the field is a metadata field.
This is needed for consumers of this API that want to filter these fields. Currently ML keeps a static list
and QL checks that the family type starts with `_`. In order to ease the addition of new metadata fields, this
change reworks the strategy in this solution and now only checks for the new flag.
Note that the new flag is also applied at the coordinator level in a best-effort to apply the logic on older nodes
in a mixed-version cluster.
* [DOCS] Clarify supported features for CCS.
* Clarify text and add subsection with title.
* Moving APIs to supported API section and paring down text.
If a search after request targets multiple indices and some of its sort
field has type `date` in one index but `date_nanos` in other indices,
then Elasticsearch won't interpret the search_after parameter correctly
in every target index. The sort value of a date field by default is a
long of milliseconds since the epoch while a date_nanos field is a long
of nanoseconds.
This commit introduces the `format` parameter in the sort field so a
sort value of a date or date_nanos will be formatted using a date format
in a search response.
The below example illustrates how to use this new parameter.
```js
{
"query": {
"match_all": {}
},
"sort": [
{
"timestamp": {
"order": "asc",
"format": "strict_date_optional_time_nanos"
}
}
]
}
```
```js
{
"query": {
"match_all": {}
},
"sort": [
{
"timestamp": {
"order": "asc",
"format": "strict_date_optional_time_nanos"
}
}
],
"search_after": [
"2015-01-01T12:10:30.123456789Z" // in `strict_date_optional_time_nanos` format
]
}
```
Closes#69192
This PR adds the special `_shard_doc` sort tiebreaker automatically to any
search requests that use a PIT. Adding the tiebreaker ensures that any
sorted query can be paginated consistently within a PIT.
Closes#56828
This change adds a new cluster privilege cancel_task that allows to:
Cancel running tasks (_tasks/_cancel).
Cancel and delete async searches.
Today the 'manage' cluster privilege is required to cancel tasks and
to delete async searches when security features are enabled.
This new focused privilege allows to handle tasks and searches only.
The change also adds the privilege to the internal 'kibana_system'
and '_async_search' roles. They both need to be able to cancel tasks
and delete async searches.
Relates #67965
Add a `max_analyzed_offset` query parameter to allow users
to limit the highlighting of text fields to a value less than or equal to the
`index.highlight.max_analyzed_offset`, thus avoiding an exception when
the length of the text field exceeds the limit. The highlighting still takes place,
but stops at the length defined by the new parameter.
Closes: #52155
(cherry picked from commit f9af60bf69)
This partially reverts #64016 and and adds #67839 and adds
additional tests that would have caught issues with the changes
in #64016. It's mostly Nik's code, I am just cleaning things up
a bit.
Co-authored-by: Nik Everett nik9000@gmail.com
Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.
Closes#68117
* [DOCS] [7.x] Updates to the runtime fields docs for 7.11+ (#67484)
* Moving examples to the page for retrieving runtime fields.
* Adding runtime_mappings to request body of search API.
* Updating runtime_mappings properties and adding runtime fields to search your data.
* Updating examples and hopefully fixing build failure.
* Fixing snippet formatting that was causing test failure.
* Adding page in Painless guide for runtime fields.
* Fixing typo.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Fixing testresponse to include _type.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
The option to use templates already defined in the cluster is not explicitly stated in the docs.
This PR adds and example to the `rank_eval` documentation.
* [DOCS] Add docs for runtime fields (#62653)
* First steps in docs for runtime fields.
* Adding new page for runtime fields.
* Adding page for runtime fields.
* Adding more to the runtime fields topic.
* Adding parameters and retrieval options for runtime fields.
* Adding TESTSETUP for index creation.
* Incorporating review feedback.
* Incorporating reviewer feedback.
* Adding examples for runtime fields.
* Adding more context and simplifying the example.
* Changing timestamp to @timestamp throughout.
* Removing duplicate @timestamp field.
* Expanding example to hopefully fix CI builds.
* Adding skip test for result.
* Adding missing callout.
* Adding TESTRESPONSEs, which are currently broken.
* Fixing TESTRESPONSEs.
* Incorporating review feedback.
* Several clarifications, better test cases, and other changes.
* Adding missing callout in example.
* Adding substitutions to TESTRESPONSE for shorter results shown.
* Shuffling some information and adding link to script-fields.
* Fixing typo.
* Updates for API redesign -- will break builds.
* Updating examples and including info about overriding fields.
* Updating examples.
* Adding info for using runtime fields in the search request.
* Adding that queries against runtime fields are expensive.
* Incorporating feedback from reviewers.
* Minor changes from reviews.
* Adding alias for test case.
* Adding aliases to PUT example.
* Fixing test cases, for real this time.
* Updating use cases and introducing overlay throughout.
* Edits, adding 'shadowing', and explaining shadowing better.
* Streamlining tests and other changes.
* Fix formatting in example for test.
* Apply suggestions from code review
* Incorporating reviewer feedback 7 Dec
* Shifting structure of mapping page to fix cross links.
* Revisions for shadowing, overview, and other sections.
* Removing dot notation section and incorporating review changes.
* Adding updated example for shadowing.
* Streamlining shadowing example and TESTRESPONSEs.
* Adding type field in responses for 7.x to fix test cases.
Currently, the 'fields' option only supports fetching mapped fields. Since
'fields' is meant to be the central place to retrieve document content, it
should allow for loading unmapped values. This change adds implementation and
tests for this feature.
Closes#63690
- Replaces more abstract docs about object structure and values source with task-based examples.
- Relocates several sections from the current `misc.asciidoc` file.
- Alphabetically sorts agg categories in the nav.
- Removes the matrix agg family. Moves the stats matrix agg under the metric agg family
Co-authored-by: debadair <debadair@elastic.co>
After #63811 it became clear to me that `postCollect` is kind of
dangerous and not all that useful. So this removes it.
The trouble with `postCollect` is that it all happened right after we
finished calling `collect` on the `LeafBucketCollectors` but before we
built the aggregation results. But in #63811 we found out that we can't
call `postCollect` on the children of `parent` or `child` aggregators
until we know which *which* aggregation results we're building.
So this removes `postCollect` and moves all of the things we did at
post-collect phase into `buildAggregations` or into hooks called in
those methods.
We've identified two important enhancements that may affect the API. We expect
any API changes from these enhancements to be minor, but want to leave open the
possibility for small breaks. For example, we may end up returning unmapped
fields by default, or omitting nested fields from the root hit. The impact to
users should be quite small.
We're tracking the issues we need to resolve before removing the 'beta' label
here: #60985.
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.
Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:
- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`
Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
