This PR adds a new `knn` option to the `_search` API to support ANN search.
It's powered by the same Lucene ANN capabilities as the old `_knn_search`
endpoint. The `knn` option can be combined with other search features like
queries and aggregations.
Addresses #87625
Currently we have two parameters that control how the source of a document
is stored, `enabled` and `synthetic`, both booleans. However, there are only
three possible combinations of these, with `enabled:false` and `synthetic:true`
being disallowed. To make this easier to reason about, this commit replaces
the `enabled` parameter with a new `mode` parameter, which can take the values
`stored`, `synthetic` and `disabled`. The `mode` parameter cannot be set
in combination with `enabled`, and we will subsequently move towards
deprecating `enabled` entirely.
* Revert "Revert "[DOCS] Add TSDS docs (#86905)" (#87702)"
This reverts commit 0c86d7b9b2.
* First fix to tests
* Add data_stream object to index template
* small rewording
* Add enable data stream object in gradle example setup
* Add bullet about data stream must be enabled in template
* [DOCS] Add TSDB docs
* Update docs/build.gradle
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Address Nik's comments, part 1
* Address Nik's comments, part deux
* Reword write index
* Add feature flags
* Wrap one more section in feature flag
* Small fixes
* set index.routing_path to optional
* Update storage reduction value
* Update create index template code example
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
This adds an example of the precision loss for `double`, `float`, and
`half_float` numbers that we can link folks to when explaining what
happened to their numbers. You can link directly to it with something
like:
```
/guide/number.html#floating_point
```
* Soft-deprecation of point/geo_point formats
Since GeoJSON and WKT are now common formats for all three types:
geo_shape, geo_point and point
We decided to soft-deprecate the other point formats by ordering:
* GeoJSON (object with keys `type` and `coordinates`)
* WKT `POINT(x y)`
* Object with keys `lat` and `lon` (or `x` and `y` for point)
* Array [lon,lat]
* String `"lat,lon"` (or `"x,y"` in point)
* String with geohash (only in `geo_point`)
The geohash is last because it is only in one field type.
The string version is second last because it is the most controversial
being the only version to reverse the coordinate order from all other
formats (for geo_point only, since the coordinates are not reversed
in point).
In addition we replaced many examples in both documentation and tests
to prioritize WKT over the plain string format.
Many remaining examples of array format or object with keys still exist
and could be replaced by, for example, GeoJSON, if we feel the need.
* Incorrect quote position
This PR adds support for a new mapping parameter to the configuration of the object mapper (root as well as individual fields), that makes it possible to store metrics data where it's common to have fields with dots in their names in the following format:
```
{
"metrics.time" : 10,
"metrics.time.min" : 1,
"metrics.time.max" : 500
}
```
Instead of expanding dotted paths the their corresponding object structure, objects can be configured to preserve dots in field names, in which case they can only hold leaf sub-fields and no further objects.
The mapping parameter is called subobjects and controls whether an object can hold other objects (defaults to true) or not. The following example shows how it can be configured in the mappings:
```
{
"mappings" : {
"properties" : {
"metrics" : {
"type" : "object",
"subobjects" : false
}
}
}
}
```
Closes#63530
There has been some confusion over the definition of a field type family. This
PR clarifies the definition in the docs: the two types should have the exact
same search behavior (including supporting the same queries/ aggs, and producing
the same response). It's not sufficient for them to just support the samme
search operations.
This change also fixes an inaccurate statement that there is only one field type
family so far.
This PR introduces the lookup runtime fields which are used to retrieve
data from the related indices. The below search request enriches its
search hits with the location of each IP address from the `ip_location`
index.
```
POST logs/_search
{
"runtime_mappings": {
"location": {
"type": "lookup",
"lookup_index": "ip_location",
"query_type": "term",
"query_input_field": "ip",
"query_target_field": "_id",
"fetch_fields": [
"country",
"city"
]
}
},
"fields": [
"timestamp",
"message",
"location"
]
}
```
Response:
```
{
"hits": {
"hits": [
{
"_index": "logs",
"_id": "1",
"fields": {
"location": [
{
"city": [ "Montreal" ],
"country": [ "Canada" ]
}
],
"message": [ "the first message" ]
}
}
]
}
}
```
Clarifies that the `orientation` mapping parameter only applies to WKT polygons. GeoJSON polygons use a default orientation of `RIGHT`, regardless of the mapping parameter.
Also notes that the document-level `orientation` parameter overrides the default orientation for both WKT and GeoJSON polygons.
Closes https://github.com/elastic/elasticsearch/issues/84009.
Similar to #82409, but for geo_point fields.
Allows searching on geo_point fields when those fields are not indexed (index: false) but just doc values are enabled.
Also adds distance feature query support for date fields (bringing date field to feature parity with runtime fields)
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
* [DOCS] Update dynamic mapping docs to clarify supported match_mapping_type
* Add ES data type column header
* Remove sentence about always choosing the larger data type
* Clarify that JSON doesn't distinguish types
* Add frame to table
Updates and reuses a warning against creating multi-level `join` fields to make it more prominent.
The current warning is low on the page, where some users may not seeing until they've already begun mapping fields.
Closes https://github.com/elastic/elasticsearch/issues/82818.
Allows searching on ip fields when those fields are not indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
Allows searching on boolean fields when those fields are not indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
Allows searching on keyword fields when those fields are not indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
Similar to #82409, but for date fields.
Allows searching on date field types (date, date_nanos) when those fields are not indexed (index: false) but just doc
values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
Allows searching on number field types (long, short, int, float, double, byte, half_float) when those fields are not
indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Note to reviewers:
I have split isSearchable into two separate methods isIndexed and isSearchable on MappedFieldType. The former one is
about whether actual indexing data structures have been used (postings or points), and the latter one on whether you
can run queries on the given field (e.g. used by field caps). For number field types, queries are now allowed whenever
points are available or when doc values are available (i.e. searchability is expanded).
Relates #81210 and #52728
Cosine similarity is not defined when one of the vectors has zero magnitude.
Before, the kNN search endpoint threw a confusing exception related to top docs
collection. Now we reject vectors early with a clear error message, failing
indexing if the vector has zero magnitude.
This commit adds docs for the new `_knn_search` endpoint.
It focuses on being an API reference and is light on details in terms of how
exactly the kNN search works, and how the endpoint contrasts with
`script_score` queries. We plan to add a high-level guide on kNN search that
will explain this in depth.
Relates to #78473.
Currently, we don't support kNN search against fields in a `nested` mapping.
Before, we were checking this at search-time. This commit moves it earlier, so
you aren't even allowed to set `index: true` if the vector is in a nested
mapping. That way, users are aware of the limitation before they start to index
documents.
Relates to #78473.
This reverts the change to use segment ordinals in composite terms aggregations due to a performance degradation when the field is high cardinality.
Co-authored-by: Mark Tozzi <mark.tozzi@elastic.co>
Removes `testenv` annotations and related code. These annotations originally let you skip x-pack snippet tests in the docs. However, that's no longer possible.
Relates to #79309, #31619
This commit updates the `dense_vector` docs to include information on the new
`index`, `similarity`, and `index_options` parameters. It also tries to clarify
the difference between `similarity` and `index_options` with the existing
parameters that have the same name.
Relates to #78473.
Adds an `experimental` annotation to the following:
* `time_series_metric` mapping parameter
* `time_series_dimension` mapping parameter
* `index.mapping.dimension_fields.limit` index setting
* `time_series_dimension` and `time_series_metric` properties in the field caps API response
