API for `/_inference/{task_type}/{inference_id}/_stream` and `/_inference/{inference_id}/_stream`
Request is `application/json`
Response is `text/event-stream`
This adds some more counts for dense_vector field mapping stats. This
allows for seeing the number of mappings with a given element type,
similarity, or index type.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add remote cluster stats to _cluster/stats
* Implement remote cluster stats polling
* Add docs for the include_remotes part
(cherry picked from commit b26d81c713)
Deprecate to, from, include_lower, include_upper range query params.
These params have been removed from our documentation in v. 0.90.4 (d6ecdecc19),
but did not got through deprecation cycle.
These params to be removed in v9.0.
Related to #81276Closes#48538
This implements the `parseBytesRef` method for the `_ts_routing_hash` field so we
can parse the values generated by the companion `format` method.
We parse the values when fetching them from the source when the field is used
as a `sort` paired with `search_after`.
Before this change a sort by and search_after `_ts_routing_hash` would yield
an `UnsupportedOperationException`
(cherry picked from commit 4e5e870370)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Here we introduce a new index-level setting, `ignore_above`, similar to what we have
for `ignore_malformed`. The setting will apply to all `keyword`, `wildcard` and `flattened`
fields. Each field mapping will still be allowed to override the index-level setting using a
mapping-level `ignore_above` value.
(cherry picked from commit 208a1fe571)
* Add support for multi-value dimensions (#112645)
Closes https://github.com/elastic/elasticsearch/issues/110387
Having this in now affords us not having to introduce version checks in
the ES exporter later. We can simply use the same serialization logic
for metric attributes as we do for other signals. This also enables us
to properly map `*.ip` fields to the ip field type as ip fields
containing a list of IPs are not converted to a comma-separated list.
(cherry picked from commit 8d223cbf7a)
# Conflicts:
# server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java
* Remove skip test for 8.x
This was just needed for 8.x to 9.0 compatibility tests
* Fix `ignore_above` for flattened fields (#112944)
Flattened fields do no actually use `ignore_above`. When retrieving source, `ignore_above` behaves
as expected. Anyway, it is ignored when fetching field values through the fields API. In that case
values are returned no matter the `ignore_above` setting. Here we implement the filtering logic
in the `ValueFecther` implementation of `RootFlattenedFieldType`.
(cherry picked from commit 91674f85a0)
* Fix
---------
Co-authored-by: Salvatore Campagna <93581129+salvatore-campagna@users.noreply.github.com>
The addition of the logger requires several updates to tests to deal with the possible warning, or muting if there is not way to specify an allowed (but not mandatory) warning
Introduces per-field param `synthetic_source_keep` that overrides the
behavior for keeping the field source in synthetic source mode: -
`none` : no source is stored - `arrays`: the incoming source is
recorded as-is for arrays of a given field - `all`: the incoming source
is recorded as is for both singleton and array values of a given field
Related to #112012
* Retrieve the source for objects and arrays within arrays in a separate parsing phase (#113027)
In synthetic source, storing array elements to `_ignored_source` may
hide other, regular elements from showing up during source synthesizing.
This is due to contents from `_ignored_source` taking precedence over
matching fields from regular source loading.
To avoid this, arrays are pre-emptively tracked and marked for source
storing, if any of their elements needs to store its source. A second
doc parsing phase is introduced that checks for fields missing values
and records their source, while skipping objects and arrays that don't
contain any such fields.
Fixes#112374
(cherry picked from commit 4ff4384550)
# Conflicts:
# muted-tests.yml
* merge
* fix compilation error, jdk-21 function
* Deprecate dot-prefixed indices and composable template index patterns (#112571)
This commit adds a module emitting a deprecation warning when a
dot-prefixed index is manually or automatically created, or when a
composable index template with an index pattern that uses a dot-prefix
is created. This pattern warns that in the future these indices will not
be allowed. In a future breaking change (10.0.0 maybe?) the deprecation
can then be changed to an exception.
These deprecations are only displayed when a non-operator user is using
the API (one that does not set the `X-elastic-product-origin` header).
* Attempt to fix build for V7 gradle stuff
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Adds an API which scans all the metadata (and optionally the raw data)
in a snapshot repository to look for corruptions or other
inconsistencies.
Closes https://github.com/elastic/elasticsearch/issues/52622 Closes
ES-8560
If a segment doesn't contain any documents with a dense_vector field,
but the mapping defines it, an NPE can occur when retrieving the
dense_vector stats.
Relates #111729
Introduce an index setting that forces storing the source of leaf field
and object arrays in synthetic source mode. Nested objects are excluded
as they already preserve ordering in synthetic source.
Next step is to introduce override params at the mapper level that will
allow disabling the source, or storing the source for arrays (if not
enabled at index level), or storing the source for both arrays and
singletons. This will happen in follow-up changes, so that we can
benchmark the impact of this change in parallel.
Related to #112012
JDK 23 removes the COMPAT locale provider, leaving CLDR as the only option. This commit configures Elasticsearch
to use the CLDR provider when on JDK 23, but still use the existing COMPAT provider when on JDK 22 and below.
This causes some differences in locale behaviour; this also adapts various tests to still work whether run on COMPAT or CLDR.
This PR adds a count of currently unassigned primary shards to both the
`/_cat/health` and `/_cluster/health` endpoints. This is to aid cluster
administrators in estimating the time remaining for a cluster to go from
RED to YELLOW status as per enchancement request #111727.
Tests and doc updates are in place with this PR and manual testing with
`./gradlew run` has been conducted on the endpoints to ensure correct
output.
## Known Limitations * Testing * Due to limitations in the YAML REST
test framework skip functionality, YAML REST tests for this endpoint are
disabled when running a mixed version cluster by using a cluster version
number synthetic feature to skip when any member of the cluster is not
at a version greater than when this change is due to be introduced
This commit adds support for the `verbose` querystring parameter to the
get data stream API (`GET /_data_stream/{name}`).
The flag defaults to "false".
When set to true, the `maximum_timestamp` for the data stream will be
retrieved and returned for each data stream retrieved. This is the same
information available from the data stream stats API (and internally
uses the same action to retrieval).
Using a normalizer for a keyword field might result in not being able
to reconstruct the original source when using synthetic source.
Here if synthetic source is enabled and a normalizer is configured
we store the original value in a stored field which is later used
at document reconstruction time to reconstruct the field value as
it was in the original document.
We use the same fallback solution we use in other places like
`ignore_malformed`.
