I forgot to link the ToDateNanos docs when I merged that function.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This adds a sentence to `redirects.asciidoc` explaining what frozen
indices were - otherwise, everything will point to the message about
the unfreeze API having gone away, which is not very helpful. Some
cross-references are updated to point to this rather than to the
notice about the removal of the unfreeze API.
ES-9736 #comment Removed `_unfreeze` REST endpoint in https://github.com/elastic/elasticsearch/pull/119227
`fold` can be surprisingly heavy! The maximally efficient/paranoid thing
would be to fold each expression one time, in the constant folding rule,
and then store the result as a `Literal`. But this PR doesn't do that
because it's a big change. Instead, it creates the infrastructure for
tracking memory usage for folding as plugs it into as many places as
possible. That's not perfect, but it's better.
This infrastructure limit the allocations of fold similar to the
`CircuitBreaker` infrastructure we use for values, but it's different
in a critical way: you don't manually free any of the values. This is
important because the plan itself isn't `Releasable`, which is required
when using a real CircuitBreaker. We could have tried to make the plan
releasable, but that'd be a huge change.
Right now there's a single limit of 5% of heap per query. We create the
limit at the start of query planning and use it throughout planning.
There are about 40 places that don't yet use it. We should get them
plugged in as quick as we can manage. After that, we should look to the
maximally efficient/paranoid thing that I mentioned about waiting for
constant folding. That's an even bigger change, one I'm not equipped
to make on my own.
* Including examples
* Using js instead of json
* Adding unified docs to main page
* Adding missing description text
* Refactoring to remove unified route
* Addign back references to the _unified route
* Update docs/reference/inference/chat-completion-inference.asciidoc
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Address feedback
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
This wires up the randomized testing for DateFormat. Prior to this PR, none of the randomized testing was hitting the one parameter version of the function, so I wired that up as well. This required some compromises on the type signatures, see comments in line.less
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
As a drive-by, this removes another usage of the trappy default master
node timeout.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
As a drive-by, this removes another usage of the trappy default master
node timeout.
Fixes two bugs in _resolve/cluster.
First, the code that detects older clusters versions and does a fallback to the _resolve/index
endpoint was using an outdated string match for error detection. That has been adjusted.
Second, upon security exceptions, the _resolve/cluster endpoint was marking the clusters as connected: true,
under the assumption that all security exceptions related to cross cluster calls and remote index access were
coming from the remote cluster, but that is not always the case. Some cross-cluster security violations can
be detected on the local querying cluster after issuing the remoteClient.execute call but before the transport
layer actually sends the request remotely. So we now mark the connected status as false for all ElasticsearchSecurityException cases. End user docs have been updated with this information.
* [Connector API] Add interface for soft-deletes
* Define connector deleted system index
* Got soft-delete logic working
* Add unit tests
* Add yaml e2e test and attempt to update permissions
* Fix permissions
* Update docs
* Fix docs
* Update docs/changelog/118282.yaml
* Change logic
* Fix tests
* Remove unnecessary privilege from yaml rest test
* Update changelog
* Update docs/changelog/118669.yaml
* Adapt yaml tests
* Undo changes to muted-tests.yml
* Fix compilation issue after other PR got merged
* Exclude soft-deleted connector from checks about index_name already in use
* Update docs/reference/connector/apis/get-connector-api.asciidoc
Co-authored-by: Tim Grein <tim@4greins.de>
* Update rest-api-spec/src/main/resources/rest-api-spec/api/connector.list.json
Co-authored-by: Tim Grein <tim@4greins.de>
* Adapt comments, add connector wire serializing test
* Introduce new transport versions for passing the delete flag
* Get rid of wire serialisation, use include_deleted instead of deleted flag
* Remove unused import
* Final tweaks
* Adapt variable name in rest layer
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Tim Grein <tim@4greins.de>
I believe this is a typo, as in our 8.16.1 cluster this field appears to be called `combined_coordinating_and_primary`
Co-authored-by: Ian Lee <IanLee1521@gmail.com>
This PR introduces a new syntactical feature to index expression resolution: The selector.
Selectors, denoted with a :: followed by a recognized suffix will allow users to specify which component of
an index abstraction they would like to operate on within an API call. In this case, an index abstraction is a
concrete index, data stream, or alias; Any abstraction that can be resolved to a set of indices/shards. We
define a component of an index abstraction to be some searchable unit of the index abstraction.
This exposes new OTel node and index based metrics for indexing failures due to version conflicts.
In addition, the /_cat/shards, /_cat/indices and /_cat/nodes APIs also expose the same metric, under the newly added column iifvc.
Relates: #107601
When originally added, the knn query didn't apply `top-k` restrictions
to the query. Instead it would allow the resulting `num_candidate` to be
combined with sibling queries without restricting to `top-size` results
ahead of time.
This honestly is confusing behavior and leads to some bugs in understand
how it all works.
This commit addresses this by eagerly gathering only `size` results when
`k==null` before combining with other queries.
To achieve the previous behavior, this can be done directly by setting
`k==num_candidates` in the query.
Late-interaction models are powerful rerankers. While their size and
overall cost doesn't lend itself for HNSW indexing, utilizing them as
second order "brute-force" reranking can provide excellent boosts in
relevance. At generally lower inference times than large cross-encoders.
This commit exposes a new experimental `rank_vectors` field that allows
for maxSim operations. This unlocks the initial, and most common use of
late-interaction dense-models.
For example, this is how you would use it via the API:
```
PUT index
{
"mappings": {
"properties": {
"late_interaction_vectors": {
"type": "rank_vectors"
}
}
}
}
```
Then to index:
```
POST index/_doc
{
"late_interaction_vectors": [[0.1, ...],...]
}
```
For querying, scoring can be exposed with scripting:
```
POST index/_search
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "maxSimDotProduct(params.query_vector, 'my_vector')",
"params": {
"query_vector": [[0.42, ...], ...]
}
}
}
}
}
```
Of course, the initial ranking should be done before re-scoring or
combining via the `rescore` parameter, or simply passing whatever first
phase retrieval you want as the inner query in `script_score`.
Shutdown metadata is keyed on node id. This makes sense since only one
node with a given node id can exist within a cluster. However, it is
possible that shutdown was initiated for one instance of a node, but
that node is restarted. This commit adds the ephemeral node id to
shutdown metadata so that nodes with the same id but different ephemeral
id can be distinguished.
* (Doc+) Enrich run on ingest+data nodes not coordinating-only
👋 howdy, team! I'm not otherwise finding it so documenting https://github.com/elastic/elasticsearch/issues/95969 in ES docs
> Currently we tell users of enrich that they should co-locate the nodes that perform the enrichment (ingest nodes) with the actual enrich data so that enrich operations don't require a remote search operation.
* feedback
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
---------
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
The `?local` parameter becomes a no-op and is marked as deprecated.
Relates #101805
Relates #107984
