JDK 23 removes the COMPAT locale provider, leaving CLDR as the only option. This commit configures Elasticsearch
to use the CLDR provider when on JDK 23, but still use the existing COMPAT provider when on JDK 22 and below.
This causes some differences in locale behaviour; this also adapts various tests to still work whether run on COMPAT or CLDR.
* Add an override to the aggs tests to override the allow list default setting. This makes it possible to run the scripted metric aggs tests on Serverless, even when we disallow these aggs per default on Serverless.
* Move the allow list tests next to the scripted metric tests since these belong together.
Introduces new cluster settings that allow only a certain set of scripts in scripted metrics aggregations:
- search.aggs.only_allowed_metric_scripts, defaults to false
- search.aggs.allowed_inline_metric_scripts, defaults to empty list
- search.aggs.allowed_stored_metric_scripts, defaults to empty list
Add support to the Unified highlighter to combine matches on multiple fields
to highlight a single field: "matched_fields".
Based on Lucene PR: https://github.com/apache/lucene/pull/13268
Lucene PR is based on the concept of masked fields where masked fields
are different from the original highlighted field. This PR in
Elasticsearch uses the already existing highlighter parameter
"matched_fields".
In order to improve the experience of cross-cluster search, we are changing
the default value of the remote cluster `skip_unavailable` setting from `false` to `true`.
This setting causes any cross-cluster _search (or _async_search) to entirely fail when
any remote cluster with `skip_unavailable=false` is either unavailable (connection to it fails)
or if the search on it fails on all shards.
Setting `skip_unavailable=true` allows partial results from other clusters to be
returned. In that case, the search response cluster metadata will show a `skipped`
status, so the user can see that no data came in from that cluster. Kibana also
now leverages this metadata in the cross-cluster search responses to allow users
to see how many clusters returned data and drill down into which clusters did not
(including failure messages).
Currently, the user/admin has to specifically set the value to `true` in the configs, like so:
```
cluster:
remote:
remote1:
seeds: 10.10.10.10:9300
skip_unavailable: true
```
even though that is probably what search admins want in the vast majority of cases.
Setting `skip_unavailable=false` should be a conscious (and probably rare) choice
by an Elasticsearch admin that a particular cluster's results are so essential to a
search (or visualization in dashboard or Discover panel) that no results at all should
be shown if it cannot return any results.
To simplify the migration away from version based skip checks in YAML specs,
this PR adds a synthetic version feature `gte_vX.Y.Z` for any version at or before 8.14.0.
New test specs for 8.14 or later are expected to use respective new cluster features,
or a test-only feature supplied via ESRestTestCase#createAdditionalFeatureSpecifications
if sufficient.
Add the ability to test for the original/old cluster features during a rolling upgrade
* Moving ALL_FEATURES to ESRestTestCase (and make it private - only usage)
We recently introduced support for index_filter to the open point in time API (see #102388).
Open point in time supports executing against remote indices, in which case it will open a
reader context against the target remote shards. With support for index_filter, shards that
cannot match the filter are not even included in the PIT id that open PIT returns.
When the following search is executed that includes such PIT id, there is one search shards call
per cluster performed, which will return all shards from the targeted indices, including those
that open PIT has filtered out. In that case, we should just ignore those shards instead of
throwing exception when those are looked up in the search context id map built from the PIT id.
Closes#102596
* Introduce Prerequisites criteria (Predicate + factory) for modular skip decisions
- Removed accessors to specific criteria from SkipSection (used only on tests), adjusted test assertions
- Moved Features check (YAML test runner features) to SkipSection build time
* Separated check for xpack/no_xpack
Check for xpack is cluster-configuration (modules installed) dependent, while Features are meant to be "static" test-runner capabilities. We separate them so checks on one (test-runner features) can be run before and separately from the other.
* Consolidate skip() methods
- Divide require and skip predicates
- Divide requires and skip parsing (distinct sections)
- Renaming SkipSection to PrerequisiteSection and related methods/fields (e.g. skip -> evaluate)
* Refactoring tests
- moving and adding VersionRange tests
- adding specific version and os skip tests
- modified parse/validate/build to make SkipSection more unit-testable
* Adding cluster feature-based skip criteria
* Updated javadoc + renaming + better skip reason message
* Pushing down node versions as strings
Deferring Version parsing to the actual places where a minimum node version/common cluster version is needed; eventually this will be completely lazy and/or replaced by other checks (e.g. features).
Combine versions, oses and features in multi-cluster YAML test contexts.
Just moving stuff around, no change in behaviour. Moving these fields showed how we are not treating correctly in derived classes where multiple clusters are tested (ex: CCR), but this is for another time.
Co-authored-by: Moritz Mack <moritz@mackmail.net>
The open point in time API accepts a list of indices and opens a point in time view against those indices.
Like we do already for field caps, this commit allows users to provide an index_filter parameter as part of
the request body, that will be used to execute the can match phase and exclude the indices that can't possibly
match such filter.
Closes#99740
Some Rest Specs contain multiple paths, where some paths are
preferred and the others exist for historical reasons.
This commit makes 2 changes:
1. Tracks when a path is deprecated in the Rest Spec, so that metadata
can be accessed in the Java model of the API
2. Adds filter (predicate) to the execution context to determine which
paths should be considered as candidates within the test suite
Tests that extends from `ESClientYamlSuiteTestCase` can override
`createRestTestExecutionContext` and pass in a predicate that skips
deprecated paths (or any other criteria), provided that it does not
skip _all_ paths in an API.
This adds a new parameter to LocalClusterSpecBuilder.user(..) to
indicate whether the user should be an operator or not.
Users added with the simplified user(username, password) method are
operators (and have the "_es_test_root" role).
We want to be able to tweak client configuration in tests extending
ESRestTestCase. This makes that simpler by allowing overriding the
general client configuration.
The PR adds enforcement for API key type at authentication time.
Concretely, new cross-cluster API keys (#95714) can only be used on the
dedicated remote cluster interface and the existing (rest) API key must
not be used for new remote cluster communication. To make cross-cluster
API keys actually usable after authentication, the PR also adds support
for resolving their roles.
This PR adds new named cluster and index priviliges for CCR actions
required with the new RCS model. The new privileges are tightly scoped
so that it is no longer necessary to grant wider named privileges, e.g.
manage. Concretely, the following privileges are added: * cluster
privilege `cross_cluster_replication` * index privilege
`cross_cluster_replication` covers index actions required to be
performed with end users * index privilege
`cross_cluster_replication_internal` covers index actions performed by
the internal user
The intention of having two index privileges for CCR is that
`cross_cluster_replication` could be granted as part of `remote_indices`
for end-users on the QC (follower cluster). QC admins or users do not
have to care about `cross_cluster_replication_internal` which will be
automatically handled by FC (leader cluster) admins once the specialized
API key is in place.
The PR also renames `cross_cluster_access` to `cross_cluster_search`
which corresponds better to the new `cross_cluster_replication`.
Renames the `cluster.remote.*.authorization` setting to
`cluster.remote.*.credentials` since the name is more precise and aligns
with the rest of the code base.
This PR adds a redacted cluster_credentials field to the
RemoteConnctionInfo API response to differenitate between basic and RCS
remote clusters. The new field is available only for RCS remote
clusters.
Rename refactor PR that uses `cross_cluster_access` in place of
`remote_access` wherever appropriate, since `cross_cluster_access` is a
more precise, clearer term. No functional changes, however I did make a
few tweaks around version handling.
The test suite has been recently expanded to including more YAML tests,
PIT and msearh (#93720), EQL (#94265) and SQL (#94416). The original 15
min timeout needs to be bumped a bit to accomodate the new tests. This
PR increases it for additional 5 minutes.
PS: It is possible that the corresponding CcsCommonYamlTestSuite also
needs to similar increase in timeout. This PR does not touch it because
we haven't observed similar timeout for it so far and also tests with
existing CCS model and security disabled run a bit faster. So it may not
be an issue yet.
Resolves: #94491
This PR adds support for CCS common YAML test suite to run SQL yaml
tests with both the existing and new remote cluster models.
Relates: #94265, #93720
PS: SQL does not send its own actions across clusters and relies only on
the generic search, field_caps actions. Hence it is automatically
supported without touching production code, i.e. changes are test only.
This PR adds sniff mode support for RCS. It's achieved by having
alternative handshake action and nodes action to return DiscoveryNode
with remote cluster server address instead of the main transport
address. The changes are only effective to the new model and how
existing remote cluster model works is fully maintained. REST tests are
updated to random between proxy and sniff mode.
An additional benefit of using the alternative handshake action is that
we can remove the special handling of handshake action on the FC side to
ensure all actions must be allowed by the remote access API key. This
will be addressed in a separate PR.
This PR ensures most search features (scroll, async search, pit, field
caps, msearch, vector tile etc) work with the new RCS model. The main
code change is tested by adapting the common yaml CCS tests to use the
new RCS model to provide a broad test coverage. The tests ensure the new
RCS model works from search's perspective. We could still use more tests
from security's perspective, e.g. DLS/FLS, in separate PRs.
Note: * Eql yaml test files are not located under `x-pack/plugin` and
this makes it hard to reuse. It should be possible to relocate them. But
I'll address it separately. * Sql yaml requires special transformation
to work. I'll also have it separately.
This PR moves kNN search and dense vector support out of an xpack plugin and
into server.
In #87625 we plan to integrate ANN search into the main `_search` endpoint as a
new top-level component called `knn`. So kNN will be a dedicated part of the
search request, and we'll have kNN logic within the search phases. The classes
and logic will live in server, matching the other search components like
suggesters, field collapsing, etc.
Currently we only test a small subset of cross cluster search functionality in
rest tests living in the 'multi-cluster-search' qa module. In order to increase
test coverage for basic CCS functionality , this change adds a new qa modula
that re-uses a subset of existing yaml rest tests and runs them in a slightly
modified fashion in a CCS scenario.
Document data and other write operations are executed agains a "remote"
cluster, while all calls to the search API aand other APIs that support CCS are
performed on a local cluster connected to the remote with all the data via CCS.
Relates to #84481
