Currently these tests run against any old cluster older than 8.0.0, but
the fix that allowed `index.mapper.dynamic` to exist is only available
in 7.17.22.
Adjust these tests to only run if old cluster is after version 7.17.21
and before 8.0.0
Currently when upgrading a 7.x cluster to 8.x with
`index.mapper.dynamic` index setting defined the following happens:
- In case of a full cluster restart upgrade, then the index setting gets archived and after the upgrade the cluster is in a green health.
- In case of a rolling cluster restart upgrade, then shards of indices with the index setting fail to allocate as nodes start with 8.x version. The result is that the cluster has a red health and the index setting isn't archived. Closing and opening the index should archive the index setting and allocate the shards.
The change is about ensuring the same behavior happens when upgrading a
cluster from 7.x to 8.x with indices that have the
`index.mapper.dynamic` index setting defined. By re-defining the
`index.mapper.dynamic `index setting with
`IndexSettingDeprecatedInV7AndRemovedInV8` property, the index is
allowed to exist in 7.x indices, but can't be defined in new indices
after the upgrade. This way we don't have to rely on index archiving and
upgrading via full cluster restart or rolling restart will yield the
same outcome.
Based on the test in #109301. Relates to #109160 and #96075
Our test utility returns the node name when starting a new node.
A lot of APIs (such as routing table or node shutdown) require a node id.
This change introduces a simple way to retrieve the node id based on its name.
To simplify the migration away from version based skip checks in YAML specs,
this PR adds a synthetic version feature `gte_vX.Y.Z` for any version at or before 8.14.0.
New test specs for 8.14 or later are expected to use respective new cluster features,
or a test-only feature supplied via ESRestTestCase#createAdditionalFeatureSpecifications
if sufficient.
This commit removes the legacy yaml rolling upgrade tests for vectors to the new rolling upgrade package.
Also, it adds rolling upgrade tests for `int8_hnsw`.
Simple test, sets up downsampling to run in the old cluster, then waits
for it to complete and verifies that downsampled indexes can get queried
in the mixed and upgraded clusters.
For float32, there is no compelling reason to use all the memory
required by default for HNSW. Using `int8_hnsw` provides a much saner
default when it comes to cost vs relevancy.
So, on all new indices that use `dense_vector` and want to index them
for fast search, we will default to `int8_hnsw`.
Users can still customize their parameters, or prefer `hnsw` over
float32 if they so desire.
The code to parse a `SnapshotInfo` object out of an `XContent` response
body is only used in tests, so this commit moves it out of the
production codebase and into the test framework.
Packaging tests have several files that may be useful in debugging
failures. Additionally, we sometimes have assertions for which we want
to catch them and emit additional debugging info. This commit guards
the common ways that Elasticsearch is started and assertions are run
with dumping all debug information available.
Today the handling of the `?after` param is kinda spread out over
`TransportGetSnapshotsAction` and `GetSnapshotsRequest` making it hard
to follow and adding unnecessary complexity to these two classes. This
commit moves it into `SnapshotSortKey` which is a better fit since the
behaviour varies so much for different sort keys.
The behaviour of the get-snapshots API varies quite considerably
depending on the sort key chosen. Today this logic is implemented using
scattered `switch` statements and other conditionals but it'd be clearer
if we delegated this stuff to the sort key instances themselves. This
commit moves the sort key enum to the top level and replaces one of the
`switch` statements with a method on the enum instances.
This commit decouples the version used in enrollment tokens from node
version, as part of the larger effort to make versioning more granular.
The changes are relatively minimal, as the version encoded into
enrollment tokens is not actually used anywhere as far as I can tell,
either in Elasticsearch or Kibana, apart from checks that it is present.
That said, I've been around the block enough times to know better than
to remove a perfectly good version field that's already in something
like this.
The difference is that our matcher uses .find() to search for a regex match anywhere in the string, whereas the hamcrest one uses .matches() to check the whole string against the regex. This leads to more specific regex checks.
I've left our own one for YAML tests, as that way we don't need to mangle the regex to add .* either side, which might be confusing in test failures.
To prevent leaking sensitive information such as credentials and keys in logs, this
commit prevents configuring some restricted loggers (currently `org.apache.http`
and `com.amazonaws.request`) at high verbosity unless the NetworkTraceFlag
(`es.insecure_network_trace_enabled`) is enabled.
Removes all the now-dead code related to reading pre-7.16 get-snapshots
requests and responses, and also moves the `XContent` response parsing
out of production and into the only test suite that uses it.
Loads of code here that is only used in tests and one duplicate unused
class that was only used as an indirection to parsing the
`AsyncSearchResponse`. Moved what I could easily move via automated
refactoring to `SearchResponseUtils` in tests and removed the duplicate
now unused class from the client codebase.
We are adding a query parameter to the field_caps api in order to filter out
fields with no values. The parameter is called `include_empty_fields` and
defaults to true, and if set to false it will filter out from the field_caps
response all the fields that has no value in the index.
We keep track of FieldInfos during refresh in order to know which field has
value in an index. We added also a system property
`es.field_caps_empty_fields_filter` in order to disable this feature if needed.
---------
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
A predicate to check whether the cluster supports a feature is available
to rest handlers defined in server. This commit adds that predicate to
plugins defining rest handlers as well.
A Lucene limitation on doc values for UTF-8 fields does not allow us to
write keyword fields whose size is larger then 32K. This limits our
ability to map more than a certain number of dimension fields for time
series indices. Before introducing this change the tsid is created as a
catenation of dimension field names and values into a keyword field.
To overcome this limitation we hash the tsid. This PR is intended to be
used as a draft to test different options.
Note that, as a side effect, this reduces the size of the tsid field as
a result of storing far less data when the tsid is hashed. Anyway, we
expect tsid hashing to affect compression of doc values and resulting in
larger storage footprint. Effect on query latency needs to be evaluated
too.
Resolves#93564
Deprecated node_version field, made it optional(unused) in new parser
Added deprecation warning handler for mixed cluster
Split tests for old vs. current format
Adds a new `RestInterceptor` interface and converts
`RestServerActionPlugin.getRestHandlerInterceptor` to return this new
type instead of a wrapping function.
This has the following benefits:
- Less object creation, there is 1 instance of the interceptor class
(see `SecurityRestFilter`) rather than an instance per handler
- More control over the sequence of steps in processing a request.
The explicit interceptor separates it from the deprecation handler
or any validation that might be needed, and the controller can be
intentional about the order in which these operations are applied.
Add the ability to test for the original/old cluster features during a rolling upgrade
* Moving ALL_FEATURES to ESRestTestCase (and make it private - only usage)
A couple of children of `BroadCastResponse` are completely redundant,
adding no extra fields or separate serialization.
Removed them and replaced their use by the broadcast response itself.
We recently introduced support for index_filter to the open point in time API (see #102388).
Open point in time supports executing against remote indices, in which case it will open a
reader context against the target remote shards. With support for index_filter, shards that
cannot match the filter are not even included in the PIT id that open PIT returns.
When the following search is executed that includes such PIT id, there is one search shards call
per cluster performed, which will return all shards from the targeted indices, including those
that open PIT has filtered out. In that case, we should just ignore those shards instead of
throwing exception when those are looked up in the search context id map built from the PIT id.
Closes#102596
* Introduce Prerequisites criteria (Predicate + factory) for modular skip decisions
- Removed accessors to specific criteria from SkipSection (used only on tests), adjusted test assertions
- Moved Features check (YAML test runner features) to SkipSection build time
* Separated check for xpack/no_xpack
Check for xpack is cluster-configuration (modules installed) dependent, while Features are meant to be "static" test-runner capabilities. We separate them so checks on one (test-runner features) can be run before and separately from the other.
* Consolidate skip() methods
- Divide require and skip predicates
- Divide requires and skip parsing (distinct sections)
- Renaming SkipSection to PrerequisiteSection and related methods/fields (e.g. skip -> evaluate)
* Refactoring tests
- moving and adding VersionRange tests
- adding specific version and os skip tests
- modified parse/validate/build to make SkipSection more unit-testable
* Adding cluster feature-based skip criteria
* Updated javadoc + renaming + better skip reason message
