#105359. we changed the bucket ordering for partial reduces which causes issues when the output is shared with a
node running on an older version. This commit reorders the output to the expected order for previous versions.
Loads of code here that is only used in tests and one duplicate unused
class that was only used as an indirection to parsing the
`AsyncSearchResponse`. Moved what I could easily move via automated
refactoring to `SearchResponseUtils` in tests and removed the duplicate
now unused class from the client codebase.
We are adding a query parameter to the field_caps api in order to filter out
fields with no values. The parameter is called `include_empty_fields` and
defaults to true, and if set to false it will filter out from the field_caps
response all the fields that has no value in the index.
We keep track of FieldInfos during refresh in order to know which field has
value in an index. We added also a system property
`es.field_caps_empty_fields_filter` in order to disable this feature if needed.
---------
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
A predicate to check whether the cluster supports a feature is available
to rest handlers defined in server. This commit adds that predicate to
plugins defining rest handlers as well.
A Lucene limitation on doc values for UTF-8 fields does not allow us to
write keyword fields whose size is larger then 32K. This limits our
ability to map more than a certain number of dimension fields for time
series indices. Before introducing this change the tsid is created as a
catenation of dimension field names and values into a keyword field.
To overcome this limitation we hash the tsid. This PR is intended to be
used as a draft to test different options.
Note that, as a side effect, this reduces the size of the tsid field as
a result of storing far less data when the tsid is hashed. Anyway, we
expect tsid hashing to affect compression of doc values and resulting in
larger storage footprint. Effect on query latency needs to be evaluated
too.
Resolves#93564
Deprecated node_version field, made it optional(unused) in new parser
Added deprecation warning handler for mixed cluster
Split tests for old vs. current format
Adds a new `RestInterceptor` interface and converts
`RestServerActionPlugin.getRestHandlerInterceptor` to return this new
type instead of a wrapping function.
This has the following benefits:
- Less object creation, there is 1 instance of the interceptor class
(see `SecurityRestFilter`) rather than an instance per handler
- More control over the sequence of steps in processing a request.
The explicit interceptor separates it from the deprecation handler
or any validation that might be needed, and the controller can be
intentional about the order in which these operations are applied.
Add the ability to test for the original/old cluster features during a rolling upgrade
* Moving ALL_FEATURES to ESRestTestCase (and make it private - only usage)
A couple of children of `BroadCastResponse` are completely redundant,
adding no extra fields or separate serialization.
Removed them and replaced their use by the broadcast response itself.
We recently introduced support for index_filter to the open point in time API (see #102388).
Open point in time supports executing against remote indices, in which case it will open a
reader context against the target remote shards. With support for index_filter, shards that
cannot match the filter are not even included in the PIT id that open PIT returns.
When the following search is executed that includes such PIT id, there is one search shards call
per cluster performed, which will return all shards from the targeted indices, including those
that open PIT has filtered out. In that case, we should just ignore those shards instead of
throwing exception when those are looked up in the search context id map built from the PIT id.
Closes#102596
* Introduce Prerequisites criteria (Predicate + factory) for modular skip decisions
- Removed accessors to specific criteria from SkipSection (used only on tests), adjusted test assertions
- Moved Features check (YAML test runner features) to SkipSection build time
* Separated check for xpack/no_xpack
Check for xpack is cluster-configuration (modules installed) dependent, while Features are meant to be "static" test-runner capabilities. We separate them so checks on one (test-runner features) can be run before and separately from the other.
* Consolidate skip() methods
- Divide require and skip predicates
- Divide requires and skip parsing (distinct sections)
- Renaming SkipSection to PrerequisiteSection and related methods/fields (e.g. skip -> evaluate)
* Refactoring tests
- moving and adding VersionRange tests
- adding specific version and os skip tests
- modified parse/validate/build to make SkipSection more unit-testable
* Adding cluster feature-based skip criteria
* Updated javadoc + renaming + better skip reason message
* Fix `require_alias` implicit true value on presence
This commit brings the `require_alias` query-string parameter into line with the rest of our parameters where its presence indicates an implicit "true" value (so a user can do `POST /_bulk?require_alias` to enable the check).
Resolves#103945
* Update docs/changelog/104099.yaml
This are to be made ref-counted shortly. There's no point in having any
pooling/leak-tracking for empty instances though. To prepare for that,
lets add some short-cuts for dealing with empty instances to make the
overall change smaller and cleanup code already.
* Pushing down node versions as strings
Deferring Version parsing to the actual places where a minimum node version/common cluster version is needed; eventually this will be completely lazy and/or replaced by other checks (e.g. features).
Combine versions, oses and features in multi-cluster YAML test contexts.
Tests that send REST requests with bodies must today build up a separate
`String` containing the body contents as JSON. This is kinda ugly, and
also means we do not cover the other supported body formats in these
tests. This commit introduces a utility to allow construction of REST
requests with `XContent` bodies directly, and generalizes things to
choose randomly between JSON and other supported body formats.
This indirection isn't really necessary now that its use in the high
level rest client went away. -> flattened the `SearchResponse` class to
make ref counting easier -> removed `InternalSearchResponse` whose
functionality was completely redundant now -> removed serialization and
most of the usage of `SearchResponseSections`, only keeping this for the
few search execution spots that currently use it and where removing it
won't get us anything and will be a huge changeset.
enables the last steps in #102030
Just moving stuff around, no change in behaviour. Moving these fields showed how we are not treating correctly in derived classes where multiple clusters are tested (ex: CCR), but this is for another time.
Co-authored-by: Moritz Mack <moritz@mackmail.net>
`cosine` is our default similarity and should provide a good experience
on speed.
`dot_product` is faster than `cosine` as it doesn't require calculating
vector magnitudes in the similarity comparison loop. Instead, it can
assume vectors have a length of `1` and use an optimized `dot_product`
calculation.
However, `cosine` as it exists today accepts vectors of any magnitude
and cannot take advantage of this.
This commit addresses this by:
- Normalizing all vectors passed when indexing via `cosine`
- Storing the calculated magnitude in an additional field (only if its `!= 1`).
- Using the `dot_product` Lucene calculation
- Normalizing query vectors when used against these new `cosine` fields
- De-normalizing vectors when accessed via scripts
- Allowing scripts to access these stored magnitudes.
* Move TestCluster plugin and module setup from Node to Cluster
* Do not materialize unused TestCluster
* Fix lazy test cluster evaluation
* Register artifact transforms ones per project
* Make task caching for TestClusterAware tasks CC compatible
* Move stateful logic out of taskgraph.whenready
We expect that the HTTP response stats should reflect the responses
received by the client, but today we record that a chunked-encoded
response was sent _after_ sending the last chunk, which might mean that
the client can receive the complete response body and then retrieve
stats that do not include that response. With this commit we record the
stats before sending the last chunk.
Closes#102547
Same as #101175, shorten `client().prepareIndex(index)` and
`client().prepareIndex().setIndex(index)` via a test utility.
Saves lots of code now and sets up some follow-up simplifcations.
* Introducing TestFeatureService to ESRestTestCase
- Added RestTestLegacyFeatures to encompass legacy (historical) features that have been removed from production code but are still needed by REST tests
- Encapsulated Mark's getHistoricalFeatures method inside a FeatureProvider (ESRestTestCaseHistoricalFeatures)
- ESRestTestCaseHistoricalFeatures is not yet used, as we need to figure out how to deal with old cluster tests
This commit upgrades the Bouncy Castle jars. Bouncy Castle is used for
some internal build concners as well as a comnand line application.
Most notably Bouncy Castle is also used as the FIPs certified JCE/JSEE provider
we use to test our ability to use a FIPs compliant crypto provider.
The following changes here are a result of the upgraded Bouncy Castle jars:
* TLSv1.3 is now supported when running in FIPs mode
* RSA PKCS#1 v1.5 is no longer allowed in FIPS mode
* Triple DES (3DES) is no longer allowed in FIPS mode
* Minor updates the security manager configuration used to test FIPs (to read permissions from the security provider)
* Minor adjustments to tests to accommodate the above changes.
* Minor adjustments to the gradle build to accommodate new dependencies
Note - update to the documentation will come in a later commit.
The open point in time API accepts a list of indices and opens a point in time view against those indices.
Like we do already for field caps, this commit allows users to provide an index_filter parameter as part of
the request body, that will be used to execute the can match phase and exclude the indices that can't possibly
match such filter.
Closes#99740
