With this, all rolling upgrade tests that involve a
`nextNodeToNextVersion` update are gradle configuration cache
compatible.
Simplify API around test cluster registry and cc compatible usage of
test cluster in TestClusterAware tasks.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
To simplify the migration away from version based skip checks in YAML specs,
this PR adds a synthetic version feature `gte_vX.Y.Z` for any version at or before 8.14.0.
New test specs for 8.14 or later are expected to use respective new cluster features,
or a test-only feature supplied via ESRestTestCase#createAdditionalFeatureSpecifications
if sufficient.
This commit removes the legacy yaml rolling upgrade tests for vectors to the new rolling upgrade package.
Also, it adds rolling upgrade tests for `int8_hnsw`.
For float32, there is no compelling reason to use all the memory
required by default for HNSW. Using `int8_hnsw` provides a much saner
default when it comes to cost vs relevancy.
So, on all new indices that use `dense_vector` and want to index them
for fast search, we will default to `int8_hnsw`.
Users can still customize their parameters, or prefer `hnsw` over
float32 if they so desire.
`cosine` is our default similarity and should provide a good experience
on speed.
`dot_product` is faster than `cosine` as it doesn't require calculating
vector magnitudes in the similarity comparison loop. Instead, it can
assume vectors have a length of `1` and use an optimized `dot_product`
calculation.
However, `cosine` as it exists today accepts vectors of any magnitude
and cannot take advantage of this.
This commit addresses this by:
- Normalizing all vectors passed when indexing via `cosine`
- Storing the calculated magnitude in an additional field (only if its `!= 1`).
- Using the `dot_product` Lucene calculation
- Normalizing query vectors when used against these new `cosine` fields
- De-normalizing vectors when accessed via scripts
- Allowing scripts to access these stored magnitudes.
Checks for soft delete support (and deprecation of non-soft delete), peer recovery retention leases and replication of closed indices are using IndexVersion in production code. Moving these checks from Version to IndexVersion in rest tests too.
* Don't update system index mappings in mixed clusters
#99668 seems to have introduced a bug where
SystemIndexMappingUpdateService updates system index mappings
even in mixed clusters. This PR restores the old version-based
check in order to be sure that there's no update until the
cluster is fully upgraded.
The timing of the mapping update seems to be causing worse
problems, corrupting persisted cluster state.
Fixes#99778, #101331
* Remove broken assertion
The compatibility versions objects are not showing up
correctly, so we shouldn't assert on them.
Similar to the TransportVersions holder class, IndexVersions is the new
place to contain all constants for IndexVersion. This commit moves all
existing constants to the new class. It is purely mechanical.
* Replaced some occurrences of Version.CURRENT toString
* Added unit tests for the 2 different warn header patterns in HeaderWarnings
* Comment to clarify non-semantic versions + revert change on Version.major message
Re-applies the changes from #99572 to move some bwc tests to a junit-based build infrastructure. Some tests that did not handle the move well have been kept in rolling-upgrade-legacy using the old gradle-based infrastructure
Two test suites did not react well to the junit-based bwc infrastructure, so those have been separated into a legacy module using the old gradle-based system until they can be looked at properly.
This unblocks the 8.11 release.
