* [Gradle] Make rolling upgrade tests configuration cache compatible (#119577)
With this, all rolling upgrade tests that involve a
`nextNodeToNextVersion` update are gradle configuration cache
compatible.
Simplify API around test cluster registry and cc compatible usage of
test cluster in TestClusterAware tasks.
(cherry picked from commit 7b6bdfa323)
# Conflicts:
# qa/ccs-rolling-upgrade-remote-cluster/build.gradle
# x-pack/plugin/sql/qa/jdbc/security/build.gradle
# x-pack/plugin/sql/qa/server/security/build.gradle
* Fix backport merge issue
This adds infrastructure to make the legacy test cluster plugin and
the legacy test cluster based test plugins generally configuration cache compatible.
(cherry picked from commit 8c20ac5884)
# Conflicts:
# qa/mixed-cluster/build.gradle
* Deprecate dot-prefixed indices and composable template index patterns (#112571)
This commit adds a module emitting a deprecation warning when a
dot-prefixed index is manually or automatically created, or when a
composable index template with an index pattern that uses a dot-prefix
is created. This pattern warns that in the future these indices will not
be allowed. In a future breaking change (10.0.0 maybe?) the deprecation
can then be changed to an exception.
These deprecations are only displayed when a non-operator user is using
the API (one that does not set the `X-elastic-product-origin` header).
* Attempt to fix build for V7 gradle stuff
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Follow up to #110552, add utility for starting tasks at the same time
via a barrier as discussed there. Also, make use of the new tooling in a
couple more spots to save LoC and thread creation.
* Mixed cluster tests with string NodeInfo version
- Move version based feature comparison to a common, deprecated method (to be replaced with real features)
- Use string comparison against old cluster version to partition new/old cluster nodes
* Don't print synthetic source in mapping for bwc tests
* Move comment.
* Don't print synthetic source in mapping for bwc tests #2
* Don't print synthetic source in mapping for bwc tests #2
* Revert "Don't print synthetic source in mapping for bwc tests #2"
This reverts commit 034262c5d2.
* Revert "Don't print synthetic source in mapping for bwc tests #2"
This reverts commit 44e815635e.
* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"
This reverts commit 9322ab9b91.
* Exclude synthetic source test from mixedClusterTests
* Update comment.
* Mute all tsdb tests in mixedClusterTests
This is an interim step to stop sporadic test failures, while we try to
fix version skip for mixed cluster tests.
* Remove old exclusion
* Add aggregations too
* Mute tests for versions between 8.7-8.10
* Remove mute
* Remove setting the number of replicas in aggregation yaml tests.
* Stop waiting for yellow/green. The create index request by default return s when at least the primary shard has been assigned.
* Adjust assertions that are sensitive to the number of shard copies.
* Muted 4 aggregation yaml tests in mixed cluster qa ([link](https://github.com/elastic/elasticsearch/pull/99413/files#diff-033fa895a9866248015cbe7871deab0b3c88fa66a5e1bc9e789079d0d1a39f24R31)). These tests assert cache hits and with replicas we currently can't reliable assert this in a multi node test cluster. Like the mentioned comment suggest if the node selector can reliable select the same node, then we can unmute. I think muting is ok for now. These yaml tests are run in aggregation module and will soon be run in serverless. I think enforcing that many yaml tests run tests against indices with no replicas is worse for test coverage than muting 4 yaml tests in this qa module. Additionally I think that testing that the request cache work should not be tested in full blown integration tests, but more targeted unit/semi integration tests.
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
As we move the aggs tests to a module, we still want to run the aggs
tests against a mixed version cluster. This adds the tests back to the
`qa:mixed-cluster` project.
The ESClientYamlSuiteTestCase is used to run yaml tests throughout
Elasticsearch. It utilizes the low level rest client in sniffing for
nodes, but the sniffer is not needed anywhere else in the test
framework.
This commit creates a new project, `:test:rest-runner` which is meant to
house the rest test running infrastructure. This has two purposes. First
is to remove the sniffer from the test framework dependencies, because
it transitively depends on Jackson. Second is to setup the runner for
future refactorings where it could be made to not depend on the entire
test framework, though how that could work is left for the future.
This adds a new sampling aggregation that performs a background sampling over all documents in an index.
The syntax is as follows:
```
{
"aggregations": {
"sampling": {
"random_sampler": {
"probability": 0.1
},
"aggs": {
"price_percentiles": {
"percentiles": {
"field": "taxful_total_price"
}
}
}
}
}
}
```
This aggregation provides fast random sampling over the entire document set in order to speed up costly aggregations.
Testing this over a variety of aggregations and data sets, the median speed up when sampling at `0.001` over millions of documents is around 70X speed improvement.
Relative error rate does rely on the size of the data and the aggregation kind. Here are some typically expected numbers when sampling over 10s of millions of documents. `p` is the configured probability and `n` is the number of documents matched by your provided filter query.
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
- Add `es.index_mode_feature_flag_registered` feature flag to data-streams module's internalClusterTest task.
- Add `es.random_sampler_feature_flag_registered` feature flag to xpack rest tests with security qa module.
Closes#83722
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
The sort by change has been merged in 7.x now and therefore
we should make 8.0 use the same BWC version.
Relates to #79392 and #79507
Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
* Do not create unused testCluster
This avoids creating test clusters that are not required during the build.
We use lazy configuration here on testClusters and only instantiate them as theyre
* Do not fail on run task (debug)
* Create more test cluster lazy
* Make more test cluster lazy
* Avoid creating unused testcluster
* Fix PluginBuildPlugin
* Fix disabling geo db download
* Fix cluster setup in repository-multi-version
* Polishing
* Fix issue with irretic groovy ogic
* Fix bwc tests
* Fix more bwcTests
* Fix more bwc tests
* Fix more bwc tests
* Fix more bwc tests
* Fix typo
* Minor polishing
* Fix rolling upgrade tests
* Fix cluster config in sql qa mixedcluster project
* Fix more bwc tests
* Clean up before review
* Document test cluster usage
* Api polising after Review
provide useCluster(Provider) method to TestClusterAware
Ideally we take this a step further and realize those test clusters only on use.
But out of scope of this PR.
* Allow gradle provider as value for nonSystemProperties
* Some simplification on test configuration
* Fix typo in rest test config
* Fix more typos
* Fix another typo
* Fix more typos
This adds a setting to enable time series mode that is hidden by the
`es.index_mode_feature_flag_registered` feature flag. All it does right
now is make sure you haven't configured index sorting and partitioning.
This gives us a place to "hang" all of our further work on time series
mode.
Time series mode will entirely take over index sorting. We don't believe
you'll ever be able to configure sorting yourself when you are in time
series mode. We don't expect time series mode to support index
partitioning when we first build it but we'd like to get there.
synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats
relates removal pr #50882
relates #51816
