#105359. we changed the bucket ordering for partial reduces which causes issues when the output is shared with a
node running on an older version. This commit reorders the output to the expected order for previous versions.
* Mixed cluster tests with string NodeInfo version
- Move version based feature comparison to a common, deprecated method (to be replaced with real features)
- Use string comparison against old cluster version to partition new/old cluster nodes
* Don't print synthetic source in mapping for bwc tests
* Move comment.
* Don't print synthetic source in mapping for bwc tests #2
* Don't print synthetic source in mapping for bwc tests #2
* Revert "Don't print synthetic source in mapping for bwc tests #2"
This reverts commit 034262c5d2.
* Revert "Don't print synthetic source in mapping for bwc tests #2"
This reverts commit 44e815635e.
* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"
This reverts commit 9322ab9b91.
* Exclude synthetic source test from mixedClusterTests
* Update comment.
* Mute all tsdb tests in mixedClusterTests
This is an interim step to stop sporadic test failures, while we try to
fix version skip for mixed cluster tests.
* Remove old exclusion
* Add aggregations too
* Mute tests for versions between 8.7-8.10
* Remove mute
* Remove setting the number of replicas in aggregation yaml tests.
* Stop waiting for yellow/green. The create index request by default return s when at least the primary shard has been assigned.
* Adjust assertions that are sensitive to the number of shard copies.
* Muted 4 aggregation yaml tests in mixed cluster qa ([link](https://github.com/elastic/elasticsearch/pull/99413/files#diff-033fa895a9866248015cbe7871deab0b3c88fa66a5e1bc9e789079d0d1a39f24R31)). These tests assert cache hits and with replicas we currently can't reliable assert this in a multi node test cluster. Like the mentioned comment suggest if the node selector can reliable select the same node, then we can unmute. I think muting is ok for now. These yaml tests are run in aggregation module and will soon be run in serverless. I think enforcing that many yaml tests run tests against indices with no replicas is worse for test coverage than muting 4 yaml tests in this qa module. Additionally I think that testing that the request cache work should not be tested in full blown integration tests, but more targeted unit/semi integration tests.
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
As we move the aggs tests to a module, we still want to run the aggs
tests against a mixed version cluster. This adds the tests back to the
`qa:mixed-cluster` project.
The ESClientYamlSuiteTestCase is used to run yaml tests throughout
Elasticsearch. It utilizes the low level rest client in sniffing for
nodes, but the sniffer is not needed anywhere else in the test
framework.
This commit creates a new project, `:test:rest-runner` which is meant to
house the rest test running infrastructure. This has two purposes. First
is to remove the sniffer from the test framework dependencies, because
it transitively depends on Jackson. Second is to setup the runner for
future refactorings where it could be made to not depend on the entire
test framework, though how that could work is left for the future.
This adds a new sampling aggregation that performs a background sampling over all documents in an index.
The syntax is as follows:
```
{
"aggregations": {
"sampling": {
"random_sampler": {
"probability": 0.1
},
"aggs": {
"price_percentiles": {
"percentiles": {
"field": "taxful_total_price"
}
}
}
}
}
}
```
This aggregation provides fast random sampling over the entire document set in order to speed up costly aggregations.
Testing this over a variety of aggregations and data sets, the median speed up when sampling at `0.001` over millions of documents is around 70X speed improvement.
Relative error rate does rely on the size of the data and the aggregation kind. Here are some typically expected numbers when sampling over 10s of millions of documents. `p` is the configured probability and `n` is the number of documents matched by your provided filter query.
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
- Add `es.index_mode_feature_flag_registered` feature flag to data-streams module's internalClusterTest task.
- Add `es.random_sampler_feature_flag_registered` feature flag to xpack rest tests with security qa module.
Closes#83722
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
The sort by change has been merged in 7.x now and therefore
we should make 8.0 use the same BWC version.
Relates to #79392 and #79507
Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
* Do not create unused testCluster
This avoids creating test clusters that are not required during the build.
We use lazy configuration here on testClusters and only instantiate them as theyre
* Do not fail on run task (debug)
* Create more test cluster lazy
* Make more test cluster lazy
* Avoid creating unused testcluster
* Fix PluginBuildPlugin
* Fix disabling geo db download
* Fix cluster setup in repository-multi-version
* Polishing
* Fix issue with irretic groovy ogic
* Fix bwc tests
* Fix more bwcTests
* Fix more bwc tests
* Fix more bwc tests
* Fix more bwc tests
* Fix typo
* Minor polishing
* Fix rolling upgrade tests
* Fix cluster config in sql qa mixedcluster project
* Fix more bwc tests
* Clean up before review
* Document test cluster usage
* Api polising after Review
provide useCluster(Provider) method to TestClusterAware
Ideally we take this a step further and realize those test clusters only on use.
But out of scope of this PR.
* Allow gradle provider as value for nonSystemProperties
* Some simplification on test configuration
* Fix typo in rest test config
* Fix more typos
* Fix another typo
* Fix more typos
This adds a setting to enable time series mode that is hidden by the
`es.index_mode_feature_flag_registered` feature flag. All it does right
now is make sure you haven't configured index sorting and partitioning.
This gives us a place to "hang" all of our further work on time series
mode.
Time series mode will entirely take over index sorting. We don't believe
you'll ever be able to configure sorting yourself when you are in time
series mode. We don't expect time series mode to support index
partitioning when we first build it but we'd like to get there.
synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats
relates removal pr #50882
relates #51816
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.
Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.
The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.
In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.
As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
