This PR increases client's port ranges for tests which are executed on
Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on the
observation where a random consecutive range of 200 ports can be
excluded on Windows test workers.
Relates to https://github.com/elastic/elasticsearch/pull/103894, which
fixed similar issues but this one seems to have been missed.
Resolves: https://github.com/elastic/elasticsearch/issues/106537
It's possible that a 7.17 master node needs to assign ML jobs
to 8.x ML nodes. Currently this does not work, as the 7.x mechanism
for detecting ML nodes does not work with 8.x ML nodes. Currently
a 7.17 master node will not assign jobs to 8.x ML nodes, so the
jobs sit in limbo until the master node is upgraded to 8.x, and
then they get assigned and pick up where they left off.
This change allows the 7.17 master node to correctly identify 8.x
ML nodes in the cluster, allowing the ML jobs to be reassigned
more quickly during a rolling upgrade from 7.17 to 8.x where
master nodes are upgraded last (as recommended).
This commit upgrades the non-FIPS jars to the lastest available versions for 7.17.
Bouncy Castle slightly changed their naming as evident by bouncycastle.org/latest_releases.html
They no longer maintain a jdk15on library and instead use strategy for [1.5->1.8) and [1.8 -> ). This commit
adopts the jdk18on libraries with the latest version.
related: #100923
* [Test] Use larger client ports range for tests running on Windows (#103894)
This PR increases client's port ranges for tests which are executed
on Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on
the observation where a random consecutive range of 200 ports can
be excluded on Windows test workers.
Closes#102349
(cherry picked from commit bdf5c7fc38)
# Conflicts:
# modules/transport-netty4/src/internalClusterTest/java/org/elasticsearch/transport/netty4/Netty4TransportMultiPortIntegrationIT.java
# x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/transport/filter/IpFilteringIntegrationTests.java
* Fix compilation error
These tests were muted both at the suite level as well as at the test level
for reasons I don't fully understand, and then were unmuted at one level
but not the other. They don't appear to fail after a few thousand runs,
so this PR unmutes them the rest of the way.
* `WaitForSnapshotStep` verifies if the index belongs to the latest snapshot of that SLM policy (#100911)
The `WaitForSnapshotStep` used to check if the SLM policy has been
executed after the index has entered the delete phase, but it did not
check if the SLM policy included this index.
The result of this is that if the user used an SLM policy that did not
include this index, when the index would enter the
`WaitForSnapshotStep`, it would wait for a snapshot to be taken, a
snapshot that would not include the index, and then ILM would delete the
index.
See the exact reproduction path:
https://github.com/elastic/elasticsearch/issues/57809
**Solution** This PR, after it finds a successful SLM run, it verifies
if the snapshot taken by SLM contains this index. If not it throws an
error, otherwise it proceeds.
ILM explain will report:
```
"step_info": {
"type": "illegal_state_exception",
"reason": "the last successful snapshot of policy 'hourly-snapshots' does not include index '.ds-my-other-stream-2023.10.16-000001'"
}
```
**Backwards compatibility concerns** In this PR, the
`WaitForSnapshotStep` changed from `ClusterStateWaitStep` to
`AsyncWaitStep`. We do not think this is gonna cause an issue. This was
tested manually by the following steps: - Run a master node with the old
version. - When ILM is executing `wait-for-snapshot`, we shutdown the
node - We start the node again with the new version os ES - ES was able
to pick up the step and continue with the new code.
We believe that this covers bwc concerns.
Fixes: https://github.com/elastic/elasticsearch/issues/57809
(cherry picked from commit 5697fcf594)
* Update gradle wrapper to 8.3 (#97838)
Gradle now fully supports compiling, testing and running on Java 20.
Among other general performance improvements this release introduces --test-dry-run command line option that allows checking if tests are filtered or not by gradle.
Required updating nebula ospackage plugin as setuid was broken in gradle 8.3.
(cherry picked from commit b23e000c30)
# Conflicts:
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ElasticsearchJavaModulePathPlugin.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/compat/compat/AbstractYamlRestCompatTestPlugin.java
# build-tools-internal/src/main/resources/minimumGradleVersion
# gradle/verification-metadata.xml
# gradle/wrapper/gradle-wrapper.jar
# gradlew
# x-pack/plugin/watcher/qa/with-monitoring/src/javaRestTest/java/org/elasticsearch/smoketest/MonitoringWithWatcherRestIT.java
* [7.17] Use patched nebula os package gradle plugin
* Update testingconvention precommit integ test
* [ML] Defend against negative datafeed start times (#100284)
A negative start time in the datafeed can cause significant disruption
to an entire cluster. This PR checks that the start time is greater
than or equal to 0 and throws an exception otherwise.
* Adjust backported test for 7.17
This PR adds a validation step to the end of an enrich policy run to ensure the integrity of the
enrich index that is about to be promoted.
(cherry picked from commit 225db3190a)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
There should be NullPointerException check and throw index not found exception to the response
so the user can understand what happens with the enrich index
---------
Co-authored-by: James Baiera <james.baiera@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit ccc896d128)
# Conflicts:
# x-pack/plugin/enrich/src/main/java/org/elasticsearch/xpack/enrich/EnrichCache.java
# x-pack/plugin/enrich/src/test/java/org/elasticsearch/xpack/enrich/EnrichCacheTests.java
Co-authored-by: puppylpg <shininglhb@163.com>
* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
This commit adjusts the behavior of the node shutdown status
calculation to avoid reporting a `STALLED` status when ILM is
in the process of doing something that might result in the
migration being temporarily stalled (at time of writing, that
means shrinking an index).
This check is only made if ILM is not stopped.
Backport of #98367
RestHandler has a number of methods that affect the behaviour of request
processing. If the handler is wrapped (e.g. SecurityRestFilter or
DeprecationRestHandler) then these methods must be delegated to the
underlying handler.
This commit introduces a new abstract base class `FilterRestHandler`
that correctly delegates these methods so that wrappers (subclasses) do
not need to implement the behaviour on a case-by-case basis
Backport of: #98861
Prior to this change NodeReplacementAllocationDecider was unconditionally skipping both replacement source and target nodes when calculation auto-expand replicas. This is fixed by autoexpanding to the replacement node if source node already had shards of the index
Backport of PR #96281 amended for 7.17.x
Closes#89527
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
This is a backport of multiple work items related to authentication enhancements for HTTP,
which were originally merged in the 8.8 - 8.9 releases.
Hence, the HTTP (only the netty4-based implementation (default), not the NIO one) authentication
implementation gets a throughput boost (especially for requests failing authn).
Relates to: ES-6188 #92220#95112
* Update certs for PKI tests and re-enable tests (#97766)
The certs for the PKI tests expired and the test was muted.
This commit follows the instructions in the read to update the certs and unmutes the test.
The certs will now expire 20 years from now.
fixes: #97756
* precommit
* Migrate to data tiers routing configures correct default for mounted indices
(cherry picked from commit eaec9602a9c4a4b092f36fd35f58d1e4b8faca52)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
* Add isExplicitDataTier method back as it's used in 7.17
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Port lucene fix github-12352 to Elasticsearch 7.17
* Update docs/changelog/96721.yaml
* Support for Byte and Short as vector tiles value tag (#97619)
Byte and short data types are not supported by the vector tiles specification. If a user tries to add one of those, it
actually gets ignored. This add s support to those values by casting them as integers. In addition it will throw an
exception if the value added is a type it cannot understand so it is not silently ignored.
# Conflicts:
# x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/vector-tile/10_basic.yml
* Update Guava dependency version to 32.0.1-jre
* Fix dependencies
* Remove direct dependencies on guava ver.31
* Revert "Remove direct dependencies on guava ver.31"
This reverts commit 1e90a271c2.
The method `verifyAutoFollowMonitoring` searches for
`ccr_auto_follow_stats` documents in `monitoring-es-*` indices to see if
one document has the field `number_of_successful_follow_indices` greater
than 0.
If such document is found, it means that x-pack monitoring successfully
indexed documents about CCR stats and that CCR at that time had at least
1 following index successfully running.
But the current `verifyAutoFollowMonitoring` method only check the first
10 docs returned by the search requests. Recent failures of
`AutoFollowIT.testAutoFollowPatterns` on 7.17 branch indicates that more
than 10 docs were found, so I suspect one of them has
`number_of_successful_follow_indices` > 0 but is not returned in the
first 10 top docs.
This pull request changes the method to only count documents with
`number_of_successful_follow_indices` > 0.
Closes#91984
File based service tokens were added to support orchestration
requirements in environments such as ECE and ECK. Outside of these
environments we recommend that API based tokens are used instead.
Resolves: #83491
Cold cache prewarming tasks are not stopped immediately when
the shard is closed, causing excessive use of disk for nothing.
This change adds a Supplier<Boolean> to prewarming logic that
can be checked before executing any consuming operation to know
if the Store is closing.
I'm not super happy with my test changes but the logic for
checking if prewarming works correctly is tricky.
Closes#95504
