This change makes it so that the reference time from which the "age" field of the IndexLifecycleExplainResponse
object is derived does not change for the duration of testConcurrentToXContent().
Co-authored-by: Keith Massey <keith.massey@elastic.co>
Different JRE vendors/version produce slightly different error
messages. We don't care which one we receive, just accept any of them.
Resolves: #115016
This PR increases client's port range for `ServerTransportFilterIntegrationTests`
which are executed on Windows. This is needed in order to avoid failures
due to some port ranges being excluded from use.
The larger ports range (300) is chosen based on the observation where a
random consecutive range of 200 ports can be excluded on Windows test workers.
apache.santuario.xmlsec version 2.1.4 is documented vulnerable.
We should update to mitigate the vulnerabilities.
But apache.santuario.xmlsec is a dependency of opensaml version 3.*.
However, in a patch release of elasticsearch (i.e. 7.17.*) it's best we avoid updating dependencies across major versions (i.e. opensaml from version 3.* to version 4.*), particularly for such a complex dependency as opensaml (we did update the opensaml dep in this way, but in a minor elasticsearch 8.* release, i.e. #98199). The latest opensaml 3.* release (i.e. 3.4.6) still requires a vulnerable apache.santuario.xmlsec dep: https://mvnrepository.com/artifact/org.opensaml/opensaml-xmlsec-impl/3.4.6).
In this case, our best hope is to find a non-vulnerable version of apache.santuario.xmlsec that is still on the same major version as the version listed in the deps of opensaml (i.e. 2.*). That's version 2.2.6: https://mvnrepository.com/artifact/org.apache.santuario/xmlsec/2.2.6 , which is not vulnerable
This PR updates apache.santuario.xmlsec from the existing 2.1.4 version to the 2.2.6 version. The release notes of the 2.2.0 version from https://santuario.apache.org/javareleasenotes.html look simple, and the dependencies differences (from https://mvnrepository.com/artifact/org.apache.santuario/xmlsec/2.1.4) are minimal as well (hopefully optional dependencies, which we don't pull in, stay optional in the same way in the new version).
So, it looks to me that the dep update is relatively safe (and it also passes CI)!
Prior to this PR, when the security-crypto threadpool queue overflows and rejects API key hashing submissions, a toxic value (specifically, a future which will never be completed) is added to the API key auth cache. This toxic cache value causes future authentication attempts with that API key to fail by timeout, because they will attempt to wait for the toxic future, until that value is invalidated and removed from the cache. Additionally, this will hold on to memory for each request that waits on the toxic future, even after the request has timed out.
This PR adds a unit test to replicate this case, and adjusts the code which submits the key hashing task to the security-crypto threadpool to properly handle this point of failure by invalidating the cached future and notifying waiting handlers that the computation has failed.
elasticsearch-certutil csr generates a private key and a certificate
signing request (CSR) file. It has always accepted the "--pass" command
line option, but ignore it and always generated an unencrypted private
key.
This commit fixes the utility so the --pass option is respected and the
private key is encrypted.
Backport of: #106105
Sometimes buildkite seems to swallow time. This commit extends some of
the authentication/session expiry times to compensate.
Relates: #109299
Backport of: #109607
This commit introduced stricter DLS rules and is a manual backport of #105709 and #105714
with additional node level settings to optionally disable the stricter DLS rules.
Since these settings are not present in 8.x the needed deprecation info API entries have also
been added to help inform any users that may have set these values to remove them before upgrading.
Fixes IpFilterRemoteAddressFilterTests testThatFilteringWorksByIp. There
were 2 "@ Before"-annotated methods that didn't play well together.
Fixes: #108441
Updating repository-hdfs, repository-gcs, and vector-tile to all use the same more recent protobuf.
Co-authored-by: Keith Massey <keith.massey@elastic.co>
* SQL: Limit how much space some string functions can use (#107333)
This will check and fail if certain functions would generate a result
exceeding a certain fixed byte size.
This prevents an operation/query to fail the entire VM.
(cherry picked from commit f1bcb338ec)
* Fix use of forbidden APIs
* Style
This PR increases client's port ranges for tests which are executed on
Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on the
observation where a random consecutive range of 200 ports can be
excluded on Windows test workers.
Relates to https://github.com/elastic/elasticsearch/pull/103894, which
fixed similar issues but this one seems to have been missed.
Resolves: https://github.com/elastic/elasticsearch/issues/106537
It's possible that a 7.17 master node needs to assign ML jobs
to 8.x ML nodes. Currently this does not work, as the 7.x mechanism
for detecting ML nodes does not work with 8.x ML nodes. Currently
a 7.17 master node will not assign jobs to 8.x ML nodes, so the
jobs sit in limbo until the master node is upgraded to 8.x, and
then they get assigned and pick up where they left off.
This change allows the 7.17 master node to correctly identify 8.x
ML nodes in the cluster, allowing the ML jobs to be reassigned
more quickly during a rolling upgrade from 7.17 to 8.x where
master nodes are upgraded last (as recommended).
This commit upgrades the non-FIPS jars to the lastest available versions for 7.17.
Bouncy Castle slightly changed their naming as evident by bouncycastle.org/latest_releases.html
They no longer maintain a jdk15on library and instead use strategy for [1.5->1.8) and [1.8 -> ). This commit
adopts the jdk18on libraries with the latest version.
related: #100923
* [Test] Use larger client ports range for tests running on Windows (#103894)
This PR increases client's port ranges for tests which are executed
on Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on
the observation where a random consecutive range of 200 ports can
be excluded on Windows test workers.
Closes#102349
(cherry picked from commit bdf5c7fc38)
# Conflicts:
# modules/transport-netty4/src/internalClusterTest/java/org/elasticsearch/transport/netty4/Netty4TransportMultiPortIntegrationIT.java
# x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/transport/filter/IpFilteringIntegrationTests.java
* Fix compilation error
These tests were muted both at the suite level as well as at the test level
for reasons I don't fully understand, and then were unmuted at one level
but not the other. They don't appear to fail after a few thousand runs,
so this PR unmutes them the rest of the way.
* `WaitForSnapshotStep` verifies if the index belongs to the latest snapshot of that SLM policy (#100911)
The `WaitForSnapshotStep` used to check if the SLM policy has been
executed after the index has entered the delete phase, but it did not
check if the SLM policy included this index.
The result of this is that if the user used an SLM policy that did not
include this index, when the index would enter the
`WaitForSnapshotStep`, it would wait for a snapshot to be taken, a
snapshot that would not include the index, and then ILM would delete the
index.
See the exact reproduction path:
https://github.com/elastic/elasticsearch/issues/57809
**Solution** This PR, after it finds a successful SLM run, it verifies
if the snapshot taken by SLM contains this index. If not it throws an
error, otherwise it proceeds.
ILM explain will report:
```
"step_info": {
"type": "illegal_state_exception",
"reason": "the last successful snapshot of policy 'hourly-snapshots' does not include index '.ds-my-other-stream-2023.10.16-000001'"
}
```
**Backwards compatibility concerns** In this PR, the
`WaitForSnapshotStep` changed from `ClusterStateWaitStep` to
`AsyncWaitStep`. We do not think this is gonna cause an issue. This was
tested manually by the following steps: - Run a master node with the old
version. - When ILM is executing `wait-for-snapshot`, we shutdown the
node - We start the node again with the new version os ES - ES was able
to pick up the step and continue with the new code.
We believe that this covers bwc concerns.
Fixes: https://github.com/elastic/elasticsearch/issues/57809
(cherry picked from commit 5697fcf594)
* Update gradle wrapper to 8.3 (#97838)
Gradle now fully supports compiling, testing and running on Java 20.
Among other general performance improvements this release introduces --test-dry-run command line option that allows checking if tests are filtered or not by gradle.
Required updating nebula ospackage plugin as setuid was broken in gradle 8.3.
(cherry picked from commit b23e000c30)
# Conflicts:
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ElasticsearchJavaModulePathPlugin.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/compat/compat/AbstractYamlRestCompatTestPlugin.java
# build-tools-internal/src/main/resources/minimumGradleVersion
# gradle/verification-metadata.xml
# gradle/wrapper/gradle-wrapper.jar
# gradlew
# x-pack/plugin/watcher/qa/with-monitoring/src/javaRestTest/java/org/elasticsearch/smoketest/MonitoringWithWatcherRestIT.java
* [7.17] Use patched nebula os package gradle plugin
* Update testingconvention precommit integ test
* [ML] Defend against negative datafeed start times (#100284)
A negative start time in the datafeed can cause significant disruption
to an entire cluster. This PR checks that the start time is greater
than or equal to 0 and throws an exception otherwise.
* Adjust backported test for 7.17
This PR adds a validation step to the end of an enrich policy run to ensure the integrity of the
enrich index that is about to be promoted.
(cherry picked from commit 225db3190a)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
There should be NullPointerException check and throw index not found exception to the response
so the user can understand what happens with the enrich index
---------
Co-authored-by: James Baiera <james.baiera@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit ccc896d128)
# Conflicts:
# x-pack/plugin/enrich/src/main/java/org/elasticsearch/xpack/enrich/EnrichCache.java
# x-pack/plugin/enrich/src/test/java/org/elasticsearch/xpack/enrich/EnrichCacheTests.java
Co-authored-by: puppylpg <shininglhb@163.com>
* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
