This commit introduced stricter DLS rules and is a manual backport of #105709 and #105714
with additional node level settings to optionally disable the stricter DLS rules.
Since these settings are not present in 8.x the needed deprecation info API entries have also
been added to help inform any users that may have set these values to remove them before upgrading.
Fixes IpFilterRemoteAddressFilterTests testThatFilteringWorksByIp. There
were 2 "@ Before"-annotated methods that didn't play well together.
Fixes: #108441
Updating repository-hdfs, repository-gcs, and vector-tile to all use the same more recent protobuf.
Co-authored-by: Keith Massey <keith.massey@elastic.co>
* SQL: Limit how much space some string functions can use (#107333)
This will check and fail if certain functions would generate a result
exceeding a certain fixed byte size.
This prevents an operation/query to fail the entire VM.
(cherry picked from commit f1bcb338ec)
* Fix use of forbidden APIs
* Style
This PR increases client's port ranges for tests which are executed on
Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on the
observation where a random consecutive range of 200 ports can be
excluded on Windows test workers.
Relates to https://github.com/elastic/elasticsearch/pull/103894, which
fixed similar issues but this one seems to have been missed.
Resolves: https://github.com/elastic/elasticsearch/issues/106537
It's possible that a 7.17 master node needs to assign ML jobs
to 8.x ML nodes. Currently this does not work, as the 7.x mechanism
for detecting ML nodes does not work with 8.x ML nodes. Currently
a 7.17 master node will not assign jobs to 8.x ML nodes, so the
jobs sit in limbo until the master node is upgraded to 8.x, and
then they get assigned and pick up where they left off.
This change allows the 7.17 master node to correctly identify 8.x
ML nodes in the cluster, allowing the ML jobs to be reassigned
more quickly during a rolling upgrade from 7.17 to 8.x where
master nodes are upgraded last (as recommended).
This commit upgrades the non-FIPS jars to the lastest available versions for 7.17.
Bouncy Castle slightly changed their naming as evident by bouncycastle.org/latest_releases.html
They no longer maintain a jdk15on library and instead use strategy for [1.5->1.8) and [1.8 -> ). This commit
adopts the jdk18on libraries with the latest version.
related: #100923
* [Test] Use larger client ports range for tests running on Windows (#103894)
This PR increases client's port ranges for tests which are executed
on Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on
the observation where a random consecutive range of 200 ports can
be excluded on Windows test workers.
Closes#102349
(cherry picked from commit bdf5c7fc38)
# Conflicts:
# modules/transport-netty4/src/internalClusterTest/java/org/elasticsearch/transport/netty4/Netty4TransportMultiPortIntegrationIT.java
# x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/transport/filter/IpFilteringIntegrationTests.java
* Fix compilation error
These tests were muted both at the suite level as well as at the test level
for reasons I don't fully understand, and then were unmuted at one level
but not the other. They don't appear to fail after a few thousand runs,
so this PR unmutes them the rest of the way.
* `WaitForSnapshotStep` verifies if the index belongs to the latest snapshot of that SLM policy (#100911)
The `WaitForSnapshotStep` used to check if the SLM policy has been
executed after the index has entered the delete phase, but it did not
check if the SLM policy included this index.
The result of this is that if the user used an SLM policy that did not
include this index, when the index would enter the
`WaitForSnapshotStep`, it would wait for a snapshot to be taken, a
snapshot that would not include the index, and then ILM would delete the
index.
See the exact reproduction path:
https://github.com/elastic/elasticsearch/issues/57809
**Solution** This PR, after it finds a successful SLM run, it verifies
if the snapshot taken by SLM contains this index. If not it throws an
error, otherwise it proceeds.
ILM explain will report:
```
"step_info": {
"type": "illegal_state_exception",
"reason": "the last successful snapshot of policy 'hourly-snapshots' does not include index '.ds-my-other-stream-2023.10.16-000001'"
}
```
**Backwards compatibility concerns** In this PR, the
`WaitForSnapshotStep` changed from `ClusterStateWaitStep` to
`AsyncWaitStep`. We do not think this is gonna cause an issue. This was
tested manually by the following steps: - Run a master node with the old
version. - When ILM is executing `wait-for-snapshot`, we shutdown the
node - We start the node again with the new version os ES - ES was able
to pick up the step and continue with the new code.
We believe that this covers bwc concerns.
Fixes: https://github.com/elastic/elasticsearch/issues/57809
(cherry picked from commit 5697fcf594)
* Update gradle wrapper to 8.3 (#97838)
Gradle now fully supports compiling, testing and running on Java 20.
Among other general performance improvements this release introduces --test-dry-run command line option that allows checking if tests are filtered or not by gradle.
Required updating nebula ospackage plugin as setuid was broken in gradle 8.3.
(cherry picked from commit b23e000c30)
# Conflicts:
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ElasticsearchJavaModulePathPlugin.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/compat/compat/AbstractYamlRestCompatTestPlugin.java
# build-tools-internal/src/main/resources/minimumGradleVersion
# gradle/verification-metadata.xml
# gradle/wrapper/gradle-wrapper.jar
# gradlew
# x-pack/plugin/watcher/qa/with-monitoring/src/javaRestTest/java/org/elasticsearch/smoketest/MonitoringWithWatcherRestIT.java
* [7.17] Use patched nebula os package gradle plugin
* Update testingconvention precommit integ test
* [ML] Defend against negative datafeed start times (#100284)
A negative start time in the datafeed can cause significant disruption
to an entire cluster. This PR checks that the start time is greater
than or equal to 0 and throws an exception otherwise.
* Adjust backported test for 7.17
This PR adds a validation step to the end of an enrich policy run to ensure the integrity of the
enrich index that is about to be promoted.
(cherry picked from commit 225db3190a)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
There should be NullPointerException check and throw index not found exception to the response
so the user can understand what happens with the enrich index
---------
Co-authored-by: James Baiera <james.baiera@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit ccc896d128)
# Conflicts:
# x-pack/plugin/enrich/src/main/java/org/elasticsearch/xpack/enrich/EnrichCache.java
# x-pack/plugin/enrich/src/test/java/org/elasticsearch/xpack/enrich/EnrichCacheTests.java
Co-authored-by: puppylpg <shininglhb@163.com>
* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
This commit adjusts the behavior of the node shutdown status
calculation to avoid reporting a `STALLED` status when ILM is
in the process of doing something that might result in the
migration being temporarily stalled (at time of writing, that
means shrinking an index).
This check is only made if ILM is not stopped.
Backport of #98367
RestHandler has a number of methods that affect the behaviour of request
processing. If the handler is wrapped (e.g. SecurityRestFilter or
DeprecationRestHandler) then these methods must be delegated to the
underlying handler.
This commit introduces a new abstract base class `FilterRestHandler`
that correctly delegates these methods so that wrappers (subclasses) do
not need to implement the behaviour on a case-by-case basis
Backport of: #98861
Prior to this change NodeReplacementAllocationDecider was unconditionally skipping both replacement source and target nodes when calculation auto-expand replicas. This is fixed by autoexpanding to the replacement node if source node already had shards of the index
Backport of PR #96281 amended for 7.17.x
Closes#89527
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
This is a backport of multiple work items related to authentication enhancements for HTTP,
which were originally merged in the 8.8 - 8.9 releases.
Hence, the HTTP (only the netty4-based implementation (default), not the NIO one) authentication
implementation gets a throughput boost (especially for requests failing authn).
Relates to: ES-6188 #92220#95112
* Update certs for PKI tests and re-enable tests (#97766)
The certs for the PKI tests expired and the test was muted.
This commit follows the instructions in the read to update the certs and unmutes the test.
The certs will now expire 20 years from now.
fixes: #97756
* precommit
* Migrate to data tiers routing configures correct default for mounted indices
(cherry picked from commit eaec9602a9c4a4b092f36fd35f58d1e4b8faca52)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
* Add isExplicitDataTier method back as it's used in 7.17
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Port lucene fix github-12352 to Elasticsearch 7.17
* Update docs/changelog/96721.yaml
* Support for Byte and Short as vector tiles value tag (#97619)
Byte and short data types are not supported by the vector tiles specification. If a user tries to add one of those, it
actually gets ignored. This add s support to those values by casting them as integers. In addition it will throw an
exception if the value added is a type it cannot understand so it is not silently ignored.
# Conflicts:
# x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/vector-tile/10_basic.yml
