* [Test] Use larger client ports range for tests running on Windows (#103894)
This PR increases client's port ranges for tests which are executed
on Windows in order to avoid failures due to some port ranges being
excluded from use. The larger ports range (300) is chosen based on
the observation where a random consecutive range of 200 ports can
be excluded on Windows test workers.
Closes#102349
(cherry picked from commit bdf5c7fc38)
# Conflicts:
# modules/transport-netty4/src/internalClusterTest/java/org/elasticsearch/transport/netty4/Netty4TransportMultiPortIntegrationIT.java
# x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/transport/filter/IpFilteringIntegrationTests.java
* Fix compilation error
In https://github.com/elastic/elasticsearch/pull/100872 Painless errors
were wrapped so as to avoid throwing Errors outside scripting. However,
one case was missed: PainlessExplainError which is used by
Debug.explain. This commit adds the explain error to those that painless
wraps.
closes#103018
* Improve painless error wrapping (#100872)
Painless sandboxes some errors from Java for which it can recover. These
errors are wrapped within a ScriptException. However, retaining the
error as a cause can be confusing when walking the error chain. This
commit wraps the error so that the real error type does not appear,
but maintains the same error message in xcontent serialized form.
* fix compile
* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
In Java 21 List now extends SequencedCollection, instead of Collection
directly. When resolving methods Painless starts at the defined type,
and iterates up through super classes and interfaces. Unfortunately if a
superinterface was not known, as it is for SequencedCollection since it
is not in the allowed list of classes, method resolution would give up.
This commit adjusts the superinterface interation to continue traversing
until the method is found or no more superinterfaces are found.
fixes#97022
(cherry picked from commit 16b45575c3)
# Conflicts:
# gradle.properties
Co-authored-by: Ryan Ernst <ryan@iernst.net>
This is a backport of multiple work items related to authentication enhancements for HTTP,
which were originally merged in the 8.8 - 8.9 releases.
Hence, the HTTP (only the netty4-based implementation (default), not the NIO one) authentication
implementation gets a throughput boost (especially for requests failing authn).
Relates to: ES-6188 #92220#95112
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
Backports the following commits to 7.17:
- Add verification metadata for dependencies (#88814)
- Add missing dependency verification checksums (#89139)
Backporting #86842 to 7.17 branch.
For example in the case that the a cluster is running out of disk
space and indices reject writes.
This can otherwise load to unnecessary error logs being printed and
just add to more instability. Instead, the GeoIpDownloader should
just try to download the files the next it runs.
Ensure projects with only yaml, java or cluster tests also apply precommit checks.
We only apply testingconventions now for projects with existing src test folder
as the TestingConventionsTask is incompatible with projects with no test sourceSet.
This is a prerequisite to port more projects away from using StandaloneRestTestPlugin
and RestTestPlugin in favor of yaml, java or cluster tests with dedicated sourceSets.
Also we fix deprecation warnings for forbiddenPattern and filePermissions tasks about
implicit non declared dependencies on resources tasks.
We need to stash thread context in DefaultRestChannel before we call
channel.sendResponse because when calling this method, our execution
might be delayed and the thread be reused for another task - like
sending another response.
And it would see thread context from the initial “delayed” work.
This commit also expands the assertions on the empty thread context
to make sure it does not contains response headers.
closes#68278
Currently, CompoundProcessor does not catch Exception and
if a processor throws an error and a method higher in the
call stack doesn't catch the exception then pipeline execution
stalls and bulk requests may not complete.
Usually these exceptions are caught by IngestService#executePipelines(...) method,
but when a processor executes async (for example: enrich processor) and the thread
that executes enrich is no longer the original write thread then there is no logic
that deals with failing pipeline execution and cleaning resources up. This then leads
to memory leaks.
Closes#84781
Also change how 'pipeline doesn't exist' error is thrown in TrackingResultProcessor.
With the change to CompoundProcessor thrown exceptions are caught and delegated to handler.
SimulateExecutionService in verbose mode ignores exceptions delegated to its handler, since
it assumes that processorResultList contains the result (successful or not successful) of every
processor in the pipeline.
In case TrackingResultProcessor for PipelineProcessor couldn't find the mentioned pipeline then
it just throws an error without updating the processorResultList. This commit addresses that.
The only place we download the database (in the traditional pull from
the configured endpoint sense) is in the `GeoIpDownloader`. The
`DatabaseNodeService` is looking up the chunks of data we save in the
`.geoip_databases` index.
This updates the logging to be clear about who does what.
In addition to the backport, this commit also adds SystemIndexMigrationTaskParams/State to writeableRegistry of transport client
This will backport the following commits from master to 7.17:
Register Feature migration persistent task state named XContent (#84192)
* Always re-run Feature migrations which have encountered errors (#83918)
This PR addressed the behavior described in #83917, in which Feature migrations
which have encountered errors are not re-run in some cases. As of this PR, Features
which have encountered errors during migration are treated the same as Features
requiring migration.
This PR also adds a test which artificially replicates #83917.
* Adjust for method signature differences in this branch
Prior to this PR, the system index migration facility did not use the
index type configured in the descriptor, instead always using `_doc`.
This PR adjusts the migration facility to use the index type specified
in the descriptor instead.
Note that this PR is only targeted at 7.17, as it is the only version impacted.
* GeometryNormalizer should not fail if it cannot compute signed area (#84051)
This commit removes the exception throwing and assumes that when the area is zero, the polygon has
the right orientation. The exception will be thrown at indexing time when the polygon is invalid .
# Conflicts:
# server/src/test/java/org/elasticsearch/common/geo/GeometryIndexerTests.java
# server/src/test/java/org/elasticsearch/common/geo/GeometryNormalizerTests.java
* spotless
The multi feature migration test works by having two system index
features that are to be upgraded. The reindexing of the system indices
of those features is done in sequence by the system index migrator, and
the test has an assertBusy that waits for all the migrations to
complete. Unfortunately assertBusy backs off exponentially, so it isn't
great for quickly resuming the test once the underlying assertion
becomes true, since the condition might change shortly after an
iteration, and then have to wait for a long backoff.
This commit adds a latch which will count down all the executions of the
test plugins, so that the assert busy will not be run until after all
the indexes have migrated.
closes#83953
* Update YAML Rest tests to check for product header on all responses (#83290)
This PR fixes a number of misuses of ThreadContext, mostly because of stashing listeners
without their accompanying contexts. The YAML Rest Test changes in the original PR were
not backported since they break BWC testing going further back into 7.x.
If rolling upgrade was used from version prior GeoIPv2 (<`7.14`) then
geoip downloader wouldn't be started so no new databases were
downloaded. This is especially troubling in `8.x` as we no longer
provide default databases inside ES so after upgrade no geoip enrichment
can take place until downloader is started with workaround (setting
`ingest.geoip.downloader.enabled` to `false` and `true` again). This is
because logic that was used to lower number of requests / cluster update
listeners at the startup was too optimistic about order of actions / who
can be elected master at what time. This change fixes that and also
cleans up logs when there are some ignorable errors and adds debug
logging on start and stop of the task to ease up troubleshooting. It
also adds rolling upgrade test to make sure the fix works.
