This is a backport of multiple work items related to authentication enhancements for HTTP,
which were originally merged in the 8.8 - 8.9 releases.
Hence, the HTTP (only the netty4-based implementation (default), not the NIO one) authentication
implementation gets a throughput boost (especially for requests failing authn).
Relates to: ES-6188 #92220#95112
* Migrate core rest tests with security to new testing framework (#92575)
# Conflicts:
# x-pack/qa/core-rest-tests-with-security/build.gradle
* Fixes
* More fixes
* More fixes
* More more fixes
* Add support for addition configuration files to test clusters framework (#92579)
This adds the ability to supply arbitrary files to the config directory
of cluster nodes. Typically, this is used for security use cases, such
as providing for SSL certificates and trust stores.
This commit adds a few other features to enable more testing ues cases
as well, such as the ability to restart a cluster, as well as explicit
ordering of test cases withing a test class. This is needed for test
suites that need to execute some tests, restart the cluster, then
execute more in a particular order.
# Conflicts:
# test/test-clusters/src/main/java/org/elasticsearch/test/cluster/local/LocalClusterHandle.java
# x-pack/plugin/security/qa/basic-enable-security/build.gradle
# x-pack/plugin/security/qa/basic-enable-security/src/javaRestTest/java/org/elasticsearch/xpack/security/EnableSecurityOnBasicLicenseIT.java
# x-pack/qa/multi-node/src/javaRestTest/java/org/elasticsearch/multi_node/GlobalCheckpointSyncActionIT.java
* Fix static initialization of random value
* Remove unused imports
* Spotless
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
This commit extends the TLS restricted trust model to allow reading from
alternative fields from the X509 certificate. Prior to this commit the only
supported (hard coded) value that could be used with restricted trust
is the SAN/otherName/CN value. This commit introduces support to read
from other fields from the X509 certificate. This commit also introduces
support to read from SAN/dnsName if configured. Any fields read from the
certificate will be used to match against the restricted trust file and if any
of the values match to the restricted trust file, then restricted trust is allowed.
Only if none of the values match then the restricted trust denied.
SAN/otherName/CN is the default, and SAN/dnsName can be used in addition
or in place of SAN/otherName/CN. The possible configuration values are:
```
*.trust_restrictions.x509_fields: ["subjectAltName.otherName.commonName", "subjectAltName.dnsName"]
```
To help support testing, all of the existing certificates have been updated
to include a SAN/dnsName that matches the SAN/otherName/CN. This
allows the tests to randomize which field(s) are used to match for restricted trust.
This also has the side effect of making this commit larger than expected in
terms of lines of change. A readme has been included with copy-able commands
to recreate the certificates as needed.
Additionally, a CCS REST test has been introduced that uses the restricted trust.
To support this new CCS REST test the private keys for the test certificates are also
included in this commit as well as the gradle configuration needed to share those
certificates across projects.
We have some test runtime resources that are generated by our Docker
based test fixtures. These resources are then placed on the runtime
test classpath to be used by the test classes themselves. The problem
here is that any build that attempts to resolve the test runtime
classpath will then trigger the starting of these Docker fixtures, which
is quite expensive. Some of our precommit checks, like forbidden apis
and third-party audit tasks do this.
This commit decouples this a bit and removes the implicit dependency
replacing it with an explicit one on the test task itself. It's slightly
less idomatic Gradle, but it's a worthwhile optimization.
This commit re-implements how the SAML tests configure their ephemeral ports.
The prior implementation had 2 tasks competing over files in the output directory
resulting in some odd behavior w.r.t. when the task was up to date or was not.
The change here removes the unnecessary task and leverages a lazy map with
the expand option to lazily populate the port mapping.
# Conflicts:
# x-pack/qa/saml-idp-tests/build.gradle
We currently use unicon/shibboleth-idp:3.4.2 to help test our SAML integration.
That container is no longer actively supported and does not support
ARM architectures.
This commit is a partial clone from Unicon/shibboleth-idp-dockerized 3.4.3.
Changes from upstream include:
Use openjdk:11.0.16-jre as the base image for support for ARM architectures
Handle missing keystore download from Jetty
Fix URL paths for artifacts to download
Changes to this repository include:
Copied required Jetty configuration files from upstream project
Updates to docker compose
Placed the missing keystore Jetty downloads in a separate location (jetty-custom)
The final result is a bit messy. Mixing cloned files with custom files and mixing
Jetty and IDP concerns. However, it is not much messier than prior and now
that we control building the image we can more easily upgrade shibboleth IDP
The upgrade to the latest version is fairly involved and as such we will need to
deviate more from the clone which should allow some additional clean up.
part of: #71378
related: #91144
supersedes: #89674
This commit updates the c2id docker image to the latest released version.
This commit also introduces a multi-stage build with the openjdk image that
supports ARM architectures. So our oidc tests now supports multiple archtectures.
related: #89526
part of: #71378
Implemented a fall-back to `localhost` when FQDN for
loopback address (`127.0.0.1`) cannot be resolved.
This can happen if test platform's DNS resolution
is not properly configured.
Closes#89324
This commit updates the OIDC test fixture to use the latest demo docker image
from https://connect2id.com. As part of this change some of the docker image demo
login has changed from prior versions and the test code is updated accordingly.
fixes#89477
reverts #89478 / e949dff.
* Skip ML tests on later glibc for incompatible BWC versions
* More flexible version parsing
* Use relaxed mode for parsing glibc version
* Missed a spot
* spotless
This deprecates the elasticsearch.rest-test plugin and elasticsearch.standalone-rest-test and ports
all usages of them in x-pack/plugins. Other usages will be removed in a few upcoming PRs to not have one >300file PR
When all usages have been addressed we're going to remove those gradle plugins from the codebase.
This PR is a subset of #85491 which got just too big to handle IMO
* Validate index format agreement for system index descriptors (#85173)
* Validate index format for system indices
* Add bwc test for watcher meta version issue
* Update docs/changelog/85173.yaml
* Add include_type_name warning handler for bwc tests
If rolling upgrade was used from version prior GeoIPv2 (<`7.14`) then
geoip downloader wouldn't be started so no new databases were
downloaded. This is especially troubling in `8.x` as we no longer
provide default databases inside ES so after upgrade no geoip enrichment
can take place until downloader is started with workaround (setting
`ingest.geoip.downloader.enabled` to `false` and `true` again). This is
because logic that was used to lower number of requests / cluster update
listeners at the startup was too optimistic about order of actions / who
can be elected master at what time. This change fixes that and also
cleans up logs when there are some ignorable errors and adds debug
logging on start and stop of the task to ease up troubleshooting. It
also adds rolling upgrade test to make sure the fix works.
Follow-up from #77144 (comment) with converting id/_id to always be strings instead of integers. This makes the type value in the Elasticsearch specification be only string instead of string | number.
this change was generated using following command on ubuntu
find . -type f -name "*.yml" -print0 | xargs -0 sed -i -r 's/([^a-zA-Z0-9_\.]id|[^a-zA-Z0-9_]_id):(\s)([0-9]+)/\1:\2"\3"/g'
relates #82681
Some previous 7.x release validate that these legacy templates exist in
the cluster (the validation executes on non-master nodes too). This can
cause the rolling upgrade test (particulary the ones that do not upgrade
the entire cluster ie. mixed cluster scenarios) to hang waiting for
watcher to be started, however it will never start because of the
legacy templates validation failing.
This is not a problem per se for "real life" rolling upgrades as this does not
block the rolling upgrades, nor does it raise any errors, it will
however stop watcher on the nodes that detect it (until the said nodes are
upgraded, at which point watcher is started).
* [7.16] [ML] Model snapshot upgrade needs a stats endpoint
Previously the ML model snapshot upgrade endpoint did not
provide a way to reliably monitor progress. This could lead
to the upgrade assistant UI thinking that a model snapshot
upgrade had finished when it actually hadn't.
This change adds a new "stats" API that allows external
interested parties to find out the status of each model
snapshot upgrade and which node (if any) each is running on.
Backport of #81641
* Fixing compilation
This removes the unnecessary watcher start call in the test to reduce
the possibility of flakiness. The call is not needed as the Watcher
plugin is active and it will install (and upgrade/remove) the templates
regardless if watcher is started.
This removes a few legacy index templates that were superseeded by
equivalent component templates or updated index templates.
(cherry picked from commit dace65b69a)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
# Conflicts:
# x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/WatcherRestartIT.java
We no longer need any ML legacy index templates, as we've switched to
either composable index templates or system indices. All the ML legacy
templates we've created over the years should be deleted, as they're
just confusing clutter within the cluster now.
* [ML] add trained model inference bwc tests (#80618)
This adds rolling upgrade tests that:
Build a classification and regression model through data frame analytics
Make sure pipelines + stats work in the old cluster
Verify through the rolling upgrade process that pipelines + stats still work
* fixing compilation
* [Transform] Fix issue if upgrade runs right after a rolling cluster upgrade (#80579)
do not fail a running transform if upgrader rewrites state inbetween
fixes#80073
* reactivate test
We used to default enctypes to des3-cbc-sha1-kd but with JDK17, weak
encryption types are disabled by default. This caused our Kerberos
integration tests to fail with an `sun.security.krb5.KrbException: no
supported default etypes for default_tkt_enctypes` exception. We have
since changed our default encryption type to aes256-cts-hmac-sha1-96 in
#78703 and we can unmute these tests now.
We're enforcing constraints on the tier setting now that were not enforced
in 7.12. Newer version masters will not work correctly with defaults send
by 7.12 so we force a tier setting for frozen tier indices in this test.
closes#79786closes#79541
* [Transform] add rolling upgrade tests for upgrade endpoint (#79721)
add rolling upgrade tests for upgrade endpoint
* always use the transform endpoint prefix
* remove mixed cluster check, because the endpoint does not exist < 7.16
Matchers is deprecated in Mockito, in favor of the newer
ArgumentMatchers class. In fact, internally Matchers just extends
ArgumentMatchers as all the methods there were moved. This commit
changes all imports of org.mockito.Matchers to
org.mockito.ArgumentMatchers.
When rewriting authentication for requests crossing nodes of different
versions, we now preserve all request headers except the authentication
one which needs to be rewritten. Previously all other request headers
were dropped and it caused issue like an operator user not being
recognised on the remote node. Other now preserved headers include audit
and system index access. This new behaviour is more correct because we
would never drop these headers if the nodes are on the same version.
Resolves: #79354
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
