Enlarge jetty restart timeframe in idpfixture in order to trying to fix flakiness we see in #103595
caused by connection issues to the idp fixtuere in the SamlAuthenticationIT tests
Add healthcheck wait for idp fixture that matches the healthcheck we had in docker compose,
being closer to what we had in docker compose to limit surface we see at #103595
Ports OIDC integration tests from the old Docker-based ES test cluster
setup to internal REST tests. The OIDC fixture itself is still
dockerized. I'm doing this in the context of bumping a version on a
dependency. The old Docker-based test does not really work on M1 Macs;
porting this suite to the new test setup (aside from being a general
improvement) also allows me to test the version update and iterate on it
locally.
This adds support for the `field` scripting API in many but not all
cases. Before this change numbers, dates, and IPs supported the `field`
API when running with _source in synthetic mode because they always have
doc values. This change adds support for `match_only_text`, `store`d
`keyword` fields, and `store`d `text` fields. Two remaining field
configurations work with synthetic _source and do not work with `field`:
* A `text` field with a sub-`keyword` field that has `doc_values` * A
`text` field with a sub-`keyword` field that is `store`d
This commit re-implements how the SAML tests configure their ephemeral ports.
The prior implementation had 2 tasks competing over files in the output directory
resulting in some odd behavior w.r.t. when the task was up to date or was not.
The change here removes the unnecessary task and leverages a lazy map with
the expand option to lazily populate the port mapping.
We currently use unicon/shibboleth-idp:3.4.2 to help test our SAML integration.
That container is no longer actively supported and does not support
ARM architectures.
This commit is a partial clone from Unicon/shibboleth-idp-dockerized 3.4.3.
Changes from upstream include:
Use openjdk:11.0.16-jre as the base image for support for ARM architectures
Handle missing keystore download from Jetty
Fix URL paths for artifacts to download
Changes to this repository include:
Copied required Jetty configuration files from upstream project
Updates to docker compose
Placed the missing keystore Jetty downloads in a separate location (jetty-custom)
The final result is a bit messy. Mixing cloned files with custom files and mixing
Jetty and IDP concerns. However, it is not much messier than prior and now
that we control building the image we can more easily upgrade shibboleth IDP
The upgrade to the latest version is fairly involved and as such we will need to
deviate more from the clone which should allow some additional clean up.
part of: #71378
related: #91144
supersedes: #89674
This commit updates the c2id docker image to the latest released version.
This commit also introduces a multi-stage build with the openjdk image that
supports ARM architectures. So our oidc tests now supports multiple archtectures.
related: #89526
part of: #71378
This commit updates the OIDC test fixture to use the latest demo docker image
from https://connect2id.com. As part of this change some of the docker image demo
login has changed from prior versions and the test code is updated accordingly.
fixes#89477
reverts #89478 / e949dff.
We have an existing QA test and fixture that integrates the OIDC realm
with the c2id OIDC server.
This commit extends that test suite and fixture to also test that the
JWT id tokens generated by c2id can be used to authenticate to the
Elasticsearch JWT realm
Co-authored-by: Justin Cranford <89857999+justincr-elastic@users.noreply.github.com>
Historically, we haven't enabled the transport TLS bootstrap
check for trial licenses because:
- We wanted to make the experience of trial license users as
easy as possible and configuring transport TLS was considered
cumbersome.
- Trial licenses have a limited lifetime so that minimizes the
impact of this potentially insecure configuration.
With security on by default project we are:
- Enabling security by default for basic and trial licenses
- We offer an easy, automated way for users to configure
transport TLS
- Enabling by default this bootstrap check for basic licenses.
It doesn't make much sense for us to enforce the bootstrap check
on basic licenses but not on trial and given that the concerns
that were driving the original decision are not there or have been
partly alleviated, this commit changes our behavior so that we
enable the TLS bootstrap check regardless of the license level.
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
This PR adds documentation for GeoIPv2 auto-update feature.
It also changes related settings names from geoip.downloader.* to ingest.geoip.downloader to have the same convention as current setting.
Relates to #68920
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This change enables GeoIP downloader by default.
It removes feature flag but adds flag that is used by tests to disable it again (as we don't want to hammer GeoIP database service with every test cluster we spin up).
Relates to #68920
Renew our smb-fixture expired test certificates that are in use in the
fixture and in various tests for LDAP and AD.
Add a README file for how these certificates can be renewed in the
future, when the need arises again.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
This finishes porting all tasks created in gradle build scripts and plugins to use
the task avoidance api (see #56610)
* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in :modules:reindex
* Declare proper task inputs
The OpenID Connect specification defines a number of ways for a
client (RP) to authenticate itself to the OP when accessing the
Token Endpoint. We currently only support `client_secret_basic`.
This change introduces support for 2 additional authentication
methods, namely `client_secret_post` (where the client credentials
are passed in the body of the POST request to the OP) and
`client_secret_jwt` where the client constructs a JWT and signs
it using the the client secret as a key.
Support for the above, and especially `client_secret_jwt` in our
integration tests meant that the OP we use ( Connect2id server )
should be able to validate the JWT that we send it from the RP.
Since we run the OP in docker and it listens on an ephemeral port
we would have no way of knowing the port so that we can configure
the ES running via the testcluster to know the "correct" Token
Endpoint, and even if we did, this would not be the Token Endpoint
URL that the OP would think it listens on. To alleviate this, we
run an ES single node cluster in docker, alongside the OP so that
we can configured it with the correct hostname and port within
the docker network.
We have some Dockerfiles that reference Ubuntu 19.04, which is not an LTS
version and has now appears to have been retired from the Ubuntu repositories.
Switch to 18.04, which is the current long-term support version. Also change a
usage of 16.04 to 18.04, for consistency.
* Document SAML APIs
This change adds documentation for the SAML APIs in Elasticsearch
and adds simple instructions on how these APIs can be used to
authenticate a user with SAML by a custom web application other
than Kibana.
Resolves: #40352
* typo
* fix links
* fix more links
* [DOCS] Fixes broken link
* Add metadata file with shorter names for docs, fix typos and mute tests
* [DOCS] Reformats the SAML APIs to match API template
* Apply suggestions from code review
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
* Address feedback and add small section on IdP-initiated SSO handling
* address feedback
* moar feedback
* Clarifications and addressing feedback
* properly resolve conflicts
* address feedback
* fix doc links
* minor fixes
* Fix reference
This commit removes the infrastructure for client feature tracking. We
introduced this functionality to support clients that do not necessarily
understand all the features that the server might support, for example,
customs in the cluster state provided by plugins that a client might not
have. This can arise in situations such as rolling upgrades from the OSS
distribution to the default distribution. With the removal of the
transport client, this infrastructure is no longer needed. This commit
removes client feature tracking from the server in 8.0.0.
This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.
This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions
This commit adds an OpenID Connect authentication realm to
elasticsearch. Elasticsearch (with the assistance of kibana or
another web component) acts as an OpenID Connect Relying
Party and supports the Authorization Code Grant and Implicit
flows as described in http://ela.st/oidc-spec. It adds support
for consuming and verifying signed ID Tokens, both RP
initiated and 3rd party initiated Single Sign on and RP
initiated signle logout.
It also adds an OpenID Connect Provider in the idp-fixture to
be used for the associated integration tests.
The code in this commit has been tracked in a feature branch
and has been previously reviewed and approved in :
#37009#37787#38474#38475#40262
This change removes the use of hardcoded port values for the
idp-fixture in favor of the mapped ephemeral ports. This should prevent
failures due to port conflicts in CI.
This change adjusts the LDAP connection timeout for retrieving
attributes while performing the SAML IT to 5 seconds, from 5 ms
that it previously was.
Resolves: #40025
The change replaces the Vagrant box based fixture with a fixture
based on docker compose and 2 docker images, one for an openldap
server and one for a Shibboleth SAML Identity Provider.
The configuration of both openldap and shibboleth is identical to
the previous one, in order to minimize required changes in the
tests
This change adds a docker compose configuration that's used with
the `elasticsearch.test.fixtures` plugin to start up the image
and check that the TCP ports are up.
We can build on this to add other checks for culster health,
run REST tests, etc.
We can add multiple containers and configurations to the compose
file (e.x. test different env vars) and form clusters.
