This commit adds compatibility tests that target ES revisions that align with specific Lucene versions. In this case, we are intending to upgrade from Lucene 10.0 to 10.1. Since no on-prem Elasticsearch release exists with 10.0, we need another method to ensure compatibility with Lucene 10.0 indicies.
The work here is a bit hacky since all our compatibility testing infrastructure is centered around versions and we're now effectively doing compatibility tests between two different revisions of Elasticsearch that both report the same version. Ideally this specific testing would be replaced by unit tests, rather that reusing our full cluster restart tests for this purpose.
We'll also want to bump the commit referenced in the CI pipelines here to align with the last commit using Lucene 10.0.
The STABLE_CONFIGURATION_CACHE feature flag enables the following:
- tasks using a shared build service without declaring the requirement via the Task.usesService method will emit a deprecation warning.
- when the configuration cache is not enabled but the feature flag is present, deprecations for the following configuration cache requirements are also enabled:
- Registering build listeners
- Using task extensions and conventions at execution time
Rephrase the authorization check in `S3HttpFixture` in terms of a
predicate provided by the caller so that there's no need for a separate
subclass that handles session tokens, and so that it can support
auto-generated credentials more naturally.
Also adapts `Ec2ImdsHttpFixture` to dynamically generate credentials
this way.
Also extracts the STS fixture in `S3HttpFixtureWithSTS` into a separate
service, similarly to #117324, and adapts this new fixture to
dynamically generate credentials too.
Relates ES-9984
The S3 and IMDS services are separate things in practice, we shouldn't
be conflating them as we do today. This commit introduces a new
independent test fixture just for the IMDS endpoint and migrates the
relevant tests to use it.
Relates ES-9984
The distribution tools are meant to be CLIs. This commit moves the
entitlements jar projects to the libs dir, under a single
libs/entitlement root directory to keep the related jars together.
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
* Replace cloud-ess docker image with wolfi-ess
We just replaced the existing implementation of cloud-ess with what was wolfi-ess which is a wolfi based ess image.
The cloud image itself will be removed in a future commit it was not used anywhere
* Switch to test cloud docker image instead of default docker in packaging pr tests.
This adds way more coverage than the default docker image which is also barely touched
* Entitlements for System.exit
* Respond to Simon's comments
* Rename trampoline -> bridge
* Require exactly one bridge jar
* Use Type helpers to generate descriptor strings
* Various cleanup from PR comments
* Remove null "receiver" for static methods
* Use List<Type> instead of voidDescriptor
* Clarifying comment
* Whoops, getMethod
* SuppressForbidden System.exit
* Spotless
* Use embedded provider plugin to keep ASM off classpath
* Oops... forgot the punchline
* Move ASM license to impl
* Use ProviderLocator and simplify bridgeJar logic
* Avoid eager resolution of configurations during task configuration
* Remove compile-time dependency agent->bridge
---------
Co-authored-by: Mark Vieira <portugee@gmail.com>
* Initial hello-world entitlements agent
* Respond to Ryan's comments
* License header
* Fix forbidden APIs setup
* Rename EntitlementAgent
* Automated refactor missed one
* Automated rename really let me down here
* Very serious test name
* README files for the new modules
* Use "tasks.named('jar')"
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Use 'tasks.named('test')'
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* More deferral of gradle tasks
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Even more deferral
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* FIx gradle syntax for javaagent arg
---------
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Update Gradle gradleEnterprise plugin (develocity now)
* Fix imports in build scan script
* Fix build scan api usage
* Dependency cleanup and API fix
* Fix API update for BuildResult in Build scans
* Fix buildkite buildscan annotations based on gradle failures
ImmutableCollections uses a seed, set early during JVM startup, which
affects the iteration order of collections. Although we do not want to
rely on the iteration order of Map and Set collections, bugs do
sometimes occur. In order to reproduce those bugs to fix them, it is
important the test seed for Elasticsearch matches the seed used in
ImmutableCollections.
Unfortunately ImmutableCollections is internal to the JDK, and the seed
used is private and final. This commit works around these limitations by
creating a patched version of ImmutableCollections which allows access
to the seed member. ESTestCase is then able to reflectively set the seed
at runtime based on the Elasticsearch seed.
Note that this only affects tests. ImmutableCollections remains is
unchanged for production code.
relates #94946
Fix incompatibility with 8.8 and our internal api usages
- Update ospackage to a version that contains a fix we provided
- Tweak build logic to avoid deprecation warnings
- Use newer permission api
- Use custom shadowplugin
- Rework ElasticsearchDistribution dependencies resolution
- Update Gradle wrapper to 8.8
This ports our krb5kdc test fixture to test container and reworks hdfs handling to also be based on test containers.
The yaml rest tests that are using hdfs required introducing variable substitution in yamlresttestparser handling.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
This introduces a new getHistoricalFeatures() method on ESRestTestCase
which returns a map of historical feature version mappings loaded from
FeatureSpecification implementations from any plugins/modules in use
by the current test suite. The mappings are generated by a new Gradle
task at build time, and then injected into the test runtime as a
System property.
This commit upgrades to OpenSAML v4.3.0
Versions of OpenSAML ≥ 4.1 have a hard dependency on the non-FIPS release of BouncyCastle.
This would prevent ES from being able to run in a JVM where BC-FIPS is configured as the security provider.
Closes: #71983
Co-authored-by: Tim Vernum tim@adjective.org
Using gradle toolchain support in gradle requires refactoring how the composite build is composed.
We added three toolchain resolver
1. Resolver for resolving defined bundled version from oracle as openjdk
2. Resolve all available jdks from Adoption
3. Resolve archived Oracle jdk distributions.
We should be able to remove the JdkDownloadPlugin altogether without having that in place, but we'll do that in a separate effort.
Fixes#95094
We need to verify, for each release, that our stable plugin APIs
are not breaking.
This commit adds some Gradle support for basic backwards compatibility
testing. On the Gradle side, we add a new qa project to test the
current commit against downloads of released versions, and against
fresh builds of snapshot versions.
As for the actual comparison, we break up the output of javap (the
decompiler) by line and create maps of classes to public class,
field, and method declarations within those class files. We then
check that the signature map from the new jar is not missing any
elements present in the old jar. This method has known limitations,
which are documented in the JarApiComparisonTask class.
Co-authored-by: Mark Vieira <portugee@gmail.com>
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
This project has a problem with availability of Docker images after
release. Disabling individual tasks is tricky because it uses test
fixtures, so instead just skip the project entirely until we can work
out a way forward.
We only rely on the checkstyle version in the buildLibs.toml gradle version catalogue with this change.
Also added some hints for gradle best practices.
This is an aftermath of #88283
Elasticsearch provides several command line tools, as well as the main script to start elasticsearch. While most of the logic is abstracted away for cli tools, the main elasticsearch script has hundreds of lines of platform specific shell code. That code is difficult to maintain because it uses many special shell features which then must also exist in other platforms (ie windows batch files). Additionally, the logic in these scripts are not easy to test, we must be on the actual platform and test with a full installation of Elasticsearch, which is relatively slow (compared to most in process tests).
This commit replaces logic of the main server script, as well as the windows service management script, with Java. The new entrypoints use the CliToolLauncher. The server cli figures out all the jvm options and such necessary, then launches the real server process. If run in the foreground, the launcher will stay alive for the lifetime of Elasticsearch; the streams are effectively inherited so all output from Elasticsearch still goes to the console. If daemonizing, the launcher waits around until Elasticsearch is "ready" (this means the Node startup completed), then detaches and exits.
Co-authored-by: William Brafford <william.brafford@elastic.co>
The "launchers" tool jar contains tools used when launching
Elasticsearch, to find the necessary jvm options and temp directory. In
preparation for #85758, this commit renames the "launchers" tool to
server-cli. The classes there will become part of the Java based launch
script, and the new naming better matches the intent of the jar, which
is to serve as the cli entrypoint for the Elasticsearch server.
relates #85758
CLI scripts have a common infrastructure in that they call to the shared
elasticsearch-cli shell script which launches them with the appropriate
java command line. However, each underlying Java class must implement
its own main method.
This commit introduces a single main method to be shared by CLIs. The
new CliToolLauncher takes in system properties to determine which tool
is being run, and a new CliToolProvider SPI allows defining and finding
the named tools.
relates #85758
Co-authored-by: William Brafford <william.brafford@elastic.co>
The ESClientYamlSuiteTestCase is used to run yaml tests throughout
Elasticsearch. It utilizes the low level rest client in sniffing for
nodes, but the sniffer is not needed anywhere else in the test
framework.
This commit creates a new project, `:test:rest-runner` which is meant to
house the rest test running infrastructure. This has two purposes. First
is to remove the sniffer from the test framework dependencies, because
it transitively depends on Jackson. Second is to setup the runner for
future refactorings where it could be made to not depend on the entire
test framework, though how that could work is left for the future.
This commit adds a jar separate from the test framework to provide
utilities for testing x-content related code. The first thing moved
there is the base schema validation test case, which also pulls along
the com.networknt dependency and jackson. For now these are direct
dependencies, though we could consider shading them in the future so as
not to expose downstream projects to them, which may have version
conflicts.
This change isolates the Jackson implementation of x-content parsers and generators to a separate classloader. The code is loaded dynamically upon accessing any x-content functionality.
The x-content implementation is embedded inside the x-content jar, as a hidden set of resource files. These are loaded through a special classloader created to initialize the XContentProvider through service loader. One caveat to this approach is that IDEs will no longer trigger building the x-content implementation when it changes. However, running any test from the command line, or running a full Build in IntelliJ will trigger the directory to be built.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
Adds a new "ConsoleLoader" that uses jANSI in a separate classloader
to determine whether standard output is a real console (that is, not
redirected to a file or /dev/null, etc)
Also updates security auto-configuration to only print out credentials
when there is a console.
