We only rely on the checkstyle version in the buildLibs.toml gradle version catalogue with this change.
Also added some hints for gradle best practices.
This is an aftermath of #88283
This changes the LoggedExec task to be configuration cache compatible. We changed the implementation
to use `ExecOperations` instead of extending `Exec` task. As double checked with the Gradle team this task
is not planned to be made configuration cache compatible out of the box anytime soon.
This is part of the effort on https://github.com/elastic/elasticsearch/issues/57918
This changes AbstractGradleFuncTest to run with configuration cache enabled by default. That gives us better test coverage on how our build logic works with configuration cache enabled and makes it explicit which parts are not yet working.
We fixed some minor configuration cache related issues in our code.
This adds the generation and upload logic of Gradle dependency graphs to snyk
We directly implemented a rest api based snyk plugin as:
the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is
a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.
Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.
As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.
Addresses #87620
- Make our build tool tests more readable
- One step closer to remove outdated test fixtures like GradleUnitTestCase and friends
Didn't port some basic setter getter tests like PluginExtensionTests as it does not add much value and we have integration test coverage for this stuff too
This is intended to help us getting closer to #57918 by implicitly
testing our build logic configuration-cache support. Plugin and Task
tests can be marked as configuration cache compatible now and we will
always run then with configuration cache enabled.
By default, gradle will fail the build if configuration cache problems
have been detected during build execution. That should be in general
better then adding explicit tests for testing configuration cache
compatibility per Test class
Bootstrap plugins were an internal mechanism added to allow a
filesystemprovider for cloud with the quota-aware-fs plugin. Since that
was removed, bootstrap plugins no longer serve a purpose. They were
never officially documented because they were for internal use only.
This commit removes the bootstrap plugins infrastructure.
This PR reworks the testing conventions precommit plugin. This plugin now:
- is compatible with yaml, java rest tests and internalClusterTest (aka different sourceSets per test type)
- enforces test base class and simple naming conventions (as it did before)
- adds one check task per test sourceSet
- uses the worker api to improve task execution parallelism and encapsulation
- is gradle configuration cache compatible
This also ports the TestingConventions integration testing to Spock and removes the build-tools-internal/test kit folder that is not required anymore. We also add some common logic for testing java related gradle plugins.
We will apply further cleanup on other tests within our test suite in a dedicated follow up cleanup
Today when using Gradle-managed test clusters we separate the log output
from each node with a separator that looks like this:
=== Log output of node `node{:qa:mixed-cluster:v8.2.0-0}` ===
However, we write this to `stderr` and yet write the rest of the log
output to `stdout`. This commit moves the separators to the same log
level as the rest of this output so that they can all be redirected to
the same place.
This PR upstreams a custom Gradle Plugin for compiling modular source
projects, that we've been using for a while now over on the modules
branch. The Plugin infers which dependencies should be on the module
path and which dependencies should be on the class path, when compilng
the main java sourceset. This supports the "bottom up" modularization of
Elasticsearch, where parent projects are modularized before child
projects. Allowing to modularize the core of the server independently of
the extension components, that can themselves be modularized later.
With this plugin various subprojects can be modularized (by adding a
module-info.java in the source root) with minimal fuss - no custom
gradle logic is required by the subproject, beyond a couple of
subproject specific compiler lint suppressions.
The inference in the plugin is driven by walking the project
dependencies, searching for the presence of a module-info.java,
afterwhich all subprojects will be put on the module path. Again, this
supports "bottom up" modularization.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
Co-authored-by: Mark Vieira <portugee@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Co-authored-by: Rene Groeschke <rene@breskeby.com>
We introduce the use of a Gradle version catalogue for handling build related dependencies.
This provides type safe accessors for dependencies and allow centralised version definitions.
Later we want to move all our dependency handling to version catalogues.
Since this is a Gradle feature we remove long term maintenance cost for custom version handling in
our build and make centralised version handling more straight forward and support better tooling
based on version catalogues
Fixes asm version alignment on the way using 9.3 everywhere in our build logic.
Also adding test coverage and fixing certain issues we stumbled into
when resolving project dependencies of compileClasspath we need to use allDependencies instead of dependencies as usually no dependencies are added directly to compileClasspath but via implementation, compileonly and api and friends
Fixed javadoc setup for projects using shadowed dependencies via shadow plugin
Fixed dealing with skipped javadoc tasks in referenced dependent projects.
In general this PR only fixes expected javadoc generation as it was intended before this PR.
This also contains some tweaks to our gradle integration test fixtures for setting up and debugging test projects
In preparation for modularizing Elasticsearch server, each plugin needs
a way to declare itself as modular. When modularized, the main class of
a plugin must be found in a named module. This commit adds a module name
property to the plugin properties template and gradle support for
setting it, as well as automatically inferring the module name when a
module-info.java exists in the source a plugin.
The main elasticsearch jar used to have optional deps. This was because
of the high level rest client, where users did not want to pull in these
additional heavyweight deps that are not actually exposed by the Java apis.
These optional deps no longer exist since we no longer ship the HLRC.
When building plugins, these deps were "upgraded" to normal compile
classpath deps, since they were in fact available at runtime for
plugins. Since they are no longer optional, they are included
transitively by the dependency on the elasticsearch jar. This commit
removes the addition of these deps in the plugin build.
Ensure projects with only yaml, java or cluster tests also apply precommit checks.
We only apply testingconventions now for projects with existing src test folder
as the TestingConventionsTask is incompatible with projects with no test sourceSet.
This is a prerequisite to port more projects away from using StandaloneRestTestPlugin
and RestTestPlugin in favor of yaml, java or cluster tests with dedicated sourceSets.
Also we fix deprecation warnings for forbiddenPattern and filePermissions tasks about
implicit non declared dependencies on resources tasks
Plugins and modules use PluginBuildPlugin in gradle to build their
plugin properties file. Internally that uses an afterEvaluate since the
plugin properties extension will not have been filled in yet when the
plugin is first applied. This commit removes the afterEvaluate in favor
of configuring the plugin properties map for filling in the template
within the pluginProperties task configuration, which is applied when
the task is actually realized in the task graph.
There are many places in Elasticsearch which must decode some stream of
bytes into characters. Most of the time this is expected to be UTF-8
encoded data, and we hardcode that charset name. However, methods in the
JDK that take a String charset name require catching
UnsupportedEncodingException. Yet most of these APIs also has a variant
of the same methods which take a known Charset instance, for which we
can use StandardCharsets.UTF_8. This commit converts most instances of
passing string charset names to use a Charset instance.
In #85349 we restored capturing the stdout and stderr of Elasticsearch
processes started by testclusters. However, since testclusters launches
Elasticsearch in the forground for each process, the log output is
duplicated to stdout. This commit changes the redirects to use a
unique file, separate from ES's log file.
closes#85553
Although most of the time the stdout and stderr of the Java process is
not needed since ES has its own log file, there are some cases where it
is very important to have the output before ES has initialized its
logging system. This commit changes the stdout/stderr of the ES process
created by testclusters to use the same log file that ES will write to.
There should not be contention with ProcessBuilder and ES trying to
write at the same time because ES swaps out the stream handles to write
to its logging system, so after that point ProcessBuilder will never see
anymore output.
relates #68333
This removes the overhead of zipping modules that are referenced in
test cluster configurations
We now Support declaring task providers as module input in testclusters
This removes the overhead of zipping up modules that are immediately unzipped again when packaging
the elasticsearch distribution.
We also move some logic for packaging the elasticsearch distribution into a plugin and remove some outdated
overhead dealing with 'meta plugins' when copying modules into a distribution.
Another follow up and related optimization out of scope of this PR is, to also not zip unzip modules declared
for usage in our test cluster setups.
This partially addresses #76726.
The ESClientYamlSuiteTestCase is used to run yaml tests throughout
Elasticsearch. It utilizes the low level rest client in sniffing for
nodes, but the sniffer is not needed anywhere else in the test
framework.
This commit creates a new project, `:test:rest-runner` which is meant to
house the rest test running infrastructure. This has two purposes. First
is to remove the sniffer from the test framework dependencies, because
it transitively depends on Jackson. Second is to setup the runner for
future refactorings where it could be made to not depend on the entire
test framework, though how that could work is left for the future.
This change isolates the Jackson implementation of x-content parsers and generators to a separate classloader. The code is loaded dynamically upon accessing any x-content functionality.
The x-content implementation is embedded inside the x-content jar, as a hidden set of resource files. These are loaded through a special classloader created to initialize the XContentProvider through service loader. One caveat to this approach is that IDEs will no longer trigger building the x-content implementation when it changes. However, running any test from the command line, or running a full Build in IntelliJ will trigger the directory to be built.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
- Rework task implementations to avoid project usage at execution time
- In general usages of getProject() within a task should be avoided as it is not compatible
with gradle configuration cache. Related to #57918
This commit changes the superuser role (as used by the "elastic"
builtin user) so that it no longer has any sort of write access to
restricted indices (system indices).
This improves the safety and security of the cluster, as it means
that there are no out-of-the-box users or roles that can write to,
delete or close the security index.
Superusers can still read from (and monitor) system indices.
Other roles (and users) can still access system indices as specified
in their descriptor. These can be custom such as the
"_es_test_root" role used in the integration test suite, or builtin
roles such as kibana_system.
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.
Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
