Fix incompatibility with 8.8 and our internal api usages
- Update ospackage to a version that contains a fix we provided
- Tweak build logic to avoid deprecation warnings
- Use newer permission api
- Use custom shadowplugin
- Rework ElasticsearchDistribution dependencies resolution
- Update Gradle wrapper to 8.8
While `LicenseHeadersTask` only uses `DocumentBuilderFactory` internally
to parse `rat.xml` files (which are the output of running Apache RAT on
Elasticsearch codebase files), it is a good practice to disable XXE
features even if it's part of checks that are run on developers
machines.
Fixing a couple of file leaks (and cleaning up one missing
try-with-resources). The directory descriptor leaks in particular
were leaking massively on every precommit run, to the point where it
slows down the whole system and/or we're running into descriptor limits.
* Use String.replace() instead of replaceAll() for non-regexp replacements
When arguments do not make use of regexp features replace() is a more efficient option, especially the char-variant.
Lots of spots where we did weird things around streams like redundant stream creation, redundant collecting
before adding all the collected elements to another collection or so, redundant streams for joining strings
and using less efficient `Collectors.toList` and in a few cases also incorrectly relying on the result being mutable.
Using gradle toolchain support in gradle requires refactoring how the composite build is composed.
We added three toolchain resolver
1. Resolver for resolving defined bundled version from oracle as openjdk
2. Resolve all available jdks from Adoption
3. Resolve archived Oracle jdk distributions.
We should be able to remove the JdkDownloadPlugin altogether without having that in place, but we'll do that in a separate effort.
Fixes#95094
* Fix usage of `IndexAccessControl` in `CustomAuthorizationEngine`
* Make version properties loader more robust
Co-authored-by: Mark Vieira <portugee@gmail.com>
The current logic in `VersionPropertiesPlugin` for determining the
location of the "elasticsearch" project workspace doesn't account well
for scenarios where the elasticsearch project itself is an included
build in a larger composite. This change accounts for this by
traversing the build hierarchy.
We use `between(x, y)` calls with `switch` statements in tests to
randomize test behaviour. However, some usages define `case` statements
that can never execute, because the `case` value is outside the range
defined by the `between` call.
Write a rule that inspects the switches and the cases, and fails on the
broken cases. This rule checks `between`, `randomIntBetween` and
`randomInt`.
This adds the generation and upload logic of Gradle dependency graphs to snyk
We directly implemented a rest api based snyk plugin as:
the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is
a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.
Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.
As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.
Addresses #87620
We introduce the use of a Gradle version catalogue for handling build related dependencies.
This provides type safe accessors for dependencies and allow centralised version definitions.
Later we want to move all our dependency handling to version catalogues.
Since this is a Gradle feature we remove long term maintenance cost for custom version handling in
our build and make centralised version handling more straight forward and support better tooling
based on version catalogues
Fixes asm version alignment on the way using 9.3 everywhere in our build logic.
Also adding test coverage and fixing certain issues we stumbled into
when resolving project dependencies of compileClasspath we need to use allDependencies instead of dependencies as usually no dependencies are added directly to compileClasspath but via implementation, compileonly and api and friends
Fixed javadoc setup for projects using shadowed dependencies via shadow plugin
Fixed dealing with skipped javadoc tasks in referenced dependent projects.
In general this PR only fixes expected javadoc generation as it was intended before this PR.
This also contains some tweaks to our gradle integration test fixtures for setting up and debugging test projects
Some architectures provide non-standard output in /proc/cpuinfo.
Specifically, AWS graviton. In case we can't properly calculate physical
cpu cores via the standard method just fallback to using the JDK-backed
implementation.
Use targetExclude for excluding file from spotless checks as formatter
warnings are not reported as Error in spotless and still can cause the
build to fail
- Rework task implementations to avoid project usage at execution time
- In general usages of getProject() within a task should be avoided as it is not compatible
with gradle configuration cache. Related to #57918
* Make ForbiddenApisPrecommitPlugin plugin Gradle 8.0 compatible
* Fix deprecations on ignoring empty folders for task inputs
* Update Gradle wrapper to 7.4 GA
Checkstyle 9.3 has been released, and contains a fix for the memory
problems we saw in previous versions. Upgrade to this version.
Note that the IDEA config has also been updated, but the plugin
doesn't yet have 9.3 as an option according to the release notes.
This shouldn't be a problem.
The contribution guide's Javadoc section suggests writing Javadoc on
test methods. However the extra Checktyle rules for IDEs flag up missing
test method Javadoc at "warning", which is too severe. Since the
Checktyle rule is not very configurable, fork it and add more
configuration parameters, then add more targetted rules so that test
method Javadoc violations are only reported at "info" level.
- Updates checkstyle to 8.45.1
- This was update was triggered by us running into checkstyle/checkstyle#9897 We do not update to latest 9.x release yet as we see a performance decrease when running precommit with this. likely related to checkstyle/checkstyle#10934
Thanks to https://bugs.eclipse.org/bugs/show_bug.cgi?id=574437,
we've run into a situation where Spotless is incorrectly formatting
a particular piece of syntax (due the underlying Eclipse bug). We
were able to turn off formatting of this syntax using `// @formatter:off`
and `// @formatter:on`, but there was a further problem. We configure
IntelliJ to use the Eclipse formatter plugin, but this doesn't
respect the `@formatter` tags since these are set at the Spotless
level, not the Eclipse formatter level. Note that these tags aren't
set in the Eclipse formatter config, because there we use `// tag::`
and `// end::` in order to avoid reformatting docs snippets, which
have a much narrower line width.
What a mess.
So, to get around all this, drop the `@formatter` tags and tweak
our custom `SnippetLengthCheck` Checkstyle rule so that
`// tag:noformat` regions are not subject to the narrower line length
check, but are still exempt from formatting.
The new rule Checks for calls to `String#formatted(Object...)` that
include format specifiers that are not locale-safe. This method always
uses the default `Locale`, and so for our purposes it is safer to use
`String#format(Locale, String, Object...)`.
Note that this rule can currently only detect violations when calling
`formatted()` on a string literal or text block. In theory, it could be
extended to detect violations in local variables or statics.
Note that this change also forbids `.formatted()` in server code, so
we are only permitted to use `.formatted()` in test code.
Change our fork of `HiddenFieldCheck` rule to allow it to ignore
shadowed variables for methods shorter than a configured minimim line
count. While this technically defeats the point of the rule, in practice
short methods are easier to verify by eye, compared to longer methods
where accidentally referencing the wrong variable is harder to spot.
Following this, revert some changes in the EQL code which were required
to pass the `HiddenField` check before the `minLineCount` change.
Additionally, introduce a mechanism for ignoring shadowed variables when
used created a new object. This allowed us to ignore variables in usages
of e.g. `ConstructingObjectParser` whose purpose, as the name suggests,
is to build objects, and object shadows variables in the course of
building objects.
Part of #19752. Fix more instances where local variable names were
shadowing field names. Also: * Remove unused bits and pieces from
`XPackClientPlugin`. * Expand the possible method names that are
skipped when checking for shadowed vars, and allow shadowed vars
in builder classes.
Part of #19752. Fix more instances where local variable names were
shadowing field names.
Also modify our fork of HiddenFieldCheck to add the ignoreConstructorBody
and ignoredMethodNames parameters, so that the check can ignore
more matches.
This is an attempt at resolving, or at least reducing the prevalence of
#77837. A more robust solution is probably required, but this should
help reduce this problem since we now apply spotless to about 90 fewer
subprojects.
Part of #19752.
Fix a number of cases of shadows vars under `client/rest-high-level`. As
part of this, fork the Checkstyle `HiddeFieldCheck` class so that it
understand the pattern of settings with no "set" prefix.
