This commit alters our backward compatibility testing logic to restrict
testing to the latest patch release for any given minor version. Since
our policy is not to do any kind of breaking changes in patches, most of
this testing is superfluous. This change reduces the number of versions
we need to test by over 70%.
(cherry picked from commit b1e4407c0a)
* Update gradle wrapper to 8.3 (#97838)
Gradle now fully supports compiling, testing and running on Java 20.
Among other general performance improvements this release introduces --test-dry-run command line option that allows checking if tests are filtered or not by gradle.
Required updating nebula ospackage plugin as setuid was broken in gradle 8.3.
(cherry picked from commit b23e000c30)
# Conflicts:
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ElasticsearchJavaModulePathPlugin.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/compat/compat/AbstractYamlRestCompatTestPlugin.java
# build-tools-internal/src/main/resources/minimumGradleVersion
# gradle/verification-metadata.xml
# gradle/wrapper/gradle-wrapper.jar
# gradlew
# x-pack/plugin/watcher/qa/with-monitoring/src/javaRestTest/java/org/elasticsearch/smoketest/MonitoringWithWatcherRestIT.java
* [7.17] Use patched nebula os package gradle plugin
* Update testingconvention precommit integ test
* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
* Fix TestResultProcessor api changes
* Fix inputs for generateProviderManifest
* Ignore tests for now until gradle has fixed reporting issue
* Fix dependency substitution in example plugins build
* Use right java bin path on windows
* Add hint to task onlyif when no docker is available
* Use all lucene index compatible versions for bwc version generation (#87133)
The updateCIBwcVersions task regenerates the list of versions that are
tested in CI. This task internally calls a filter method which filters
out incompatible versions for the local system. That means it can change
depending on the type of system the task is run on. This commit adds a
new variant of the internal method to return all actual index compatible
versions, and renames the existing method to make it clear it is for
running actual tests on the local system.
* spotless
* Port javadoc configuration logic into a binary plugin (#86471)
Also adding test coverage and fixing certain issues we stumbled into
when resolving project dependencies of compileClasspath we need to use allDependencies instead of dependencies as usually no dependencies are added directly to compileClasspath but via implementation, compileonly and api and friends
Fixed javadoc setup for projects using shadowed dependencies via shadow plugin
Fixed dealing with skipped javadoc tasks in referenced dependent projects.
In general this PR only fixes expected javadoc generation as it was intended before this PR.
This also contains some tweaks to our gradle integration test fixtures for setting up and debugging test projects
# Conflicts:
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/fixtures/AbstractRestResourcesFuncTest.groovy
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/InternalYamlRestTestPluginFuncTest.groovy
# build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/YamlRestCompatTestPluginFuncTest.groovy
# build-tools/src/integTest/groovy/org/elasticsearch/gradle/TestClustersPluginFuncTest.groovy
# build-tools/src/testFixtures/groovy/org/elasticsearch/gradle/fixtures/AbstractGradleFuncTest.groovy
* Fix merge conflicts for backporting
This is an attempt at resolving, or at least reducing the prevalence of
#77837. A more robust solution is probably required, but this should
help reduce this problem since we now apply spotless to about 90 fewer
subprojects.
Our spotless configuration wasn't being applied to `build-tools`
and `build-tools-internal`. Move the Spotless configuration to a
Java plugin in `build-conventions`, and apply it everywhere.
This resulted in a lot more Java files being subject to formatting,
so I added more exclusions to the list.
Also remove the `paddedCell` stuff, we've never needed it.
- Use file property and conventions to avoid afterEvaluate hook
- Simplify root build script
- One little step closer to configuration cache compliance
* Resolve system properties in build scripts via provider factory (#76199)
This allows tracking system properties used in the build configuration and brings us
one step closer to be gradle configuration cache compliant.
* Fix system property resolution at configuration time (#78669)
This fixes system property resolution when running the build with --configuration-cache
One step closer to make use of configuration cache
This introduces the infrastructure for auto backporting source to java 8 compatibility code by using the rewrite library (https://github.com/openrewrite/). We introduce a custom rewrite plugin as the original rewrite plugin uses tons of deprecated api and non efficient gradle api. Also we introduce a rewrite task per project rewrite task makes error analysis easier IMO.
We have the configuration of the backporting in a rewrite.yml file that contains three types of backports
java.util.Map.of(...) to org.elasticsearch.core.Map(...)
java.util.List.of(...) to org.elasticsearch.core.List(...)
java.util.Set.of(...) to org.elasticsearch.core.Set(...)
The configuration uses custom rules I wrote (https://github.com/breskeby/java-recipes) as the build-in rewrite rules shipped or provided with rewrite do not provide proper handling of dealing with classes of the same name but different package. These rules are resolved from maven central.
An example of how all this backporting works in action can be seen in the functional test provided as part of this PR.
Our release pipeline is getting increasingly complex. This is leading
to lots of issues with tasks "leaking" into those builds causing
issues. We rely currently on `assemble` as the task to "build
everything", which it is, but we don't necessarily want to build
_everything_ in our release pipeline in a single go.
This change introduces an explicit task to be used by our
release builds which filters out ML and Docker build as those are
doing in separate release stages. As necessary going forward we
should tweak the dependencies here, instead of making our release
configuration more and more fragile.
The feature usage tracking data currently contains an opaque "name"
attribute which identifies the feature that was used. This name needs to
be unique enough that certain features can be identified independently
of others. For example, distinguishing machine learning jobs from
trained models. Yet both those examples are all "machine learning".
This commit adds a "family" attribute so that similar tracked features
can be grouped together. The output format of the feature usage api is
essentially the same; it is still a flat list of features and their last
used times. The family attribute can be used on the receiving end to
group many features.
Part of #67335.
Add tasks for generating release notes, using information stored in files
in the repository:
* `generateReleaseNotes` - generates new release notes, release
highlights and breaking changes
* `validateChangelogs` - validates that all the changelog YAML files are
well-formed (confirm to schema, have required fields depending on the
`type` value)
I also changed `Version` to allow a `v` prefix in relaxed mode
- No manual show task dependencies logic needed. we can use build scans instead for that.
- Move common plugin configuration into according plugins
- Reduce overall callbacks in root script, remove unneeded logic in afterEvaluate hooks
- Fix new introduced deprecated usages
- Update to newer ospackage snapshot to include provided PR for fixing deprecated usage
This gradle release comes with improvements on incremental compilation which we should benefit from
backport of #72861 to 7.x branch
This moves the public build api and plugins into a separete included build called 'build-tools' and we remove the duplication of included buildSrc twice (as build-tools).
The elasticsearch internal build logic is kept in build-tools-internal as included build which allows us better handling of this project that its just being an buildSrc project (e.g. we can reference tasks directly from the root build etc.)
Convention logic applied to both projects will live in a new build-conventions project.
back porting #72470 to 7.x
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and
related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
* Disable Test rerun plugin
We disable this plugin for now as there are issues we need to shake out
See #72169 and #72818
* Tweak rerun build scan configuration
backports #72361 to 7.x branch
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.
As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
backports #72030 to 7.x
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
