Commit graph

84 commits

Author SHA1 Message Date
Mark Vieira
315fa321ae
Improve efficiency of packer cache script (#94701) 2023-03-23 13:29:45 -07:00
Mark Vieira
3add0b2343
Fix reference to location of read-only Gradle dependency cache (#92852)
Environment variable of course only works in the context of a shell.
Instead of using `$HOME` just reference the `user.home` system property
here.
2023-01-11 17:49:34 -05:00
Mark Vieira
e7b29abf11
Use 'main' when referring to default branch (#84463)
Replace any old references of master with main. These are mostly
cosmetic, as the name of the default branch doesn't actually affect the
build in any meaningful way.
2022-03-01 12:28:45 -08:00
Rene Groeschke
5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Mark Vieira
6f995e65bc Disable fail on deprecation for packer cache script 2021-04-14 13:07:40 -07:00
Mark Vieira
a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Rene Groeschke
2a8885ae91
Wire local unreleased bwc versions more efficient for tests (#62473)
For testing against the local distribution we already avoid the packaging/unpackaging 
cycle of es distributions when setting up test clusters. This PR adopts the usage of the
expanded created distributions for unreleased bwc versions (versions that are checkout 
from a branch and build from source in the :distribution:bwc:minor / :distribution:bwc:bugfix). 
This makes the setup of bwc based cross version tests a bit faster by avoiding 
the unpackaging overhead. We still assemble both in the bwcBuild tasks atm 
which will be addressed in a later issue.

This reworks the :distribution:bwc project:

- Convert all the custom logic from build script logic (groovy) into gradle binary plugins (java)
- Tried to make the bwc setup logic a bit more readable
- Add basic functional test coverage for the bwc logic this PR tweaked.
- Extracted a general internal BWC Git plugin out of the bwc setup plugin to improve maintenance 
and testability
- Changed the InternalDistributionPlugin to resolve the extracted distro instead on relying 
on unpacking the distribution archive
2020-10-01 10:45:21 +02:00
Rene Groeschke
72ff11d080
Simplify distribution download and extraction (Take two) (#61276)
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.

This has the benefit of

- handcrafted extract tasks on the root project are not required. The general tight coupling to the root project 
has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.

DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
2020-08-27 14:21:41 +02:00
Mark Vieira
ad9bc452a8
Revert "Simplify distribution download and extraction (#60969)"
This reverts commit b61f71875d.
2020-08-17 08:53:00 -07:00
Rene Groeschke
b61f71875d
Simplify distribution download and extraction (#60969)
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.

This has the benefit of

- handcrafted extract tasks on the root project are not required. The general tight coupling to the root project 
has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.

DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
2020-08-17 08:27:47 +02:00
Rene Groeschke
a72760e55b
Replace immediate task creations by using task avoidance api (#60071)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 11:29:14 +02:00
Mark Vieira
d17c8cfa58
Add compatibility testing for JDBC driver (#60430)
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.

Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
2020-07-29 14:11:16 -07:00
Rene Groeschke
a57dd54306
Update Gradle wrapper to 6.5 (#57580)
* Update Gradle wrapper to 6.5
* Fix groovy incompatibility issue after gradle update
* Fix Gstring String incompatibility
2020-06-04 10:32:47 +02:00
Rene Groeschke
731b282c9f
Improvement usage of gradle task avoidance api (#56627)
Use gradle task avoidance api wherever it is possible as a drop in replacement in the es build
2020-05-19 20:01:49 +02:00
Rene Groeschke
fb86184dbb
Move bwcVersions extension property to BuildParams (#56206)
- resolved in GlobalBuildInfoPlugin
- propagate exception if version file cannot be resolved
- add null check when setting bwcVersions
2020-05-07 12:08:17 +02:00
Ryan Ernst
6cb17a3a71
Simplify java home verification (#55635)
* Simplify java home verification

At one time, all uses of java home were found through the getJavaHome
utility method on BuildPlugin. However, that was changed many
refactorings ago, but the complex support for registering a java home
version needed that fails at configuration time still exists. The only
remaining use of grabbing java home is within bwc tests, and must be at
runtime since that is when we have the checkout and know what version is
needed.

This commit consolidates the java home finding method into a utility
unassociated with BuildPlugin.

* fix checkstyle

* address feedback
2020-04-27 12:42:50 -07:00
Mark Vieira
3b5fb808f4
Improve total build configuration time (#54611)
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:

 - Removing the usage of the 'nebula.info-scm' plugin. This plugin
   leverages jgit to load read various pieces of VCS information. This
   is mostly overkill and we have our own minimal implementation for
   determining the current commit id.
 - Removing unnecessary build dependencies such as perforce and jgit
   now that we don't need them. This reduces our classpath considerably.
 - Expanding the usage lazy task creation, particularly in our
   distribution projects. The archives and packages projects create
   lots of tasks with very complex configuration. Avoiding the creation
   of these tasks at configuration time gives us a nice boost.
2020-04-08 15:31:24 -07:00
Mark Vieira
faca8404df
Enable the remote build cache in BWC distribution builds (#54691) 2020-04-02 17:04:56 -07:00
Mark Vieira
c4aff0c959
Tag "nested" Gradle build scans (#51440) 2020-01-24 11:00:14 -08:00
Mark Vieira
bdd7bda47b
Upgrade to Gradle 6.1 (#50453) 2020-01-16 11:08:07 -08:00
Mark Vieira
3050172936
Don't pass OPENSHIFT_IP env variable when building old BWC branches (#50153)
This commit tweaks the workaround introduced in #49211 to support
Gradle 6.0. In the workaround, we specifically override the address
the Gradle daemon binds to by passing the desired address via the
OPENSHIFT_IP environment variable. This works fine for builds using
Gradle 6.0, but for older Gradle versions this causes issues with
inter-daemon communication, specifically when we build BWC branches
not on Gradle 6.0. The fix here is to strip that environment variable
out when building the target BWC branch if that branch is on an
older Gradle version.

This is all temporary and will be removed when this bug fix is released
in Gradle 6.1.

Closes #50025
2019-12-12 14:52:36 -08:00
Rory Hunter
3a3e5f6176
Apply 2-space indent to all gradle scripts (#48849)
Closes #48724. Update `.editorconfig` to make the Java settings the default
for all files, and then apply a 2-space indent to all `*.gradle` files.
Then reformat all the files.
2019-11-13 10:14:04 +00:00
Mark Vieira
af6af346f7
Introduce type-safe and consistent pattern for handling build globals (#48778)
This commit introduces a consistent, and type-safe manner for handling
global build parameters through out our build logic. Primarily this
replaces the existing usages of extra properties with static accessors.
It also introduces and explicit API for initialization and mutation of
any such parameters, as well as better error handling for uninitialized
or eager access of parameter values.

Closes #42042
2019-11-01 09:54:22 -07:00
Mark Vieira
deef0d9117
Disable bwc distribution caching in master branch (#46686)
This commit disables caching of BWC snapshot distributions in the "trunk" (aka master) branch.
Since the previous major release branches move quickly we rarely get cache hits for these 
tasks, and the artifacts themselves are very large. This means the overhead here is high and
savings basically zero. We conditionally disable task output caching in this scenario in CI to 
avoid excessive build cache overhead as well as causing too much turn in the cache itself which
would lead to lots of cache entry evictions.
2019-09-18 11:51:44 -07:00
Mark Vieira
3074e51202
Log actual checkout hash instead of generic refspec (#46637) 2019-09-13 16:07:19 -07:00
Alpar Torok
b633cd5ec5
Add input and outut tracking of built bwc versions (#45694)
* Add input and outut tracking of built bwc versions

This PR adds tracking of the bwc versions git has as input and all the
expected files as output.
The effect is that `gradlew` is not called at all when the git has
doesn't change and the version was allready built.
Previusly gradlew would be called for the bwc version and it would have
to configure the project and go trough up to date checks to figure out
that nothing changed.
This helps when working on bwc tests locally needing to run the test
multiple times.
This should also help in CI not re-build bwc versions across different
runs.

* Enable caching of bwc builds
2019-08-20 10:01:03 +03:00
Mark Vieira
3ebb7ae1d6
Rename system property to change bwc checkout behavior (#45574) 2019-08-16 08:43:14 -07:00
Alpar Torok
38a9f6f395
Change how we pick bwc versions to check out (#45189)
Prior to this PR we always checked out the latest bwc branches and had
an external mechanism to store the bwc versions used for every CI run so
we could both reproduce those builds and run additional tests using the
same combination.

This adds complexities in setting up and maintaining CI and makes it
difficult to set up multi jobs.

This change replaces that mechanism with a time based approach
that looks at the commit date of the current revision and picks the
newest on the bwc branch that's still older than that.
It also makes sure there are no merge commits in this interval.

This new behavior will is ment to be enabled in CI only, for everything
except PR checks that will still use last available bwc revision.
2019-08-07 16:37:15 +03:00
Alpar Torok
c9d04ccb3a Make packer cache branches explicit (#41990)
Before this change we would recurse to cache bwc versions.
This proved to be problematic due to  the number of steps it was
generating taking too long.
Also this required tricky maintenance to break the recursion for old
branches we don't really care about.

With this change we now cache specific branches only.
2019-05-22 12:00:51 -07:00
Ryan Ernst
bc0b0f55a2
Hide bwc build output on success (#42102)
Previously we used LoggedExec for running the internal bwc builds.
However, this had bad performance implications as all the output was
buffered into memory, thus we changed back to normal Exec. This commit
adds a `spoolOutput` setting to LoggedExec which can be used for
commands with large amounts of output, and switches the bwc builds to
use this flag.
2019-05-16 09:48:57 -04:00
Ryan Ernst
a0b30065f9
Use separate tasks per bwc artifact (#40703)
This commit changes the bwc builds from a single task for a branch to a
task for each bwc artifact. This reduces the bwc build time when only
needing a specific artifact, for example when running cluster restart
tests on a mac, the windows artifacts or rpm/debs are not needed.
2019-04-03 16:23:05 -07:00
Jay Modi
834cc35193
Use ephemeral ports for idp-fixture (#40333)
This change removes the use of hardcoded port values for the
idp-fixture in favor of the mapped ephemeral ports. This should prevent
failures due to port conflicts in CI.
2019-03-26 08:40:00 -06:00
Alpar Torok
195aa64b3f Fix building bwc versions (#40361)
Account for openjdk when figuring out what java version to
use for building bwc versions.
2019-03-22 11:27:01 -06:00
Alpar Torok
fdb918a999
Fix verify versions (#39624)
closes #38708
2019-03-07 13:54:20 +02:00
Alpar Torok
bb5e339f10
Simulate BWC tests that start with the current version (#39102)
* Include current version in bwc tests
2019-03-04 11:21:29 +02:00
Jason Tedor
26cf6f2e04
Conditionally build BWC projects in parallel (#39396)
This commit sets the BWC projects to build in parallel if Gradle was
invoked with parallal project execution enabled. This substantially
speeds up the time of building the BWC projects since there are many
dependent projects needed to build a BWC version.
2019-02-27 08:04:19 -05:00
Jason Tedor
83e9d0b9c6 Add 7.1 and 8.0 version constants to master (#38514)
* Add 7.1 and 8.0 version constants to master

This commit adds the 7.1 and 8.0 version constants to the master branch,
and bumps the master branch to 8.0.0.

Fixing many BWC and other tests  affected by the version bumps.
2019-02-08 08:50:46 +02:00
Ryan Ernst
8e5f9c4b14
Add OS/architecture classifier to distributions (#37881)
This commit adds classifiers to the distributions indicating the
OS (for archives) and platform. The current OSes are for windows, darwin (ie
macos) and linux. This change will allow future OS/architecture specific
changes to the distributions. Note the docs using distribution links
have been updated, but will be reworked in a followup to make OS
specific instructions for the archives.
2019-01-29 11:18:30 -08:00
Alpar Torok
7b516f99b9
Resolve BWC dependencies too when resolving deps (#37412)
This will allow those to be cached too leading to faster builds and
fewer opportunity for network failures
2019-01-25 14:14:12 +02:00
Ryan Ernst
d9d13f3414
Use project dependency instead of substitutions for distributions (#37730)
Currently integration tests which use either bwc snapshot versions or
the current version of elasticsearch depend on project substitutions to
link to the build of those artifacts. Likewise, vagrant tests use
dependency substitutions to get to bwc snapshots of rpm and debs.
This commit changes those to depend on the relevant project/configuration
and removes the dependency substitutions for distributions we do not
publish.
2019-01-23 23:41:21 -08:00
Ryan Ernst
722b850efd
Build: Make assemble a noop in bwc projects (#37245)
This commit makes the assemble tasks in the bwc projects noops by
setting the dependsOn directly. While we can not remove things from
dependsOn, we can still completely override the dependencies.

closes #33581
2019-01-09 10:57:26 -08:00
Alpar Torok
e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Alpar Torok
59b0900174
Upgrade to Gradle 5.0 (#34263) 2018-12-05 14:06:11 +02:00
Alpar Torok
f22700812e
Introduce build qualifier parameter (#35155)
* Introduce property to set version qualifier

- VersionProperties.elasticsearch is now a string which can have qualifier
and snapshot too
- The Version class in the build no longer cares about snapshot and
qualifier.
2018-11-02 05:27:40 +02:00
Alpar Torok
f8378d91a6
Implement VersionCollection in Java (#34050) 2018-11-01 17:43:57 +02:00
Ryan Ernst
4b7257d971
Build: Pass offline flag through to bwc build (#34367)
This commit makes the bwc build aware of the offline flag passed to the outer build.

closes #34361
2018-10-09 17:07:51 -04:00
Alpar Torok
76310eb3f1
BWC Build: Read CI properties to determine java version (#34295) 2018-10-05 10:46:00 +03:00
Jason Tedor
9d36cbaf16
Set BWC builds for 6.x to use JDK 11
The BWC builds for the 6.x branch should be using JDK 11. This commit
fixes the BWC builds to specify that they use JDK 11 instead of JDK 10
which is now incompatible with the 6.x build.
2018-10-03 09:52:21 -04:00
Alpar Torok
9f96d2ce17
Don't build bwc on assemble (#33372)
Gradle triggers the build of artifacts even if assemble is disabled.
Most users will not need bwc distributions after running `./gradlew
assemble` so instead of forcing them to add `-x buildBwcVersion`, we
detect this and skip the configuration of the artifacts.
2018-09-05 08:24:44 +03:00
Alpar Torok
7f7e8fd733
Disable assemble task instead of removing it (#33348) 2018-09-04 07:32:14 +03:00