Our benchmarks have demonstrated that Elasticsearch performs better with `-XX:G1HeapRegionSize=4M` and `-XX:G1ReservePercent=15` options for the small heap sizes. With this commit we ergonomically choose different G1GC options for heap sizes smaller than (not including) 8GB.
Co-authored-by: Daniel Mitterdorfer daniel.mitterdorfer@elastic.co
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.
Brings us one step closer to #57918
For testing against the local distribution we already avoid the packaging/unpackaging
cycle of es distributions when setting up test clusters. This PR adopts the usage of the
expanded created distributions for unreleased bwc versions (versions that are checkout
from a branch and build from source in the :distribution:bwc:minor / :distribution:bwc:bugfix).
This makes the setup of bwc based cross version tests a bit faster by avoiding
the unpackaging overhead. We still assemble both in the bwcBuild tasks atm
which will be addressed in a later issue.
This reworks the :distribution:bwc project:
- Convert all the custom logic from build script logic (groovy) into gradle binary plugins (java)
- Tried to make the bwc setup logic a bit more readable
- Add basic functional test coverage for the bwc logic this PR tweaked.
- Extracted a general internal BWC Git plugin out of the bwc setup plugin to improve maintenance
and testability
- Changed the InternalDistributionPlugin to resolve the extracted distro instead on relying
on unpacking the distribution archive
- Extract distribution archives defaults into plugin
- Added basic test coverage
- avoid packaging/unpackaging cycle when relying on locally build distributions
- Provide DSL for setting up distribution archives
- Cleanup archives build script
This commit removes `integTest` task from all es-plugins.
Most relevant projects have been converted to use yamlRestTest, javaRestTest,
or internalClusterTest in prior PRs.
A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)
related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
This commit adds external test modules. These are modules meant for
external systems to test edge cases in elasticsearch, but only within
snapshots. They are not meant to be used in production, so protections
are also added from their accidental inclusion in release builds.
Note that this commit does not actually add any new modules, it only
adds the infrastructure for the new modules, under
`test/external-modules`.
PR #61474 reworked deprecation logging to rely more heavily on log4j. Unfortunately,
the changes required to log4j's configuration were not applied to the version we ship
with the Docker image.
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.
This has the benefit of
- handcrafted extract tasks on the root project are not required. The general tight coupling to the root project
has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.
DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
- Added test coverage
- Removes build script cluttering
- Splits archive building and archive checking logic
- only rely on boost for now for ML licenses(tbd)
- Use Gradle build-in untar and unzip support
Part of #46106. Simplify the implementation of deprecation logging by
relying of log4j more completely, and implementing additional behaviour
through custom appenders and filters.
Closes#60864. Tweak the JDK directories' permissions in the ES
Docker image so that ES can run under a different user and group.
These changes assume that the image is being run with bind-mounted
config, data and logs directories, and reads and writes to these
locations will still fail when both the UID and GID are not the
default. Everything should be OK when running with the default GID
of zero, however.
This PR resurrects support for building Docker images based on one of Red Hat's UBI images. It also adds support for
running the existing Docker tests against the image. The image is named `elasticsearch-ubi8:<version>`.
I also changed the Docker build file uses enums instead strings in a lot of places, for added rigour.
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.
This has the benefit of
- handcrafted extract tasks on the root project are not required. The general tight coupling to the root project
has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.
DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.
Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
For systemd, while we are starting up, we notify the system every 15
seconds that we are still in the middle of starting up. However, if
initial startup before plugin initialization is slower than 15 seconds,
we won't ever get the chance to run the first timeout extension. This
commit sets the initial timeout to 75 seconds, up from the default 30
seconds used by systemd.
closes#60140
* Split internal distribution logic into separate plugin
* Remove moved func test from DistributionDownloadPluginIT
* Assert internal plugin is only appied on internal build
* Extract common abstract test specification for plugin func tests
* Provide proper failure if unexpected non jdk bundled bwc version is requested
- Fix duplicate path deprecation by removing duplicate test resources
- fix deprecated non annotated input property in LazyPropertyList
- fix deprecated usage of AbstractArchiveTask.version
- Resolve correct test resources
- Fixes how libs in distribution are resolved
- Related to #57920
- Required minor rework on common repository setup to allow distribution projects
to resolve thirdparty artifacts
- Use Default configurations when resolving tools for distribution packaging
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.
The remaining cases in modules, plugins, and x-pack will be handled in followups.
This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.
The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.
Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).
As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.
Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
- required as java library will by default not have build jar file
- jar file is now explicit input of the task and gradle will ensure its properly build
The plugin installer currently checks the fake plugins installed contain
a single jar file. This commit adds another test for a plugin which
contains multiple jar files, ensuring all jars exist in the installed
plugin.
The windows service script does a little munging of the parsed JVM
option string, converting whitespaces to semicolons. We recently added
an optional Java 14 JDK flag to our system JVM flags. On earlier JDKs,
the windows service batch script would encounter a double whitespace
when this option was missing and convert it into double semicolons.
Double semicolons, in turn, don't work in the arguments to the windows
service command, and led to a lot of JVM options being dropped,
including "es.path.conf", which is required for startup.
This commit puts in a double defense. First, it removes any empty-string
options from the system options list in the Java Options Parser code.
Second, it munges out double semicolons if they do appear in the parsed
option output.
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
* Move classes from build scripts to buildSrc
- move Run task
- move duplicate SanEvaluator
* Remove :run workaround
* Some little cleanup on build scripts on the way
In #51459 DEBUG-level logging was removed from the default log4j
configuration. However, our docker build has its own log4j configuration
which was missed in that change. This commit removes the same from the
docker log4j configuration.
relates #51459
relates #51198
With the removal of support for older OSes, we no longer have any
supported systems which use SysV init. This commit removes support for
that legacy init system.
relates #51480
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
This commit tightens certain dependency license checks in our build.
Firstly, the build will not fail if it cannot accurately identify the
type of license in one of our LICENSE.txt files. Secondly, dependencies
for licenses identified as requiring source redistribution will fail if
a corresponding SOURCES.txt file does not exist. This file should
include a hyperlink to a source artifact for the given dependency to be
used for redistribution during the release process.
Docker informed us that for official multi-arch Docker builds, there
needs to be a single Dockerfile and build context that can be used for
each supported architecture. Therefore, rework the build to move the
relevant architecture logic into the Dockerfile, and merge the aarch64
/ x64 docker context builds.
