Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
Related to #52610 this PR introduces a rerun of all tests for a test task if the test jvm
has crashed because of a system exit. We furthermore log potential tests that caused
the System.exit based on which tests have been active at the time of the system exit.
We also modified the build scan logic to track unexpected test jvm exists
with the tag `unexpected-test-jvm-exit`
- Update gradle wrapper to gradle 7.0
- Remove deprecated usages to make build 7.0 compatible
- Fix excludes in docs snippet tasks (See https://github.com/gradle/gradle/issues/16160 for details)
- Fix deprecation warnings in 7.0
- Add explicit dependencies that have been missed
- Make extract native licenses tasks output dir more explicit
- Use a snapshot of the ospackage plugin that includes a fix for 7.0 already
- fix test runtime classpath setup in repository-hdfs
- Make task dependency explicit to fix further deprecation warnings
- Remove manual check for http repo usages that has been deprecated in gradle 7.0
- Update spock to latest 2.0 milestone required for groovy 3
This commit adds the rest endpoints for the node shutdown API. These APIs are behind the
`es.shutdown_feature_flag_enabled` feature flag for now, as development is ongoing.
Currently these APIs do not do anything, returning immediately. We plan to implement them for real
in subsequent work.
Relates to #70338
It turns out you can configure custom Checkstyle rules using the plugin
for IntelliJ, you just have to tell the plugin where to find the code.
Therefore, update the contributing docs with the necessary process.
Also fork the Checkstyle rule `MissingJavadocTypeCheck` so that we can
disable it for test classes. While it's useful to describe test cases in
test methods' Javadoc, this is somewhat less true for test classes
themselves.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Now that Checkstyle can be made useful in an IDE, add extra rules only when
checking in the IDE so that a contributor is given extra help when editing
files, without having the checkstyle task spam the console when running gradle.
Apart from the `BooleanNegation` rule below, the new rules have the `warning`
severity level. The new Javadoc rules reflect the guidelines in
`CONTRIBUTING.md` that we've had for some time.
* I upgraded Checkstyle, so now it interprets the config file in the same was
as the IntelliJ plugin. That means that I could move the `LineLength` rule
up a level and remove its special handling in the `:configureIdeCheckstyle`
task
* I added the `SuppressWarningsFilter` and `SuppressWarningsHolder` rules so
that the `@SuppressWarnings` annotation can be used to silence Checkstyle
checks. We shouldn't typically need this, but it seemed like a useful thing
to configure. In contrast to the suppressions file, this approach makes the
suppression obvious.
* I changed the `:configureIdeCheckstyle` task to pull in rules from
`checkstyle_ide_fragment.xml` and merged them into the generated config.
The rules are as follows:
* `BooleanNegation` - a task defined using `DescendantToken` that detects
cases of `!` being used negate a boolean expression. This ought to be in
the main config, but at present we have a number of violations in the
source
* `MissingJavadocMethod` - requires that public methods are documented. I
set `minLineCount` to 2 so that the rule doesn't trigger on simple
methods.
* `MissingJavadocPackage` - require Javadoc in `package-info.java`
* `MissingJavadocType` - require types to be documented
* `JavadocMethod` - require params and return type to be documeted
We would attempt to match a string that starts with lower case
'oracle' but runtimeJavaDetails is ORACLE/Oracle Corporation.
Changed the matching to be a little more inclusive so that it will
not easily break by future changes to our build tooling or jdk
reported vendor string.
resolves: #66821
- Updates our build to use the latest Gradle 6.8 release which is the last release
before the major 7.0 release.
- Resolve available gradle versions using built-in toolchain support
- Fixes deprecated usage of JavaInstallationRegistry
- We replace jdk handling in our build to rely on jvm detection provided by the gradle build tool itself.
As we rely on environment variables pointing to jdks we wire this into the gradle jdk detection mechanism
This PR conditionally allow SunRsaSign to be used as a security provider when
the runtime java is an implementation by Oracle. It is necessary because:
* Oracle JVM mandates Security Provider verification
* The verification class (javax.crypto.JarVerifier) uses hardcoded certificates
with md5WithRsa signature.
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
Storing the generated config in the `build` dir meant that it could be
removed for various reasons, which causes e.g. IntelliJ to pop up
dialog, and is a mild annoyance. Instead, write the file to the project
root.
Also inline the path to the suppressions config, which removes a step
from the setup instructions.
With a few changes, our checkstyle config can be used by e.g. IntelliJ's
Checkstyle plugin. Add a task to generate this file automatically, and
create the necessary IntelliJ config to use it.
Also add a line-length setting to .editorconfig.
- Build scans allow to be configured in a background thread which lowers the impact
of the configuration on the overall build configuration time.
- The configuration time was already quite low but as its such a low hanging fruit we
take advantage of that option in our ES build too.
Bouncy Castle's BC-FJA-1.0.2 has been certified for a while now
but we had noticed that it seems to be rather entropy hungry and
ES would start very slowly ( and tests would take forever )
because of blocking calls to /dev/random.
We verified that this is resolved when enabling hw RNG or a
software one like haveged. While rng-tools should be suggested for
production uses, our ephemeral workers have haveged installed
which should work just fine for CI.
Resolves: #47455
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.
Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
* Split internal distribution logic into separate plugin
* Remove moved func test from DistributionDownloadPluginIT
* Assert internal plugin is only appied on internal build
* Extract common abstract test specification for plugin func tests
* Provide proper failure if unexpected non jdk bundled bwc version is requested
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
* Move classes from build scripts to buildSrc
- move Run task
- move duplicate SanEvaluator
* Remove :run workaround
* Some little cleanup on build scripts on the way
This commit moves the configuration of all test jvms for fips to a
script plugin. Fips testing is something very specific to the
Elasticsearch build and does not need to be passed on to plugin authors.
