In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
This commit adds a new command line option to the run task to include
plugins. The option takes in a comma separated list of plugins from the
plugins directory. For example:
```
./gradlew run --with-plugins=analysis-icu,analysis-nori
```
The JDK team has completely disabled the Java SecurityManager from Java 24. Elasticsearch has always used the Java SecurityManager as an additional protection mechanism; in order to retain this second line of defense, the Elasticsearch Core/Infra team has been working on the Entitlements project.
Similar to SecurityManager, Entitlements only allow calling specific methods in the JDK when the caller has a matching policy attached. In other words, if some code (in the main Elasticsearch codebase, in a plugin/module, or in a script) attempts to perform a "privileged" operation and it is not entitled to do so, a NotEntitledException will be thrown.
This PR includes the minimal set of changes to always use Entitlements, regardless of system properties or Java version.
Relates to ES-10921
In case when file with `.attach_pid` in name was stored in distribution
and then deleted, the exception could stop copying/linking files
without any sign of issue. The files were then missing in the cluster
used in the test causing them sometimes to fail (depending on which
files haven't been copied).
When using `Files.walk` it is impossible to catch the IOException and
continue walking through files conditionally. It has been replaced with
FileVisitor implementation to be able to continue if the exception is
caused by files left temporarily by JVM but no longer available.
This commit adds compatibility tests that target ES revisions that align with specific Lucene versions. In this case, we are intending to upgrade from Lucene 10.0 to 10.1. Since no on-prem Elasticsearch release exists with 10.0, we need another method to ensure compatibility with Lucene 10.0 indicies.
The work here is a bit hacky since all our compatibility testing infrastructure is centered around versions and we're now effectively doing compatibility tests between two different revisions of Elasticsearch that both report the same version. Ideally this specific testing would be replaced by unit tests, rather that reusing our full cluster restart tests for this purpose.
We'll also want to bump the commit referenced in the CI pipelines here to align with the last commit using Lucene 10.0.
With this, all rolling upgrade tests that involve a
`nextNodeToNextVersion` update are gradle configuration cache
compatible.
Simplify API around test cluster registry and cc compatible usage of
test cluster in TestClusterAware tasks.
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
Also provide caching support for buildparams provider
* Extract BuildParameterExtension public api into interface
* Make tests better readable
* Fix test flakyness
* do not track certain env vars for LoggedExec
* Fix some more tasks on build cacheability
* Some more cleanup on task inputs
* Mark more tasks as cacheable
replace deprecated gradle api
fix permission api usage in debian and rpm package creation
remove deprecated usage of #ProjectDependency..getDependencyProject()
improves gradle configuration cache reading in our weekly benchmark by almost 30%
This adds infrastructure to make the legacy test cluster plugin and
the legacy test cluster based test plugins generally configuration cache compatible.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
* Refactor: treat "maybe" JVM options uniformly
* WIP
* Get entitlement running with bridge all the way through, with qualified
exports
* Cosmetic changes to SystemJvmOptions
* Disable entitlements by default
* Bridge module comments
* Fixup forbidden APIs
* spotless
* Rename EntitlementChecker
* Fixup InstrumenterTests
* exclude recursive dep
* Fix some compliance stuff
* Rename asm-provider
* Stop using bridge in InstrumenterTests
* Generalize readme for asm-provider
* InstrumenterTests doesn't need EntitlementCheckerHandle
* Better javadoc
* Call parseBoolean
* Add entitlement to internal module list
* Docs as requested by Lorenzo
* Changes from Jack
* Rename ElasticsearchEntitlementChecker
* Remove logging javadoc
* exportInitializationToAgent should reference EntitlementInitialization, not EntitlementBootstrap.
They're currently in the same module, but if that ever changes, this code would have become wrong.
* Some suggestions from Mark
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
REST tests extending JsonLogsIntegTestCase migrated to the new REST
testing framework, using 'elasticsearch.internal-java-rest-test' Gradle
plugin
Explicit handling of test single-cluster config by checking if cluster
is explicitly configured as single-node to avoid an incorrect configuration
which may cause split brain.
Replaced `ElasticsearchJavaPlugin` with `ElasticsearchJavaBasePlugin` in
`RestTestBasePlugin` for better granularity.
Additionally, updated the `DistributionDownloadPlugin` to set configurations as
non-consumable and the `InternalDistributionArchiveSetupPlugin` to mark
certain configurations as consumable. This ensures that configurations
are correctly utilized during build setup and execution phases.
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
The `java.security.manager=allow` system property is required when
running tests on newer Java versions with the security manager
deprecated. As such, it should be set in our
`GradleTestPolicySetupPlugin` so that it's done for external plugin
authors.
* Fallback to bundled jdk for runtime jdk instead of build jvm
* Rework bundled jdk resolution to be lazy
* Use fixed runtime jdk for gradle build integ tests
We do not implicitly rely on all different kind of env variables when leveraging Gradle Configuration Cache
this makes reusing config cache more reliable and improves cache hits
* configure default env variables for logged exec task
* Fix antfixturestop constructor
* Make LoggedExec cc compatible (fix integtests)
* Fix spotless
We throw an error when we detect resource leaks in cluster logs but we
don't provide any indication of which cluster, or which log. This change
logs the path to the log file in question so you can investigate.
Fix incompatibility with 8.8 and our internal api usages
- Update ospackage to a version that contains a fix we provided
- Tweak build logic to avoid deprecation warnings
- Use newer permission api
- Use custom shadowplugin
- Rework ElasticsearchDistribution dependencies resolution
- Update Gradle wrapper to 8.8
We should make sure to find leaks reported by both of these, these days
our `LeakTracker` will likely be more sensitive that Netty's in some
cases since our objects refer to Netty objects and thus get collected
first.
Adding support for MDX files in our :docs project. We parse those *.mdx files
like we do for asciidoc files for code snippets and generate yaml specs from them that
we test as part of our integration tests.
By default:
When searching for doc sources in the docs folder we fail the build if we detect multiple files of
the same name but different extension. E.g. having painless-field-context.mdx
and painless-field-context.asciidoc in the same source folder will fail the build.
Migration Mode:
To allow easier migration from asciidoc to mdx the build supports a kind of migration mode.
When running the build with -Dgradle.docs.migration=true (e.g. ./gradlew buildRestTests -Dgradle.docs.migration=true)
Duplicate doc source files (asciidoc and mdx) are allowed
The Generated yaml rest specs for duplicates will have the extension *.mdx.yml or *asciidoc.yml.
The generated yaml rest specs for duplicates are compared to each other to ensure they produce the same yml output.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
Sometimes subprojects are created by build tests. There exists a utility
function to define the subproject build file. Yet some tests add the
project to settings and then configure through the subprojects directive
in the root build file. This commit cleans up two such cases to use
explicit subproject build files.
On Serverless it is not possible to configure deprecation indexing (it
is always off). This commit updates the behaviour of
`ElasticsearchCluster` to no longer attempt to configure deprecation
indexing on stateless nodes.
Fixing a couple of file leaks (and cleaning up one missing
try-with-resources). The directory descriptor leaks in particular
were leaking massively on every precommit run, to the point where it
slows down the whole system and/or we're running into descriptor limits.
* Use single-char variant of String.indexOf() where possible
indexOf(char) is more efficient than searching for the same one-character String.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
