Adding support for MDX files in our :docs project. We parse those *.mdx files
like we do for asciidoc files for code snippets and generate yaml specs from them that
we test as part of our integration tests.
By default:
When searching for doc sources in the docs folder we fail the build if we detect multiple files of
the same name but different extension. E.g. having painless-field-context.mdx
and painless-field-context.asciidoc in the same source folder will fail the build.
Migration Mode:
To allow easier migration from asciidoc to mdx the build supports a kind of migration mode.
When running the build with -Dgradle.docs.migration=true (e.g. ./gradlew buildRestTests -Dgradle.docs.migration=true)
Duplicate doc source files (asciidoc and mdx) are allowed
The Generated yaml rest specs for duplicates will have the extension *.mdx.yml or *asciidoc.yml.
The generated yaml rest specs for duplicates are compared to each other to ensure they produce the same yml output.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
Sometimes subprojects are created by build tests. There exists a utility
function to define the subproject build file. Yet some tests add the
project to settings and then configure through the subprojects directive
in the root build file. This commit cleans up two such cases to use
explicit subproject build files.
On Serverless it is not possible to configure deprecation indexing (it
is always off). This commit updates the behaviour of
`ElasticsearchCluster` to no longer attempt to configure deprecation
indexing on stateless nodes.
Fixing a couple of file leaks (and cleaning up one missing
try-with-resources). The directory descriptor leaks in particular
were leaking massively on every precommit run, to the point where it
slows down the whole system and/or we're running into descriptor limits.
* Use single-char variant of String.indexOf() where possible
indexOf(char) is more efficient than searching for the same one-character String.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Use String.replace() instead of replaceAll() for non-regexp replacements
When arguments do not make use of regexp features replace() is a more efficient option, especially the char-variant.
Prevent invalid and miss-configuration of APM agent using an explicit allow-list of setting keys.
Additionally, configuration defaults of APMAgentSettings are consolidated in APMJvmOptions to keep defaults in a single location.
(ES-6916)
RunTaks is used by both statefull and serverless.
when --with-apm-server is used, `telemetry.metrics.enabled` is always turned to true
and server_url is set to localhost
However when --with-apm-server is NOT used, then we should disable `telemetry.metrics.enabled`
only if that setting was not explicitly used. This explicte use is for instance when
running gradlew run and configuring ES to send to ESS cluster
a follow up from https://github.com/elastic/elasticsearch/pull/101941/files#r1388403805
When a test cluster in serverless is run with --with-apm-server
3 nodes were attempting to start apm server. This was throwing an exception.
There is no harm in reusing the already running apm server,
and in fact is more realistic
* Move TestCluster plugin and module setup from Node to Cluster
* Do not materialize unused TestCluster
* Fix lazy test cluster evaluation
* Register artifact transforms ones per project
* Make task caching for TestClusterAware tasks CC compatible
* Move stateful logic out of taskgraph.whenready
With apm metrics being enabled by default in serverless, the run task should only allow for apm integration and starting the server when --with-apm-server is switch on
When the flag is not present, the apm metrics should be disabled and apm server not started
* Make YamlRestTest and ElasticDistributionPlugin cc compatible
These changes are addressing configuration cache incompatibilities of running
rest tests with the new ClassRule based Elasticsearch test cluster setup.
This allows running those tests with configuration cache enabled:
`./gradlew :modules:ingest-user-info:yamlRestTest --configuration-cache`
Gradle now fully supports compiling, testing and running on Java 20.
Among other general performance improvements this release introduces --test-dry-run command line option that allows checking if tests are filtered or not by gradle.
Required updating nebula ospackage plugin as setuid was broken in gradle 8.3.
Lots of spots where we did weird things around streams like redundant stream creation, redundant collecting
before adding all the collected elements to another collection or so, redundant streams for joining strings
and using less efficient `Collectors.toList` and in a few cases also incorrectly relying on the result being mutable.
Using gradle toolchain support in gradle requires refactoring how the composite build is composed.
We added three toolchain resolver
1. Resolver for resolving defined bundled version from oracle as openjdk
2. Resolve all available jdks from Adoption
3. Resolve archived Oracle jdk distributions.
We should be able to remove the JdkDownloadPlugin altogether without having that in place, but we'll do that in a separate effort.
Fixes#95094
Version.java currently contains mappings to the Lucene version for each
Elasticsearch version. The only use of this in the build logic is for
filtering based on index compatibility. However, that compatibility can
be inferred based on the Elasticsearch major version since there is a
one to one mapping between Elasticsearch major and Lucene major. This
commit removes Lucene version extraction from the build bwc logic.
Fixes#82794. Upgrade the spotless plugin, which addresses the issue
around formatting `instanceof` expressions. Formatting of statements
including lambdas seems to have improved too.
- Remove custom checksum build logic in wrapper task
- Adjust jdk home handling adjusting the change in behaviour in gradle. Requires providing canonical paths for provisioned jdk homes.
- Fix test by add workaround to bug in configuration cache
This updates the gradle wrapper to 7.6.1. This patch release contains a
fix for incremental compilation of java modules we have raised against
gradle 7.6
see https://github.com/gradle/gradle/issues/23067
in example plugins a pluginApiVersion is more appropriate than elasticsearchVersion as it better implies that it can be different then the version of ES cluster
The previous model relied on treating the server's certificate
configuration as a trust anchor. This isn't guaranteed to work,
which lead to needing to support "certificate_authorities" as an
alternative, which in turn polluted the node's config with settings
that only existed to enable tests to run.
The new model ties the "wait-for-health" HTTP client to the leaf
certificates themselves. This means that it will always connect to a
node that has the exact certificates it expects, and doesn't rely on
knowing the issuer of the node's certificate.
This PR adds settings and infrastructure to support a new Remote Cluster port,
to be used in Remote Cluster Security 2.0. Specifically, this commit adds new
settings that allow opening a new Remote Cluster port, which will eventually
exclusively support the new cross-cluster authentication method. Since support
for that new authentication method is still under construction, these settings
are hidden behind a feature flag.
The new settings cover all Transport profile settings, to ensure that users will
not have to be exposed to Transport Profiles directly to make use of RCS2.0
functionality. This includes all core settings, as well as IP filter and SSL
configuration.
Co-authored-by: Yang Wang <yang.wang@elastic.co>
Refactoring that drops the api suffix from package name
This will have to be followed up by a plugins/examples fix in imports
Also set an artifact group name to `org.elasticsearch.plugin` in the plugin-api and plugin-analysis-api
new stable plugins require generated named_components.json file which contains all analysis components implemented by this plugin. The generation is currently done in build-tools by elasticsearch.stable-esplugin
However this makes the generation only available for plugins using gradle. Plugin developers using maven or other building tooling will not be able to use it.
This commits refactors the scanning logic into libs:plugin-scanner which will allow for plugin install command to perform the scanning too.
relates #88980
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
* Fix TestResultProcessor api changes
* Fix inputs for generateProviderManifest
* Ignore tests for now until gradle has fixed reporting issue
* Fix dependency substitution in example plugins build
* Use right java bin path on windows
* Add hint to task onlyif when no docker is available
