This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
(cherry picked from commit ba61f8c7f7)
# Conflicts:
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/distribution/DockerCloudElasticsearchDistributionType.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/distribution/DockerUbiElasticsearchDistributionType.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/Fixture.java
# plugins/repository-hdfs/hadoop-client-api/build.gradle
# server/src/main/java/org/elasticsearch/inference/ChunkingOptions.java
# x-pack/plugin/kql/build.gradle
# x-pack/plugin/migrate/build.gradle
# x-pack/plugin/security/qa/security-basic/build.gradle
Wholesale fix of every `TRAPPY_IMPLICIT_DEFAULT_MASTER_NODE_TIMEOUT` in
`o.e.snapshots` and `o.e.repositories`, just pulling them up to the REST
layer (where they become API params), the test suite (where they become
`TEST_REQUEST_TIMEOUT`), or some other place where an explicit value is
available.
Relates #107984
This ports our krb5kdc test fixture to test container and reworks hdfs handling to also be based on test containers.
The yaml rest tests that are using hdfs required introducing variable substitution in yamlresttestparser handling.
We have the pattern of getting a response only for its `TotalHits` in a couple of spots,
some of them still leaking the `SearchResponse`.
This PR dries up those that I could quickly identify to use the new utilty method to simplify this pattern
or just moves the code to `assertHitCount` where possible.
Reworking forbiddenApis check to use gradle worker api exposed a bug in
how we resolve krb5kdc keytab information. This fixes the depenendency to krb5kdc keytab configuration and
its builtBy task.
This also changes the usage of krb5kdc keytab files to be passed directly to task classpath as
they are only required at runtime and directly having them as part of javaRestTestRuntimeOnly would mean precommit
requires krb5kdc compose up which we definitely not want
Today many blob store repository tests specify that the operations they
perform have purpose `OperationPurpose#SNAPSHOT`, but most of these
tests do not care about the purpose of these operations. This commit
switches them to using a random purpose to highlight that the purpose is
unimportant to the test.
Same as #101175, shorten `client().prepareIndex(index)` and
`client().prepareIndex().setIndex(index)` via a test utility.
Saves lots of code now and sets up some follow-up simplifcations.
Cleaning this up a little even though it's still quite horrible.
`.get()` in this API actually means `actionGet()` so to speak.
I think a good first step to cleaning this up is to at least reduce
the duplication though and save 1k lines.
Just found that we have a lot of inconsistency and needless verbosity
here in tests. We can just use `assertAcked` in a couple spots
to save `.get`, `.actionGet` etc., especially with the signature
change I added here.
Today blobstore stats are collected against each HTTP operation, e.g.
Get, List. This is not granular enough because the same HTTP operration
can be performed for different purposes, e.g. cluster state, indices or
translog. This PR adds a new Purpose enum to provide further breakdown
for the same HTTP operation.
Relates: ES-6800
Allows users of the HDFS repository plugin to configure the replication factor for files created by the repository implementation at repository creation time.
---------
Co-authored-by: James Baiera <james.baiera@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
Today the blob store register supports recording only a `long`,
represented as an 8-byte blob. We need to store a little more data in
the register, so this commit generalises things to work with a
`BytesReference` directly.
currently when Elasticsearch starts up there are warnings on the console complaining about SLF4J providers not found.
this is emitted twice
1 - when repository-azure module is loaded
2 - when xpack-security (depending on transport-netty) module is loaded
This commits adds a slf4j-nop to fix that warning
related to #93714
and #93878
Fixes#82794. Upgrade the spotless plugin, which addresses the issue
around formatting `instanceof` expressions. Formatting of statements
including lambdas seems to have improved too.
Further work towards the S3 compare-and-exchange implementation showed
that we would like this API to permit async operations. This commit
moves to an async API.
Also, this change made it fairly awkward to use an exception to deliver
to the caller the indication that the current value could not be read,
so this commit adjusts things to use `OptionalLong` throughout as
suggested in the discussion on #93955.
Only for testing purposes through the `FsRepository` for now and rather simple,
but should get the job done and technically be correct for a compliant NFS implementation.
Co-authored-by: David Turner <david.turner@elastic.co>
Some optimisations that I found when reusing searchable snapshot code elsewhere:
* Add an efficient input stream -> byte buffer path that avoids allocations + copies for heap buffers, this is non-trivial in its effects IMO
* Also at least avoid allocations and use existing thread-local buffer when doing input stream -> direct bb
* move `readFully` to lower level streams class to enable this
* Use same thread local direct byte buffer for frozen and caching index input instead of constantly allocating new heap buffers and writing those to disk inefficiently
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
As discussed, we can be up to twice as fast without increasing CPU use
much on high latency blob stores so increasing the pool size to 10 here
to better utilize larger data nodes.
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
This adds the generation and upload logic of Gradle dependency graphs to snyk
We directly implemented a rest api based snyk plugin as:
the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is
a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.
Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.
As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.
Addresses #87620
Since Java 9, the JDK has provided a means of parsing Java versions and
getting the current Java version. That class obviates the need for the
JavaVersion class of Elasticsearch. This commit removes the JavaVersion
class in favor of Runtime.Version.
Note that most of the changes here simply removed logic around
versioning because this change is intended only for the master branch,
where Java 17 is required.
Ensure that the temporary file stream is closed
before renaming it to the final name.
This issue can impact HDFS repository operations,
as we use the atomic writes primitive to ensure
that writes to metadata files are atomic.
- use separate sourceSets for different type of tests
- remove usage of RestBuildPlugin
- Make fixture dependencies gradle idiomatic
Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
Fixes split packages between server and the LLRC (and HLRC), by renaming
the server package to a more appropriate name that represents the fact
that is in an internal client. That is, rename server's
org.elasticsearch.client to org.elasticsearch.client.internal.
