* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
Backports the following commits to 7.17:
- Add verification metadata for dependencies (#88814)
- Add missing dependency verification checksums (#89139)
When a gzip-encoded response is decompressed the response should no more
have a content-encoding header and content-length should be set to
"unknown". GzipDecompressingEntity correctly does this for the entity
but the response still reported the original response's content-encoding
and content-length headers.
Thanks to https://bugs.eclipse.org/bugs/show_bug.cgi?id=574437,
we've run into a situation where Spotless is incorrectly formatting
a particular piece of syntax (due the underlying Eclipse bug). We
were able to turn off formatting of this syntax using `// @formatter:off`
and `// @formatter:on`, but there was a further problem. We configure
IntelliJ to use the Eclipse formatter plugin, but this doesn't
respect the `@formatter` tags since these are set at the Spotless
level, not the Eclipse formatter level. Note that these tags aren't
set in the Eclipse formatter config, because there we use `// tag::`
and `// end::` in order to avoid reformatting docs snippets, which
have a much narrower line width.
What a mess.
So, to get around all this, drop the `@formatter` tags and tweak
our custom `SnippetLengthCheck` Checkstyle rule so that
`// tag:noformat` regions are not subject to the narrower line length
check, but are still exempt from formatting.
Matchers is deprecated in Mockito, in favor of the newer
ArgumentMatchers class. In fact, internally Matchers just extends
ArgumentMatchers as all the methods there were moved. This commit
changes all imports of org.mockito.Matchers to
org.mockito.ArgumentMatchers.
Securemock is a wrapper around Mockito that monkey patches internals of
Mockito to work with the SecurityManager. However, the library has not
been updated in several years due to the complicated nature of this
monkey patching. This has left us with an ancient version of Mockito,
missing out on updates to the library in the last half decade.
While Securemock currently works with Mockito 1.x, in 2.x an official
means of plugging into mockito was added, MockMaker. This commit removes
securemock as a dependnecy of the test framework, replacing it with a
modern version of Mockito, and implementing a MockMaker that integrates
with SecurityManager.
Note that while there is a newer version of Mockito available, 4.0, it
has several deprecations removed that are used throughout Elasticsearch.
Those can be addressed in followups, and then a subsequent upgrade to
4.0 should be possible.
relates #79567closes#40334
A recent change for the deprecation logs provided the capability to emit deprecation's at critical vs. warning levels, #77482.
However deprecated settings always log at critical level without the ability to express that the setting deprecation is only a
warning.
This commit exposes the ability to set the deprecation level when deprecating a setting.
Relates #78781
This adds support for the headers necessary for REST version compatibility to the High Level Rest
Client (HLRC).
Compatibility mode can be turned on either with the .setAPICompatibilityMode(true) when creating
the client, or by setting the ELASTIC_CLIENT_APIVERSIONING to true similar to our other
Elasticsearch clients.
Resolves#77859
- Use file property and conventions to avoid afterEvaluate hook
- Simplify root build script
- One little step closer to configuration cache compliance
* Reformatting to keep Checkstyle after formatting
* Configure spotless everywhere, and disable the tasks if necessary
* Add XContentBuilder helpers, fix test
Follow-up to #73434
Ensures that High Level Rest Client is running against a verified
Elasticsearch. When the first request is send on HLRC, a request to the
info endpoint is made first to verify the product identification and
version.
backport of #72861 to 7.x branch
This moves the public build api and plugins into a separete included build called 'build-tools' and we remove the duplication of included buildSrc twice (as build-tools).
The elasticsearch internal build logic is kept in build-tools-internal as included build which allows us better handling of this project that its just being an buildSrc project (e.g. we can reference tasks directly from the root build etc.)
Convention logic applied to both projects will live in a new build-conventions project.
This relaxes the assertion to account for the case when the client
executes the request successfully before we cancel the underlying http
request.
(cherry picked from commit 4b02e84058)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Instead of using the same handler with new latches each time, test now registers completely new handler for every request. This prevents race condition that led to deadlock and timeout.
Local testing shows around 500k iterations without a failure.
Closes#45577
This commit introduces data_content, data_hot, data_warm,
data_cold, and data_frozen to the low level REST client (LLRC).
Since the LLRC only cares about dedicated masters this change
simply makes the LLRC aware of the new roles and does not have
any functional impact. The tests have been adjusted accordingly.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Adds a X-Elastic-Client-Meta header to http requests sent by RestClient. This
header contains information about the runtime environment that is meant to
allow analyzing usage context by collecting this information on the receiving
side of requests, like a proxy server in front of ES.
Using a custom header allows client applications to change the User-Agent
header for their own purpose without losing this information.
Backport of #66303
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
There is little evidence of this endpoint being used
and there is quite a lot of code complexity associated
with the various formats that can be used to upload
data and the different errors that can occur when direct
data upload is open to end users.
In a future release we can make this endpoint internal
so that only datafeeds can use it, and remove all the
options and formats that are not used by datafeeds.
End users will have to store their input data for
anomaly detection in Elasticsearch indices (which we
believe all do today) and use a datafeed to feed it
to anomaly detection jobs.
Backport of #66347
Adds a `RestClient.setCompressionEnabled()` setting that will gzip-
compress request bodies and add a `Accept-Encoding: gzip` header so
that the ES server can send compressed responses.
* Move tasks in build scripts to task avoidance api (#64046)
- Some trivial cleanup on build scripts
- Change task referencing in build scripts to use task avoidance api
where replacement is trivial.
This removes the assertion that the header warnings we parse in the
rest client reponses conform to RFC 7234 because we are not in full control
of the warnings that could be present in the responses (ie. proxies might
emit warnings that don't comply).
We still maintain this assertion on the ES side (see `HeaderWarning#addWarning`)
for the warnings we emit.
(cherry picked from commit 1259a46cbe)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
The domain part of a Cloud-Id can contain an optional custom port, e.g.
cloud.example.org:9443. This feature is used for Elastic Cloud
Enterprise installations that can't use the default port 443.
This change fixes RestClient.build() to correctly handle custom ports.
Splitting DeprecationLogger into two. HeaderWarningLogger - responsible for adding a response warning headers and ThrottlingLogger - responsible for limiting the duplicated log entries for the same key (previously deprecateAndMaybeLog).
Introducing A ThrottlingAndHeaderWarningLogger which is a base for other common logging usages where both response warning header and logging throttling was needed.
relates #55699
relates #52369
backports #55941
Different kinds of requests may need different request options from the client
default. Users can optionally set RequestConfig on a single request's
RequestOptions to override the default. Without this, socketTimeout can only
set at RestClient initialization.
Co-authored-by: weizijun <weizijun1989@gmail.com>
* Replace compile configuration usage with api (#58451)
- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
- required as java library will by default not have build jar file
- jar file is now explicit input of the task and gradle will ensure its properly build
* Fix compile usages in 7.x branch
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
Added support for decompression at LLRC and added integration test
(cherry picked from commit 2621452473)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Hakky54 <hakangoudberg@hotmail.com>
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
I've noticed that a lot of our tests are using deprecated static methods
from the Hamcrest matchers. While this is not a big deal in any
objective sense, it seems like a small good thing to reduce compilation
warnings and be ready for a new release of the matcher library if we
need to upgrade. I've also switched a few other methods in tests that
have drop-in replacements.
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
We mades roles pluggable, but never updated the client to account for
this. This means that when speaking to a modern cluster, application
logs are spammed with warning messages around unrecognized roles. This
commit addresses this by accounting for the fact that roles can extend
beyond master/data/ingest now.
This commit adds examples in our documentation for
- An HLRC instance authenticating to an elasticsearch cluster using
an elasticsearch token service access token or an API key
- An HLRC instance connecting to an elasticsearch cluster that is
setup for TLS on the HTTP layer when the CA certificate of the
cluster is available either as a PEM file or a keystore
- An HLRC instance connecting to an elasticsearch cluster that
requires client authentication where the client key and certificate
are available in a keystore
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
