This commit upgrades the Bouncy Castle jars. Bouncy Castle is used for
some internal build concners as well as a comnand line application.
Most notably Bouncy Castle is also used as the FIPs certified JCE/JSEE provider
we use to test our ability to use a FIPs compliant crypto provider.
The following changes here are a result of the upgraded Bouncy Castle jars:
* TLSv1.3 is now supported when running in FIPs mode
* RSA PKCS#1 v1.5 is no longer allowed in FIPS mode
* Triple DES (3DES) is no longer allowed in FIPS mode
* Minor updates the security manager configuration used to test FIPs (to read permissions from the security provider)
* Minor adjustments to tests to accommodate the above changes.
* Minor adjustments to the gradle build to accommodate new dependencies
Note - update to the documentation will come in a later commit.
Another round of automated fixes to this, marking things that can be
made static as static. Saves some JIT cycles but also turns some lambdas
from capturing to non-capturing and makes the "utilityness" of some
classes visible.
Adds @SuppressWarnings("this-escape") to all necessary places to that
Elasticsearch can compile with -Werror on JDK21
No investigation has been done to determine whether any of the cases
are a potential source of errors - we have simply suppressed all
existing occurrences.
Resolves: #99845
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
This commit changes access to the latest TransportVersion constant to
use a static method instead of a public static field. By encapsulating
the field we will be able to (in a followup) lazily determine what the
latest is, outside of clinit.
We can dry things up a little here and also making things a little faster
(in case we missed a corner case where a list setting is hot) with the optimized
string list setting constructor.
The vast majority of this commit is doing a simple s/Version/TransportVersion/, however there are a couple of areas of complexity.
Handshakes proved to be a particularly tricky issue, as they have complex usage of Version.minimumCompatibilityVersion. In lieu of solving this problem now, I've replaced uses of minimumCompatibilityVersion with TransportVersion.minimumCompatibilityVersion and TransportVersion.calculateMinimumCompatVersion.
As a later piece of work, code using both those methods will need to be refactored to do something else, and the methods removed.
Co-authored-by: Przemyslaw Gomulka <przemyslaw.gomulka@elastic.co>
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
This is a result of structural search/replace in intellij. This only affects log methods with a signature
logger.info((Supplier) ()-> ParametrizedMessage) logger.info((Supplier) ()-> ParametrizedMessage, Throwable)
relates #86549
Most of the Jackson uses, eg in x-content and azure, have already been
upgraded. This commit upgrades the rest of the uses. Note that it does
not yet upgrade the aws sdk, this should also be done on its own.
Most classes under elasticsearch-core had been moved to the o.e.core
package. However, a couple io related classes remained in an "internal"
package. This commit moves Streams and IOUtils to the core package, as
they are no more "internal" than the rest of the classes in core.
This removes the overhead of zipping up modules that are immediately unzipped again when packaging
the elasticsearch distribution.
We also move some logic for packaging the elasticsearch distribution into a plugin and remove some outdated
overhead dealing with 'meta plugins' when copying modules into a distribution.
Another follow up and related optimization out of scope of this PR is, to also not zip unzip modules declared
for usage in our test cluster setups.
This partially addresses #76726.
This change isolates the Jackson implementation of x-content parsers and generators to a separate classloader. The code is loaded dynamically upon accessing any x-content functionality.
The x-content implementation is embedded inside the x-content jar, as a hidden set of resource files. These are loaded through a special classloader created to initialize the XContentProvider through service loader. One caveat to this approach is that IDEs will no longer trigger building the x-content implementation when it changes. However, running any test from the command line, or running a full Build in IntelliJ will trigger the directory to be built.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
Removing the mock nio transport and replacing its usage with the netty transport to make tests
with a more realistic transport implementation. This way improves the real world coverage for
the Netty transport, makes our tests more realistic and saves lots of code.
In particular, coverage on the rather complicated throttling/chunking in the netty message handler
is really ice to have.
The downside of this change is that we lose the slow transport thread warnings that the mock transport
outputs. This isn't a big deal these days in my opinion as we have slow logging in other places
now that makes up for this (we didn't when initially adding the slow logging) and that contains
far more detailed information on what exactly was slow.
Other than that, the mock transport does not come with any features we don't also have in the Netty
transport at this point.
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
Originally we tried to a log4j update in #47298, but we were unable to
that due to the `DeprecationLoggerTests.testLogPermissions` test
failing. The test relied on mocking and got removed in
https://github.com/elastic/elasticsearch/pull/61474/files#diff-70de5a6ba5c637e7f19c51341417760d6e957beb5a1fa5703049095ea2719ee0L47
Now we should be able to the upgrade and then we can address the Security
Manager permission questions raised in #47298 separately.
* Initialize pattern layout with AccessController.doPrivileged
We need the `getClassLoader` permissions
* Disable the SecurityManager for command testing because of `CommandLoggingConfigurator`
which fails under the `SecurityManager`
This commit updates all deprecation message (except for REST
compatible API messages) in 8.0+ to be emit at warning level.
Currently none of these have been removed in future versions (yet) so they
should be logged at warning, not critical.
This commit also changes the default assertWarning to assert at warning level
and introduces a new assertCriticalWarning to assert critical warnings.
A recent change for the deprecation logs provided the capability to emit deprecation's at critical vs. warning levels, #77482.
However deprecated settings always log at critical level without the ability to express that the setting deprecation is only a
warning.
This commit exposes the ability to set the deprecation level when deprecating a setting.
Closes#78781
This commit removes the dependency on the Joda library. It removes
many remaining references to joda, though not all because some comments
are worthwhile for historical reasoning.
This commit switches the security and identity-provider plugins to use
v4.0.1 of the OpenSAML library (upgraded from v3.4).
In order to facilitate this upgrade the following changes are also
made:
- Common Codec is upgraded to 1.14 across all modules
- Guava is upgraded to v28.2 in the 2 affected modules
Relates: #71983
This commit changes default deprecation logger level to CRITICAL, where default means deprecations emitted by DeprecationLogger#critical method.
It also introduces WARN deprecations which are emitted by DeprecationLogger#warn Those log lines emitted at WARN are meant to indicate that a functionality is deprecated but will not break at next major version.
relates #76754
Today `AbstractRefCounted` has a `name` field which is only used to
construct the exception message when calling `incRef()` after it's been
closed. This isn't really necessary, the stack trace will identify the
reference in question and give loads more useful detail besides. It's
also slightly irksome to have to name every single implementation.
This commit drops the name and the constructor parameter, and also
introduces a handy factory method for use when there's no extra state
needed and you just want to run a method or lambda when all references
are released.
This introduces a basic public yaml rest test plugin that is supposed to be used by external
elasticsearch plugin authors. This is driven by #76215
- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins
Co-authored-by: Mark Vieira <portugee@gmail.com>
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
The recent upgrade of the Azure SDK has caused a few test failures that
have been difficult to debug and do not yet have a fix. In particular, a
change to the netty reactor resolving
(https://github.com/reactor/reactor-netty/issues/1655). We need to wait
for a fix for that issue, so this reverts commit
6c4c4a0ecb.
relates #73493
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.
closes#66555closes#67214
Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.
closes#66555closes#67214
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
Today when creating an internal test cluster, we allow the test to
supply the node settings that are applied. The extension point to
provide these settings has a single integer parameter, indicating the
index (zero-based) of the node being constructed. This allows the test
to make some decisions about the settings to return, but it is too
simplistic. For example, imagine a test that wants to provide a setting,
but some values for that setting are not valid on non-data nodes. Since
the only information the test has about the node being constructed is
its index, it does not have sufficient information to determine if the
node being constructed is a non-data node or not, since this is done by
the test framework externally by overriding the final settings with
specific settings that dicate the roles of the node. This commit changes
the test framework so that the test has information about what settings
are going to be overriden by the test framework after the test provide
its test-specific settings. This allows the test to make informed
decisions about what values it can return to the test framework.
