This commit upgrades the Bouncy Castle jars. Bouncy Castle is used for
some internal build concners as well as a comnand line application.
Most notably Bouncy Castle is also used as the FIPs certified JCE/JSEE provider
we use to test our ability to use a FIPs compliant crypto provider.
The following changes here are a result of the upgraded Bouncy Castle jars:
* TLSv1.3 is now supported when running in FIPs mode
* RSA PKCS#1 v1.5 is no longer allowed in FIPS mode
* Triple DES (3DES) is no longer allowed in FIPS mode
* Minor updates the security manager configuration used to test FIPs (to read permissions from the security provider)
* Minor adjustments to tests to accommodate the above changes.
* Minor adjustments to the gradle build to accommodate new dependencies
Note - update to the documentation will come in a later commit.
* Avoid "this-escape" by making classes final
The "this-escape" compiler warning is intended to alert
developers to potential bugs in object initialization due to
subclassing. This class of bugs cannot occur when a class is
final. Here, we take cases where a class has no implementations
but generates a "this-escape" warning, and we make those
classes final rather than suppressing the compiler warning.
This makes the remaining suppressions more meaningful, since
they now indicate places where we may want to look for
initialization bugs.
In a few cases, making a class final meant changing some of its
protected fields and methods to private or default
accessibility.
Some classes with no implementations are mocked in testing.
Since making those classes final would involve non-trivial
rewrites of tests, I've left them alone.
* Spotless, remove redundant modifiers, clean up "protected" usage
* Revert a few more mocked classes
Adds @SuppressWarnings("this-escape") to all necessary places to that
Elasticsearch can compile with -Werror on JDK21
No investigation has been done to determine whether any of the cases
are a potential source of errors - we have simply suppressed all
existing occurrences.
Resolves: #99845
Lots of spots where we did weird things around streams like redundant stream creation, redundant collecting
before adding all the collected elements to another collection or so, redundant streams for joining strings
and using less efficient `Collectors.toList` and in a few cases also incorrectly relying on the result being mutable.
The error message should refer to the setting of ssl.key instead of
ssl.keystore.path
PS: Labelling this as non-issue since we don't consider error message as
actual production changes (that need go into the release notes).
A follow up to #91946 with the
minor requested changes.
Changes included here are:
* reuse of variables
* additional unit test
* convert to use enumeration instead of set of strings
This commit extends the TLS restricted trust model to allow reading from
alternative fields from the X509 certificate. Prior to this commit the only
supported (hard coded) value that could be used with restricted trust
is the SAN/otherName/CN value. This commit introduces support to read
from other fields from the X509 certificate. This commit also introduces
support to read from SAN/dnsName if configured. Any fields read from the
certificate will be used to match against the restricted trust file and if any
of the values match to the restricted trust file, then restricted trust is allowed.
Only if none of the values match then the restricted trust denied.
SAN/otherName/CN is the default, and SAN/dnsName can be used in addition
or in place of SAN/otherName/CN. The possible configuration values are:
*.trust_restrictions.x509_fields: ["subjectAltName.otherName.commonName", "subjectAltName.dnsName"]
To help support testing, all of the existing certificates have been updated
to include a SAN/dnsName that matches the SAN/otherName/CN. This
allows the tests to randomize which field(s) are used to match for restricted trust.
This also has the side effect of making this commit larger than expected in
terms of lines of change. A readme has been included with copy-able commands
to recreate the certificates as needed.
Additionally, a CCS REST test has been introduced that uses the restricted trust.
To support this new CCS REST test the private keys for the test certificates are also
included in this commit as well as the gradle configuration needed to share those
certificates across projects.
This PR represents the initial phase of Modularizing Elasticsearch (with
Java Modules).
This initial phase modularizes the core of the Elasticsearch server
with Java Modules, which is then used to load and configure extension
components atop the server. Only a subset of extension components are
modularized at this stage (other components come in a later phase).
Components are loaded dynamically at runtime with custom class loaders
(same as is currently done). Components with a module-info.class are
defined to a module layer.
This architecture is somewhat akin to the Modular JDK, where
applications run on the classpath. In the analogy, the Elasticsearch
server modules are the platform (thus are always resolved and present),
while components without a module-info.class are non-modular code
running atop the Elasticsearch server modules. The extension components
cannot access types from non-exported packages of the server modules, in
the same way that classpath applications cannot access types from
non-exported packages of modules from the JDK. Broadly, the core
Elasticseach java modules simply "wrap" the existing packages and export
them. There are opportunites to export less, which is best done in more
narrowly focused follow-up PRs.
The Elasticsearch distribution startup scripts are updated to put jars
on the module path (the class path is empty), so the distribution will
run the core of the server as java modules. A number of key components
have been retrofitted with module-info.java's too, and the remaining
components can follow later. Unit and functional tests run as
non-modular (since they commonly require package-private access), while
higher-level integration tests, that run the distribution, run as
modular.
Co-authored-by: Chris Hegarty <christopher.hegarty@elastic.co>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Just some quick static analysis+fixing here. Not much in terms of code changes
besides adding the `static` keywords with the exception of some simplifications
to some of the search objects that don't need the search controller instance
passed down in many spots.
This was done mostly automatically by the IDE but some quick manual inspection shows
quite a few spots where this should make things behave better via things like making lambdas
non-capturing.
Since Java 9, the JDK has provided a means of parsing Java versions and
getting the current Java version. That class obviates the need for the
JavaVersion class of Elasticsearch. This commit removes the JavaVersion
class in favor of Runtime.Version.
Note that most of the changes here simply removed logic around
versioning because this change is intended only for the master branch,
where Java 17 is required.
For compatibility reasons we support reading PEM files that are
encrypted using DES or TripleDES (DESede). This is necessary because
many historical versions of OpenSSL would default to using these
algorithms and there are a lot of PEM files that require DES parsing
support.
However, there is a risk that this DES code could be copied to other
places, or used as an example of how to perform encryption.
This commit extracts the DES identifiers into appropriately named and
documents constants to explain that their use is for compatibility
only and that DES (and DESede) should not be considered safe for
general encryption needs.
Note: This DES support is for decrypting the contents of a
password-protected PEM file (that is, for a private key). The
encryption format of the file contents does not affect how the
key-pair is used for on-the-wire encryption.
When running elasticsearch-reconfigure-node to allow a node that
was installed via a package(RPM/DEB) to enroll to an existing
secured cluster, we should ensure that the file ownership is
proper so that elasticsearch can actually read the files when it
starts after reconfiguration.
This commits sets the group owner of the keystore files to
`elasticsearch` which is the group that we create during
installation.
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.
Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
This commit adds support for decrypting PKCS#8 encoded private keys
that have been encrypted using a PBES2 based scheme (AES only).
Unfortunately `java.crypto.EncryptedPrivateKeyInfo` doesn't make this
easy as the underlying encryption algorithm is hidden within the
`AlgorithmParameters`, and can only be extracted by calling
`toString()` on the parameters object.
See: https://datatracker.ietf.org/doc/html/rfc8018#appendix-A.4
See: AlgorithmParameters#toString()
See: com.sun.crypto.provider.PBES2Parameters#toString()
Resolves: #78901, #32021
This introduces a new CLI tool `elasticsearch-enroll-node`.
It takes an Enrollment Token as a parameter and using the
information in it, it attempts to
- Communicate with an existing node of the cluster
- Receive necessary key/certificate material
- Persist said material and configuration
This tool needs to be run before the first time the current
node starts and if it doesn't have any explicit security
related configuration already defined.
* Reformatting to keep Checkstyle after formatting
* Configure spotless everywhere, and disable the tasks if necessary
* Add XContentBuilder helpers, fix test
* Tweaks
* Add a TODO
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit extends the SSL diagnostics message to include descriptions of the
- The KeyUsage and ExtendedKeyUsage of the peer's certificate
- The CipherSuite & Protocol (TLS/SSL version) of the current session
These can be helpful in diagnosing SSL errors.
Co-authored-by: Tim Vernum <tim@adjective.org>
This commit adds the concept of a KeyStore filter to the SSL
configuration library.
Such a filter it applied to a KeyStore before it is used to construct
a KeyManager, in order to modify the entries in the keystore
(typically to remove entries that should not be used as SSL
client/server keys).
This commit upgrades the existing SSPL licensed "ssl-config" library
to include additional features that are supported by the X-Pack SSL
library.
This commit does not make any changes to X-Pack to use these new
features - it introduces them in preparation for their future use by
X-Pack.
The reindex module is updated to reflect API changes in ssl-config
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
The org.elasticsearch.bootstrap package exists in server with classes
for starting up Elasticsearch. The elasticsearch-core jar has a handful
of classes that were split out from there, namely java version parsing
and jarhell. This commit moves those classes to a new
org.elasticsearch.jdk package so as to not split the server owned
bootstrap package.
relates #73784
Part 8.
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
ESRestTestCase rest clients could only be configured to trust
the certificate authorities that were contained in a truststore. In
certain cases (like in fips mode where JKS/PKCS12 keystores) cannot
be used, it's beneficial to be able to trust specific certificate
authorities (indicated by the CA PEM endoded certificate)
This commit changes the SSL Diagnostic warning to include additional
details about trusted certificate issuers when the provide certificate
chain does not match any trust anchors.
- If there are no trusted issuers, this is explicitly called out
- If there is one trusted issuer, it is listed by name (DN) and fingerprint
- If there are between 2 and 10 trusted issuers, then they are listed
by name (DN)
- If there are more than 10 trusted issuers, the number of issuers is
included in the message (but no other details).
- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
- required as java library will by default not have build jar file
- jar file is now explicit input of the task and gradle will ensure its properly build
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
SSLReloadDuringStartupIntegTests was recently introduced but it is
failing in FIPS mode because of the use of JKS keystore. This
change mutes it in FIPS mode. It also adjusts
PemUtilsTests#readEcCurveTests to be more robust in general and
also work in FIPS mode.
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
We were creating PemKeyConfig objects using different private
keys but always using testnode.crt certificate that uses the
RSA public key. The PemKeyConfig was built but we would
then later fail to handle SSL connections during the TLS
handshake eitherway.
This became obvious in FIPS tests where the consistency
checks that FIPS 140 mandates kick in and failed early
becausethe private key was of different type than the
public key
