This PR updates `bc-fips` and `bctls-fips` dependencies to the latest
minor versions.
(cherry picked from commit 6ea3e01958)
Co-authored-by: Slobodan Adamović <slobodanadamovic@users.noreply.github.com>
* Simplify java version checker (#112955)
The version checker ensures custom JDKs used to run Elasticsearch meet
the minimum Java version. It uses a multi-release jar to pass through
the "ok" versions, falling back to a java 8 version to print an error.
However, a multi-release jar is not necessary, we can instead do simple
string version parsing for the check, special casing Java 1.8.
* adjust for java 17
JDK 23 removes the COMPAT locale provider, leaving CLDR as the only option. This commit configures Elasticsearch
to use the CLDR provider when on JDK 23, but still use the existing COMPAT provider when on JDK 22 and below.
This causes some differences in locale behaviour; this also adapts various tests to still work whether run on COMPAT or CLDR.
When the server CLI is preparing to start Elasticsearch, it may receive
a SIGTERM. That signal causes the close method of the CLI to be invoked,
which checks for a non-null server process, and stops it if it exists.
Since shutdown occurs in a different thread, it is possible close is
invoked, sees the server process is null, so skips calling close, but
then before close finishes the server process is started.
Normally the above case is ok; the close method will cause Java to exit,
and the child process will be sent a SIGTERM. However, in the case of
serverless, close is wrapped so that the CLI can wait on uploading
possible heapdumps. This presents the possibility that close thinks the
child process has been stopped, but it hasn't, so the wrapped close
waits indefinitely on the child process that was started up during the
race.
This commit guards against the startup race by locking the critical
sections that create the child process and stop it. Either the child
process is created first, and will then be stopped immediately, or close
is invoked first, marking that the CLI is shutting down, and creating
the child process will be skipped.
Native libraries in Java are loaded by calling System.loadLibrary. This
method inspects paths in the java.library.path to find the requested
library. Elasticsearch previously used this to find libsystemd, but now
the only remaining use is to set the additional platform directory in
which Elasticsearch keeps its own native libraries.
One issue with setting java.library.path is that its not set for the cli
process, which makes loading the native library infrastructure from clis
difficult. This commit reworks how Elasticsearch native libraries are
found in order to avoid needing to set java.library.path. There are two
cases. The simplest is production, where the working directory is the
Elasticsearch installation directory, so the platform specific directory
can be constructed. The second case is for tests where we don't have an
installtion. We already pass in java.library.path there, so this change
renames the system property to be a test specific property that the new
loading infrastructure looks for.
This commit moves the file preallocation functionality into
NativeAccess. The code is basically the same. One small tweak is that
instead of breaking Java access boundaries in order to get an open file
handle, the new code uses posix open directly.
relates #104876
* Mechanical package change in IntelliJ
* A couple of manual fixups
* Export plugins.loading to deprecation
* Put plugin-cli in a module so can export PluginsUtils to it.
The workaround requires to JDK args, but SystemJvmOptions actually
operates on individual JDK args. This commit adjusts SystemJvmOptions to
allow adding sets of jdk args together.
This removes getWriter() and getErrorWriter() from Terminal in order to prepare for using log4j in server-cli.
Additionally, ensure IO failures on the pump thread are handled once completed (closed) and
catch throwables in Command#main to print them (formatted) via Terminal.
This commit bumps Tika to 2.9.2 and manually bumps the transitive versions
to match 2.9.2's parent POM. This commit also centralizes the dependency
versions so that you only need to look at 1 list to see the full set of dependencies
to manually check.
This commit adds an optimised int8 vector distance implementation for aarch64. Additional platforms like, say, x64, will be added as a follow-up.
The vector distance implementation outperforms Lucene's Pamana Vector implementation for binary comparisons by approx 5x (depending on the number of dimensions). It does so by means of compiler intrinsics built into a separate native library and link by Panama's FFI. Comparisons are performed on off-heap mmap'ed vector data.
The implementation is currently only used during merging of scalar quantized segments, through a custom format ES814HnswScalarQuantizedVectorsFormat, but its usage will likely be expanded over time.
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Mark Vieira <portugee@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
On some systems Java appears to return amd64 (even if not an amd
processor), but on others it returns x86_64. This commit handles the
latter case to correctly associate the arch with the appropriate
platform dir.
For small nodes, we need a bit more wiggle room for new size
and concurrent GC threads in order to stay below real memory
circuit breaker limits on dedicated search nodes.
ES-8087
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
The difference is that our matcher uses .find() to search for a regex match anywhere in the string, whereas the hamcrest one uses .matches() to check the whole string against the regex. This leads to more specific regex checks.
I've left our own one for YAML tests, as that way we don't need to mangle the regex to add .* either side, which might be confusing in test failures.
Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
When hotspot encounters an error, it will emite a log file which can be
used for reproducing the error. This file is dumped to /tmp by default.
This commit configures the replay file to be alongside the hs_err file.
In certain circumstances if Elasticsearch encounters an error while
starting up, the server cli may exit with no error. This commit fixes
the cli to always check and wait on the Elasticsearch process and exit
with the same exit code.
relates #104055
Telemetry / APM settings are renamed from "tracing.apm.{name}" to "telemetry.tracing.{name}" for tracing related settings. General APM settings are renamed to "telemetry.{name}". The old legacy settings are kept for now and applied as fallback.
Prevent invalid and miss-configuration of APM agent using an explicit allow-list of setting keys.
Additionally, configuration defaults of APMAgentSettings are consolidated in APMJvmOptions to keep defaults in a single location.
(ES-6916)
We're leaking quite a few of these parsers. That doesn't seem to be much
of a problem but results in some memory inefficiencies in Jackson here
and there. This PR bulk fixes a bunch of instances that I could easily
automatically fix. I'll open a follow-up for closing the parser on the
document parsing context which also suffers from this but is non-trivial
to fix.
* Refactoring: move command line and environment build to a separate class
* Refactoring: adding a builder
* Moving tmp dir setup and JVM option parsing outside of builder
This commit upgrades the Bouncy Castle jars. Bouncy Castle is used for
some internal build concners as well as a comnand line application.
Most notably Bouncy Castle is also used as the FIPs certified JCE/JSEE provider
we use to test our ability to use a FIPs compliant crypto provider.
The following changes here are a result of the upgraded Bouncy Castle jars:
* TLSv1.3 is now supported when running in FIPs mode
* RSA PKCS#1 v1.5 is no longer allowed in FIPS mode
* Triple DES (3DES) is no longer allowed in FIPS mode
* Minor updates the security manager configuration used to test FIPs (to read permissions from the security provider)
* Minor adjustments to tests to accommodate the above changes.
* Minor adjustments to the gradle build to accommodate new dependencies
Note - update to the documentation will come in a later commit.
