* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This change enhances the dense_vector section of the Nodes stats and Index stats APIs so that they report the desired size of off-heap memory for all indexed vectors. The dense_vector section of the Custer stats API remains unchanged.
The retrieval mechanism and structure of the new stats is the same across the various three stats APIs, but more fine-grained information is disclosed as when moving from Cluster -> Node -> Index API.
For Node stats, we aggregate the total byte sizes for all vectors, categorised by the data type. For example:
"dense_vector" : {
"value_count" : 5,
"off_heap" : {
"total_size_in_bytes" : 27,
"total_veb_size_in_bytes" : 3,
"total_vec_size_in_bytes" : 23,
"total_veq_size_in_bytes" : 0,
"total_vex_size_in_bytes" : 1
}
}
Index stats: same as Node stats with included field break down . For example:
"dense_vector" : {
"value_count" : 5,
"off_heap" : {
"total_size_in_bytes" : 27,
"total_veb_size_in_bytes" : 3,
"total_vec_size_in_bytes" : 23,
"total_veq_size_in_bytes" : 0,
"total_vex_size_in_bytes" : 1,
"fielddata" : {
"bar" : {
"veb_size_in_bytes" : 3,
"vec_size_in_bytes" : 14,
"vex_size_in_bytes" : 1
},
"foo" : {
"vec_size_in_bytes" : 9
}
}
}
The implementation accesses the actual statistics through reflection. This will be completely removed when Lucene exposes this, which is expected in Lucene 10.3
Now that entitlements are always used, there is no need to run tests
with security manager (a future enhancement will run tests with
entitlements). This commit removes setting up security manager from
tests.
With the change to using the logs dir as the working dir of the
Elasticsearch process we need to ensure the logs dir exists within the
CLI instead of later during startup.
relates #124966
In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
The JDK team has completely disabled the Java SecurityManager from Java 24. Elasticsearch has always used the Java SecurityManager as an additional protection mechanism; in order to retain this second line of defense, the Elasticsearch Core/Infra team has been working on the Entitlements project.
Similar to SecurityManager, Entitlements only allow calling specific methods in the JDK when the caller has a matching policy attached. In other words, if some code (in the main Elasticsearch codebase, in a plugin/module, or in a script) attempts to perform a "privileged" operation and it is not entitled to do so, a NotEntitledException will be thrown.
This PR includes the minimal set of changes to always use Entitlements, regardless of system properties or Java version.
Relates to ES-10921
Information about the system, and in particular the running JVM, is
logged relatively late in startup. When problems occur in low level
initialization, it is often useful to have this information. This commit
moves logging of system information to right after logging has been
initialized. Note that it no longer utilizes the JvmInfo class as most
was already easily available and the static init of JvmInfo does a lot.
By default Lucene only uses vectorization up to the JDK version
available at the time of its release. When testing against EA versions,
we want to still use vectorization. This commit adds a sysprop to force
lucene to use the current version for vectorization.
* Remove @UpdateForV9 annotation in RuntimeVersionFeature
Re ES-10338 the Core/Infra team agreed that this is not blocked on security manager removal. We should go ahead and remove these 2 instances of the @UpdateForV9 annotation, the code will be cleaned up soon with the removal of Security Manager.
* Remove @UpdateForV9 annotation from SystemJvmOptions
re ES-10338 the Core/Infra team agreed this is not blocked on security manager removal. We should go ahead and remove this instance of the @UpdateForV9 annotation.
The node environment has many paths. The accessors for these currently
use a "file" suffix, but they are always directories. This commit
renames the accessors to make it clear these paths are directories.
Since entitlements are still being developed, we are not yet ready to
enable them for all JDK versions. But we must use them for JDK 24+ since
the security manager is not useable there. This commit tweaks the logic
for determining whether to use entitlements to take into account the
runtime version.
* Refactor: separate check method name vs signature parsing
* Cosmetic: change checker comment format
* Entitlements for JDK-wide global state
* [CI] Auto commit changes from spotless
* Comment explaining entitlement add-exports
* @SuppressForbidden
* Refactor: rename dummy subclases
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Refactor: treat "maybe" JVM options uniformly
* WIP
* Get entitlement running with bridge all the way through, with qualified
exports
* Cosmetic changes to SystemJvmOptions
* Disable entitlements by default
* Bridge module comments
* Fixup forbidden APIs
* spotless
* Rename EntitlementChecker
* Fixup InstrumenterTests
* exclude recursive dep
* Fix some compliance stuff
* Rename asm-provider
* Stop using bridge in InstrumenterTests
* Generalize readme for asm-provider
* InstrumenterTests doesn't need EntitlementCheckerHandle
* Better javadoc
* Call parseBoolean
* Add entitlement to internal module list
* Docs as requested by Lorenzo
* Changes from Jack
* Rename ElasticsearchEntitlementChecker
* Remove logging javadoc
* exportInitializationToAgent should reference EntitlementInitialization, not EntitlementBootstrap.
They're currently in the same module, but if that ever changes, this code would have become wrong.
* Some suggestions from Mark
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
Regardless of JDK version, ES should always use CLDR locale database from 9.0.0.
This also removes IsoCalendarDataProvider used to override week-date calculations for the root locale only.
JDK 23 removes the COMPAT locale provider, leaving CLDR as the only option. This commit configures Elasticsearch
to use the CLDR provider when on JDK 23, but still use the existing COMPAT provider when on JDK 22 and below.
This causes some differences in locale behaviour; this also adapts various tests to still work whether run on COMPAT or CLDR.
When the server CLI is preparing to start Elasticsearch, it may receive
a SIGTERM. That signal causes the close method of the CLI to be invoked,
which checks for a non-null server process, and stops it if it exists.
Since shutdown occurs in a different thread, it is possible close is
invoked, sees the server process is null, so skips calling close, but
then before close finishes the server process is started.
Normally the above case is ok; the close method will cause Java to exit,
and the child process will be sent a SIGTERM. However, in the case of
serverless, close is wrapped so that the CLI can wait on uploading
possible heapdumps. This presents the possibility that close thinks the
child process has been stopped, but it hasn't, so the wrapped close
waits indefinitely on the child process that was started up during the
race.
This commit guards against the startup race by locking the critical
sections that create the child process and stop it. Either the child
process is created first, and will then be stopped immediately, or close
is invoked first, marking that the CLI is shutting down, and creating
the child process will be skipped.
Native libraries in Java are loaded by calling System.loadLibrary. This
method inspects paths in the java.library.path to find the requested
library. Elasticsearch previously used this to find libsystemd, but now
the only remaining use is to set the additional platform directory in
which Elasticsearch keeps its own native libraries.
One issue with setting java.library.path is that its not set for the cli
process, which makes loading the native library infrastructure from clis
difficult. This commit reworks how Elasticsearch native libraries are
found in order to avoid needing to set java.library.path. There are two
cases. The simplest is production, where the working directory is the
Elasticsearch installation directory, so the platform specific directory
can be constructed. The second case is for tests where we don't have an
installtion. We already pass in java.library.path there, so this change
renames the system property to be a test specific property that the new
loading infrastructure looks for.
This commit moves the file preallocation functionality into
NativeAccess. The code is basically the same. One small tweak is that
instead of breaking Java access boundaries in order to get an open file
handle, the new code uses posix open directly.
relates #104876
The workaround requires to JDK args, but SystemJvmOptions actually
operates on individual JDK args. This commit adjusts SystemJvmOptions to
allow adding sets of jdk args together.
This removes getWriter() and getErrorWriter() from Terminal in order to prepare for using log4j in server-cli.
Additionally, ensure IO failures on the pump thread are handled once completed (closed) and
catch throwables in Command#main to print them (formatted) via Terminal.
This commit adds an optimised int8 vector distance implementation for aarch64. Additional platforms like, say, x64, will be added as a follow-up.
The vector distance implementation outperforms Lucene's Pamana Vector implementation for binary comparisons by approx 5x (depending on the number of dimensions). It does so by means of compiler intrinsics built into a separate native library and link by Panama's FFI. Comparisons are performed on off-heap mmap'ed vector data.
The implementation is currently only used during merging of scalar quantized segments, through a custom format ES814HnswScalarQuantizedVectorsFormat, but its usage will likely be expanded over time.
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Mark Vieira <portugee@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
On some systems Java appears to return amd64 (even if not an amd
processor), but on others it returns x86_64. This commit handles the
latter case to correctly associate the arch with the appropriate
platform dir.
