* Guard systemd library lookup from unreadable directories (#108931)
When scanning the library path we may come across directories that are
unreadable. If that happens, the recursive walk of the library path
directories will throw a fatal IOException. This commit guards the walk
of the library paths to first check for readability of each directory we
are about to traverse.
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Linux systems with multiarch (e.g. i386 & x86_64) libraries
may have libsystemd.0 in two subdirectories of an entry in
java.library.path. For example, libsystemd.so.0 may be found
in both /usr/lib/i386-linux-gnu and /usr/lib/x86_64-linux-gnu.
Instead of attempting to load any library found, attempt all
and stop as soon as one is successfully loaded.
JNA supports two types of mapping to native methods, proxying and direct
method mapping. Proxying is nicer for unit testing, but unfortunately
the proxied methods are lazily loaded. NativeAccess expects that methods
are linked during static init, before SecurityManager is initialized.
For any native methods called after security manager init, the proxied
method will fail.
This commit changes the zstd bindings to use direct method mapping so
that calling zstd methods does not fail when using JNA (pre Java 21).
closes#107504closes#107770
This commit adds an optimised int8 vector distance implementation for aarch64. Additional platforms like, say, x64, will be added as a follow-up.
The vector distance implementation outperforms Lucene's Pamana Vector implementation for binary comparisons by approx 5x (depending on the number of dimensions). It does so by means of compiler intrinsics built into a separate native library and link by Panama's FFI. Comparisons are performed on off-heap mmap'ed vector data.
The implementation is currently only used during merging of scalar quantized segments, through a custom format ES814HnswScalarQuantizedVectorsFormat, but its usage will likely be expanded over time.
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Mark Vieira <portugee@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
CloseableByteBuffer is backed by native memory segments, but the
interfaces for compress and decompress methods of zstd take ByteBuffer.
Although both Jna and the Jdk can deal with turning the native
ByteBuffer back into an address to pass to the native method, the jdk
may have a more significant cost to that action.
This commit changes the signature of compress and decompress to take in
CloseableByteBuffer so that each implementation can do its own
unwrapping to get the appropriate native address.
relates #103374
The jdk implementation of CloseableByteBuffer currently uses a shared
arena. The assumption was that a buffer might be shared across threads.
However, in practice for compression/decompression that is not true, and
the shared arena has a noticeable impact on deallocation when the buffer
is closed. This commit switches to a confined arena, limtting buffer
creation and compress/decompress calls to a single thread.
relates #103374
Since mrjars may use preview apis, forbidden apis must know about any
preview apis from the jdk. However, we do not run forbidden apis with
the preview enabled flag, nor in a separate jvm, so it does not know
about these classes. Thus we ignore missing classes on source sets added
by the mrjar plugin.
This commit configures all sourcesets added by mrjar plugin to ignore
forbidden apis missing classes.
The distributions already have correct permissions set on native
libraries copied to them. However, the build itself to extract the
native libs relies on the upstream file permissions. This commit sets
explicit permissions on the copy task which extracts native libraries.
Similar to https://github.com/elastic/elasticsearch/pull/106360, the
methods for allocating a native string changed between Java 21 and 22.
This commit adds another util method to handle the differences and uses
it in the jdk systemd impl.
The foreign memory API changed between Java 21 and 22 in how to decode a
string from native memory. This commit adds an multi-release class to
handle the two different methods on MemorySegment to decode a string.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
Sometimes libs have subprojects that may not be java projects. This commit adjusts the shared
configuration for libs to only affect direct subprojects of :lib.
Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
