* Use directory name as project name for libs (#115720)
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
* fixes
Recent refactoring of native library paths broke jna loading zstd. This
commit fixes jna to set the jna.library.path during init so that jna
calls to load libraries still work.
Native libraries in Java are loaded by calling System.loadLibrary. This
method inspects paths in the java.library.path to find the requested
library. Elasticsearch previously used this to find libsystemd, but now
the only remaining use is to set the additional platform directory in
which Elasticsearch keeps its own native libraries.
One issue with setting java.library.path is that its not set for the cli
process, which makes loading the native library infrastructure from clis
difficult. This commit reworks how Elasticsearch native libraries are
found in order to avoid needing to set java.library.path. There are two
cases. The simplest is production, where the working directory is the
Elasticsearch installation directory, so the platform specific directory
can be constructed. The second case is for tests where we don't have an
installtion. We already pass in java.library.path there, so this change
renames the system property to be a test specific property that the new
loading infrastructure looks for.
Systemd notification now happens by directly communicating with the
systemd socket. This commit removes the native access to libsystemd,
which is no longer used.
The libsystemd library function sd_notify is just a thin wrapper
around opeing and writing to a unix filesystem socket. This commit
replaces using libsystemd with opening the socket provided by systemd
directly.
relates #86475
Memory locking on Windows with the bundled jdk was broken by native
access refactoring. This commit fixes the linking issue, as well as adds
a packaging test to ensure memory locking is invoked on all supported
platforms.
The VectorSystemPropertyTests need to run a child process. Normally this
isn't possible since we run with security manager, but the
`@WithoutSecurityManager` annotation causes the test suite to run with
the security manager disabled. However, that annotation only works with
ESTestCase. This commit changes the base class of the test suite to
ESTestCase.
closes#110949
The preallocate tests assumed that preallocation was using the fallback
implementation which calls setLength on Windows. However, that fallback
only happens inside the SharedBytes class, so windows doesn't actually
do anything when tryPreallocate is called. This commit skips the test on
windows.
closes#110948
When JNA loads libraries it creates a proxy object for the library.
Unfortunately it doesn't actually inspect any of the methods, those get
bound lazily at runtime when the method is called through the proxy. For
fstat64 we need to know at load time whether the symbol exists, so that
we can fallback to an alternate function if it doesn't.
This commit looks up the NativeLibrary object from JNA for libc and
checks if fstat64 exists during load time.
This commit moves the file preallocation functionality into
NativeAccess. The code is basically the same. One small tweak is that
instead of breaking Java access boundaries in order to get an open file
handle, the new code uses posix open directly.
relates #104876
This commit moves the system call filtering initialization into
NativeAccess. The code is essentially unmodified from its existing
state, now existing within the *NativeAccess implementations.
relates #104876
* Add vec_caps and inner implementation for AVX-512-F (without VNNI)
* select FNNI function name based on vec_caps; native templated implementation for manual unrolling
* Switched compiler to clang for x64, as gcc has a bug
This commit extends the custom SIMD optimized SQ vector scorer to include search time scoring.
When run on JDK22+ vector scoring with be done with the custom scorer. The implementation uses the JDK 22+ on-heap ALLOW_HEAP_ACCESS Linker.Option so that the native code can access the query vector directly.
Elasticsearch uses a couple windows specific functions, specifically
gettting a short path, and registering a console control handler for
shutdown notification. This commit moves this functionality from the
existing jna natives into NativeAccess.
relates #104876
When scanning the library path we may come across directories that are
unreadable. If that happens, the recursive walk of the library path
directories will throw a fatal IOException. This commit guards the walk
of the library paths to first check for readability of each directory we
are about to traverse.
Linux systems with multiarch (e.g. i386 & x86_64) libraries
may have libsystemd.0 in two subdirectories of an entry in
java.library.path. For example, libsystemd.so.0 may be found
in both /usr/lib/i386-linux-gnu and /usr/lib/x86_64-linux-gnu.
Instead of attempting to load any library found, attempt all
and stop as soon as one is successfully loaded.
This commit updates the native vector provider to reflect that Lucene's scalar quantization is unsigned int7, with a range of values from 0 to 127 inclusive. Stride has been pushed down into native, to allow other platforms to more easily select there own stride length.
Previously the implementation supports signed int8. We might want the more general signed int8 implementation in the future, but for now unsigned int7 is sufficient, and allows to provide more efficient implementations on x64.
JNA supports two types of mapping to native methods, proxying and direct
method mapping. Proxying is nicer for unit testing, but unfortunately
the proxied methods are lazily loaded. NativeAccess expects that methods
are linked during static init, before SecurityManager is initialized.
For any native methods called after security manager init, the proxied
method will fail.
This commit changes the zstd bindings to use direct method mapping so
that calling zstd methods does not fail when using JNA (pre Java 21).
closes#107504closes#107770
This commit adds an optimised int8 vector distance implementation for aarch64. Additional platforms like, say, x64, will be added as a follow-up.
The vector distance implementation outperforms Lucene's Pamana Vector implementation for binary comparisons by approx 5x (depending on the number of dimensions). It does so by means of compiler intrinsics built into a separate native library and link by Panama's FFI. Comparisons are performed on off-heap mmap'ed vector data.
The implementation is currently only used during merging of scalar quantized segments, through a custom format ES814HnswScalarQuantizedVectorsFormat, but its usage will likely be expanded over time.
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Mark Vieira <portugee@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
CloseableByteBuffer is backed by native memory segments, but the
interfaces for compress and decompress methods of zstd take ByteBuffer.
Although both Jna and the Jdk can deal with turning the native
ByteBuffer back into an address to pass to the native method, the jdk
may have a more significant cost to that action.
This commit changes the signature of compress and decompress to take in
CloseableByteBuffer so that each implementation can do its own
unwrapping to get the appropriate native address.
relates #103374
The jdk implementation of CloseableByteBuffer currently uses a shared
arena. The assumption was that a buffer might be shared across threads.
However, in practice for compression/decompression that is not true, and
the shared arena has a noticeable impact on deallocation when the buffer
is closed. This commit switches to a confined arena, limtting buffer
creation and compress/decompress calls to a single thread.
relates #103374
Since mrjars may use preview apis, forbidden apis must know about any
preview apis from the jdk. However, we do not run forbidden apis with
the preview enabled flag, nor in a separate jvm, so it does not know
about these classes. Thus we ignore missing classes on source sets added
by the mrjar plugin.
This commit configures all sourcesets added by mrjar plugin to ignore
forbidden apis missing classes.
The distributions already have correct permissions set on native
libraries copied to them. However, the build itself to extract the
native libs relies on the upstream file permissions. This commit sets
explicit permissions on the copy task which extracts native libraries.
Similar to https://github.com/elastic/elasticsearch/pull/106360, the
methods for allocating a native string changed between Java 21 and 22.
This commit adds another util method to handle the differences and uses
it in the jdk systemd impl.
The foreign memory API changed between Java 21 and 22 in how to decode a
string from native memory. This commit adds an multi-release class to
handle the two different methods on MemorySegment to decode a string.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
Sometimes libs have subprojects that may not be java projects. This commit adjusts the shared
configuration for libs to only affect direct subprojects of :lib.
Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
