* Guard systemd library lookup from unreadable directories (#108931)
When scanning the library path we may come across directories that are
unreadable. If that happens, the recursive walk of the library path
directories will throw a fatal IOException. This commit guards the walk
of the library paths to first check for readability of each directory we
are about to traverse.
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Offsets in memory segments should be computed as longs to avoid integer overflow on large segments.
---------
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Linux systems with multiarch (e.g. i386 & x86_64) libraries
may have libsystemd.0 in two subdirectories of an entry in
java.library.path. For example, libsystemd.so.0 may be found
in both /usr/lib/i386-linux-gnu and /usr/lib/x86_64-linux-gnu.
Instead of attempting to load any library found, attempt all
and stop as soon as one is successfully loaded.
JNA supports two types of mapping to native methods, proxying and direct
method mapping. Proxying is nicer for unit testing, but unfortunately
the proxied methods are lazily loaded. NativeAccess expects that methods
are linked during static init, before SecurityManager is initialized.
For any native methods called after security manager init, the proxied
method will fail.
This commit changes the zstd bindings to use direct method mapping so
that calling zstd methods does not fail when using JNA (pre Java 21).
closes#107504closes#107770
This commit adds an optimised int8 vector distance implementation for aarch64. Additional platforms like, say, x64, will be added as a follow-up.
The vector distance implementation outperforms Lucene's Pamana Vector implementation for binary comparisons by approx 5x (depending on the number of dimensions). It does so by means of compiler intrinsics built into a separate native library and link by Panama's FFI. Comparisons are performed on off-heap mmap'ed vector data.
The implementation is currently only used during merging of scalar quantized segments, through a custom format ES814HnswScalarQuantizedVectorsFormat, but its usage will likely be expanded over time.
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Mark Vieira <portugee@gmail.com>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
CloseableByteBuffer is backed by native memory segments, but the
interfaces for compress and decompress methods of zstd take ByteBuffer.
Although both Jna and the Jdk can deal with turning the native
ByteBuffer back into an address to pass to the native method, the jdk
may have a more significant cost to that action.
This commit changes the signature of compress and decompress to take in
CloseableByteBuffer so that each implementation can do its own
unwrapping to get the appropriate native address.
relates #103374
The jdk implementation of CloseableByteBuffer currently uses a shared
arena. The assumption was that a buffer might be shared across threads.
However, in practice for compression/decompression that is not true, and
the shared arena has a noticeable impact on deallocation when the buffer
is closed. This commit switches to a confined arena, limtting buffer
creation and compress/decompress calls to a single thread.
relates #103374
Since mrjars may use preview apis, forbidden apis must know about any
preview apis from the jdk. However, we do not run forbidden apis with
the preview enabled flag, nor in a separate jvm, so it does not know
about these classes. Thus we ignore missing classes on source sets added
by the mrjar plugin.
This commit configures all sourcesets added by mrjar plugin to ignore
forbidden apis missing classes.
The distributions already have correct permissions set on native
libraries copied to them. However, the build itself to extract the
native libs relies on the upstream file permissions. This commit sets
explicit permissions on the copy task which extracts native libraries.
Similar to https://github.com/elastic/elasticsearch/pull/106360, the
methods for allocating a native string changed between Java 21 and 22.
This commit adds another util method to handle the differences and uses
it in the jdk systemd impl.
The foreign memory API changed between Java 21 and 22 in how to decode a
string from native memory. This commit adds an multi-release class to
handle the two different methods on MemorySegment to decode a string.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
With this commit we use `writeRawValue` instead of `writeRaw` when
serializing raw strings as XContent. The latter method does not consider
context (e.g. is the value being written as part of an array and
requires a comma separator?) whereas the former does. This ensures that
pre-rendered double values as we use them in the flamegraph response are
rendered correctly as XContent.
Closes#106103
Sometimes libs have subprojects that may not be java projects. This commit adjusts the shared
configuration for libs to only affect direct subprojects of :lib.
Libs were meant to be a way to break up code from server without
creating full fledged modules. They still exist on the system classpath,
but we did not want to introduce a spaghetti of jars depending on each
other. The check that ensures libs don't depend on each other was added
before Elasticsearch was modularized. Since it now runs modular, the
cross module dependencies are easy to visualize with module-info, and
the module system protects us from circular deps. Additionally, the
number of exceptions to the no-cross-lib-deps rule has grown
considerably.
Given all of the above, the check on cross lib dependencies no longer
provides much benefit, and is more of a hinderance. This commit removes
the check.
The response of the flamegraph is quite large: A typical response can
easily reach 50MB (uncompressed). In order to reduce memory pressure and
also to start sending the response sooner, we chunk the response.
However, this leads to many chunks that are very small and lead to high
overhead. In our experiments, just the serialization takes more than
500ms.
With this commit we take the following measures:
1. We split the response into chunks only when it makes sense and
otherwise send one larger chunk.
2. Serialization of doubles is very expensive: Just the serialization of
annual CO2 tons takes around 80ms in our test setup. Therefore, we
apply a custom serialization that is both faster than the builtin
serialization as well reduces the amount of bytes sent over the wire
because we round to four decimal places (which is more than sufficient for
our purposes).
Just a suggetion. I think this would save us a bit of memory here and
there. We have loads of places where the always true lambdas are used
with `Predicate.or/and`. Found this initially when looking into field
caps performance where we used to heavily compose these but many spots
in security and index name resolution gain from these predicates.
The better toString also helps in some cases at least when debugging.
* Use String.replace() instead of replaceAll() for non-regexp replacements
When arguments do not make use of regexp features replace() is a more efficient option, especially the char-variant.
Introducing a plain version of `AbstractRefCounted` as a compromise.
This saves a bunch of allocations and a circular reference to the object
holding the ref counted instance, making smaller SearchHit instances
etc. cheaper. We could get an even more direct solution here by making
these extend `AbstractRefCounted` but that would lose us the ability to
leak-track in tests, so doing it this way (same way Netty does it on
their end) as a compromise.
This is mainly added as a prerequisite to slicing doc sources out of
bulk index requests. The few usages for it added in this PR have limited
performance impact but demonstrate correct functioning of the
implementations.
Co-authored-by: David Turner <david.turner@elastic.co>
Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
Preallocate opens a FileInputStream in order to get a native file
desctiptor to pass to native functions. However, getting at the file
descriptor requires breaking modular access. This commit adds native
posix functions for opening/closing and retrieving stats on a file in
order to avoid requiring additional permissions.
Qualfied exports in the boot layer only work when they are to other boot
modules. Yet Elasticsearch has dynamically loaded modules as in plugins.
For this purpose we have ModuleQualifiedExportsService. This commit
moves loading of ModuleQualfiedExportService instances in the boot layer
into core so that it can be reused by ProviderLocator when a qualified
export applies to an embedded module.
There's no need for this helper to take more than one argument. Almost
all the usages only passed in a single argument, and the few cases that
supplied more than one can be rewritten as a single argument to save
allocating all those extra lambdas.
x-content embeds its jackson implementation inside its jar. This commit
formalizes the setup for this embedding with a gradle plugin so that it
can be reused by other libs.
It's less code and it actually inlines (avoiding virtual calls in most
cases) to just do the null check here instead of delegating to IOUtils
and then catching the impossible IOException. Also, no need to use
`Releaseables` in 2 spots where try-with-resources works as well and
needs less code.
Noticed this when I saw that we had a lot of strange CPU overhead in
this call in some hot loops like translog writes.
Our readEnum code instantiates/clones enum value arrays on read.
Normally, this doesn't matter much but the two spots adjusted here are
visibly hot during bulk indexing, causing GBs of allocations during e.g.
the http_logs indexing run.
This commit adds an elasticsearch gradle plugin which sets up a project
to build an MRJAR. By applying the plugin, the src dir is checked for
any directories matching mainXX where XX is the java version number.
A source set is automatically setup, an appropriate compiler tied to
that source set, and it's output placed in the correct part of the final
jar. Additionally, the sourceset allows use of preview features in that
verison of Java, and the preview bits are stripped from the resulting
class files.
* Add extract match ranges functionality to Grok.
* TestGrokPatternAction and Request
* TestGrokPattern response
* Update docs/changelog/104394.yaml
* Polish validation error message
* Improve test_grok_pattern API
* Add explicit CharSet
* Add endpoint to operator constants
* Add TransportTestGrokPatternActionTests
* REST API spec
* One more TransportTestGrokPatternActionTest
* Fix API spec
* Refactor REST API spec
* Polish code
* Replace TransportTestGrokPatternActionTests by a YAML REST test
* Add ecs_compatibility
* Always return arrays in the API
* Documentation
* YAML test for ecs_compatibility
* Rename doc fileø
* serverless scope
* Fix docs (hopefully)
* Update docs/reference/rest-api/index.asciidoc
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Add "text structure APIs" header in docs TOC
* Move file
* Remove test grok from main index
* typo
* Nested APIs underneath text structure
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Remove the rough limit on string length from Jackson 2.15. The limit was already relaxed for JSON in #96031, this extends that change to other XContent types.
Refs: #96031Fixes: #104009
This is needed for the search response pooling work. Also, the one usage
in `ReleasableBytesReference` actually makes outright sense. We
shouldn't be ref-counting on a global constant, that just needlessly
introduces contention that isn't entirely obvious. This change required
a couple tests to be adjusted where we were checking release mechanics
on noop instances.
Today we `decRef()` the per-node responses just after adding them to the
`responses` collection, but in fact we should keep them alive until
we've constructed the final response. This commit does that.
JDK 22 may return a console even if the terminal is redirected. These cases are detected using the new Console#isTerminal() to maintain the current behavior (closes#98033).
We're leaking quite a few of these parsers. That doesn't seem to be much
of a problem but results in some memory inefficiencies in Jackson here
and there. This PR bulk fixes a bunch of instances that I could easily
automatically fix. I'll open a follow-up for closing the parser on the
document parsing context which also suffers from this but is non-trivial
to fix.
In several places we acquire a ref to a resource that we are certain is
not closed, so this commit adds a utility for asserting this to be the
case. This also helps a little with mocks since boolean methods like
`tryIncRef()` return `false` on mock objects by default, but void
methods like `mustIncRef()` default to being a no-op.
This commit upgrades the Bouncy Castle jars. Bouncy Castle is used for
some internal build concners as well as a comnand line application.
Most notably Bouncy Castle is also used as the FIPs certified JCE/JSEE provider
we use to test our ability to use a FIPs compliant crypto provider.
The following changes here are a result of the upgraded Bouncy Castle jars:
* TLSv1.3 is now supported when running in FIPs mode
* RSA PKCS#1 v1.5 is no longer allowed in FIPS mode
* Triple DES (3DES) is no longer allowed in FIPS mode
* Minor updates the security manager configuration used to test FIPs (to read permissions from the security provider)
* Minor adjustments to tests to accommodate the above changes.
* Minor adjustments to the gradle build to accommodate new dependencies
Note - update to the documentation will come in a later commit.
This allows to replace deep copying of blocks by simply calling
Block::incRef - the block then has to be closed (or decRefed) one
additional time for each call to incRef (and tryIncRef, if successfull).
