This commit updates the low-level bounds checks in JDKVectorLibrary and add benchmark, so that we can more easily bench the low-level operations.
Note: I added the mr-jar gradle plugin to the benchmarks so that we can compile with preview features in Java 21, namely MemorySegment.
When the entitelement agent fails to load the underlying exception can
be cryptic. In some casees it may be that the path the agent jar is bad.
This commit expands the exception message to show the agent path
that we tried to load.
Java class transformers swallow exceptions, so any instrumentation
failures, for example due to a java version mismatch, will silently
proceed with startup, which then will cryptically fail the entitlement
self test. This commit logs exceptions that occur during
instrumentation, as well as plumb through the fact that any occured so
that bootstrap can fail rather than allow startup to proceed.
Keep better track of shard contexts using RefCounted, so they can be released more aggressively during operator processing. For example, during TopN, we can potentially release some contexts if they don't pass the limit filter.
This is done in preparation of TopN fetch optimization, which will delay the fetching of additional columns to the data node coordinator, instead of doing it in each individual worker, thereby reducing IO. Since the node coordinator would need to maintain the shard contexts for a potentially longer duration, it is important we try to release what we can eariler.
An even more advanced optimization is to delay fetching to the main cluster coordinator, but that would be more involved, since we need to first figure out how to transport the shard contexts between nodes.
Summary of main changes:
DocVector now maintains a RefCounted instance per shard.
Things which can build or release DocVectors (e.g., LuceneSourceOperator, TopNOperator), can also hold RefCounted instances, so they can pass them to DocVector and also ensure contexts aren't released if they can still be potentially used later.
Driver's main loop iteration (runSingleLoopIteration), now closes its operators even between different operator processing. This is extra aggressive, and was mostly done to improve testability.
Added a couple of tests to TopNOperator and a new integration test EsqlTopNShardManagementIT, which uses the pausable plugin framework to check that TopNOperator releases things as early as possible..
Some features are unavailable in serverless and are thus not worth the
investment to make fully project-aware. This new annotation can be used
to clearly mark blocks of code that are intentionally not made properly
project-aware, in case we need to revisit them in the future.
Using a static `diff` or epsilon just doesn't work for this test as the
scores can be very large, but relatively close.
Maybe there is a simpler way, but my mind wasn't wanting to "math" very
much.
For example, the seed that this previously failed on had scores like
`1.726524E9` and `1.7265239E9`, which, given their size, are really
close together (within 128). But a static epsilon wouldn't capture that.
closes: https://github.com/elastic/elasticsearch/issues/128485
Follow-up to #126492 to apply the json parsing optimization to strings
containing unicode characters and some backslash-escaped characters.
Supporting backslash-escaped strings is tricky as it requires modifying the
string. There are two types of modification: some just remove the backslash
(e.g. \", \\), and some replace the whole escape sequence with a new
character (e.g. \n, \r, \u00e5). In this implementation, the optimization
only supports the first case--removing the backslash. This is done by
making a copy of the data, skipping the backslash. It should still be more
optimized than full String decoding, but it won't be as fast as
non-backslashed strings where we can directly reference the input bytes.
Relates to #129072.
* Support multiple plugin source paths
* Refactor: remove unncessary PathLookup method.
It's only called in one place, and there's no need to override it for testing.
Removing it just makes things simpler.
* Refactor: local var for pathLookup
* Fix bugs in test build info parsing
* Fix representative_class in test
* Move BridgeUtilTests.
Tests in org.elasticsearch.entitlement.bridge are going to be uniquely hard to
test once we patch the bridge into java.base, due to Java's prohibition on
split packages.
Let's just move this guy to another package.
* Upcast (?!) Java23EntitlementChecker to EntitlementChecker
* Empty TestPathLookup
* Create PolicyManager during bootstrap, allowing us to share initialization
* Use empty component path list instead of null
* Downcast to the class of the check method.
In our unit test, we have a mock checker that doesn't extend
EntitlementChecker, so downcasting to that would require us to needlessly
rework the unit test.
* Fix javadoc typos
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
When parsing documents, we receive the document as UTF-8 encoded data which
we then parse and convert the fields to java-native UTF-16 encoded Strings.
We then convert these strings back to UTF-8 for storage in lucene.
This patch skips the redundant conversion, instead passing lucene a
direct reference to the received UTF-8 bytes when possible.
This PR is a precursor to #126492.
It does three things:
1. Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
2. Refactor the Text class to use a new EncodedBytes record instead of
the elasticsearch BytesReference.
3. Add the XContentString interface, with the Text class implementing
that interface.
These changes were originally implemented in #127666 and #128316,
however they were reverted in #128484 due to problems caused by the
mutable nature of java ByteBuffers. This is resolved by instead using a
new immutable EncodedBytes record.
Instead of waiting for the next run of the `ClusterStateObserver` (which
might be arbitrarily far in the future, but bound by the timeout if one
is set), we notify the listener immediately that the task has been
cancelled. While doing so, we ensure we invoke the listener only once.
Fixes#117971
Entitlement instrumentation works by reflectively calling back into the
entitlements lib to grab the checker. It must be fully in place before
any classes are instrumented. This commit fixes a bug that was
introduced by refactoring which caused the checker to not be set until
after all classes were instrumented. In some situations this could lead
the checker to being null when it is grab (and statically cached) by the
entitlement bridge.
* Rename and encapsulate InitializeArgs
* Move ElasticsearchEntitlementChecker out of api package.
It's an implementation detail that doesn't need to be exposed to the rest of
the system.
* Stub TestPathLookup (not yet implemented)
* Revert "Fix the Text class package change in example plugins (#128316)"
This reverts commit cc486480e3.
* Revert "Update Text class to use native java ByteBuffer (#127666)"
This reverts commit db0c3c7a28.
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
* Use package to suppress warning for entitlement self-test
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Entitlements do a stack walk to find the calling class. When method
refences are used in a lambda, the frame ends up hidden in the stack
walk. In the case of using a method reference with
AccessController.doPrivileged, the call looks like it is the jdk itself,
so the call is trivially allowed. This commit adds hidden frames to the
stack walk so that the lambda frame created for the method reference is
included. Several internal packages are then necessary to filter out of
the stack.
This PR is a precursor to #126492.
It does three things:
- Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
- Refactor the Text class to use a java-native ByteBuffer instead
of the elasticsearch BytesReference.
- Add the XContentString interface, with the Text class implementing
that interface.
While backporting entitlement initialization refactorings, I realized there is a mismatch in getVersionSpecificCheckerClass signature, and also that this function in the backports is used in more places (DynamicInstrumentation), making it "strange" to have this in EntitlementInitialization. This PR extracts the function to a separate static class (package-private) and makes the signature uniform with backports.
This will need to be backported manually to the 8.x branches, and will make the backported version of DynamicInstrumentation cleaner.
Moves FilesEntitlements validation to a separate class. This is the final PR to make EntitlementsInitialization a simpler "orchestrator" of the various steps in the initialization phase.
Our path comparison for file access is string based, due to the fact that we need to support Paths created for different file systems/platforms.
However, Windows files and paths are (sort of) case insensitive.
This PR fixes the problem by abstracting String comparison operations and making them case sensitive or not based on the host OS.
