On x64, we are testing if we support vector capabilities (1 = "basic" = AVX2, 2 = "advanced" = AVX-512) in order to enable and choose a native implementation for some vector functions, using CPUID.
However, under some circumstances, this is not sufficient: the OS on which we are running also needs to support AVX/AVX2 etc; basically, it needs to acknowledge it knows about the additional register and that it is able to handle them e.g. in context switches. To do that we need to a) test if the CPU has xsave feature and b) use the xgetbv to test if the OS set it (declaring it supports AVX/AVX2/etc).
In most cases this is not needed, as all modern OSes do that, but for some virtualized situations (hypervisors, emulators, etc.) all the component along the chain must support it, and in some cases this is not a given.
This PR introduces a change to the x64 version of vec_caps to check for OS support too, and a warning on the Java side in case the CPU supports vector capabilities but those are not enabled at OS level.
Tested by passing noxsave to my linux box kernel boot options, and ensuring that the avx flags "disappear" from /proc/cpuinfo, and we fall back to the "no native vector" case.
Fixes#126809
In Jackson 2.15 a maximum string length of 50k characters was
introduced. We worked around that by override the length to max int on
all parsers created by xcontent. Jackson 2.17 introduced a similar limit
on field names. This commit mimics the workaround for string length by
overriding the max name length to be unlimited.
relates #58952
These entitlements should not be necessary. Server does not directly
create any outbound connections, and netty is not a dependency of server
so it should not exist within the server policy.
This commit cleans up how entitlement test methods are discovered. It
also adds another robustness check to ensure an annotation doesn't exist
on a private method.
In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
This PR expands the README for the entitlements package; it adds information for Elasticsearch developers on how the system is structured, how to build policies, what to do when new code "does not work" because it's bumping into a NotEntitledException, and how to apply a policy patch in case of an emergency.
This PR fixes 2 issues discovered around PolicyUtils (and PolicyUtilsTests) when running CI on Windows:
- in tests, absolute paths like always are different, this fix updates the tests to account for the difference.
- on Windows, Files.move was failing because we were parsing the Entitlement policy but not closing the stream in a timely manner. This causes plugin installation (and related CI tests) to fail 70% of the time. Fixed by closing the stream properly
Fixes#126176
This PR replaces the parsing and formatting of SecurityManager policies with the parsing and formatting of Entitlements policy during plugin installation.
Relates to ES-10923
The transform API for instrumentation should return null when no
transformation occurs. This commit fixes our entitlement transformer to
return null instead of the original buffer.
closes#125920
* Use ALL-UNNAMED for the unnamed module in warnings
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Using ASM CheckClassAdapter was key to diagnose the issue we had with incorrect signatures for some check methods.
In this PR I polished up the code I used to pinpoint the issue, and made it available via a system property so it can be turned on if we need it (and it's always on for Entitlements IT tests too).
It is also turned on in case we get VerifyErrors during retransformClasses early in the Entitlement agent bootstrap: retransformClasses runs in the native part of the JVM, so the VerifyError it produces is not so readable (e.g. it lacks a full stack trace and a description); in case this happens, we re-apply the transformation with verification turned on to get a meaningful error before dying.
With #124904 we introduced server policy patching via system properties; now that we have the possibility to merge policies and scopes, it's worth applying the same mechanism to plugin policies too.
This PR changes the behaviour of plugin policy overrides; now they are not replacing the policy entirely, but patch it by adding the entitlements and scopes specified in the "patch policy"
* Use child loggers broadly in PolicyManager
* Pass ModuleEntitlements to notEntitled
* Store logger name instead of object.
Some of our unit tests check for equality of `ModuleEntitlements` objects,
and they are entitled to do so (no pun intended).
* Alright, let's cache 'em
* Memoize at point of creation, not point of use
* Explanatory comments
The ingest-attachment module uses Tika to parse some content; Tika in turn uses some libraries from java.desktop to perform its tasks.
In turn, the JDK loads one (or more) native libraries for its implementation as part of class initialization. This means we need to grant load_native_libraries to java.desktop so that because AWT can load libraries for itself.
This PR adds the ability to override entitlement policies for Elasticsearch plugins and modules via a system property.
The system property is in the form es.entitlements.policy.<plugin name>, and accepts a versioned policy:
versions:
- version1
- versionN
policy:
<a standard entitlement policy>
For example:
versions:
- 9.1.0
policy:
ALL-UNNAMED:
- set_https_connection_properties
- outbound_network
- files:
- relative_path: ".config/gcloud"
relative_to: home
mode: read
The versioned policy needs to be base64 encoded. For example, to pass the above policy to a test cluster via gradle run:
./gradlew run --debug-jvm -Dtests.jvm.argline="-Des.entitlements.policy.repository-gcs=dmVyc2lvbnM6CiAgLSA5LjEuMApwb2xpY3k6CiAgQUxMLVVOTkFNRUQ6CiAgICAtIHNldF9odHRwc19jb25uZWN0aW9uX3Byb3BlcnRpZXMKICAgIC0gb3V0Ym91bmRfbmV0d29yawogICAgLSBmaWxlczoKICAgICAgLSByZWxhdGl2ZV9wYXRoOiAiLmNvbmZpZy9nY2xvdWQiCiAgICAgICAgcmVsYXRpdmVfdG86IGhvbWUKICAgICAgICBtb2RlOiByZWFkCg=="
The versions listed in the policy are string-matched against Build.version().current(); it is possible to specify any number of versions. If the list is empty/there is no versions field, the policy is assumed to match any Elasticsearch versions.
The override policy specified for any given plugin replaces the embedded policy for that plugin. See how EntitlementsAllowedViaOverrideIT replaces an empty policy for the entitlement-test-plugin with a policy that allows load_native_libraries and access to files in the test read_dir.
Also tested manually with an override with a different version, with an override with an invalid policy and with a valid override (see command above).
Relates to ES-11009
When enabling FIPS `javax.net.ssl.trustStore` is not necessarily set.
This change adds FIPS entitlements based on
`org.bouncycastle.fips.approved_only=true`, which enforces usage of FIPS
approved functionality only.
Additionally, this PR grants read access to a custom trust store if
provided via `javax.net.ssl.trustStore`, otherwise read access to the
default JDK trust store is granted.
Relates to ES-11025.
Writing tests for #123861, turns out that #124195 is not enough.
We really need new IT test cases for "always allowed" actions: in order to be sure they are allowed, we need to setup the plugin with no policy.
This PR adds test cases for that, plus the support for writing test functions that accept one Environment parameter: many test paths we test and allow/deny are relative to paths in Environment, so it's useful to have access to it (see readAccessConfigDirectory as an example)
* Refactor: findRequestingFrame
* INFO instead of WARN for NotEntitledException.
Some of these are expected, so an INFO seems more appropriate.
The stack trace tends to attract attention even when entitlements are not the
cause of a problem, so let's avoid the stack trace, but still include stack
frame info from the frame of interest.
* Use child loggers for Not Entitled logs
* Use warn, and include compoenent name
* Fix ALL_UNNAMED
* Mute entitlement warnings from repositories
* PR feedback
* Common out the Not Entitled prefix.
We're alerting on this, so let's not rely on every caller of notEntitled to remember it.
This will rethrow the `NoSuchFileException` when encountering an invalid
symbolic link when following links during file (read) entitlement
checks.
Relates to https://github.com/elastic/elasticsearch/pull/124133
(ES-11019)
Even if the contract for JDK methods using the SecurityManager states that the exception throw is of type SecurityException, many libraries (including our own, apparently!) violates that and use the type actually thrown by SecurityManager, AccessControlException. A prime example is the GCS/CSP libraries.
In order to maintain compatibility for them, we need to inherit from the more specific AccessControlException; this is less desirable, as AccessControlException is marked as deprecated for removal alongside the other SecurityManager classes, but we discussed and found this is the best short term solution.
More work will be needed -- again, this is a short term solution.
Replaces #123984
Our check methods injected by the instrumenter receive "this" as the second parameter.
For internal classes like AbstractDelegateHttpsURLConnection we generally use a base type; in this case we were using javax.net.ssl.HttpsURLConnection, which is incorrect as AbstractDelegateHttpsURLConnection derives from java.net.HttpURLConnection.
This was not failing in our tests because we don't actually use that parameter in that check function.
Also, it was not failing on transform, just on retransformClasses, and only in JDK 24. Apparently, JDK 24 introduced new validation there (to be confirmed).
And it was failing just on cloud as the APM agent there (which is loaded before our agent) connects to a https endpoint - our IT tests, and ./gradlew run --with-apm-server, use a http endpoint. Using https makes the JVM load AbstractDelegateHttpsURLConnection, making it one of the classes we need to retransform, triggering the VerifyError.
Makes the execution and use of enrich policies project-aware.
Note: this does not make the enrich cache project-aware. That is to be
handled in a follow-up PR.
