This change refactors the known directory resolution such as modules,
plugins, lib, etc. into a PathLookup. This is one of the steps towards
allowing unit tests to provide their own PathLookup for resolution so
we can enable entitlements there.
ES-11584
We already had logging in FileAccessTree as result of debugging the \\pipe\ failures a while ago; this PR slightly improves the logs to provide more information.
Design and implementation of Entitlement with this level of detail needs to stay close to the code, and take advantage of javadoc features like linking and class-references to help us with refactorings and future code changes.
The bulk of the information went into the package-info file for the main library, but I split up some parts and referenced them from the main doc, where I thought it made sense (mainly: the bridge sub-project for some implementation details, PolicyManager, EntitlementInitialization and FileAccessTree); this way they still can be reached from the "overview" while being closer to where the information really belongs.
Relates to ES-11284
These entitlements should not be necessary. Server does not directly
create any outbound connections, and netty is not a dependency of server
so it should not exist within the server policy.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
With this PR we restrict the paths we allow access to, forbidding plugins to specify/request entitlements for reading or writing to specific protected directories.
I added this validation to EntitlementInitialization, as I wanted to fail fast and this is the earliest occurrence where we have all we need: PathLookup to resolve relative paths, policies (for plugins, server, agents) and the Paths for the specific directories we want to protect.
Relates to ES-10918
This commit cleans up how entitlement test methods are discovered. It
also adds another robustness check to ensure an annotation doesn't exist
on a private method.
This PR expands the README for the entitlements package; it adds information for Elasticsearch developers on how the system is structured, how to build policies, what to do when new code "does not work" because it's bumping into a NotEntitledException, and how to apply a policy patch in case of an emergency.
This PR fixes 2 issues discovered around PolicyUtils (and PolicyUtilsTests) when running CI on Windows:
- in tests, absolute paths like always are different, this fix updates the tests to account for the difference.
- on Windows, Files.move was failing because we were parsing the Entitlement policy but not closing the stream in a timely manner. This causes plugin installation (and related CI tests) to fail 70% of the time. Fixed by closing the stream properly
Fixes#126176
This PR replaces the parsing and formatting of SecurityManager policies with the parsing and formatting of Entitlements policy during plugin installation.
Relates to ES-10923
The transform API for instrumentation should return null when no
transformation occurs. This commit fixes our entitlement transformer to
return null instead of the original buffer.
closes#125920
* Move remaining check methods in RestEntitlementsCheckAction (#125351)
All future check methods should be found reflectively with the
EntitlementTest annotation.
* Fix compilation after #125442
Using ASM CheckClassAdapter was key to diagnose the issue we had with incorrect signatures for some check methods.
In this PR I polished up the code I used to pinpoint the issue, and made it available via a system property so it can be turned on if we need it (and it's always on for Entitlements IT tests too).
It is also turned on in case we get VerifyErrors during retransformClasses early in the Entitlement agent bootstrap: retransformClasses runs in the native part of the JVM, so the VerifyError it produces is not so readable (e.g. it lacks a full stack trace and a description); in case this happens, we re-apply the transformation with verification turned on to get a meaningful error before dying.
* Use ALL-UNNAMED for the unnamed module in warnings
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
With #124904 we introduced server policy patching via system properties; now that we have the possibility to merge policies and scopes, it's worth applying the same mechanism to plugin policies too.
This PR changes the behaviour of plugin policy overrides; now they are not replacing the policy entirely, but patch it by adding the entitlements and scopes specified in the "patch policy"
* [Entitlelments] Server policy patching via system property (#124904)
This PR adds a mechanism to patch the server layer policy by merging entitlements from a policy provided via a system property (merge, not replace).
* change to support 8.x language level
* Use child loggers broadly in PolicyManager
* Pass ModuleEntitlements to notEntitled
* Store logger name instead of object.
Some of our unit tests check for equality of `ModuleEntitlements` objects,
and they are entitled to do so (no pun intended).
* Alright, let's cache 'em
* Memoize at point of creation, not point of use
* Explanatory comments
The ingest-attachment module uses Tika to parse some content; Tika in turn uses some libraries from java.desktop to perform its tasks.
In turn, the JDK loads one (or more) native libraries for its implementation as part of class initialization. This means we need to grant load_native_libraries to java.desktop so that because AWT can load libraries for itself.
This PR adds the ability to override entitlement policies for Elasticsearch plugins and modules via a system property.
The system property is in the form es.entitlements.policy.<plugin name>, and accepts a versioned policy:
versions:
- version1
- versionN
policy:
<a standard entitlement policy>
For example:
versions:
- 9.1.0
policy:
ALL-UNNAMED:
- set_https_connection_properties
- outbound_network
- files:
- relative_path: ".config/gcloud"
relative_to: home
mode: read
The versioned policy needs to be base64 encoded. For example, to pass the above policy to a test cluster via gradle run:
./gradlew run --debug-jvm -Dtests.jvm.argline="-Des.entitlements.policy.repository-gcs=dmVyc2lvbnM6CiAgLSA5LjEuMApwb2xpY3k6CiAgQUxMLVVOTkFNRUQ6CiAgICAtIHNldF9odHRwc19jb25uZWN0aW9uX3Byb3BlcnRpZXMKICAgIC0gb3V0Ym91bmRfbmV0d29yawogICAgLSBmaWxlczoKICAgICAgLSByZWxhdGl2ZV9wYXRoOiAiLmNvbmZpZy9nY2xvdWQiCiAgICAgICAgcmVsYXRpdmVfdG86IGhvbWUKICAgICAgICBtb2RlOiByZWFkCg=="
The versions listed in the policy are string-matched against Build.version().current(); it is possible to specify any number of versions. If the list is empty/there is no versions field, the policy is assumed to match any Elasticsearch versions.
The override policy specified for any given plugin replaces the embedded policy for that plugin. See how EntitlementsAllowedViaOverrideIT replaces an empty policy for the entitlement-test-plugin with a policy that allows load_native_libraries and access to files in the test read_dir.
Also tested manually with an override with a different version, with an override with an invalid policy and with a valid override (see command above).
Relates to ES-11009
Writing tests for #123861, turns out that #124195 is not enough.
We really need new IT test cases for "always allowed" actions: in order to be sure they are allowed, we need to setup the plugin with no policy.
This PR adds test cases for that, plus the support for writing test functions that accept one Environment parameter: many test paths we test and allow/deny are relative to paths in Environment, so it's useful to have access to it (see readAccessConfigDirectory as an example)
When enabling FIPS `javax.net.ssl.trustStore` is not necessarily set.
This change adds FIPS entitlements based on
`org.bouncycastle.fips.approved_only=true`, which enforces usage of FIPS
approved functionality only.
Additionally, this PR grants read access to a custom trust store if
provided via `javax.net.ssl.trustStore`, otherwise read access to the
default JDK trust store is granted.
Relates to ES-11025.
* Refactor: findRequestingFrame
* INFO instead of WARN for NotEntitledException.
Some of these are expected, so an INFO seems more appropriate.
The stack trace tends to attract attention even when entitlements are not the
cause of a problem, so let's avoid the stack trace, but still include stack
frame info from the frame of interest.
* Use child loggers for Not Entitled logs
* Use warn, and include compoenent name
* Fix ALL_UNNAMED
* Mute entitlement warnings from repositories
* PR feedback
* Common out the Not Entitled prefix.
We're alerting on this, so let's not rely on every caller of notEntitled to remember it.
This will rethrow the `NoSuchFileException` when encountering an invalid
symbolic link when following links during file (read) entitlement
checks.
Relates to https://github.com/elastic/elasticsearch/pull/124133
(ES-11019)
* Add exclusive access files for security module (#123676)
This commit fills out missing entitlements for the security module.
Specifically they are config files which require exclusive access.
* Use a consistent ordering in policy manager exclulsive tests
* use better assertion
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Make NotEntitledException inherit from AccessControlException for compatibility purposes (#124321)
Even if the contract for JDK methods using the SecurityManager states that the exception throw is of type SecurityException, many libraries (including our own, apparently!) violates that and use the type actually thrown by SecurityManager, AccessControlException. A prime example is the GCS/CSP libraries.
In order to maintain compatibility for them, we need to inherit from the more specific AccessControlException; this is less desirable, as AccessControlException is marked as deprecated for removal alongside the other SecurityManager classes, but we discussed and found this is the best short term solution.
More work will be needed -- again, this is a short term solution.
Replaces #123984
* cleanup exceptions
Our check methods injected by the instrumenter receive "this" as the second parameter.
For internal classes like AbstractDelegateHttpsURLConnection we generally use a base type; in this case we were using javax.net.ssl.HttpsURLConnection, which is incorrect as AbstractDelegateHttpsURLConnection derives from java.net.HttpURLConnection.
This was not failing in our tests because we don't actually use that parameter in that check function.
Also, it was not failing on transform, just on retransformClasses, and only in JDK 24. Apparently, JDK 24 introduced new validation there (to be confirmed).
And it was failing just on cloud as the APM agent there (which is loaded before our agent) connects to a https endpoint - our IT tests, and ./gradlew run --with-apm-server, use a http endpoint. Using https makes the JVM load AbstractDelegateHttpsURLConnection, making it one of the classes we need to retransform, triggering the VerifyError.
