With this PR we restrict the paths we allow access to, forbidding plugins to specify/request entitlements for reading or writing to specific protected directories.
I added this validation to EntitlementInitialization, as I wanted to fail fast and this is the earliest occurrence where we have all we need: PathLookup to resolve relative paths, policies (for plugins, server, agents) and the Paths for the specific directories we want to protect.
Relates to ES-10918
* Rename environment dir accessors (#121803)
The node environment has many paths. The accessors for these currently use a "file" suffix, but they are always directories. This commit renames the accessors to make it clear these paths are directories.
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Instrument write access to System properties by means of the `WriteSystemPropertiesEntitlement`.
`System.setProperties(Properties)` is always denied.
Part of #ES-10359
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
(cherry picked from commit ba61f8c7f7)
Shell utility in hdfs tries to execute a local script statically to
determine whether setsid is available. With the security manager this
doesn't work, but hdfs catches the SecurityException and assumes false.
With entitlements this doesn't work since hdfs does not know about our
NotEntitledException.
This commit reworks the patching of hdfs-client-api to use asm. It then
adds patching of hdfs' Shell class to replace the method that tries to
execute.
This method is never called against a general `BlobStore`, we only use
it in certain implementations for which a bulk delete at the `BlobStore`
level makes sense. This commit removes the unused interface method.
Wholesale fix of every `TRAPPY_IMPLICIT_DEFAULT_MASTER_NODE_TIMEOUT` in
`o.e.snapshots` and `o.e.repositories`, just pulling them up to the REST
layer (where they become API params), the test suite (where they become
`TEST_REQUEST_TIMEOUT`), or some other place where an explicit value is
available.
Relates #107984
This ports our krb5kdc test fixture to test container and reworks hdfs handling to also be based on test containers.
The yaml rest tests that are using hdfs required introducing variable substitution in yamlresttestparser handling.
We have the pattern of getting a response only for its `TotalHits` in a couple of spots,
some of them still leaking the `SearchResponse`.
This PR dries up those that I could quickly identify to use the new utilty method to simplify this pattern
or just moves the code to `assertHitCount` where possible.
Reworking forbiddenApis check to use gradle worker api exposed a bug in
how we resolve krb5kdc keytab information. This fixes the depenendency to krb5kdc keytab configuration and
its builtBy task.
This also changes the usage of krb5kdc keytab files to be passed directly to task classpath as
they are only required at runtime and directly having them as part of javaRestTestRuntimeOnly would mean precommit
requires krb5kdc compose up which we definitely not want
Today many blob store repository tests specify that the operations they
perform have purpose `OperationPurpose#SNAPSHOT`, but most of these
tests do not care about the purpose of these operations. This commit
switches them to using a random purpose to highlight that the purpose is
unimportant to the test.
Same as #101175, shorten `client().prepareIndex(index)` and
`client().prepareIndex().setIndex(index)` via a test utility.
Saves lots of code now and sets up some follow-up simplifcations.
Cleaning this up a little even though it's still quite horrible.
`.get()` in this API actually means `actionGet()` so to speak.
I think a good first step to cleaning this up is to at least reduce
the duplication though and save 1k lines.
Just found that we have a lot of inconsistency and needless verbosity
here in tests. We can just use `assertAcked` in a couple spots
to save `.get`, `.actionGet` etc., especially with the signature
change I added here.
Today blobstore stats are collected against each HTTP operation, e.g.
Get, List. This is not granular enough because the same HTTP operration
can be performed for different purposes, e.g. cluster state, indices or
translog. This PR adds a new Purpose enum to provide further breakdown
for the same HTTP operation.
Relates: ES-6800
Allows users of the HDFS repository plugin to configure the replication factor for files created by the repository implementation at repository creation time.
---------
Co-authored-by: James Baiera <james.baiera@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
Today the blob store register supports recording only a `long`,
represented as an 8-byte blob. We need to store a little more data in
the register, so this commit generalises things to work with a
`BytesReference` directly.
currently when Elasticsearch starts up there are warnings on the console complaining about SLF4J providers not found.
this is emitted twice
1 - when repository-azure module is loaded
2 - when xpack-security (depending on transport-netty) module is loaded
This commits adds a slf4j-nop to fix that warning
related to #93714
and #93878
Fixes#82794. Upgrade the spotless plugin, which addresses the issue
around formatting `instanceof` expressions. Formatting of statements
including lambdas seems to have improved too.
Further work towards the S3 compare-and-exchange implementation showed
that we would like this API to permit async operations. This commit
moves to an async API.
Also, this change made it fairly awkward to use an exception to deliver
to the caller the indication that the current value could not be read,
so this commit adjusts things to use `OptionalLong` throughout as
suggested in the discussion on #93955.
Only for testing purposes through the `FsRepository` for now and rather simple,
but should get the job done and technically be correct for a compliant NFS implementation.
Co-authored-by: David Turner <david.turner@elastic.co>
Some optimisations that I found when reusing searchable snapshot code elsewhere:
* Add an efficient input stream -> byte buffer path that avoids allocations + copies for heap buffers, this is non-trivial in its effects IMO
* Also at least avoid allocations and use existing thread-local buffer when doing input stream -> direct bb
* move `readFully` to lower level streams class to enable this
* Use same thread local direct byte buffer for frozen and caching index input instead of constantly allocating new heap buffers and writing those to disk inefficiently
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
As discussed, we can be up to twice as fast without increasing CPU use
much on high latency blob stores so increasing the pool size to 10 here
to better utilize larger data nodes.
