Snapshot builds of elasticsearch-java are no longer available.
Using the current major highest version should be safe according to the
compatibility guarantees described on https://github.com/elastic/elasticsearch-java?tab=readme-ov-file#compatibility
Repo https://snapshots.elastic.co/maven/ is no longer needed in a
composite build. Previously it was only used to provide snapshot version
of `elasticsearch-java` client which is no longer available in snapshot
version. We keep the https://snapshots.elastic.co/maven/ repo to be
used when for non-composite builds when any other dependencies
snapshot versions can be fetched.
In order to support a future TransportRequest variant that accepts the
response type, TransportRequest needs to be an interface. This commit
adds AbstractTransportRequest as a concrete implementation and makes
TransportRequest a simple interface that joints together the parent
interfaces from TransportMessage.
Note that this was done entirely in Intellij using structural find and
replace.
This PR implements authorization logic for failure store access. It
builds on https://github.com/elastic/elasticsearch/pull/122715.
Access to the failure store is granted by two privileges:
`read_failure_store` and `manage_failure_store`. Either of these
privileges lets a user access a failure store via the `::failures`
selector, as well as access its backing failure indices.
`read_failure_store` grants read access (for example to search documents
in a failure store), `manage_failure_store` grants access to write
operations, such as rollover. Users with only `read` or `manage` on a
data stream do not get failure store access. Vice versa, users with
`read_failure_store` and `manage_failure_store` do not get access to
regular data in a data stream.
The PR implements this by making authorization logic selector-aware. It
involves two main changes:
1. Index permission groups now compare the selector under which an index resource is accessed to the selector associated with the group.
2. The `AuthorizedIndices` interface likewise uses selectors to decide which indices to treat as authorized. This part of the change requires a sizable refactor and changes to the interface.
The high-level behavior for selector-aware search is as follows:
For a user with `read_failure_store` over data stream `logs`:
- `POST /logs::failures/_search` returns the documents in the failure store.
- `POST /logs/_search` returns a 403.
- `POST /logs/_search?ignore_unavailable=true` and `POST /*/_search` return an empty result.
Similarly, for a user with `read` over data stream `logs`:
- `POST /logs::failures/_search` returns a 403.
- `POST /logs/_search` returns documents in the data stream.
- `POST /logs::failures/_search?ignore_unavailable=true` and `POST /*::failures/_search` return an empty result.
A user with both `read` and `read_failure_store` over data stream `logs`
gets access to both `POST /logs::failures/_search` and `POST
/logs/_search`.
The index privilege `all` automatically grants access to both data and
the failures store, as well as all hypothetical future selectors.
Resolves: ES-10873
We have some incredibly deep callstacks in security that seem to visibly
raise context switch costs, make profiling more complicated and
generally make the code rather hard to follow.
Since the methods adjusted here return a result synchronously we can
both save overhead and make things a little easier to follow by using
promises as returns in place of consuming callbacks.
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
The "is authorized" check for explicit names must take
into account that a name can be authorized even if it doesn't exist.
To that end, the "is authorized" check for explicit names,
during request rewriting, should not fail if the index is unavailable.
This PR is a refactoring that allows changing the predicate
without changing the wildcard expansion test.
This PR brings the constructor call for `GetUserPrivilegesResponse`
up-to-date with `main`, to fix plugin compilation.
Passing an empty set for `remoteIndices` since the relevant
functionality for remote privilege still sits behind a feature flag.
Will update the code accordingly with real remote indices permissions in
the future (will track in a Jira issue).
Fixes: https://github.com/elastic/elasticsearch/issues/91415
The auditable field was meant to determine whether authorization should
be audited or not. However, in practice, this field is always true and
what actually enabling auditing is by using different AuditTrail
implementations. The field is hence not necessary and only adds clutter
to the code. It is also arguable whether auditable or not belongs to the
AuthorizationInfo class. As a result, this PR removes this field from
AuthorizationInfo and its subclass IndexAuthorizationInfo.
Relates: https://github.com/elastic/elasticsearch/pull/91180/files#r1011344119
* Fix usage of `IndexAccessControl` in `CustomAuthorizationEngine`
* Make version properties loader more robust
Co-authored-by: Mark Vieira <portugee@gmail.com>
Boolean-only privilege checks, i.e. the ones currently used in the
"profile has privilege" API, now benefit from a performance improvement,
because the check will now stop upon first encountering a privilege that
is NOT granted over a resource (and return `false` overall). Previously,
all the privileges were always checked over all the resources in order
to assemble a comprehensive response with all the privileges that are
not granted.
This introduces a new Security API `_security/profile/_has_privileges`
that can be used to verify which Users have the requested privileges,
given their associated User Profiles. Multiple profile uids can be specified
in a single has privileges request.
This is analogous to the existing Has privileges API. It also uses the same
format for specifying the privileges to be checked, and should be used in
the same situations (ie to run an authorization preflight check or to verify
privileges over application resources). However, unlike the existing
has privilege API, this can be used to check the privileges of multiple
users (not only of the currently authenticated one), but the users must
have an existing profile, and the response is binary only (either it has or
it does not have the requested privileges).
Calling this API requires the `manage_user_profile` cluster privilege.
In #86206, we closed down Authentication constructors to favour
dedicated convenient methods for instantiation. The constructor usages
in the example project were however left out (another refactor fallout).
Relates: #86206Resolves: #86378
This fixes two classpath issues for integration tests in the example
build
1. adds log4j to the runtime classpath where needed
2. removes exclude of jakarta json which was added as a workaround for
https://github.com/elastic/elasticsearch-java/issues/47
This commit detects a specific case when a child action (e.g. a shard
level action, or a phased action) acts on the same indices (or a
subset of the indices) or that parent request, and we can retain the
original authorization result.
The optimization is only effective for the invocation of the child
action on the same node as the parent - if the transport action needs
to be executed on a remote node then that authorization will not be
optimized and will perform the full check as existed before this
change.
This change is primarily benefitial for actions where a single parent
action on a coordinating node triggers the execution of multiple
children (e.g. a child action per shard) as it allows the
coordinating node to trigger those action and allow the load
to be passed to the remote nodes as quickly as possible rather than
having authorization on the coordinating node become a bottleneck.
This changes the result of AuthorizationEngine.loadAuthorizedIndices
(and dependent methods) from List<String> to Set<String>.
This has the following performance benefits:
1. `contains` checks are faster
2. RBACEngine always formed this collections as a Set, so this
change reduces unnecessary copying.
An additional performance improvement was added when resolve authorized
index names for data streams.
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
For all OSS plugins (except repository-* and discovery-*) integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.
This commit does NOT convert the discovery-* and repository-* since they
are bit more complex then the rest of tests and this PR is large enough.
Those plugins will be addressed in a future PR(s).
This commit also fixes a minor issue that did not copy the rest api
for projects that only had YAML TEST tests.
related: #56841
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.
The remaining cases in modules, plugins, and x-pack will be handled in followups.
This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.
The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.
Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).
As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.
Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.
This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
In order to prepare the `AliasOrIndex` abstraction for the introduction of data streams,
the abstraction needs to be made more flexible, because currently it really can be only
an alias or an index.
* Renamed `AliasOrIndex` to `IndexAbstraction`.
* Introduced a `IndexAbstraction.Type` enum to indicate what a `IndexAbstraction` instance is.
* Replaced the `isAlias()` method that returns a boolean with the `getType()` method that returns the new Type enum.
* Moved `getWriteIndex()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface.
* Moved `getAliasName()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface and renamed it to `getName()`.
* Removed unnecessary casting to `IndexAbstraction.Alias` by just checking the `getType()` method.
Relates to #53100
The current implementations make it difficult for
adding new privileges (example: a cluster privilege which is
more than cluster action-based and not exposed to the security
administrator). On the high level, we would like our cluster privilege
either:
- a named cluster privilege
This corresponds to `cluster` field from the role descriptor
- or a configurable cluster privilege
This corresponds to the `global` field from the role-descriptor and
allows a security administrator to configure them.
Some of the responsibilities like the merging of action based cluster privileges
are now pushed at cluster permission level. How to implement the predicate
(using Automaton) is being now enforced by cluster permission.
`ClusterPermission` helps in enforcing the cluster level access either by
performing checks against cluster action and optionally against a request.
It is a collection of one or more permission checks where if any of the checks
allow access then the permission allows access to a cluster action.
Implementations of cluster privilege must be able to provide information
regarding the predicates to the cluster permission so that can be enforced.
This is enforced by making implementations of cluster privilege aware of
cluster permission builder and provide a way to specify how the permission is
to be built for a given privilege.
This commit renames `ConditionalClusterPrivilege` to `ConfigurableClusterPrivilege`.
`ConfigurableClusterPrivilege` is a renderable cluster privilege exposed
as a `global` field in role descriptor.
Other than this there is a requirement where we would want to know if a cluster
permission is implied by another cluster-permission (`has-privileges`).
This is helpful in addressing queries related to privileges for a user.
This is not just simply checking of cluster permissions since we do not
have access to runtime information (like request object).
This refactoring does not try to address those scenarios.
Relates #44048
Test clusters currently has its own set of logic for dealing with
finding different versions of Elasticsearch, downloading them, and
extracting them. This commit converts testclusters to use the
DistributionDownloadPlugin.
testclusters detect from settings that security is enabled
if a user is not specified using the DSL introduced in this PR, a default one is created
the appropriate wait conditions are used authenticating with the first user defined in the DSL ( or the default user ).
an example DSL to create a user is user username:"test_user" password:"x-pack-test-password" role: "superuser" all keys are optional and default to the values shown in this example
* Un-mute and fix BuildExamplePluginsIT
There doesn't seem to be anything wrong with the test iteself.
I think the failure were CI performance related, but while it was muted,
some failures managed to sneak in.
Closes#38784
* PR review
For some users, the built in authorization mechanism does not fit their
needs and no feature that we offer would allow them to control the
authorization process to meet their needs. In order to support this,
a concept of an AuthorizationEngine is being introduced, which can be
provided using the security extension mechanism.
An AuthorizationEngine is responsible for making the authorization
decisions about a request. The engine is responsible for knowing how to
authorize and can be backed by whatever mechanism a user wants. The
default mechanism is one backed by roles to provide the authorization
decisions. The AuthorizationEngine will be called by the
AuthorizationService, which handles more of the internal workings that
apply in general to authorization within Elasticsearch.
In order to support external authorization services that would back an
authorization engine, the entire authorization process has become
asynchronous, which also includes all calls to the AuthorizationEngine.
The use of roles also leaked out of the AuthorizationService in our
existing code that is not specifically related to roles so this also
needed to be addressed. RequestInterceptor instances sometimes used a
role to ensure a user was not attempting to escalate their privileges.
Addressing this leakage of roles meant that the RequestInterceptor
execution needed to move within the AuthorizationService and that
AuthorizationEngines needed to support detection of whether a user has
more privileges on a name than another. The second area where roles
leaked to the user is in the handling of a few privilege APIs that
could be used to retrieve the user's privileges or ask if a user has
privileges to perform an action. To remove the leakage of roles from
these actions, the AuthorizationService and AuthorizationEngine gained
methods that enabled an AuthorizationEngine to return the response for
these APIs.
Ultimately this feature is the work included in:
#37785#37495#37328#36245#38137#38219Closes#32435
