This PR makes RepositoriesService project aware so that the basic Put,
Get, Delete and Verify repository actions are now project scoped.
It intentionally leaves the following aspects out of scope for the
current changes: * Repository stats reporting * Repository clean-up,
analysis and integrity verification * Repository usages for searchable
snapshots and CCR
They will be worked on separately. One main reason for leaving them out
is that they are not needed by OBS which is currently blocked by
repository/snapshot changes. They may also have their own complexities,
e.g. stats reporting.
Resolves: ES-10478
* Enable And Disable Endpoint
* Status Endpoint
* Integration Tests
* REST Spec
* REST Spec tests
* Some documentation
* Update docs/changelog/129474.yaml
* Fix failing security test
* PR Fixes
* PR Fixes - Add missing feature flag name to YAML spec
* PR Fixes - Fix support for timeout and master_timeout parameters
* PR Fixes - Make the REST handler validation happy with the new params
* Delete docs/changelog/129474.yaml
* PR Fixes - Switch to local metadata action type and improve request handling
* PR Fixes - Make enable / disable endpoint cancellable
* PR Fixes - Switch timeout param name for status endpoint
* PR Fixes - Switch timeout param name for status endpoint in spec
* PR Fixes - Enforce local only use for status action
* PR Fixes - Refactor StreamsMetadata into server
* PR Fixes - Add streams module to multi project YAML test suite
* PR Fixes - Add streams cluster module to multi project YAML test suite
* Add multi-project support for more stats APIs
This affects the following APIs:
- `GET _nodes/stats`:
- For `indices`, it now prefixes the index name with the project ID (for non-default projects). Previously, it didn't tell you which project an index was in, and it failed if two projects had the same index name.
- For `ingest`, it now gets the pipeline and processor stats for all projects, and prefixes the pipeline ID with the project ID. Previously, it only got them for the default project.
- `GET /_cluster/stats`:
- For `ingest`, it now aggregates the pipeline and processor stats for all projects. Previously, it only got them for the default project.
- `GET /_info`:
- For `ingest`, same as for `GET /_nodes/stats`.
This is done by making `IndicesService.stats()` and `IngestService.stats()` include project IDs in the `NodeIndicesStats` and `IngestStats` objects they return, and making those stats objects incorporate the project IDs when converting to XContent.
The transitive callers of these two methods are rather extensive (including all callers to `NodeService.stats()`, all callers of `TransportNodesStatsAction`, and so on). To ensure the change is safe, the callers were all checked out, and they fall into the following cases:
- The behaviour change is one of the desired enhancements described above.
- There is no behaviour change because it was getting node stats but neither `indices` nor `ingest` stats were requested.
- There is no behaviour change because it was getting `indices` and/or `ingest` stats but only using aggregate values.
- In `MachineLearningUsageTransportAction` and `TransportGetTrainedModelsStatsAction`, the `IngestStats` returned will return stats from all projects instead of just the default with this change, but they have been changed to filter the non-default project stats out, so this change is a noop there. (These actions are not MP-ready yet.)
- `MonitoringService` will be affected, but this is the legacy monitoring module which is not in use anywhere that MP is going to be enabled. (If anything, the behaviour is probably improved by this change, as it will now include project IDs, rather than producing ambiguous unqualified results and failing in the case of duplicates.)
* Update test/external-modules/multi-project/build.gradle
Change suggested by Niels.
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
* Respond to review comments
* fix merge weirdness
* [CI] Auto commit changes from spotless
* Fix test compilation following upstream change to base class
* Update x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/datatiers/DataTierUsageFixtures.java
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
* Make projects-by-index map nullable and omit in single-project; always include project prefix in XContent in multip-project, even if default; also incorporate one other review comment
* Add a TODO
* update IT to reflect changed behaviour
* Switch to using XContent.Params to indicate whether it is multi-project or not
* Refactor NodesStatsMultiProjectIT to common up repeated assertions
* Defer use of ProjectIdResolver in REST handlers to keep tests happy
* Include index UUID in "unknown project" case
* Make the index-to-project map empty rather than null in the BWC deserialization case.
This works out fine, for the reasons given in the comment. As it happens, I'd already forgotten to do the null check in the one place it's actively used.
* remove a TODO that is done, and add a comment
* fix typo
* Get REST YAML tests working with project ID prefix TODO finish this
* As a drive-by, fix and un-suppress one of the health REST tests
* [CI] Auto commit changes from spotless
* TODO ugh
* Experiment with different stashing behaviour
* [CI] Auto commit changes from spotless
* Try a more sensible stash behaviour for assertions
* clarify comment
* Make checkstyle happy
* Make the way `Assertion` works more consistent, and simplify implementation
* [CI] Auto commit changes from spotless
* In RestNodesStatsAction, make the XContent params to channel.request(), which is the value it would have had before this change
---------
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Mainly moves project setup from `MultipleProjectsClientYamlSuiteTestCase` to
`ESRestTestCase`. This allows both Java REST tests and YAML tests to be run
in MP mode by passing the system property `-Dtests.multi_project.enabled=true`.
Future work will add the required gradle changes to be able to run any REST test in
MP mode more easily.
In this PR we introduce the data stream API in the `es-rest-api` using
the feature flag feature. This enabled us to use the `yamlRestTests`
tests instead of the `javaRestTests`.
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
The multi-project module is actually a testing module, but was included
in the modules/ tree for convenience. This commit moves it to
`test/external-modules/` which is where it belongs.
The yaml test
reindex/30_search/Sorting deprecated wait_for_completion false
does not work correctly with security enabled.
This test should be permanently added to the "unsupported" list.
Relates: https://github.com/elastic/elasticsearch/issues/120252
This updates `TransportResizeAction` to be project aware.
Although these actions are not supported on Serverless projects, we make
them work correctly with projects so that we can work towards removing
the deprecated `getProject` method.
This updates the cluster stats action to take all projects into account
when calculating the statistics. This is true even if a specific project
is requested.
The only exception is the "health status" because we only have
primitives for calculating the health of a single project. The stats
response will always give the health of the targeted project.
This PR fixes two more cluster-blocks usages and unmuted tests that can
pass. Some of the tests can already pass without the fix so that this PR
also unmutes them.
This PR makes open/close indices and add index blocks work by replaces a
large percentage of usages for the deprecated methods with the
corresponding project-aware versions. The ones that are no longer used
are removed. Some deprecated methods still remain after this PR. They
require even more cascading changes in areas that have not been touched,
such as snapshot/restore, watcher, security index settings etc. I prefer
to leave them out since this PR is already big enough. The main changes
are in MetadataIndexStateService class where both routing table and
metadata need to be specific to the right project.
This PR adds a project layer to ClusterBlocks so that it works with
multiple projects. Previously the index level blocks are tracked with a
map of "index-name to set-of-blocks". This is changed to be a map of
"project-id to project-level-blocks", where project-level-blocks
currently contains just the index level blocks. This structure is
expandable in that if we want to support project-wide blocks (as opposed
to cluster-wide blocks), they can be added to the project-level-blocks.
One alternative that has been considered but decided against is to
update the map of "index-name to set-of-blocks" to "index-object/uuid to
set-of-blocks". This has a few issues: (1) ClusterBlocks has no
references to Metadata so it cannot serve a request for index blocks
given an index name (bwc); (2) Similarly, when deserializing from an
old node, which provides only index names, there is no easy way to
lookup for the associated index UUID; (3) The structure is rather rigid
and cannot be easily expanded for project-level blocks if they become a
thing; (4) No easy way to get all indices blocks for a project.
I chose to keep all existing ClusterBlocks methods that take index name
as argument to avoid too many cascading changes. I marked all of them as
deprecated for removal so that they can be easily spotted, replaced and
removed in subsequent PRs. That said, there are still test failures
where cascading changes are necessary.
Relates: ES-10162
This PR makes put-alias project aware so that the aliases API works in a
multi-project setup. This is achieved by adding a new project-id field
to IndicesAliasesClusterStateUpdateRequest.
The alias actions are also used by MetadataRolloverService. These usages
are NOT updated in this PR and still assumes default project. They will
be handled separately.
