A predicate to check whether the cluster supports a feature is available
to rest handlers defined in server. This commit adds that predicate to
plugins defining rest handlers as well.
This PR adds a mappings version number to system index descriptor, which is intended to replace the use of Version to signal changes in the mappings for the descriptor. This value is required for managed system indices.
Previously, most of the system index descriptors automatically incremented their mapping version with each release, which meant that the mappings would stay up-to-date with any additive changes. Now, developers will need to increment the mapping version whenever there's a change.
* Add MappingsVersion inner class to SystemIndexDescriptor
* Add mappings version to metadata in all system index mappings
* Rename version meta key ('system' -> 'managed')
* Update mappings for ML indices if required
* Trigger ML index mappings updates based on new index mappings version.
---------
Co-authored-by: Ed Savage <ed.savage@elastic.co>
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
When creating an index with a system index descriptor, ignore all
templates. Update tests that check template behavior.
Co-authored-by: Nikola Grcevski <nikola.grcevski@elastic.co>
* Assert that some getters are not accessed for unmanaged indices
There are many properties in SystemIndexDescriptor that should only be
accessed when dealing with managed system indices. Any code that accesses
the getters for these properties should be inside of a block that checks
whether an index is managed or not. Here, we add assertions to verify
that this is the case, or to point us to places where our code may be
buggy.
The check removed by this commit checks specifically for requests for
aliases that *would* be system aliases, if they existed. I'm not sure
why we were doing this, as we don't try to do this anywhere else. The
only test that this seems to make fail is the test explicitly checking
that behavior, which I don't think is really what we want. So I'm just
removing it.
Relates https://github.com/elastic/elasticsearch/issues/81589
This commit changes the superuser role (as used by the "elastic"
builtin user) so that it no longer has any sort of write access to
restricted indices (system indices).
This improves the safety and security of the cluster, as it means
that there are no out-of-the-box users or roles that can write to,
delete or close the security index.
Superusers can still read from (and monitor) system indices.
Other roles (and users) can still access system indices as specified
in their descriptor. These can be custom such as the
"_es_test_root" role used in the integration test suite, or builtin
roles such as kibana_system.
Fixes split packages between server and the LLRC (and HLRC), by renaming
the server package to a more appropriate name that represents the fact
that is in an internal client. That is, rename server's
org.elasticsearch.client to org.elasticsearch.client.internal.
System indices should be hidden from users. Since they are already restricted
indices, a users that can't view restricted indices already can't see or access
them, but they should also be hidden for superusers or users that are otherwise
granted advanced privileges.
To the greatest degree possible, we apply hidden settings in the transport
layer, so that the system can create an index or alias that is set to visible,
for example, when operating in a mixed cluster mode. However, in the case of
aliases created by templates, we hide the alias in the service layer.
This change has broken a number of tests that were relaying unnecessarily on
wildcard searches. In general, the fix for these issues was to apply
expand_wildcards=open,hidden to the request.
* Force system indices to be hidden in IndexMetadata
* Hide system data streams
* Update feature migration tests
* ML datafeed config defaults to searching hidden indices
* Prevent unmanaged system indices from becoming visible
* Change validation in TransportUpdateSettingsAction
* Validate index creation settings in transport action
* Make sure system data stream backing indices are hidden
* Make sure transport request adds hidden index setting if missing
* Validate and set default for autocreated system indices
* Add some code to hide system aliases
* Hide system aliases in create index service
* Hide system aliases when adding them via alias endpoints
* Check system indices when simulating and validating templates
* Add known issue for reenabling tests
* Update docs/changelog/79512.yaml
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
* Implement and test get feature upgrade status API
* Add integration test for feature upgrade endpoint
* Use constant enum for statuses
* Add unit tests for transport class methods
* Do not create unused testCluster
This avoids creating test clusters that are not required during the build.
We use lazy configuration here on testClusters and only instantiate them as theyre
* Do not fail on run task (debug)
* Create more test cluster lazy
* Make more test cluster lazy
* Avoid creating unused testcluster
* Fix PluginBuildPlugin
* Fix disabling geo db download
* Fix cluster setup in repository-multi-version
* Polishing
* Fix issue with irretic groovy ogic
* Fix bwc tests
* Fix more bwcTests
* Fix more bwc tests
* Fix more bwc tests
* Fix more bwc tests
* Fix typo
* Minor polishing
* Fix rolling upgrade tests
* Fix cluster config in sql qa mixedcluster project
* Fix more bwc tests
* Clean up before review
* Document test cluster usage
* Api polising after Review
provide useCluster(Provider) method to TestClusterAware
Ideally we take this a step further and realize those test clusters only on use.
But out of scope of this PR.
* Allow gradle provider as value for nonSystemProperties
* Some simplification on test configuration
* Fix typo in rest test config
* Fix more typos
* Fix another typo
* Fix more typos
Introduces functionality to generate and set a password for the
elastic user during the initilization of the Security plugin if
- `bootstrap.password` is not alredy set in the keystore
- the security index doesn't already exist
- and the password for the elastic user is not yet set ( the doc
for the user doesn't exist in the security index )
This change updates the way we handle net new system indices, which are
those that have been newly introduced and do not require any BWC
guarantees around non-system access. These indices will not be included
in wildcard expansions for user searches and operations. Direct access
to these indices will also not be allowed for user searches.
The first index of this type is the GeoIp index, which this change sets
the new flag on.
Closes#72572
