* Update Gradle Wrapper to 8.2 (#96686)
- Convention usage has been deprecated and was fixed in our build files
- Fix test dependencies and deprecation
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
Since https://github.com/elastic/elasticsearch/pull/89951 we use
ML snapshot builds in all elasticsearch-ci jobs, even the
release-tests ones.
The test of the reported ML native code version in the HLRC tests
needs to account for this.
This change is 7.17 only, as the HLRC is removed in 8.x.
Test had been muted because of issue #67749
This PR unmutes test RollupIT.testPutStartAndGetRollupJob() because most probably it has
been fixed by #86992Closes#67749
The High Level REST Client provides a MountSnapshotRequest#ignoredIndexSettings(String[]) method to define some index settings to ignore when mounting a snapshot as a searchable snapshot index.
Sadly the client generates a wrong request body field ignored_index_settings instead of ignore_index_settings. This wasn't caught until #75982 was reported because the parser of Mount API request ignores unknown fields in request body.
We fixed the wrong leniency of the request parser on the Elasticsearch side (#88987) starting version 8.5.0, and we decided to continue to ignore the ignored_index_settings generated by the HLRC bug to avoid breaking HLRC client usages.
This change fixes the request body generated by the HLRC to pass the correct field name. This fix is for versions 7.17.6+ (we do not expect to release new versions of HLRC in 8.x).
It also renames the HLRC methods to expose the change to client users.
Co-authored-by: bellengao <bellengao@tencent.com>
This expands the example in the High Level Rest Client's Put Role
documentation to include an example of creating a role with index
privileges, include FLS and DLS Relates: #81354
Thanks to https://bugs.eclipse.org/bugs/show_bug.cgi?id=574437,
we've run into a situation where Spotless is incorrectly formatting
a particular piece of syntax (due the underlying Eclipse bug). We
were able to turn off formatting of this syntax using `// @formatter:off`
and `// @formatter:on`, but there was a further problem. We configure
IntelliJ to use the Eclipse formatter plugin, but this doesn't
respect the `@formatter` tags since these are set at the Spotless
level, not the Eclipse formatter level. Note that these tags aren't
set in the Eclipse formatter config, because there we use `// tag::`
and `// end::` in order to avoid reformatting docs snippets, which
have a much narrower line width.
What a mess.
So, to get around all this, drop the `@formatter` tags and tweak
our custom `SnippetLengthCheck` Checkstyle rule so that
`// tag:noformat` regions are not subject to the narrower line length
check, but are still exempt from formatting.
* Change HLRC's LifecyclePolicy to allow all valid ILM actions (#81483)
This commit changes the High Level Rest Client's `LifecyclePolicy`
implementation to allow all the same valid actions for each phase that
ILM supports. It was previously missing actions in the hot and cold
phases, as well as missing the frozen phase entirely. This also adds a
test that uses the "real" `TimeseriesLifecycleType.ORDERED_*ACTIONS`
lists so that if an action is added or removed in the future the test
will fail. Resolves#81461
# Conflicts:
# client/rest-high-level/src/test/java/org/elasticsearch/client/indexlifecycle/LifecyclePolicyTests.java
# x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/TimeseriesLifecycleType.java
* Fix test
Do not fail the transform if pit search fails with index not found as a result of an index that got deleted via ILM,
if that index is part of a search that selects indices using a wildcard, e.g. logs-*. If pit search fails, the search
is retried using search without a pit context. The 2nd search might fail if the source targets an explicit index.
In addition the usage of the pit API can not be disabled by transform.
fixes elastic#81252
relates elastic#81256
* [7.16] Add replicated field to get data stream api response.
Backporting #80988 to 7.16 branch.
Internally we already kept track of whether a data stream is replicated by CCR.
It is part of the `DataStream` class. This just adds it to the xcontent serialization
of the get data stream api response class.
Relates to elastic/kibana#118899
* adjust to 7.16 reality
* Revert "Allow deprecation warning for the return_200_for_cluster_health_timeout parameter (#80178) (#80445)"
This reverts commit 5a5300753d.
* Revert "[7.x] Use query param instead of a system property for opting in for new cluster health response code (#79397) (#79435)"
This reverts commit d8c14281
* Revert "[7.x] Deprecate returning 408 for a server timeout on `_cluster/health` (#78180) (#78940)"
This reverts commit 03bd55d1
This deprecates estimated_heap_memory_usage_bytes on model put and replaces it with model_size_bytes.
On GET, both fields are returned (unless storing in the index) and are populated with the same field.
For the ml/info API, both fields are returned as well.
Part of #19752.
Fix a number of cases of shadows vars under `client/rest-high-level`. As
part of this, fork the Checkstyle `HiddeFieldCheck` class so that it
understand the pattern of settings with no "set" prefix.
Matchers is deprecated in Mockito, in favor of the newer
ArgumentMatchers class. In fact, internally Matchers just extends
ArgumentMatchers as all the methods there were moved. This commit
changes all imports of org.mockito.Matchers to
org.mockito.ArgumentMatchers.
* [Transform] Improve transform upgrader (#79601)
This changes fixes some issues with the upgrader. It inlines the index name resolver as wildcard
deletes are not supported if action.destructive_requires_name is set to true(default). It changes
response to always return all counters in order to avoid an empty response. After documentation has
been created, it switches the deprecation URL to the upgrade API docs.
* change url to 7.16 specific one
* fix 7.x specific test
* fix put mapping call
Securemock is a wrapper around Mockito that monkey patches internals of
Mockito to work with the SecurityManager. However, the library has not
been updated in several years due to the complicated nature of this
monkey patching. This has left us with an ancient version of Mockito,
missing out on updates to the library in the last half decade.
While Securemock currently works with Mockito 1.x, in 2.x an official
means of plugging into mockito was added, MockMaker. This commit removes
securemock as a dependnecy of the test framework, replacing it with a
modern version of Mockito, and implementing a MockMaker that integrates
with SecurityManager.
Note that while there is a newer version of Mockito available, 4.0, it
has several deprecations removed that are used throughout Elasticsearch.
Those can be addressed in followups, and then a subsequent upgrade to
4.0 should be possible.
relates #79567closes#40334
This PR adds a framework for migrating system indices as necessary prior
to Elasticsearch upgrades. This framework uses REST APIs added in
another commit:
- GET _migration/system_features
This API, which gets the status of "features" (plugins which own system
indices) with regards to whether they need to be upgraded or not. As of
this PR, this API also reports errors encountered while migrating system
indices alongside the index that was being processed when this occurred.
As an example of this error reporting:
```json
{
"feature_name": "logstash_management",
"minimum_index_version": "8.0.0",
"upgrade_status": "ERROR",
"indices": [
{
"index": ".logstash",
"version": "8.0.0",
"failure_cause": {
"error": {
"root_cause": [
{
"type": "runtime_exception",
"reason": "whoopsie",
"stack_trace": "<omitted for brevity>"
}
],
"type": "runtime_exception",
"reason": "whoopsie",
"stack_trace": "<omitted for brevity>"
}
}
}
]
}
```
- POST _migration/system_features
This API starts the migration process. The API for this has no changes,
but when called, any system indices which need to be migrated will be
migrated, with status information stored in the cluster state for later
use by the GET _migration/system_features API.
This change the way we store mappings in `Template` during serialization
to disk - instead storing it as map we use byte array that we already
have. This avoids deserialization-serialization cycle during storing
cluster state on disk.
# Conflicts:
# server/src/main/java/org/elasticsearch/cluster/coordination/ElasticsearchNodeCommand.java
Backport #79351 to 7.x:
The original change was implemented in #78940, but we have decided to move from a system property
to a request parameter, so Cloud users/clients have an easier way to opt-in for the new status code.
Relates #70849
Backport #79351 to 7.x:
The original change was implemented in #78940, but we have decided to move from a system property
to a request parameter, so Cloud users/clients have an easier way to opt-in for the new status code.
Relates #70849
This PR changes uses of transient cluster settings to
persistent cluster settings.
The PR also deprecates the transient settings usage.
Relates to #49540
This change ensures that the HLRC can parse the search response
that the get async search returns.
The aggregations must be typed by keys but the param to enable this
mode was missing.
Closes#77608
* [Transform] add transform upgrade endpoint (#77566)
Add an _upgrade endpoint to bulk upgrade transforms. _upgrade rewrites all transforms and its
artifacts into the latest format to the latest storage(index). If all transforms are upgraded old
indices and outdated documents get deleted. Using the dry_run option it is possible to check if
upgrades are necessary without applying changes.
* fix merge conflicts
* 7.x requires a different license check
Today the search action send the full list of original indices on every shard request.
This change restricts the list to the concrete index of the shard or the alias that was used to resolve it.
Relates #78314
Migrate to persistent cluster settings in Java tests
We are deprecating transient settings, therefore this
PR changes uses of transient cluster settings to
persistent cluster settings.
* Implement GET API for System Feature Upgrades (#78642)
* Implement and test get feature upgrade status API
* Add integration test for feature upgrade endpoint
* Use constant enum for statuses
* Add unit tests for transport class methods
* Fix bwc tests for 7.x
Backports #78180 to 7.x.
Add a deprecation warning and a system property es.cluster_health.request_timeout_200 to opt in for returning 200 which will be the default in 8.0.0
Fixes#70849
This is mostly motivated by the ILM steps changes, that showed up as hot in
profiling since instantiating a `Settings` instance turns out to be somewhat expensive.
Also cleans up a couple of other spots and some duplication as well.
