This commit clarifies the force merge behavior for the `forcemerge` action either as a standalone
ILM action, or as part of the searchable snapshot action. When using a searchable snapshot it can be
confusing especially because the force merge can occur in a phase prior to the phase configured in
the ILM policy (since ILM doesn't migrate the index prior to the searchable snapshot mounting).
Add a new rollover condition with the name `max_primary_shard_docs`.
Triggers rollover when the largest primary shard in the index reaches a certain number of documents.
This enhances the migrate to data tiers routing API to also iterate over
the existing legacy, composable, and component templates and look if
they define a custom node attribute routing in their settings for either
`index.routing.allocation.require.{nodeAttrName}` or
`index.routing.allocation.include.{nodeAttrName}`. If any does, we
update them to remove all the routings settings for the provided
`nodeAttrName`.
eg. any template with the following setting configuration:
```
"settings": {
index.routing.allocation.require.data: "warm",
index.routing.allocation.include.data: "rack1",
index.routing.allocation.exclude.data: "rack2,rack3"
}
```
will have its settings updated to:
```
"settings": {}
```
The migrate to data tiers routing API required ILM to be stopped. This
is fine for "live" runs, but for dry runs this isn't a requirement.
This changes the dry_run to allow the API to run irrespective of the ILM
status.
As outlined in elastic/elasticsearch#81604, including the `searchable_snapshot` action in both the hot and cold phases can result in indices not automatically migrating to the cold tier during the cold phase.
This adds a related warning.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* Expose the index age in ILM explain output.
ILM already exposes the `age` that ILM will use to transition to the next phase, based on that phase's `min_age`. The `index_age` is based only on the index creation date and it's used to trigger a rollover.
Resolves#64429
The searchable snapshot action mounts snapshots as indices
with a different prefix depending of the phase. This commit
tries to mention them in the docs.
Makes several changes to consolidate snapshot and backup-related docs.
Highlights:
* Adds info about supported ESS snapshot repository types
* Adds docs for Kibana's Snapshot and Restore feature
* Combines tutorial pages related to taking and managing snapshots
* Consolidates explanations of the snapshot process
* Incorporates SLM into the snapshot tutorial
* Removes duplicate "back up a cluster" pages
Removes `testenv` annotations and related code. These annotations originally let you skip x-pack snippet tests in the docs. However, that's no longer possible.
Relates to #79309, #31619
Changes:
* Removes several `[testenv="gold+"]` attributes from the docs. `gold+` is not a valid [subscription level](https://www.elastic.co/subscriptions) or testenv value.
* Moves two `[testenv="basic"]` attributes to the file header. This makes the `testenv` placement consistent and fixes the yml file generated from `docs/reference/snapshot-restore/register-repository.asciidoc`.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This PR changes uses of transient cluster settings to
persistent cluster settings.
The PR also deprecates the transient settings usage.
Relates to #49540
* Add docs on searchable snaps costs
Adds a note on why searchable snapshots is cheaper, including warnings
that it might be more expensive too.
* Split into sections
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* data -> the shard contents
* More wording tweaks
* Apply suggestions from code review
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* Make the ILM `freeze` action a no-op
This changes the ILM `freeze` action to not actually freeze the index, instead performing no
operation.
Relates to #70192
* zoop -> noop in documentation anchor
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
PRs #73062 and #73043 repurposed the `alias` anchor for a new guide for index
and data stream aliases. Previously, this anchor was used for our field alias
documentation.
Repurposing the anchor has caused continuity errors for users selecting
different versions of the ES docs. It could also cause confusion for users with
a `/current/` link to the `alias` page.
This updates the anchor for the alias guide and adds a redirect page to
disambiguate the `alias` anchor.
It also fixes a bread crumb issue for redirects following the 'Modifying your
Data' redirect page.
Closes#77034.
We added configuration to AllocateAction to set the total shards per node property on the index. This makes it possible that a user could set this to a value lower than the total number of shards in the index that is about to be shrunk, meaning that all of the shards could not be moved to a single node in the ShrinkAction. This commit unsets the total shards per node property so that we fall back to the default value (-1, unlimited) in the ShrinkAction to avoid this.
Relates to #44070
This adds a new optional field to the allocate ILM action called "total_shards_per_node". If present, the
value of this field is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation
takes place.
Relates to #44070
* Allow ILM move-to-step without `action` or `name`
This commit enhances ILM's move-to-step API to allow dropping the `name`, or dropping both the
`action` and `name`. For example:
```json
POST /_ilm/move/foo-1
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "check-rollover-ready"
},
"next_step": {
"phase": "warm",
"action": "forcemerge"
}
}
```
Will move to the first step in the `forcemerge` action in the `warm` phase (without having to know
the specific step name).
Another example:
```json
POST /_ilm/move/foo-1
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "check-rollover-ready"
},
"next_step": {
"phase": "warm"
}
}
```
Will move to the first step in the `warm` phase (without having to know the specific action name).
Bear in mind that the execution order is still entirely an implementation detail, so "first" in the
above sentences means the first step that ILM would execute.
Resolves#58128
* Apply Andrei's wording change (thanks!)
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
* Log index and policy name when the concrete step key can't be resolved
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
This adds support for a `dry_run` parameter for the
`_ilm/migrate_to_data_tiers` API. This defaults to `false`, but when
configured to `true` it will simulate the migration of elasticsearch
entities to data tiers based routing, returning the entites that need to
be updated (indices, ILM policies and the legacy index template that'd
be deleted, if any was configured in the request).
To switch an index's lifecycle policy, you must first remove the existing
policy. Otherwise, phase execution for the index may silently fail.
Closes#70151
This adds the _ilm/migrate_to_data_tiers API to expose the service for
migrating the elasticsearch abstractions (indices, ILM policies and an
optional legacy template to delete) to data tiers routing allocation
(away from custom node attributes)
This commit adds the "in_use_by" object to the response for ILM policies. This map shows the
indices, data streams, and composable templates that use the ILM policy.
An example output may look like:
```json
{
"logs" : {
"version" : 1,
"modified_date" : "2021-06-23T18:42:08.381Z",
"policy" : {
...
},
"in_use_by" : {
"indices" : [".ds-logs-foo-barbaz-2021.06.23-000001", ".ds-logs-foo-other-2021.06.23-000001"],
"data_streams" : ["logs-foo-barbaz", "logs-foo-other"],
"composable_templates" : ["logs"]
}
}
}
```
Resolves#73869
Relates to #70755.
The main changes of this PR are:
Add an optional _meta field to ILM policy.
Add some test code about the change.
Update the doc of Create or update lifecycle policy API.
* [DOCS] Document how to migrate to node roles from node attrs. Closes#65855
* [DOCS] Incorporated review comments
* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
