mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-25 07:37:19 -04:00
222 lines
7.8 KiB
Text
222 lines
7.8 KiB
Text
[role="xpack"]
|
|
[[index-lifecycle-error-handling]]
|
|
== Troubleshooting {ilm} errors
|
|
|
|
When {ilm-init} executes a lifecycle policy, it's possible for errors to occur
|
|
while performing the necessary index operations for a step.
|
|
When this happens, {ilm-init} moves the index to an `ERROR` step.
|
|
If {ilm-init} cannot resolve the error automatically, execution is halted
|
|
until you resolve the underlying issues with the policy, index, or cluster.
|
|
|
|
For example, you might have a `shrink-index` policy that shrinks an index to four shards once it
|
|
is at least five days old:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/shrink-index
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"min_age": "5d",
|
|
"actions": {
|
|
"shrink": {
|
|
"number_of_shards": 4
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST
|
|
|
|
There is nothing that prevents you from applying the `shrink-index` policy to a new
|
|
index that has only two shards:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /my-index-000001
|
|
{
|
|
"settings": {
|
|
"index.number_of_shards": 2,
|
|
"index.lifecycle.name": "shrink-index"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
After five days, {ilm-init} attempts to shrink `my-index-000001` from two shards to four shards.
|
|
Because the shrink action cannot _increase_ the number of shards, this operation fails
|
|
and {ilm-init} moves `my-index-000001` to the `ERROR` step.
|
|
|
|
You can use the <<ilm-explain-lifecycle,{ilm-init} Explain API>> to get information about
|
|
what went wrong:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
GET /my-index-000001/_ilm/explain
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
Which returns the following information:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"indices" : {
|
|
"my-index-000001" : {
|
|
"index" : "my-index-000001",
|
|
"managed" : true,
|
|
"index_creation_date_millis" : 1541717265865,
|
|
"time_since_index_creation": "5.1d",
|
|
"policy" : "shrink-index", <1>
|
|
"lifecycle_date_millis" : 1541717265865,
|
|
"age": "5.1d", <2>
|
|
"phase" : "warm", <3>
|
|
"phase_time_millis" : 1541717272601,
|
|
"action" : "shrink", <4>
|
|
"action_time_millis" : 1541717272601,
|
|
"step" : "ERROR", <5>
|
|
"step_time_millis" : 1541717272688,
|
|
"failed_step" : "shrink", <6>
|
|
"step_info" : {
|
|
"type" : "illegal_argument_exception", <7>
|
|
"reason" : "the number of target shards [4] must be less that the number of source shards [2]"
|
|
},
|
|
"phase_execution" : {
|
|
"policy" : "shrink-index",
|
|
"phase_definition" : { <8>
|
|
"min_age" : "5d",
|
|
"actions" : {
|
|
"shrink" : {
|
|
"number_of_shards" : 4
|
|
}
|
|
}
|
|
},
|
|
"version" : 1,
|
|
"modified_date_in_millis" : 1541717264230
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
|
|
|
|
<1> The policy being used to manage the index: `shrink-index`
|
|
<2> The index age: 5.1 days
|
|
<3> The phase the index is currently in: `warm`
|
|
<4> The current action: `shrink`
|
|
<5> The step the index is currently in: `ERROR`
|
|
<6> The step that failed to execute: `shrink`
|
|
<7> The type of error and a description of that error.
|
|
<8> The definition of the current phase from the `shrink-index` policy
|
|
|
|
To resolve this, you could update the policy to shrink the index to a single shard after 5 days:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/shrink-index
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"min_age": "5d",
|
|
"actions": {
|
|
"shrink": {
|
|
"number_of_shards": 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
[discrete]
|
|
=== Retrying failed lifecycle policy steps
|
|
|
|
Once you fix the problem that put an index in the `ERROR` step,
|
|
you might need to explicitly tell {ilm-init} to retry the step:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST /my-index-000001/_ilm/retry
|
|
--------------------------------------------------
|
|
// TEST[skip:we can't be sure the index is ready to be retried at this point]
|
|
|
|
{ilm-init} subsequently attempts to re-run the step that failed.
|
|
You can use the <<ilm-explain-lifecycle,{ilm-init} Explain API>> to monitor the progress.
|
|
|
|
[discrete]
|
|
=== Common {ilm-init} errors
|
|
|
|
Here's how to resolve the most common errors reported in the `ERROR` step.
|
|
|
|
TIP: Problems with rollover aliases are a common cause of errors.
|
|
Consider using <<data-streams, data streams>> instead of managing rollover with aliases.
|
|
|
|
[discrete]
|
|
==== Rollover alias [x] can point to multiple indices, found duplicated alias [x] in index template [z]
|
|
|
|
The target rollover alias is specified in an index template's `index.lifecycle.rollover_alias` setting.
|
|
You need to explicitly configure this alias _one time_ when you
|
|
<<ilm-gs-alias-bootstrap, bootstrap the initial index>>.
|
|
The rollover action then manages setting and updating the alias to
|
|
<<rollover-index-api-desc, roll over>> to each subsequent index.
|
|
|
|
Do not explicitly configure this same alias in the aliases section of an index template.
|
|
|
|
[discrete]
|
|
==== index.lifecycle.rollover_alias [x] does not point to index [y]
|
|
|
|
Either the index is using the wrong alias or the alias does not exist.
|
|
|
|
Check the `index.lifecycle.rollover_alias` <<indices-get-settings, index setting>>.
|
|
To see what aliases are configured, use <<cat-alias, _cat/aliases>>.
|
|
|
|
[discrete]
|
|
==== Setting [index.lifecycle.rollover_alias] for index [y] is empty or not defined
|
|
|
|
The `index.lifecycle.rollover_alias` setting must be configured for the rollover action to work.
|
|
|
|
Update the index settings to set `index.lifecycle.rollover_alias`.
|
|
|
|
[discrete]
|
|
==== Alias [x] has more than one write index [y,z]
|
|
|
|
Only one index can be designated as the write index for a particular alias.
|
|
|
|
Use the <<indices-aliases, aliases>> API to set `is_write_index:false` for all but one index.
|
|
|
|
[discrete]
|
|
==== index name [x] does not match pattern ^.*-\d+
|
|
|
|
The index name must match the regex pattern `^.*-\d+` for the rollover action to work.
|
|
The most common problem is that the index name does not contain trailing digits.
|
|
For example, `my-index` does not match the pattern requirement.
|
|
|
|
Append a numeric value to the index name, for example `my-index-000001`.
|
|
|
|
[discrete]
|
|
==== CircuitBreakingException: [x] data too large, data for [y]
|
|
|
|
This indicates that the cluster is hitting resource limits.
|
|
|
|
Before continuing to set up {ilm-init}, you'll need to take steps to alleviate the resource issues.
|
|
For more information, see <<circuit-breaker-errors>>.
|
|
|
|
[discrete]
|
|
==== High disk watermark [x] exceeded on [y]
|
|
|
|
This indicates that the cluster is running out of disk space.
|
|
This can happen when you don't have {ilm} set up to roll over from hot to warm nodes.
|
|
|
|
Consider adding nodes, upgrading your hardware, or deleting unneeded indices.
|
|
|
|
[discrete]
|
|
==== security_exception: action [<action-name>] is unauthorized for user [<user-name>] with roles [<role-name>], this action is granted by the index privileges [manage_follow_index,manage,all]
|
|
|
|
This indicates the ILM action cannot be executed because the user used by ILM to perform the action doesn't have the proper privileges. This can happen when user's privileges has been dropped after updating the ILM policy.
|
|
ILM actions are run as though they were performed by the last user who modify the policy. The account used to create or modify the policy from should have permissions to perform all operations that are part of that policy.
|