mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-29 01:44:36 -04:00
9170113036
The `shards_availability` indicator diagnoses the condition where indices need to be restored from snapshot. Starting with 8.0 using feature_states when restoring from snapshot is mandatory. This adds support for the `FEATURE_STATE` affected resource to aid with building up the snapshot restore API call (which will need to include all the indices and feature states reported by the restore-from-snapshot diagnosis). Note that the health API will not report any indices that are part of a feature state.
486 lines
16 KiB
Text
486 lines
16 KiB
Text
// tag::cloud[]
|
|
In order to restore the indices and data streams that are missing data:
|
|
|
|
**Use {kib}**
|
|
|
|
//tag::kibana-api-ex[]
|
|
. Log in to the {ess-console}[{ecloud} console].
|
|
+
|
|
|
|
. On the **Elasticsearch Service** panel, click the name of your deployment.
|
|
+
|
|
|
|
NOTE: If the name of your deployment is disabled your {kib} instances might be
|
|
unhealthy, in which case please contact https://support.elastic.co[Elastic Support].
|
|
If your deployment doesn't include {kib}, all you need to do is
|
|
{cloud}/ec-access-kibana.html[enable it first].
|
|
+
|
|
. Open your deployment's side navigation menu (placed under the Elastic logo in the upper left corner)
|
|
and go to **Dev Tools > Console**.
|
|
+
|
|
[role="screenshot"]
|
|
image::images/kibana-console.png[{kib} Console,align="center"]
|
|
|
|
. To view the affected indices using the <<cat-indices,cat indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _cat/indices?v&health=red&h=index,status,health
|
|
----
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
index status health
|
|
.ds-my-data-stream-2022.06.17-000001 open red
|
|
kibana_sample_data_flights open red
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The `red` health of the indices above indicates that these indices are missing primary shards,
|
|
meaning they are missing data.
|
|
+
|
|
. In order to restore the data we need to find a snapshot that contains these two indices. To find
|
|
such a snapshot use the
|
|
<<get-snapshot-api,get snapshot API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _snapshot/my_repository/*?verbose=false
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"snapshots" : [
|
|
{
|
|
"snapshot" : "snapshot-20200617", <1>
|
|
"uuid" : "dZyPs1HyTwS-cnKdH08EPg",
|
|
"repository" : "my_repository", <2>
|
|
"indices" : [ <3>
|
|
".apm-agent-configuration",
|
|
".apm-custom-link",
|
|
".ds-ilm-history-5-2022.06.17-000001",
|
|
".ds-my-data-stream-2022.06.17-000001",
|
|
".geoip_databases",
|
|
".kibana-event-log-8.2.2-000001",
|
|
".kibana_8.2.2_001",
|
|
".kibana_task_manager_8.2.2_001",
|
|
"kibana_sample_data_ecommerce",
|
|
"kibana_sample_data_flights",
|
|
"kibana_sample_data_logs"
|
|
],
|
|
"data_streams" : [ ],
|
|
"state" : "SUCCESS" <4>
|
|
}
|
|
],
|
|
"total" : 1,
|
|
"remaining" : 0
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
<1> The name of the snapshot.
|
|
+
|
|
<2> The repository of the snapshot.
|
|
+
|
|
<3> The indices backed up in the snapshot.
|
|
+
|
|
<4> If the snapshot was successful.
|
|
|
|
. The snapshot `snapshot-20200617` contains the two indices we want to restore.
|
|
You might have multiple snapshots from which you could restore the target indices. Choose the latest snapshot.
|
|
+
|
|
. Now that we found a snapshot, we will proceed with the data stream preparation for restoring the lost data. We will
|
|
check the index metadata to see if any index is part of a data stream:
|
|
+
|
|
[source,console]
|
|
----
|
|
GET kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001?features=settings&flat_settings
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
{
|
|
".ds-my-data-stream-2022.06.17-000001" : { <1>
|
|
"aliases" : { },
|
|
"mappings" : { },
|
|
"settings" : { <2>
|
|
"index.creation_date" : "1658406121699",
|
|
"index.hidden" : "true",
|
|
"index.lifecycle.name" : "my-lifecycle-policy",
|
|
"index.number_of_replicas" : "1",
|
|
"index.number_of_shards" : "1",
|
|
"index.provided_name" : ".ds-my-data-stream-2022.06.17-000001",
|
|
"index.routing.allocation.include._tier_preference" : "data_hot",
|
|
"index.uuid" : "HmlFXp6VSu2XbQ-O3hVrwQ",
|
|
"index.version.created" : "8020299"
|
|
},
|
|
"data_stream" : "my-data-stream" <3>
|
|
},
|
|
"kibana_sample_data_flights" : { <4>
|
|
"aliases" : { },
|
|
"mappings" : { },
|
|
"settings" : {
|
|
"index.creation_date" : "1655121541454",
|
|
"index.number_of_replicas" : "0",
|
|
"index.number_of_shards" : "1",
|
|
"index.provided_name" : "kibana_sample_data_flights",
|
|
"index.routing.allocation.include._tier_preference" : "data_content",
|
|
"index.uuid" : "jMOlwKPPSzSraeeBWyuoDA",
|
|
"index.version.created" : "8020299"
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
<1> The name of an index.
|
|
+
|
|
<2> The settings of this index that contains the metadata we are looking for.
|
|
+
|
|
<3> This indicates that this index is part of a data stream and displays the data stream name.
|
|
+
|
|
<4> The name of the other index we requested.
|
|
+
|
|
The response above shows that `kibana_sample_data_flights` is not part of a data stream because it doesn't have a
|
|
field called `data_stream` in the settings.
|
|
+
|
|
On the contrary, `.ds-my-data-stream-2022.06.17-000001` is part of the data stream called `my-data-stream`. When you
|
|
find an index like this, which belongs to a data stream, you need to check if data are still being indexed.
|
|
You can see that by checking the `settings`, if you can find this property: `"index.lifecycle.indexing_complete" : "true"`,
|
|
it means that indexing is completed in this index and you can continue to the next step.
|
|
+
|
|
If `index.lifecycle.indexing_complete` is not there or is configured to `false` you need to rollover the data stream so you can restore the missing data without blocking
|
|
the ingestion of new data. The following command will achieve that.
|
|
+
|
|
[source,console]
|
|
----
|
|
POST my-data-stream/_rollover
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
|
|
. Now that the data stream preparation is done, we will close the target indices by using the
|
|
<<indices-close, close indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
POST kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001/_close
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
You can confirm that they are closed with
|
|
the <<cat-indices, cat indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _cat/indices?v&health=red&h=index,status,health
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
index status health
|
|
.ds-my-data-stream-2022.06.17-000001 close red
|
|
kibana_sample_data_flights close red
|
|
----
|
|
+
|
|
. The indices are closed, now we can restore them from snapshots without causing
|
|
any complications using the <<restore-snapshot-api, restore snapshot API>>:
|
|
+
|
|
[source,console]
|
|
----
|
|
POST _snapshot/my_repository/snapshot-20200617/_restore
|
|
{
|
|
"indices": "kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001", <1>
|
|
"include_aliases": true <2>
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
<1> The indices to restore.
|
|
+
|
|
<2> We also want to restore the aliases.
|
|
+
|
|
NOTE: If any <<feature-state,feature states>> need to be restored we'll need to specify them using the
|
|
`feature_states` field and the indices that belong to the feature states we restore must not be specified under `indices`.
|
|
The <<health-api, Health API>> returns both the `indices` and `feature_states` that need to be restored for the restore from snapshot diagnosis. e.g.:
|
|
+
|
|
[source,console]
|
|
----
|
|
POST _snapshot/my_repository/snapshot-20200617/_restore
|
|
{
|
|
"feature_states": [ "geoip" ],
|
|
"indices": "kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001",
|
|
"include_aliases": true
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
|
|
. Finally we can verify that the indices health is now `green` via the <<cat-indices,cat indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _cat/indices?v&index=.ds-my-data-stream-2022.06.17-000001,kibana_sample_data_flightsh=index,status,health
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
index status health
|
|
.ds-my-data-stream-2022.06.17-000001 open green
|
|
kibana_sample_data_flights open green
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
As we can see above the indices are `green` and open. The issue is resolved.
|
|
|
|
For more guidance on creating and restoring snapshots see
|
|
<<snapshot-restore, this guide>>.
|
|
|
|
|
|
//end::kibana-api-ex[]
|
|
// end::cloud[]
|
|
|
|
// tag::self-managed[]
|
|
In order to restore the indices that are missing shards:
|
|
|
|
. View the affected indices using the <<cat-indices,cat indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _cat/indices?v&health=red&h=index,status,health
|
|
----
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
index status health
|
|
.ds-my-data-stream-2022.06.17-000001 open red
|
|
kibana_sample_data_flights open red
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The `red` health of the indices above indicates that these indices are missing primary shards,
|
|
meaning they are missing data.
|
|
+
|
|
. In order to restore the data we need to find a snapshot that contains these two indices. To find
|
|
such a snapshot use the
|
|
<<get-snapshot-api,get snapshot API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _snapshot/my_repository/*?verbose=false
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"snapshots" : [
|
|
{
|
|
"snapshot" : "snapshot-20200617", <1>
|
|
"uuid" : "dZyPs1HyTwS-cnKdH08EPg",
|
|
"repository" : "my_repository", <2>
|
|
"indices" : [ <3>
|
|
".apm-agent-configuration",
|
|
".apm-custom-link",
|
|
".ds-ilm-history-5-2022.06.17-000001",
|
|
".ds-my-data-stream-2022.06.17-000001",
|
|
".geoip_databases",
|
|
".kibana-event-log-8.2.2-000001",
|
|
".kibana_8.2.2_001",
|
|
".kibana_task_manager_8.2.2_001",
|
|
"kibana_sample_data_ecommerce",
|
|
"kibana_sample_data_flights",
|
|
"kibana_sample_data_logs"
|
|
],
|
|
"data_streams" : [ ],
|
|
"state" : "SUCCESS" <4>
|
|
}
|
|
],
|
|
"total" : 1,
|
|
"remaining" : 0
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
<1> The name of the snapshot.
|
|
+
|
|
<2> The repository of the snapshot.
|
|
+
|
|
<3> The indices backed up in the snapshot.
|
|
+
|
|
<4> If the snapshot was successful.
|
|
|
|
. The snapshot `snapshot-20200617` contains the two indices we want to restore.
|
|
You might have multiple snapshots from which you could restore the target indices. Choose the latest snapshot.
|
|
+
|
|
. Now that we found a snapshot, we will proceed with the data stream preparation for restoring the lost data. We will
|
|
check the index metadata to see if any index is part of a data stream:
|
|
+
|
|
[source,console]
|
|
----
|
|
GET kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001?features=settings&flat_settings
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
{
|
|
".ds-my-data-stream-2022.06.17-000001" : { <1>
|
|
"aliases" : { },
|
|
"mappings" : { },
|
|
"settings" : { <2>
|
|
"index.creation_date" : "1658406121699",
|
|
"index.hidden" : "true",
|
|
"index.lifecycle.name" : "my-lifecycle-policy",
|
|
"index.number_of_replicas" : "1",
|
|
"index.number_of_shards" : "1",
|
|
"index.provided_name" : ".ds-my-data-stream-2022.06.17-000001",
|
|
"index.routing.allocation.include._tier_preference" : "data_hot",
|
|
"index.uuid" : "HmlFXp6VSu2XbQ-O3hVrwQ",
|
|
"index.version.created" : "8020299"
|
|
},
|
|
"data_stream" : "my-data-stream" <3>
|
|
},
|
|
"kibana_sample_data_flights" : { <4>
|
|
"aliases" : { },
|
|
"mappings" : { },
|
|
"settings" : {
|
|
"index.creation_date" : "1655121541454",
|
|
"index.number_of_replicas" : "0",
|
|
"index.number_of_shards" : "1",
|
|
"index.provided_name" : "kibana_sample_data_flights",
|
|
"index.routing.allocation.include._tier_preference" : "data_content",
|
|
"index.uuid" : "jMOlwKPPSzSraeeBWyuoDA",
|
|
"index.version.created" : "8020299"
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
<1> The name of an index.
|
|
+
|
|
<2> The settings of this index that contains the metadata we are looking for.
|
|
+
|
|
<3> This indicates that this index is part of a data stream and displays the data stream name.
|
|
+
|
|
<4> The name of the other index we requested.
|
|
+
|
|
The response above shows that `kibana_sample_data_flights` is not part of a data stream because it doesn't have a
|
|
field called `data_stream` in the settings.
|
|
+
|
|
On the contrary, `.ds-my-data-stream-2022.06.17-000001` is part of the data stream called `my-data-stream`. When you
|
|
find an index like this, which belongs to a data stream, you need to check if data are still being indexed.
|
|
You can see that by checking the `settings`, if you can find this property: `"index.lifecycle.indexing_complete" : "true"`,
|
|
it means that indexing is completed in this index and you can continue to the next step.
|
|
+
|
|
If `index.lifecycle.indexing_complete` is not there or is configured to `false` you need to rollover the data stream so you can restore the missing data without blocking
|
|
the ingestion of new data. The following command will achieve that.
|
|
+
|
|
[source,console]
|
|
----
|
|
POST my-data-stream/_rollover
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
|
|
. Now that the data stream preparation is done, we will close the target indices by using the
|
|
<<indices-close, close indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
POST kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001/_close
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
You can confirm that they are closed with
|
|
the <<cat-indices, cat indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _cat/indices?v&health=red&h=index,status,health
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
index status health
|
|
.ds-my-data-stream-2022.06.17-000001 close red
|
|
kibana_sample_data_flights close red
|
|
----
|
|
+
|
|
. The indices are closed, now we can restore them from snapshots without causing
|
|
any complications using the <<restore-snapshot-api, restore snapshot API>>:
|
|
+
|
|
[source,console]
|
|
----
|
|
POST _snapshot/my_repository/snapshot-20200617/_restore
|
|
{
|
|
"indices": "kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001", <1>
|
|
"include_aliases": true <2>
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
<1> The indices to restore.
|
|
+
|
|
<2> We also want to restore the aliases.
|
|
+
|
|
NOTE: If any <<feature-state,feature states>> need to be restored we'll need to specify them using the
|
|
`feature_states` field and the indices that belong to the feature states we restore must not be specified under `indices`.
|
|
The <<health-api, Health API>> returns both the `indices` and `feature_states` that need to be restored for the restore from snapshot diagnosis. e.g.:
|
|
+
|
|
[source,console]
|
|
----
|
|
POST _snapshot/my_repository/snapshot-20200617/_restore
|
|
{
|
|
"feature_states": [ "geoip" ],
|
|
"indices": "kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001",
|
|
"include_aliases": true
|
|
}
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
|
|
. Finally we can verify that the indices health is now `green` via the <<cat-indices,cat indices API>>.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET _cat/indices?v&index=.ds-my-data-stream-2022.06.17-000001,kibana_sample_data_flightsh=index,status,health
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
The response will look like this:
|
|
+
|
|
[source,console-result]
|
|
----
|
|
index status health
|
|
.ds-my-data-stream-2022.06.17-000001 open green
|
|
kibana_sample_data_flights open green
|
|
----
|
|
// TEST[skip:illustration purposes only]
|
|
+
|
|
As we can see above the indices are `green` and open. The issue is resolved.
|
|
|
|
For more guidance on creating and restoring snapshots see
|
|
<<snapshot-restore, this guide>>.
|
|
// end::self-managed[]
|