[float] [[saved-object-migrations]] === Saved object migrations Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <> are compatible with the new version. NOTE: To help you prepare for the upgrade to 7.0.0, 6.7.0 includes an https://www.elastic.co/guide/en/kibana/6.7/upgrade-assistant.html[*Upgrade Assistant*]. To access the assistant, go to *Management > 7.0 Upgrade Assistant*. WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}. WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly. [float] [[upgrade-migrations-process]] ==== Background Saved objects are stored in two indices: * `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`. * `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`. The index aliases `.kibana` and `.kibana_task_manager` always point to the most up-to-date saved object indices. When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic. Before you upgrade, shut down old nodes to prevent losing acknowledged writes. To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions. .Saved object indices and aliases per {kib} version [options="header"] |======================= |Upgrading from version | Outdated index (alias) | 6.0.0 through 6.4.x | `.kibana` `.kibana_task_manager_7.12.0_001` (`.kibana_task_manager` alias) | 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias) | 7.4.0 through 7.11.x | `.kibana_N` (`.kibana` alias) `.kibana_task_manager_N` (`.kibana_task_manager` alias) |======================= ==== Upgrading multiple {kib} instances When upgrading several {kib} instances connected to the same {es} cluster, ensure that all outdated instances are shut down before starting the upgrade. Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel, which allows all instances to participate in the upgrade migration in parallel. For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects, you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to complete the upgrade migration before bringing up the remaining instances. [float] [[preventing-migration-failures]] ==== Preventing migration failures Review the common causes of {kib} upgrade failures and how to prevent them. [float] ===== timeout_exception or receive_timeout_transport_exception There is a known issue in 7.12.0 for users who tried the {fleet} beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index, which causes {kib} to log errors such as: [source,sh] -------------------------------------------- Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] -------------------------------------------- For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue]. [float] ===== Corrupt saved objects To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster, especially when there are custom integrations that create saved objects in your environment. Saved objects that are corrupted through manual editing or integrations cause migration failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`. For a successful upgrade migration, you must fix or delete corrupt documents. For example, you receive the following error message: [source,sh] -------------------------------------------- Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. -------------------------------------------- To delete the documents that cause migrations to fail, take the following steps: . Remove the write block which the migration system has placed on the previous index: + [source,sh] -------------------------------------------- PUT .kibana_7.12.1_001/_settings { "index": { "blocks.write": false } } -------------------------------------------- . Delete the corrupt document: + [source,sh] -------------------------------------------- DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275 -------------------------------------------- . Restart {kib}. + The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**. Be sure you have a snapshot before you delete the corrupt document. If you are unable to restore from a snapshot, it is recommended to also delete the `temp` and `target` indices the migration creates before you restart {kib} and retry the snapshot restore. [float] ===== User defined index templates that cause new `.kibana*` indices to have incompatible settings or mappings Matching index templates that specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades. To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates. NOTE: In {kib} 6.5.0 and earlier, {kib} creates a `kibana_index_template:.kibana` index template and uses a `.kibana` index pattern. You do not need to change or remove the index template. [float] ===== An unhealthy {es} cluster Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has: * Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices * Sufficient heap size * A "green" cluster status [float] ===== Different versions of {kib} connected to the same {es} index When you perform an upgrade migration of different {kib} versions, the migration can fail. Ensure that all {kib} instances are running the same version, configuration, and plugins. [float] ===== Incompatible `xpack.tasks.index` configuration setting In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`, upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration setting prevents upgrade migrations from starting. [float] [[resolve-migrations-failures]] ==== Resolving migration failures If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to perform the migration again when the process restarts. Do not delete any saved objects indices to to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and later does not require deleting indices to release a failed migration lock. If upgrade migrations fail repeatedly, refer to <>. When you address the root cause for the migration failure, {kib} automatically retries the migration. If you're unable to resolve a failed migration, contact Support. [float] [[upgrade-migrations-rolling-back]] ==== Rolling back to a previous version of {kib} If you've followed <> and <>, and {kib} is still unable to successfully upgrade, rollback {kib} until you're able to identify and fix the root cause. WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}. Any changes made after an upgrade are lost when you roll back to a previous version. To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version. [float] ===== Roll back by restoring a backup snapshot . Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state. By default, snapshots include the `kibana` feature state. . To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. . To delete all saved object indices, use `DELETE /.kibana*`. . {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot. . Start all {kib} instances on the older version you want to rollback to. [float] ===== (Not recommended) Roll back without a backup snapshot . To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. . {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state. . Delete the version-specific indices created by the failed upgrade migration. + For example, to rollback from a failed upgrade to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`. . Inspect the output of `GET /_cat/aliases`. + If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually. Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`. . To remove the write block from the roll back indices, use `PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}` . Start {kib} on the older version you want to rollback to. [float] [[upgrade-migrations-old-indices]] ==== Handling old `.kibana_N` indices After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc). {kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to. The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.