kibana/docs/setup/upgrade/upgrade-migrations.asciidoc

[float]
[[saved-object-migrations]]
=== Saved object migrations

Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.

NOTE: To help you prepare for the upgrade to 7.0.0, 6.7.0 includes an https://www.elastic.co/guide/en/kibana/6.7/upgrade-assistant.html[*Upgrade Assistant*]. 
To access the assistant, go to *Management > 7.0 Upgrade Assistant*.

WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.

WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.

[float]
[[upgrade-migrations-process]]
==== Background

Saved objects are stored in two indices:

* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.

The index aliases `.kibana` and `.kibana_task_manager` always point to
the most up-to-date saved object indices.

When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions.

.Saved object indices and aliases per {kib} version
[options="header"]
|=======================
|Upgrading from version | Outdated index (alias)
| 6.0.0 through 6.4.x    | `.kibana`

`.kibana_task_manager_7.12.0_001` (`.kibana_task_manager` alias)
| 6.5.0 through 7.3.x    | `.kibana_N` (`.kibana` alias)
| 7.4.0 through 7.11.x
| `.kibana_N` (`.kibana` alias)

`.kibana_task_manager_N` (`.kibana_task_manager` alias)
|=======================

==== Upgrading multiple {kib} instances
When upgrading several {kib} instances connected to the same {es} cluster,
ensure that all outdated instances are shut down before starting the upgrade.

Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel, 
which allows all instances to participate in the upgrade migration in parallel.

For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
complete the upgrade migration before bringing up the remaining instances.

[float]
[[preventing-migration-failures]]
==== Preventing migration failures
Review the common causes of {kib} upgrade failures and how to prevent them.

[float]
===== timeout_exception or receive_timeout_transport_exception
There is a known issue in 7.12.0 for users who tried the {fleet} beta.
Upgrade migrations fail because of a large number of documents in the `.kibana` index, which causes {kib} to log errors such as:


[source,sh]
--------------------------------------------
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]

Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
--------------------------------------------

For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue].

[float]
===== Corrupt saved objects
To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster, 
especially when there are custom integrations that create saved objects in your environment.

Saved objects that are corrupted through manual editing or integrations cause migration
failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`.
For a successful upgrade migration, you must fix or delete corrupt documents.

For example, you receive the following error message:

[source,sh]
--------------------------------------------
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
--------------------------------------------

To delete the documents that cause migrations to fail, take the following steps:

. Remove the write block which the migration system has placed on the previous index:
+
[source,sh]
--------------------------------------------
PUT .kibana_7.12.1_001/_settings
{
  "index": {
    "blocks.write": false
  }
}
--------------------------------------------

. Delete the corrupt document:
+
[source,sh]
--------------------------------------------
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
--------------------------------------------

. Restart {kib}.
+
The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**.

Be sure you have a snapshot before you delete the corrupt document. If you are unable to restore from a snapshot, it is recommended to also delete the `temp` and `target` indices the migration creates before you restart {kib} and retry the snapshot restore.

[float]
===== User defined index templates that cause new `.kibana*` indices to have incompatible settings or mappings
Matching index templates that specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades.

To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates.

NOTE: In {kib} 6.5.0 and earlier, {kib} creates a `kibana_index_template:.kibana` index template
and uses a `.kibana` index pattern. You do not need to change or remove the index template.

[float]
===== An unhealthy {es} cluster
Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has:

 * Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices
 * Sufficient heap size
 * A "green" cluster status

[float]
===== Different versions of {kib} connected to the same {es} index
When you perform an upgrade migration of different {kib} versions, the migration can fail.
Ensure that all {kib} instances are running the same version, configuration, and plugins.

[float]
===== Incompatible `xpack.tasks.index` configuration setting
In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`,
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration setting prevents upgrade migrations from starting.

[float]
[[resolve-migrations-failures]]
==== Resolving migration failures

If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
perform the migration again when the process restarts. Do not delete any saved objects indices to
to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
later does not require deleting indices to release a failed migration lock.

If upgrade migrations fail repeatedly, refer to
<<preventing-migration-failures, preventing migration failures>>.
When you address the root cause for the migration failure,
{kib} automatically retries the migration.
If you're unable to resolve a failed migration, contact Support.

[float]
[[upgrade-migrations-rolling-back]]
==== Rolling back to a previous version of {kib}

If you've followed <<preventing-migration-failures, preventing migration failures>>
and <<resolve-migrations-failures, resolving migration failures>>, and
{kib} is still unable to successfully upgrade, rollback {kib} until
you're able to identify and fix the root cause.

WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with
your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}.
Any changes made after an upgrade are lost when you roll back to a previous version.

To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version.

[float]
===== Roll back by restoring a backup snapshot

. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
   By default, snapshots include the `kibana` feature state.
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
. To delete all saved object indices, use `DELETE /.kibana*`.
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
. Start all {kib} instances on the older version you want to rollback to.

[float]
===== (Not recommended) Roll back without a backup snapshot

. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state.
. Delete the version-specific indices created by the failed upgrade migration.
+
For example, to rollback from a failed upgrade
to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`.
. Inspect the output of `GET /_cat/aliases`.
+
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`.
. To remove the write block from the roll back indices, use
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
. Start {kib} on the older version you want to rollback to.

[float]
[[upgrade-migrations-old-indices]]
==== Handling old `.kibana_N` indices

After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.