mirror of
https://github.com/elastic/kibana.git
synced 2025-04-23 17:28:26 -04:00
* [DOCS] Adds redirect for Standard Upgrade doc
* [DOCS] Updates Upgrade docs
* [DOCS] Incorporates review comments
* [DOCS] Adds info on saved objects
* [DOCS] Fixes broken link
* [DOCS] Fixes placement of section on unknown saved objects
* [DOCS] Removes duplicate topic
(cherry picked from commit a13dafc7dd)
This commit is contained in:
gchaps
GitHub
parent
21e54e0195
commit
fc3359ea86
9 changed files with 261 additions and 247 deletions
|
|
@ -36,9 +36,3 @@ The services that core provides are:
|
|||
* <<logging-service, Logging service>>
|
||||
* <<saved-objects-service, Saved Objects service>>
|
||||
* <<ui-settings-service, UI settings service>>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -40,6 +40,8 @@ include::core/http-service.asciidoc[leveloffset=+1]
|
|||
|
||||
include::core/logging-service.asciidoc[leveloffset=+1]
|
||||
|
||||
include::core/logging-configuration-migration.asciidoc[leveloffset=+1]
|
||||
|
||||
include::core/saved-objects-service.asciidoc[leveloffset=+1]
|
||||
|
||||
include::core/uisettings-service.asciidoc[leveloffset=+1]
|
||||
|
|
|
|||
|
|
@ -386,16 +386,16 @@ This content has moved. Refer to <<managing-data-views>>.
|
|||
|
||||
This content has moved. Refer to <<kibana-role-management>>.
|
||||
|
||||
[role="exclude",id="logging-configuration-changes"]
|
||||
== Logging configuration changes
|
||||
|
||||
This content has moved. Refer to <<logging-config-changes>>.
|
||||
|
||||
[role="exclude",id="upgrade-migrations"]
|
||||
== Upgrade migrations
|
||||
|
||||
This content has moved. Refer to <<saved-object-migrations>>.
|
||||
|
||||
[role="exclude",id="upgrade-standard"]
|
||||
== Standard Upgrade
|
||||
|
||||
This content has moved. Refer to {stack-ref}/upgrading-kibana.html[Upgrade Kibana].
|
||||
|
||||
[role="exclude",id="upgrade-assistant"]
|
||||
== Upgrade Assistant
|
||||
|
||||
|
|
|
|||
|
|
@ -1,31 +1,72 @@
|
|||
[[upgrade]]
|
||||
== Upgrade {kib}
|
||||
|
||||
To upgrade from 7.16.0 or earlier to {version},
|
||||
To upgrade from 7.16.0 or earlier to {version},
|
||||
**you must first upgrade to {prev-major-last}**, which enables you to use the *Upgrade Assistant* to
|
||||
{stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare for the upgrade].
|
||||
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*.
|
||||
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*.
|
||||
|
||||
Rolling upgrades are unsupported in {kib}. To upgrade,
|
||||
Rolling upgrades are unsupported in {kib}. To upgrade,
|
||||
you must shut down all {kib} instances, install the new software, and restart {kib}.
|
||||
Upgrading while older {kib} instances are running can cause data loss or upgrade failures.
|
||||
|
||||
[WARNING]
|
||||
====
|
||||
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>.
|
||||
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>.
|
||||
In case of an upgrade failure, you can roll back to an
|
||||
earlier version of {kib}. To roll back, you **must** have a
|
||||
{ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature
|
||||
state. By default, snapshots include the `kibana` feature state.
|
||||
====
|
||||
|
||||
For more information about upgrading,
|
||||
For more information about upgrading,
|
||||
refer to {stack-ref}/upgrading-elastic-stack.html[Upgrading to Elastic {version}.]
|
||||
|
||||
IMPORTANT: You can upgrade to pre-release versions for testing,
|
||||
IMPORTANT: You can upgrade to pre-release versions for testing,
|
||||
but upgrading from a pre-release to the General Available version is unsupported.
|
||||
You should use pre-release versions only for testing in a temporary environment.
|
||||
|
||||
include::upgrade/upgrade-migrations.asciidoc[leveloffset=-1]
|
||||
[float]
|
||||
=== Upgrading multiple {kib} instances
|
||||
When upgrading several {kib} instances connected to the same {es} cluster,
|
||||
ensure that all outdated instances are shut down before starting the upgrade.
|
||||
|
||||
include::upgrade/logging-configuration-changes.asciidoc[]
|
||||
Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel,
|
||||
which allows all instances to participate in the upgrade migration in parallel.
|
||||
|
||||
For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
|
||||
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
|
||||
complete the upgrade migration before bringing up the remaining instances.
|
||||
|
||||
[float]
|
||||
[[preventing-migration-failures]]
|
||||
=== Preparing for migration
|
||||
|
||||
There are extra steps you can follow to ensure you are ready for migration.
|
||||
|
||||
[float]
|
||||
==== Ensure your {es} cluster is healthy
|
||||
Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has:
|
||||
|
||||
* Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices
|
||||
* Sufficient heap size
|
||||
* A "green" cluster status
|
||||
|
||||
[float]
|
||||
==== Ensure that all {kib} instances are the same
|
||||
When you perform an upgrade migration of different {kib} versions, the migration can fail.
|
||||
Ensure that all {kib} instances are running the same version, configuration, and plugins.
|
||||
|
||||
[float]
|
||||
==== Back up your data
|
||||
Be sure to have a {ref}/snapshot-restore.html[snapshot] of all your data before attempting a migration.
|
||||
If something goes wrong during migration, you can restore from the snapshot and try again.
|
||||
|
||||
Review the <<resolve-migrations-failures,common causes of {kib} upgrade failures>> and how to prevent them.
|
||||
|
||||
|
||||
include::upgrade/saved-objects-migration.asciidoc[]
|
||||
|
||||
include::upgrade/resolving-migration-failures.asciidoc[]
|
||||
|
||||
include::upgrade/rollback-migration.asciidoc[]
|
||||
|
|
|
|||
125
docs/setup/upgrade/resolving-migration-failures.asciidoc
Normal file
125
docs/setup/upgrade/resolving-migration-failures.asciidoc
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
[[resolve-migrations-failures]]
|
||||
=== Resolve migration failures
|
||||
|
||||
Migrating {kib} primarily involves migrating saved object documents to be compatible
|
||||
with the new version.
|
||||
|
||||
[float]
|
||||
==== Resolve saved object migration failures
|
||||
|
||||
If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
|
||||
perform the migration again when the process restarts. Do not delete any saved objects indices to
|
||||
to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
|
||||
later does not require deleting indices to release a failed migration lock.
|
||||
|
||||
If upgrade migrations fail repeatedly, refer to
|
||||
<<preventing-migration-failures, preparing for migration>>.
|
||||
When you address the root cause for the migration failure,
|
||||
{kib} automatically retries the migration.
|
||||
If you're unable to resolve a failed migration, contact Support.
|
||||
|
||||
|
||||
[float]
|
||||
[[upgrade-migrations-old-indices]]
|
||||
==== Handle old `.kibana_N` indices
|
||||
|
||||
After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
|
||||
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
|
||||
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.
|
||||
|
||||
[float]
|
||||
==== Handle known issues with {fleet} beta
|
||||
If you see a`timeout_exception` or `receive_timeout_transport_exception` error,
|
||||
it might be from a known known issue in 7.12.0 if you tried the {fleet} beta.
|
||||
Upgrade migrations fail because of a large number of documents in the `.kibana` index,
|
||||
which causes {kib} to log errors such as:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
|
||||
|
||||
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
|
||||
--------------------------------------------
|
||||
|
||||
For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue].
|
||||
|
||||
|
||||
[float]
|
||||
==== Handle corrupt saved objects
|
||||
To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster,
|
||||
especially when there are custom integrations that create saved objects in your environment.
|
||||
|
||||
Saved objects that are corrupted through manual editing or integrations cause migration
|
||||
failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`.
|
||||
For a successful upgrade migration, you must fix or delete corrupt documents.
|
||||
|
||||
For example, you receive the following error message:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
|
||||
--------------------------------------------
|
||||
|
||||
To delete the documents that cause migrations to fail, take the following steps:
|
||||
|
||||
. Remove the write block which the migration system has placed on the previous index:
|
||||
+
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
PUT .kibana_7.12.1_001/_settings
|
||||
{
|
||||
"index": {
|
||||
"blocks.write": false
|
||||
}
|
||||
}
|
||||
--------------------------------------------
|
||||
|
||||
. Delete the corrupt document:
|
||||
+
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
|
||||
--------------------------------------------
|
||||
|
||||
. Restart {kib}.
|
||||
+
|
||||
The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**.
|
||||
|
||||
[float]
|
||||
[[unknown-saved-object-types]]
|
||||
==== Handle documents for unknown saved objects
|
||||
Migrations will fail if saved objects belong to an unknown
|
||||
saved object type. Unknown saved objects are typically caused by
|
||||
to the {es} index, or by disabling a plugin that had previously
|
||||
created a saved object.
|
||||
|
||||
We recommend using the {kibana-ref-all}/7.17/upgrade-assistant.html[Upgrade Assistant]
|
||||
to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved
|
||||
object types will also log the following warning message:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes.
|
||||
--------------------------------------------
|
||||
|
||||
If you fail to remedy this, your upgrade to 8.0+ will fail with a message like:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index.
|
||||
--------------------------------------------
|
||||
|
||||
[float]
|
||||
==== Handle incompatible settings or mappings
|
||||
Matching index templates that specify `settings.refresh_interval` or
|
||||
`mappings` are known to interfere with {kib} upgrades.
|
||||
This can happen when index templates are defined manually.
|
||||
|
||||
To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates.
|
||||
|
||||
[float]
|
||||
==== Handle incompatible `xpack.tasks.index` configuration setting
|
||||
In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks`
|
||||
with the configuration setting `xpack.tasks.index: ".tasks"`,
|
||||
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration
|
||||
setting prevents upgrade migrations from starting.
|
||||
41
docs/setup/upgrade/rollback-migration.asciidoc
Normal file
41
docs/setup/upgrade/rollback-migration.asciidoc
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
[[upgrade-migrations-rolling-back]]
|
||||
=== Roll back to a previous version of {kib}
|
||||
|
||||
If you've followed <<preventing-migration-failures, preparing for migration>>
|
||||
and <<resolve-migrations-failures, resolving migration failures>>, and
|
||||
{kib} is still unable to successfully upgrade, rollback {kib} until
|
||||
you're able to identify and fix the root cause.
|
||||
|
||||
WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with
|
||||
your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}.
|
||||
Any changes made after an upgrade are lost when you roll back to a previous version.
|
||||
|
||||
To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version.
|
||||
|
||||
[float]
|
||||
==== Roll back by restoring a backup snapshot
|
||||
|
||||
. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
|
||||
By default, snapshots include the `kibana` feature state.
|
||||
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
|
||||
. To delete all saved object indices, use `DELETE /.kibana*`.
|
||||
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
|
||||
. Start all {kib} instances on the older version you want to rollback to.
|
||||
|
||||
[float]
|
||||
==== (Not recommended) Roll back without a backup snapshot
|
||||
|
||||
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
|
||||
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state.
|
||||
. Delete the version-specific indices created by the failed upgrade migration.
|
||||
+
|
||||
For example, to rollback from a failed upgrade
|
||||
to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`.
|
||||
. Inspect the output of `GET /_cat/aliases`.
|
||||
+
|
||||
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
|
||||
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
|
||||
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`.
|
||||
. To remove the write block from the roll back indices, use
|
||||
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
|
||||
. Start {kib} on the older version you want to rollback to.
|
||||
39
docs/setup/upgrade/saved-objects-migration.asciidoc
Normal file
39
docs/setup/upgrade/saved-objects-migration.asciidoc
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
[[saved-object-migrations]]
|
||||
=== Migrate saved objects
|
||||
|
||||
Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.
|
||||
|
||||
NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*].
|
||||
To access the assistant, go to *Stack Management > Upgrade Assistant*.
|
||||
|
||||
WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.
|
||||
|
||||
WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
|
||||
|
||||
[float]
|
||||
[[upgrade-migrations-process]]
|
||||
==== How saved objects migrations work
|
||||
|
||||
Saved objects are stored in two indices:
|
||||
|
||||
* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
|
||||
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.
|
||||
|
||||
The index aliases `.kibana` and `.kibana_task_manager` always point to
|
||||
the most up-to-date saved object indices.
|
||||
|
||||
When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
|
||||
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
|
||||
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
|
||||
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions.
|
||||
|
||||
.Saved object indices and aliases per {kib} version
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Upgrading from version | Outdated index (alias)
|
||||
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias)
|
||||
| 7.4.0 through 7.11.x
|
||||
| `.kibana_N` (`.kibana` alias)
|
||||
|
||||
`.kibana_task_manager_N` (`.kibana_task_manager` alias)
|
||||
|=======================
|
||||
|
|
@ -1,228 +0,0 @@
|
|||
[float]
|
||||
[[saved-object-migrations]]
|
||||
=== Saved object migrations
|
||||
|
||||
Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.
|
||||
|
||||
NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*].
|
||||
To access the assistant, go to *Stack Management > Upgrade Assistant*.
|
||||
|
||||
WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.
|
||||
|
||||
WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
|
||||
|
||||
[float]
|
||||
[[upgrade-migrations-process]]
|
||||
==== Background
|
||||
|
||||
Saved objects are stored in two indices:
|
||||
|
||||
* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
|
||||
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.
|
||||
|
||||
The index aliases `.kibana` and `.kibana_task_manager` always point to
|
||||
the most up-to-date saved object indices.
|
||||
|
||||
When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
|
||||
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
|
||||
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
|
||||
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions.
|
||||
|
||||
.Saved object indices and aliases per {kib} version
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Upgrading from version | Outdated index (alias)
|
||||
|
||||
`.kibana_task_manager_7.12.0_001` (`.kibana_task_manager` alias)
|
||||
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias)
|
||||
| 7.4.0 through 7.11.x
|
||||
| `.kibana_N` (`.kibana` alias)
|
||||
|
||||
`.kibana_task_manager_N` (`.kibana_task_manager` alias)
|
||||
|=======================
|
||||
|
||||
==== Upgrading multiple {kib} instances
|
||||
When upgrading several {kib} instances connected to the same {es} cluster,
|
||||
ensure that all outdated instances are shut down before starting the upgrade.
|
||||
|
||||
Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel,
|
||||
which allows all instances to participate in the upgrade migration in parallel.
|
||||
|
||||
For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
|
||||
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
|
||||
complete the upgrade migration before bringing up the remaining instances.
|
||||
|
||||
[float]
|
||||
[[preventing-migration-failures]]
|
||||
==== Preventing migration failures
|
||||
Review the common causes of {kib} upgrade failures and how to prevent them.
|
||||
|
||||
[float]
|
||||
====== Documents found for unknown saved object types
|
||||
Migrations will fail if saved objects belong to an unknown
|
||||
saved object type. Unknown saved objects are typically caused by
|
||||
to the {es} index, or by disabling a plugin that had previously
|
||||
created a saved object.
|
||||
|
||||
We recommend using the https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[Upgrade Assistant]
|
||||
to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved
|
||||
object types will also log the following warning message:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes.
|
||||
--------------------------------------------
|
||||
|
||||
If you fail to remedy this, your upgrade to 8.0+ will fail with a message like:
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index.
|
||||
--------------------------------------------
|
||||
|
||||
[float]
|
||||
===== timeout_exception or receive_timeout_transport_exception
|
||||
There is a known issue in 7.12.0 for users who tried the {fleet} beta.
|
||||
Upgrade migrations fail because of a large number of documents in the `.kibana` index, which causes {kib} to log errors such as:
|
||||
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
|
||||
|
||||
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
|
||||
--------------------------------------------
|
||||
|
||||
For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue].
|
||||
|
||||
[float]
|
||||
===== Corrupt saved objects
|
||||
To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster,
|
||||
especially when there are custom integrations that create saved objects in your environment.
|
||||
|
||||
Saved objects that are corrupted through manual editing or integrations cause migration
|
||||
failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`.
|
||||
For a successful upgrade migration, you must fix or delete corrupt documents.
|
||||
|
||||
For example, you receive the following error message:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
|
||||
--------------------------------------------
|
||||
|
||||
To delete the documents that cause migrations to fail, take the following steps:
|
||||
|
||||
. Remove the write block which the migration system has placed on the previous index:
|
||||
+
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
PUT .kibana_7.12.1_001/_settings
|
||||
{
|
||||
"index": {
|
||||
"blocks.write": false
|
||||
}
|
||||
}
|
||||
--------------------------------------------
|
||||
|
||||
. Delete the corrupt document:
|
||||
+
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
|
||||
--------------------------------------------
|
||||
|
||||
. Restart {kib}.
|
||||
+
|
||||
The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**.
|
||||
|
||||
Be sure you have a snapshot before you delete the corrupt document. If you are unable to restore from a snapshot, it is recommended to also delete the `temp` and `target` indices the migration creates before you restart {kib} and retry the snapshot restore.
|
||||
|
||||
[float]
|
||||
===== User defined index templates that cause new `.kibana*` indices to have incompatible settings or mappings
|
||||
Matching index templates that specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades.
|
||||
|
||||
To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates.
|
||||
|
||||
[float]
|
||||
===== An unhealthy {es} cluster
|
||||
Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has:
|
||||
|
||||
* Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices
|
||||
* Sufficient heap size
|
||||
* A "green" cluster status
|
||||
|
||||
[float]
|
||||
===== Different versions of {kib} connected to the same {es} index
|
||||
When you perform an upgrade migration of different {kib} versions, the migration can fail.
|
||||
Ensure that all {kib} instances are running the same version, configuration, and plugins.
|
||||
|
||||
[float]
|
||||
===== Incompatible `xpack.tasks.index` configuration setting
|
||||
In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`,
|
||||
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration setting prevents upgrade migrations from starting.
|
||||
|
||||
[float]
|
||||
[[resolve-migrations-failures]]
|
||||
==== Resolving migration failures
|
||||
|
||||
If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
|
||||
perform the migration again when the process restarts. Do not delete any saved objects indices to
|
||||
to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
|
||||
later does not require deleting indices to release a failed migration lock.
|
||||
|
||||
If upgrade migrations fail repeatedly, refer to
|
||||
<<preventing-migration-failures, preventing migration failures>>.
|
||||
When you address the root cause for the migration failure,
|
||||
{kib} automatically retries the migration.
|
||||
If you're unable to resolve a failed migration, contact Support.
|
||||
|
||||
[float]
|
||||
[[upgrade-migrations-rolling-back]]
|
||||
==== Rolling back to a previous version of {kib}
|
||||
|
||||
If you've followed <<preventing-migration-failures, preventing migration failures>>
|
||||
and <<resolve-migrations-failures, resolving migration failures>>, and
|
||||
{kib} is still unable to successfully upgrade, rollback {kib} until
|
||||
you're able to identify and fix the root cause.
|
||||
|
||||
WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with
|
||||
your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}.
|
||||
Any changes made after an upgrade are lost when you roll back to a previous version.
|
||||
|
||||
To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version.
|
||||
|
||||
[float]
|
||||
===== Roll back by restoring a backup snapshot
|
||||
|
||||
. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
|
||||
By default, snapshots include the `kibana` feature state.
|
||||
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
|
||||
. To delete all saved object indices, use `DELETE /.kibana*`.
|
||||
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
|
||||
. Start all {kib} instances on the older version you want to rollback to.
|
||||
|
||||
[float]
|
||||
===== (Not recommended) Roll back without a backup snapshot
|
||||
|
||||
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
|
||||
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state.
|
||||
. Delete the version-specific indices created by the failed upgrade migration.
|
||||
+
|
||||
For example, to rollback from a failed upgrade
|
||||
to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`.
|
||||
. Inspect the output of `GET /_cat/aliases`.
|
||||
+
|
||||
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
|
||||
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
|
||||
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`.
|
||||
. To remove the write block from the roll back indices, use
|
||||
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
|
||||
. Start {kib} on the older version you want to rollback to.
|
||||
|
||||
[float]
|
||||
[[upgrade-migrations-old-indices]]
|
||||
==== Handling old `.kibana_N` indices
|
||||
|
||||
After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
|
||||
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
|
||||
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.
|
||||
Loading…
Add table
Add a link
Reference in a new issue