github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 17:28:26 -04:00
Code Issues Projects Releases Packages Wiki Activity

[DOCS] Edits to upgrade docs (#125019) (#125114)

Browse source 
* [DOCS] Edits to upgrade docs

* Apply suggestions from code review

Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>

Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 5fde0a07e3)
This commit is contained in:
gchaps 2022-02-09 09:19:46 -08:00 • committed by GitHub
parent fab5310059
commit 784810cfb9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 55 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

93
docs/setup/upgrade/upgrade-migrations.asciidoc
View file

@ -23,7 +23,10 @@ Saved objects are stored in two indices:
The index aliases `.kibana` and `.kibana_task_manager` will always point to
the most up-to-date saved object indices.
The first time a newer {kib} starts, it will first perform an upgrade migration before starting plugins or serving HTTP traffic. To prevent losing acknowledged writes old nodes should be shutdown before starting the upgrade. To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later will add a write block to the outdated index. Table 1 lists the saved objects indices used by previous versions of {kib}.
When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous versions of {kib}.
.Saved object indices and aliases per {kib} version
[options="header"]
@ -40,11 +43,15 @@ The first time a newer {kib} starts, it will first perform an upgrade migration
|=======================
==== Upgrading multiple {kib} instances
When upgrading several {kib} instances connected to the same {es} cluster, ensure that all outdated instances are shutdown before starting the upgrade.
When upgrading several {kib} instances connected to the same {es} cluster,
ensure that all outdated instances are shut down before starting the upgrade.
Kibana does not support rolling upgrades. However, once outdated instances are shutdown, all upgraded instances can be started in parallel in which case all instances will participate in the upgrade migration in parallel.
{kib} does not support rolling upgrades. However, once outdated instances are shut down,
all upgraded instances can be started in parallel, in which case all instances will participate in the upgrade migration in parallel.
For large deployments with more than 10 {kib} instances and more than 10 000 saved objects, the upgrade downtime can be reduced by bringing up a single {kib} instance and waiting for it to complete the upgrade migration before bringing up the remaining instances.
For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
complete the upgrade migration before bringing up the remaining instances.
[float]
[[preventing-migration-failures]]
@ -53,9 +60,9 @@ This section highlights common causes of {kib} upgrade failures and how to preve
[float]
===== timeout_exception or receive_timeout_transport_exception
There is a known issue in v7.12.0 for users who tried the fleet beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index.
There is a known issue in 7.12.0 for users who tried the {fleet} beta.
Upgrade migrations fail because of a large number of documents in the `.kibana` index, which causes {kib} to log errors such as:
This can cause Kibana to log errors like:
[source,sh]
--------------------------------------------
@ -68,11 +75,12 @@ Instructions to work around this issue are in https://github.com/elastic/kibana/
[float]
===== Corrupt saved objects
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment.
We highly recommend testing your {kib} upgrade in a development cluster to find and remedy problems
caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment.
Saved objects that were corrupted through manual editing or integrations will cause migration
failures with a log message like `Unable to migrate the corrupt Saved Object document ...`.
Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.
For a successful upgrade migration, you must fix or delete corrupt documents.
For example, given the following error message:
@ -81,7 +89,7 @@ For example, given the following error message:
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
--------------------------------------------
The following steps must be followed to delete the document that is causing the migration to fail:
To delete the documents that cause migrations to fail, take the following steps:
. Remove the write block which the migration system has placed on the previous index:
+
@ -104,15 +112,15 @@ DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab
. Restart {kib}.
+
In this example, the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**.
In this example, the dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **is no longer available**.
Be sure you have a snapshot before you delete the corrupt document. If restoring from a snapshot is not an option, it is recommended to also delete the `temp` and `target` indices the migration created before restarting {kib} and retrying.
[float]
===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings
Matching index templates which specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades.
Matching index templates that specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades.
Prevention: narrow down the index patterns of any user-defined index templates to ensure that these won't apply to new `.kibana*` indices.
Prevention: Narrow down the {data-sources} of any user-defined index templates to ensure that these won't apply to new `.kibana*` indices.
NOTE: {kib} < 6.5 creates it's own index template called `kibana_index_template:.kibana`
and uses an index pattern of `.kibana`. This index template will not interfere and does not need to be changed or removed.
@ -127,19 +135,21 @@ Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensu
[float]
===== Different versions of {kib} connected to the same {es} index
When different versions of {kib} are attempting an upgrade migration in parallel this can lead to migration failures. Ensure that all {kib} instances are running the same version, configuration and plugins.
When you perform an upgrade migration of different {kib} versions, the migration can fail.
Ensure that all {kib} instances are running the same version, configuration, and plugins.
[float]
===== Incompatible `xpack.tasks.index` configuration setting
For {kib} versions prior to 7.5.1, if the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`, upgrade migrations will fail. {kib} 7.5.1 and later prevents this by refusing to start with an incompatible configuration setting.
For {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`,
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration setting prevents upgrade migrations from starting.
[float]
[[resolve-migrations-failures]]
==== Resolving migration failures
If {kib} terminates unexpectedly while migrating a saved object index it will automatically attempt to
perform the migration again once the process has restarted. Do not delete any saved objects indices to
attempt to fix a failed migration. Unlike previous versions, {kib} version 7.12.0 and
If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
perform the migration again when the process restarts. Do not delete any saved objects indices to
attempt to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
later does not require deleting any indices to release a failed migration lock.
If upgrade migrations fail repeatedly, follow the advice in
@ -154,41 +164,48 @@ If you're unable to resolve a failed migration following these steps, please con
If you've followed the advice in <<preventing-migration-failures, preventing migration failures>>
and <<resolve-migrations-failures, resolving migration failures>> and
{kib} is still not able to upgrade successfully, you might choose to rollback {kib} until
If {kib} is still unable to upgrade successfully, rollback {kib} until
you're able to identify and fix the root cause.
WARNING: Before rolling back {kib}, ensure that the version you wish to rollback to is compatible with
your {es} cluster. If the version you're rolling back to is not compatible, you will have to also rollback {es}.
Any changes made after an upgrade will be lost when rolling back to a previous version.
WARNING: Before rolling back {kib}, ensure that the version you want to rollback to is compatible with
your {es} cluster. If the version you're rolling back to is not compatible, you must also rollback {es}.
Any changes made after an upgrade are lost when rolling back to a previous version.
In order to rollback after a failed upgrade migration, the saved object indices have to be
To rollback after a failed upgrade migration, the saved object indices have to be
rolled back to be compatible with the previous {kib} version.
[float]
===== Rollback by restoring a backup snapshot:
===== Rollback by restoring a backup snapshot
1. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
Snapshots include this feature state by default.
2. Shutdown all {kib} instances to be 100% sure that there are no instances currently performing a migration.
3. Delete all saved object indices with `DELETE /.kibana*`
4. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
5. Start up all {kib} instances on the older version you wish to rollback to.
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
. Delete all saved object indices with `DELETE /.kibana*`.
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
. Start all {kib} instances on the older version you want to rollback to.
[float]
===== (Not recommended) Rollback without a backup snapshot:
===== (Not recommended) Rollback without a backup snapshot
1. Shutdown all {kib} instances to be 100% sure that there are no {kib} instances currently performing a migration.
2. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. Snapshots include this feature state by default.
3. Delete the version specific indices created by the failed upgrade migration. For example, if you wish to rollback from a failed upgrade to v7.12.0 `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`
4. Inspect the output of `GET /_cat/aliases`.
If either the `.kibana` and/or `.kibana_task_manager` alias is missing, these will have to be created manually.
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the feature state.
. Delete the version-specific indices created by the failed upgrade migration.
+
For example, to rollback from a failed upgrade
to v7.12.0: `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`
. Inspect the output of `GET /_cat/aliases`.
+
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
For example. if the `.kibana` alias was missing and the latest index is `.kibana_3` create a new alias with `POST /.kibana_3/_aliases/.kibana`.
5. Remove the write block from the rollback indices. `PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
6. Start up {kib} on the older version you wish to rollback to.
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias with `POST /.kibana_3/_aliases/.kibana`.
. To remove the write block from the rollback indices:
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
. Start {kib} on the older version you want to rollback to.
[float]
[[upgrade-migrations-old-indices]]
==== Handling old `.kibana_N` indices
After migrations have completed, there will be multiple {kib} indices in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc). {kib} only uses the index that the `.kibana` and `.kibana_task_manager` alias points to. The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.
After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.