[[red-yellow-cluster-status]] === Red or yellow cluster health status A red or yellow cluster health status indicates one or more shards are not assigned to a node. * **Red health status**: The cluster has some unassigned primary shards, which means that some operations such as searches and indexing may fail. * **Yellow health status**: The cluster has no unassigned primary shards but some unassigned replica shards. This increases your risk of data loss and can degrade cluster performance. When your cluster has a red or yellow health status, it will continue to process searches and indexing where possible, but may delay certain management and cleanup activities until the cluster returns to green health status. For instance, some <> actions require the index on which they operate to have a green health status. In many cases, your cluster will recover to green health status automatically. If the cluster doesn't automatically recover, then you must <> the remaining problems so management and cleanup activities can proceed. See https://www.youtube.com/watch?v=v2mbeSd1vTQ[this video] for a walkthrough of monitoring allocation health. **** If you're using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to https://www.elastic.co/guide/en/cloud/current/ec-autoops.html[Monitor with AutoOps]. **** [discrete] [[diagnose-cluster-status]] ==== Diagnose your cluster status **Check your cluster status** Use the <>. [source,console] ---- GET _cluster/health?filter_path=status,*_shards ---- A healthy cluster has a green `status` and zero `unassigned_shards`. A yellow status means only replicas are unassigned. A red status means one or more primary shards are unassigned. **View unassigned shards** To view unassigned shards, use the <>. [source,console] ---- GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state ---- Unassigned shards have a `state` of `UNASSIGNED`. The `prirep` value is `p` for primary shards and `r` for replicas. To understand why an unassigned shard is not being assigned and what action you must take to allow {es} to assign it, use the <>. [source,console] ---- GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.* { "index": "my-index", "shard": 0, "primary": false } ---- // TEST[s/^/PUT my-index\n/] [discrete] [[fix-red-yellow-cluster-status]] ==== Fix a red or yellow cluster status A shard can become unassigned for several reasons. The following tips outline the most common causes and their solutions. [discrete] [[fix-cluster-status-reenable-allocation]] ===== Re-enable shard allocation You typically disable allocation during a <> or other cluster maintenance. If you forgot to re-enable allocation afterward, {es} will be unable to assign shards. To re-enable allocation, reset the `cluster.routing.allocation.enable` cluster setting. [source,console] ---- PUT _cluster/settings { "persistent" : { "cluster.routing.allocation.enable" : null } } ---- See https://www.youtube.com/watch?v=MiKKUdZvwnI[this video] for walkthrough of troubleshooting "no allocations are allowed". [discrete] [[fix-cluster-status-recover-nodes]] ===== Recover lost nodes Shards often become unassigned when a data node leaves the cluster. This can occur for several reasons, ranging from connectivity issues to hardware failure. After you resolve the issue and recover the node, it will rejoin the cluster. {es} will then automatically allocate any unassigned shards. To avoid wasting resources on temporary issues, {es} <> by one minute by default. If you've recovered a node and don’t want to wait for the delay period, you can call the <> with no arguments to start the allocation process. The process runs asynchronously in the background. [source,console] ---- POST _cluster/reroute ---- [discrete] [[fix-cluster-status-allocation-settings]] ===== Fix allocation settings Misconfigured allocation settings can result in an unassigned primary shard. These settings include: * <> index settings * <> cluster settings * <> cluster settings To review your allocation settings, use the <> and <> APIs. [source,console] ---- GET my-index/_settings?flat_settings=true&include_defaults=true GET _cluster/settings?flat_settings=true&include_defaults=true ---- // TEST[s/^/PUT my-index\n/] You can change the settings using the <> and <> APIs. [discrete] [[fix-cluster-status-allocation-replicas]] ===== Allocate or reduce replicas To protect against hardware failure, {es} will not assign a replica to the same node as its primary shard. If no other data nodes are available to host the replica, it remains unassigned. To fix this, you can: * Add a data node to the same tier to host the replica. * Change the `index.number_of_replicas` index setting to reduce the number of replicas for each primary shard. We recommend keeping at least one replica per primary. [source,console] ---- PUT _settings { "index.number_of_replicas": 1 } ---- // TEST[s/^/PUT my-index\n/] [discrete] [[fix-cluster-status-disk-space]] ===== Free up or increase disk space {es} uses a <> to ensure data nodes have enough disk space for incoming shards. By default, {es} does not allocate shards to nodes using more than 85% of disk space. To check the current disk space of your nodes, use the <>. [source,console] ---- GET _cat/allocation?v=true&h=node,shards,disk.* ---- If your nodes are running low on disk space, you have a few options: * Upgrade your nodes to increase disk space. * Delete unneeded indices to free up space. If you use {ilm-init}, you can update your lifecycle policy to use <> or add a delete phase. If you no longer need to search the data, you can use a <> to store it off-cluster. * If you no longer write to an index, use the <> or {ilm-init}'s <> to merge its segments into larger ones. + [source,console] ---- POST my-index/_forcemerge ---- // TEST[s/^/PUT my-index\n/] * If an index is read-only, use the <> or {ilm-init}'s <> to reduce its primary shard count. + [source,console] ---- POST my-index/_shrink/my-shrunken-index ---- // TEST[s/^/PUT my-index\n{"settings":{"index.number_of_shards":2,"blocks.write":true}}\n/] * If your node has a large disk capacity, you can increase the low disk watermark or set it to an explicit byte value. + [source,console] ---- PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "30gb" } } ---- // TEST[s/"30gb"/null/] [discrete] [[fix-cluster-status-jvm]] ===== Reduce JVM memory pressure Shard allocation requires JVM heap memory. High JVM memory pressure can trigger <> that stop allocation and leave shards unassigned. See <>. [discrete] [[fix-cluster-status-restore]] ===== Recover data for a lost primary shard If a node containing a primary shard is lost, {es} can typically replace it using a replica on another node. If you can't recover the node and replicas don't exist or are irrecoverable, <> will report `no_valid_shard_copy` and you'll need to do one of the following: * restore the missing data from <> * index the missing data from its original data source * accept data loss on the index-level by running <> * accept data loss on the shard-level by executing <> allocate_stale_primary or allocate_empty_primary command with `accept_data_loss: true` + WARNING: Only use this option if node recovery is no longer possible. This process allocates an empty primary shard. If the node later rejoins the cluster, {es} will overwrite its primary shard with data from this newer empty shard, resulting in data loss. + [source,console] ---- POST _cluster/reroute { "commands": [ { "allocate_empty_primary": { "index": "my-index", "shard": 0, "node": "my-node", "accept_data_loss": "true" } } ] } ---- // TEST[s/^/PUT my-index\n/] // TEST[catch:bad_request] See https://www.youtube.com/watch?v=6OAg9IyXFO4[this video] for a walkthrough of troubleshooting `no_valid_shard_copy`.