mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-30 10:23:41 -04:00
84 lines
No EOL
2.4 KiB
Text
84 lines
No EOL
2.4 KiB
Text
[[disk-usage-exceeded]]
|
|
=== Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block
|
|
|
|
This error indicates a data node is critically low on disk space and has reached
|
|
the <<cluster-routing-flood-stage,flood-stage disk usage watermark>>. To prevent
|
|
a full disk, when a node reaches this watermark, {es} blocks writes to any index
|
|
with a shard on the node. If the block affects related system indices, {kib} and
|
|
other {stack} features may become unavailable.
|
|
|
|
{es} will automatically remove the write block when the affected node's disk
|
|
usage goes below the <<cluster-routing-watermark-high,high disk watermark>>. To
|
|
achieve this, {es} automatically moves some of the affected node's shards to
|
|
other nodes in the same data tier.
|
|
|
|
To verify that shards are moving off the affected node, use the <<cat-shards,cat
|
|
shards API>>.
|
|
|
|
[source,console]
|
|
----
|
|
GET _cat/shards?v=true
|
|
----
|
|
|
|
If shards remain on the node, use the <<cluster-allocation-explain,cluster
|
|
allocation explanation API>> to get an explanation for their allocation status.
|
|
|
|
[source,console]
|
|
----
|
|
GET _cluster/allocation/explain
|
|
{
|
|
"index": "my-index",
|
|
"shard": 0,
|
|
"primary": false,
|
|
"current_node": "my-node"
|
|
}
|
|
----
|
|
// TEST[s/^/PUT my-index\n/]
|
|
// TEST[s/"primary": false,/"primary": false/]
|
|
// TEST[s/"current_node": "my-node"//]
|
|
|
|
To immediately restore write operations, you can temporarily increase the disk
|
|
watermarks and remove the write block.
|
|
|
|
[source,console]
|
|
----
|
|
PUT _cluster/settings
|
|
{
|
|
"persistent": {
|
|
"cluster.routing.allocation.disk.watermark.low": "90%",
|
|
"cluster.routing.allocation.disk.watermark.high": "95%",
|
|
"cluster.routing.allocation.disk.watermark.flood_stage": "97%"
|
|
}
|
|
}
|
|
|
|
PUT */_settings?expand_wildcards=all
|
|
{
|
|
"index.blocks.read_only_allow_delete": null
|
|
}
|
|
----
|
|
// TEST[s/^/PUT my-index\n/]
|
|
|
|
As a long-term solution, we recommend you add nodes to the affected data tiers
|
|
or upgrade existing nodes to increase disk space. To free up additional disk
|
|
space, you can delete unneeded indices using the <<indices-delete-index,delete
|
|
index API>>.
|
|
|
|
[source,console]
|
|
----
|
|
DELETE my-index
|
|
----
|
|
// TEST[s/^/PUT my-index\n/]
|
|
|
|
When a long-term solution is in place, reset or reconfigure the disk watermarks.
|
|
|
|
[source,console]
|
|
----
|
|
PUT _cluster/settings
|
|
{
|
|
"persistent": {
|
|
"cluster.routing.allocation.disk.watermark.low": null,
|
|
"cluster.routing.allocation.disk.watermark.high": null,
|
|
"cluster.routing.allocation.disk.watermark.flood_stage": null
|
|
}
|
|
}
|
|
---- |