[[size-your-shards]] == Size your shards Each index in {es} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using <> then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from _oversharding_. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable. [discrete] [[create-a-sharding-strategy]] === Create a sharding strategy The best way to prevent oversharding and other shard-related issues is to create a sharding strategy. A sharding strategy helps you determine and maintain the optimal number of shards for your cluster while limiting the size of those shards. Unfortunately, there is no one-size-fits-all sharding strategy. A strategy that works in one environment may not scale in another. A good sharding strategy must account for your infrastructure, use case, and performance expectations. The best way to create a sharding strategy is to benchmark your production data on production hardware using the same queries and indexing loads you'd see in production. For our recommended methodology, watch the https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing[quantitative cluster sizing video]. As you test different shard configurations, use {kib}'s {kibana-ref}/elasticsearch-metrics.html[{es} monitoring tools] to track your cluster's stability and performance. The following sections provide some reminders and guidelines you should consider when designing your sharding strategy. If your cluster is already oversharded, see <>. [discrete] [[shard-sizing-considerations]] === Sizing considerations Keep the following things in mind when building your sharding strategy. [discrete] [[single-thread-per-shard]] ==== Searches run on a single thread per shard Most searches hit multiple shards. Each shard runs the search on a single CPU thread. While a shard can run multiple concurrent searches, searches across a large number of shards can deplete a node's <>. This can result in low throughput and slow search speeds. [discrete] [[each-shard-has-overhead]] ==== Each index, shard, segment and field has overhead Every index and every shard requires some memory and CPU resources. In most cases, a small set of large shards uses fewer resources than many small shards. Segments play a big role in a shard's resource usage. Most shards contain several segments, which store its index data. {es} keeps some segment metadata in heap memory so it can be quickly retrieved for searches. As a shard grows, its segments are <> into fewer, larger segments. This decreases the number of segments, which means less metadata is kept in heap memory. Every mapped field also carries some overhead in terms of memory usage and disk space. By default {es} will automatically create a mapping for every field in every document it indexes, but you can switch off this behaviour to <>. Moreover every segment requires a small amount of heap memory for each mapped field. This per-segment-per-field heap overhead includes a copy of the field name, encoded using ISO-8859-1 if applicable or UTF-16 otherwise. Usually this is not noticeable, but you may need to account for this overhead if your shards have high segment counts and the corresponding mappings contain high field counts and/or very long field names. [discrete] [[shard-auto-balance]] ==== {es} automatically balances shards within a data tier A cluster's nodes are grouped into <>. Within each tier, {es} attempts to spread an index's shards across as many nodes as possible. When you add a new node or a node fails, {es} automatically rebalances the index's shards across the tier's remaining nodes. [discrete] [[shard-size-best-practices]] === Best practices Where applicable, use the following best practices as starting points for your sharding strategy. [discrete] [[delete-indices-not-documents]] ==== Delete indices, not documents Deleted documents aren't immediately removed from {es}'s file system. Instead, {es} marks the document as deleted on each related shard. The marked document will continue to use resources until it's removed during a periodic <>. When possible, delete entire indices instead. {es} can immediately remove deleted indices directly from the file system and free up resources. [discrete] [[use-ds-ilm-for-time-series]] ==== Use data streams and {ilm-init} for time series data <> let you store time series data across multiple, time-based backing indices. You can use <> to automatically manage these backing indices. One advantage of this setup is <>, which creates a new write index when the current one meets a defined `max_primary_shard_size`, `max_age`, `max_docs`, or `max_size` threshold. When an index is no longer needed, you can use {ilm-init} to automatically delete it and free up resources. {ilm-init} also makes it easy to change your sharding strategy over time: * *Want to decrease the shard count for new indices?* + Change the <> setting in the data stream's <>. * *Want larger shards or fewer backing indices?* + Increase your {ilm-init} policy's <>. * *Need indices that span shorter intervals?* + Offset the increased shard count by deleting older indices sooner. You can do this by lowering the `min_age` threshold for your policy's <>. Every new backing index is an opportunity to further tune your strategy. [discrete] [[shard-size-recommendation]] ==== Aim for shards of up to 200M documents, or with sizes between 10GB and 50GB There is some overhead associated with each shard, both in terms of cluster management and search performance. Searching a thousand 50MB shards will be substantially more expensive than searching a single 50GB shard containing the same data. However, very large shards can also cause slower searches and will take longer to recover after a failure. There is no hard limit on the physical size of a shard, and each shard can in theory contain up to just over two billion documents. However, experience shows that shards between 10GB and 50GB typically work well for many use cases, as long as the per-shard document count is kept below 200 million. You may be able to use larger shards depending on your network and use case, and smaller shards may be appropriate for {enterprise-search-ref}/index.html[Enterprise Search] and similar use cases. If you use {ilm-init}, set the <>'s `max_primary_shard_size` threshold to `50gb` to avoid shards larger than 50GB. To see the current size of your shards, use the <>. [source,console] ---- GET _cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb ---- // TEST[setup:my_index] The `pri.store.size` value shows the combined size of all primary shards for the index. [source,txt] ---- index prirep shard store .ds-my-data-stream-2099.05.06-000001 p 0 50gb ... ---- // TESTRESPONSE[non_json] // TESTRESPONSE[s/\.ds-my-data-stream-2099\.05\.06-000001/my-index-000001/] // TESTRESPONSE[s/50gb/.*/] [discrete] [[shard-count-recommendation]] ==== Master-eligible nodes should have at least 1GB of heap per 3000 indices The number of indices a master node can manage is proportional to its heap size. The exact amount of heap memory needed for each index depends on various factors such as the size of the mapping and the number of shards per index. As a general rule of thumb, you should have fewer than 3000 indices per GB of heap on master nodes. For example, if your cluster has dedicated master nodes with 4GB of heap each then you should have fewer than 12000 indices. If your master nodes are not dedicated master nodes then the same sizing guidance applies: you should reserve at least 1GB of heap on each master-eligible node for every 3000 indices in your cluster. Note that this rule defines the absolute maximum number of indices that a master node can manage, but does not guarantee the performance of searches or indexing involving this many indices. You must also ensure that your data nodes have adequate resources for your workload and that your overall sharding strategy meets all your performance requirements. See also <> and <>. To check the configured size of each node's heap, use the <>. [source,console] ---- GET _cat/nodes?v=true&h=heap.max ---- // TEST[setup:my_index] You can use the <> to check the number of shards per node. [source,console] ---- GET _cat/shards?v=true ---- // TEST[setup:my_index] [discrete] [[shard-count-per-node-recommendation]] ==== Add enough nodes to stay within the cluster shard limits The <> prevent creation of more than 1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen node. Make sure you have enough nodes of each type in your cluster to handle the number of shards you need. [discrete] [[field-count-recommendation]] ==== Allow enough heap for field mappers and overheads Mapped fields consume some heap memory on each node, and require extra heap on data nodes. Ensure each node has enough heap for mappings, and also allow extra space for overheads associated with its workload. The following sections show how to determine these heap requirements. [discrete] ===== Mapping metadata in the cluster state Each node in the cluster has a copy of the <>. The cluster state includes information about the field mappings for each index. This information has heap overhead. You can use the <> to get the heap overhead of the total size of all mappings after deduplication and compression. [source,console] ---- GET _cluster/stats?human&filter_path=indices.mappings.total_deduplicated_mapping_size* ---- // TEST[setup:node] This will show you information like in this example output: [source,console-result] ---- { "indices": { "mappings": { "total_deduplicated_mapping_size": "1gb", "total_deduplicated_mapping_size_in_bytes": 1073741824 } } } ---- // TESTRESPONSE[s/"total_deduplicated_mapping_size": "1gb"/"total_deduplicated_mapping_size": $body.$_path/] // TESTRESPONSE[s/"total_deduplicated_mapping_size_in_bytes": 1073741824/"total_deduplicated_mapping_size_in_bytes": $body.$_path/] [discrete] ===== Retrieving heap size and field mapper overheads You can use the <> to get two relevant metrics for each node: * The size of the heap on each node. * Any additional estimated heap overhead for the fields per node. This is specific to data nodes, where apart from the cluster state field information mentioned above, there is additional heap overhead for each mapped field of an index held by the data node. For nodes which are not data nodes, this field may be zero. [source,console] ---- GET _nodes/stats?human&filter_path=nodes.*.name,nodes.*.indices.mappings.total_estimated_overhead*,nodes.*.jvm.mem.heap_max* ---- // TEST[setup:node] For each node, this will show you information like in this example output: [source,console-result] ---- { "nodes": { "USpTGYaBSIKbgSUJR2Z9lg": { "name": "node-0", "indices": { "mappings": { "total_estimated_overhead": "1gb", "total_estimated_overhead_in_bytes": 1073741824 } }, "jvm": { "mem": { "heap_max": "4gb", "heap_max_in_bytes": 4294967296 } } } } } ---- // TESTRESPONSE[s/"USpTGYaBSIKbgSUJR2Z9lg"/\$node_name/] // TESTRESPONSE[s/"name": "node-0"/"name": $body.$_path/] // TESTRESPONSE[s/"total_estimated_overhead": "1gb"/"total_estimated_overhead": $body.$_path/] // TESTRESPONSE[s/"total_estimated_overhead_in_bytes": 1073741824/"total_estimated_overhead_in_bytes": $body.$_path/] // TESTRESPONSE[s/"heap_max": "4gb"/"heap_max": $body.$_path/] // TESTRESPONSE[s/"heap_max_in_bytes": 4294967296/"heap_max_in_bytes": $body.$_path/] [discrete] ===== Consider additional heap overheads Apart from the two field overhead metrics above, you must additionally allow enough heap for {es}'s baseline usage as well as your workload such as indexing, searches and aggregations. 0.5GB of extra heap will suffice for many reasonable workloads, and you may need even less if your workload is very light while heavy workloads may require more. [discrete] ===== Example As an example, consider the outputs above for a data node. The heap of the node will need at least: * 1 GB for the cluster state field information. * 1 GB for the additional estimated heap overhead for the fields of the data node. * 0.5 GB of extra heap for other overheads. Since the node has a 4GB heap max size in the example, it is thus sufficient for the total required heap of 2.5GB. If the heap max size for a node is not sufficient, consider <>, or scaling up the cluster, or redistributing index shards. Note that the above rules do not necessarily guarantee the performance of searches or indexing involving a very high number of indices. You must also ensure that your data nodes have adequate resources for your workload and that your overall sharding strategy meets all your performance requirements. See also <> and <>. [discrete] [[avoid-node-hotspots]] ==== Avoid node hotspots If too many shards are allocated to a specific node, the node can become a hotspot. For example, if a single node contains too many shards for an index with a high indexing volume, the node is likely to have issues. To prevent hotspots, use the <> index setting to explicitly limit the number of shards on a single node. You can configure `index.routing.allocation.total_shards_per_node` using the <>. [source,console] -------------------------------------------------- PUT my-index-000001/_settings { "index" : { "routing.allocation.total_shards_per_node" : 5 } } -------------------------------------------------- // TEST[setup:my_index] [discrete] [[avoid-unnecessary-fields]] ==== Avoid unnecessary mapped fields By default {es} <> for every field in every document it indexes. Every mapped field corresponds to some data structures on disk which are needed for efficient search, retrieval, and aggregations on this field. Details about each mapped field are also held in memory. In many cases this overhead is unnecessary because a field is not used in any searches or aggregations. Use <> instead of dynamic mapping to avoid creating fields that are never used. If a collection of fields are typically used together, consider using <> to consolidate them at index time. If a field is only rarely used, it may be better to make it a <> instead. You can get information about which fields are being used with the <> API, and you can analyze the disk usage of mapped fields using the <> API. Note however that unnecessary mapped fields also carry some memory overhead as well as their disk usage. [discrete] [[reduce-cluster-shard-count]] === Reduce a cluster's shard count If your cluster is already oversharded, you can use one or more of the following methods to reduce its shard count. [discrete] [[create-indices-that-cover-longer-time-periods]] ==== Create indices that cover longer time periods If you use {ilm-init} and your retention policy allows it, avoid using a `max_age` threshold for the rollover action. Instead, use `max_primary_shard_size` to avoid creating empty indices or many small shards. If your retention policy requires a `max_age` threshold, increase it to create indices that cover longer time intervals. For example, instead of creating daily indices, you can create indices on a weekly or monthly basis. [discrete] [[delete-empty-indices]] ==== Delete empty or unneeded indices If you're using {ilm-init} and roll over indices based on a `max_age` threshold, you can inadvertently create indices with no documents. These empty indices provide no benefit but still consume resources. You can find these empty indices using the <>. [source,console] ---- GET _cat/count/my-index-000001?v=true ---- // TEST[setup:my_index] Once you have a list of empty indices, you can delete them using the <>. You can also delete any other unneeded indices. [source,console] ---- DELETE my-index-000001 ---- // TEST[setup:my_index] [discrete] [[force-merge-during-off-peak-hours]] ==== Force merge during off-peak hours If you no longer write to an index, you can use the <> to <> smaller segments into larger ones. This can reduce shard overhead and improve search speeds. However, force merges are resource-intensive. If possible, run the force merge during off-peak hours. [source,console] ---- POST my-index-000001/_forcemerge ---- // TEST[setup:my_index] [discrete] [[shrink-existing-index-to-fewer-shards]] ==== Shrink an existing index to fewer shards If you no longer write to an index, you can use the <> to reduce its shard count. {ilm-init} also has a <> for indices in the warm phase. [discrete] [[combine-smaller-indices]] ==== Combine smaller indices You can also use the <> to combine indices with similar mappings into a single large index. For time series data, you could reindex indices for short time periods into a new index covering a longer period. For example, you could reindex daily indices from October with a shared index pattern, such as `my-index-2099.10.11`, into a monthly `my-index-2099.10` index. After the reindex, delete the smaller indices. [source,console] ---- POST _reindex { "source": { "index": "my-index-2099.10.*" }, "dest": { "index": "my-index-2099.10" } } ---- [discrete] [[troubleshoot-shard-related-errors]] === Troubleshoot shard-related errors Here’s how to resolve common shard-related errors. [discrete] ==== this action would add [x] total shards, but this cluster currently has [y]/[z] maximum shards open; The <> cluster setting limits the maximum number of open shards for a cluster. This error indicates an action would exceed this limit. If you're confident your changes won't destabilize the cluster, you can temporarily increase the limit using the <> and retry the action. [source,console] ---- PUT _cluster/settings { "persistent" : { "cluster.max_shards_per_node": 1200 } } ---- This increase should only be temporary. As a long-term solution, we recommend you add nodes to the oversharded data tier or <>. To get a cluster's current shard count after making changes, use the <>. [source,console] ---- GET _cluster/stats?filter_path=indices.shards.total ---- When a long-term solution is in place, we recommend you reset the `cluster.max_shards_per_node` limit. [source,console] ---- PUT _cluster/settings { "persistent" : { "cluster.max_shards_per_node": null } } ----