We encountered a case where a substantial fraction of the heap usage was
due to per-segment-per-field `FieldInfo` objects, particularly
`FieldInfo#name`. This commit adds a note to the sizing docs about this
overhead.
When using LVM or software raid on Linux the kernel, or specific
distribution rules, may use higher ergonomic defaults for the
readahead of resulting block device(s). This can adversely affect
search performance due to high page cache thrashing, in search
heavy scenarios when mmap is involved.
Add a clarification section in the docs raising awareness about this
value and preferring the lower default.
This guidance does not apply any longer.
The overhead per shard has been significantly reduced in recent versions
and removed rule of thumb will be too pessimistic in many if not
most cases and might be too optimistic in other specific ones.
=> Replace guidance with rule of thumb per field count on data nodes and
rule of thumb by index count (which is far more relevant nowadays than
shards) for master nodes.
relates #77466
Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
* correct way of getting node heap size
in [[shard-count-recommendation]], we explain that the number of shards should be at most 20 shards per GB of heap.
but the command to get relevant heap size should be _cat/nodes?v=true&h=heap.max and not _cat/nodes?v=true&h=heap.current . The latter gives the current memory consumption, which is alway moving. Here we need to consider the max allocated heap size (-Xmx)
* Adds heap.max to valid columns
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Today we don't really say anything about the requirements for the data
path in terms of correctness, and we specifically say to avoid NFS for
performance reasons. This isn't wholly accurate: some NFS
implementations work just fine. This commit documents a more balanced
position on local vs remote storage.
Asciidoc source blocks are to be delimited with four dashes. This adds missing delimiters, and updates some that contained only three dashes. It matters for parsing purposes.
Today the _Size your shards_ docs focus on shard size and count, but in
fact index count and field count are also important. This commit expands
these docs a bit to cover this observation too.
Today we indicate that the `unassigned.reason` field in various APIs
indicates the reason why a shard is unassigned. This isn't really true,
it tells you some information about the event that caused the shard to
_become_ unassigned (or which most recently changed its routing table
entry while remaining unassigned) but tells you almost nothing about why
the shard _is now_ unassigned and how to fix it. That's what the
allocation explain API is for. This commit clarifies this point in the
docs.
Closes#80892
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Update ES quick start for security ON by default
* Remove code.asciidoc, which is part of the overall doc build now
* Update node names for cleanup
* Add note with links to tools
* Add --net elastic network
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR changes uses of transient cluster settings to
persistent cluster settings.
The PR also deprecates the transient settings usage.
Relates to #49540
In production, we recommend you run {es} on a dedicated host or as a primary service.
This adds that best practice to our setup documentation.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
The current shrink API snippet doesn't show you how to remove replicas or reduce primary shards.
Rather than duplicate those instructions from the shrink API docs, this removes the snippet. A link to the shrink API and shrink ILM action docs is already provided.
It also updates a delete index API snippet to avoid wildcards. Wildcard expansion for the delete index API is disabled by default in 8.0.
* Mention match_only_text in disk usage docs
Previously we explained how to manually disable norms, freqs, and positions. We
now have a ready-made solution in the new `match_only_text` field type.
* Fixing typo and minor grammar changes
Co-authored-by: Adam Locke <adam.locke@elastic.co>
On the "Size your shards" page, the shard size recommendation assumes a time
series use case. Similarly, users shouldn't count nearly empty and rarely used
Enterprise Search system indices against the recommended shard count limit.
Closes#76328.
Changes:
* Revises the size your shards guide to use a 50GB shard guideline. This better aligns with our default in the ILM policy UI.
* Updates the language to indicate that the 50GB shard guideline is not a hard limit. Larger shards may work depending on the network and use case.
Reverts some changes added in #71367.
Today's network config docs are split into "Network", "HTTP" and
"Transport" pages, with unclear relationships between them. We often
encounter users with weird configs that indicate they don't really
understand how these settings all relate. In fact these pages are all
very interrelated, and the HTTP and Transport pages are almost all only
for advanced users. This commit brings these docs into a single page and
rewords some things to try and guide users away from the advanced
settings unless their configuration needs all the extra complexity.
It also adds a section entitled "Binding and publishing" which clarifies
the meanings of the `bind_host` and `publish_host` parameters. This is
also a common source of confusion amongst users.
It also clarifies that many of these settings accept a list of
addresses, and warns that this may not be what you want. Closes#67956.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
