From 65b4fe28d4b55b6f3640b078352ab5c62efbf74d Mon Sep 17 00:00:00 2001 From: Stef Nestor Date: Thu, 27 Apr 2023 12:08:56 -0500 Subject: [PATCH] [+DOC] Troubleshooting / Mapping Explosion (#95397) * [+DOC] Troubleshooting / Mapping Explosion --------- Co-authored-by: Steffanie Nestor Co-authored-by: Amy Jonsson --- .../common-issues/mapping-explosion.asciidoc | 120 ++++++++++++++++++ .../fix-common-cluster-issues.asciidoc | 9 ++ 2 files changed, 129 insertions(+) create mode 100644 docs/reference/troubleshooting/common-issues/mapping-explosion.asciidoc diff --git a/docs/reference/troubleshooting/common-issues/mapping-explosion.asciidoc b/docs/reference/troubleshooting/common-issues/mapping-explosion.asciidoc new file mode 100644 index 000000000000..2d5b4f86e3c7 --- /dev/null +++ b/docs/reference/troubleshooting/common-issues/mapping-explosion.asciidoc @@ -0,0 +1,120 @@ +[[mapping-explosion]] +=== Mapping explosion + +{es}'s search and {kibana-ref}/discover.html[{kib}'s discover] Javascript rendering are +dependent on the search's backing indices total amount of +<>, of all mapping depths. When this total +amount is too high or is exponentially climbing, we refer to it as +experiencing mapping explosion. Field counts going this high are uncommon +and usually suggest an upstream document formatting issue as +link:https://www.elastic.co/blog/found-crash-elasticsearch#mapping-explosion[shown in this blog]. + +Mapping explosion may surface as the following performance symptoms: + +* <> reporting high heap or CPU on the main node +and/or nodes hosting the indices shards. This may potentially +escalate to temporary node unresponsiveness and/or main overwhelm. + +* <> reporting long search durations only related to +this index or indices, even on simple searches. + +* <> reporting long index durations only related to +this index or indices. This usually relates to <> +reporting that the coordinating node is waiting for all other nodes to +confirm they are on mapping update request. + +* Discover's **Fields for wildcard** page-loading API command or {kibana-ref}/console-kibana.html[Dev Tools] page-refreshing Autocomplete API commands are taking a long time (more than 10 seconds) or +timing out in the browser's Developer Tools Network tab. + +* Discover's **Available fields** taking a long time to compile Javascript in the browser's Developer Tools Performance tab. This may potentially escalate to temporary browser page unresponsiveness. + +* Kibana's {kibana-ref}/alerting-getting-started.html[alerting] or {security-guide}/detection-engine-overview.html[security rules] may error `The content length (X) is bigger than the maximum allowed string (Y)` where `X` is attempted payload and `Y` is {kib}'s {kibana-ref}/settings.html#server-maxPayload[`server-maxPayload`]. + +* Long {es} start-up durations. + +[discrete] +[[prevent]] +==== Prevent or prepare + +<> cannot be field-reduced once initialized. +{es} indices default to <> which +doesn't normally cause problems unless it's combined with overriding +<>. The +default `1000` limit is considered generous, though overriding to `10000` +doesn't cause noticable impact depending on use case. However, to give +a bad example, overriding to `100000` and this limit being hit +by mapping totals would usually have strong performance implications. + +If your index mapped fields expect to contain a large, arbitrary set of +keys, you may instead consider: + +* Using the <> data type. Please note, +however, that flattened objects is link:https://github.com/elastic/kibana/issues/25820[not fully supported in {kib}] yet. For example, this could apply to sub-mappings like { `host.name` , +`host.os`, `host.version` }. Desired fields are still accessed by +<>. + +* Using the <>. This is helpful when you're +interested in storing but not searching a group of fields. This is commonly +used for unknown upstream scenarios which may induce however many fields. +For example, this is recommended when sub-mappings start showing new, +unexpected fields like { `o365.a01`, `o365.a02`, `o365.b01`, `o365.c99`}. + +* Setting <> to disable a particular field's +searchability. This cannot effect current index mapping, but can apply +going forward via an <>. + +Modifying to the <> data type would not resolve the core +issue. + +[discrete] +[[check]] +==== Check for issue + +To confirm the field totals of an index to check for mapping explosion: + +* Check {es} cluster logs for errors `Limit of total fields [X] in index [Y] has been exceeded` where `X` is the value of `index.mapping.total_fields.limit` and `Y` is your index. The correlated ingesting source log error would be `Limit of total fields [X] has been exceeded while adding new fields [Z]` where `Z` is attempted new fields. + +* For top-level fields, poll <> for `fields=*`. + +* Search the output of <> for `"type"`. + +* If you're inclined to use the link:https://stedolan.github.io/jq[third-party tool JQ], you can process the <> `mapping.json` output. ++ +[source, sh] +---- +$ cat mapping.json | jq -c 'to_entries[]| .key as $index| [.value.mappings| to_entries[]|select(.key=="properties") | {(.key):([.value|..|.type?|select(.!=null)]|length)}]| map(to_entries)| flatten| from_entries| ([to_entries[].value]|add)| {index: $index, field_count: .}' +---- + +You can use <> to find fields which are never or rarely populated as easy wins. + +[discrete] +[[complex]] +==== Complex explosions + +Mapping explosions also covers when an individual index field totals are within limits but combined indices fields totals are very high. It's very common for symptoms to first be noticed on a {kibana-ref}/data-views.html[data view] and be traced back to an individual index or a subset of indices via the +<>. + +However, though less common, it is possible to only experience mapping explosions on the combination of backing indices. For example, if a <>'s backing indices are all at field total limit but each contain unique fields from one another. + +This situation most easily surfaces by adding a {kibana-ref}/data-views.html[data view] and checking its **Fields** tab for its total fields count. This statistic does tells you overall fields and not only where <>, but serves as a good baseline. + +If your issue only surfaces via a {kibana-ref}/data-views.html[data view], you may consider this menu's **Field filters** if you're not using <>. Alternatively, you may consider a more targeted index pattern or using a negative pattern to filter-out problematic indices. For example, if `logs-*` has too high a field count because of problematic backing indices `logs-lotsOfFields-*`, then you could update to either `logs-*,-logs-lotsOfFields-*` or `logs-iMeantThisAnyway-*`. + + +[discrete] +[[resolve]] +==== Resolve + +Mapping explosion is not easily resolved, so it is better prevented via the above. Encountering it usually indicates unexpected upstream data changes or planning failures. If encountered, we recommend reviewing your data architecture. The following options are additional to the ones discussed earlier on this page; they should be applied as best use-case applicable: + +* Disable <>. + +* <> into an index with a corrected mapping, +either via <> or <>. + +* If index is unneeded and/or historical, consider <>. + +* {logstash-ref}/plugins-inputs-elasticsearch.html[Export] and {logstash-ref}/plugins-outputs-elasticsearch.html[re-import] data into a mapping-corrected index after {logstash-ref}/plugins-filters-prune.html[pruning] +problematic fields via Logstash. + +<> would not resolve the core issue. diff --git a/docs/reference/troubleshooting/fix-common-cluster-issues.asciidoc b/docs/reference/troubleshooting/fix-common-cluster-issues.asciidoc index 9d04f1ab969a..47244ac17e99 100644 --- a/docs/reference/troubleshooting/fix-common-cluster-issues.asciidoc +++ b/docs/reference/troubleshooting/fix-common-cluster-issues.asciidoc @@ -40,10 +40,17 @@ misconfigured allocation settings to lack of disk space. A cluster in which nodes leave unexpectedly is unstable and can create several issues. + +<>:: +A cluster in which an index or index pattern as exploded with a high count of +mapping fields which causes performance look-up issues for Elasticsearch and +Kibana. + <>:: Hot spotting may occur in {es} when resource utilizations are unevenly distributed across nodes. + include::common-issues/disk-usage-exceeded.asciidoc[] include::common-issues/circuit-breaker-errors.asciidoc[] include::common-issues/high-cpu-usage.asciidoc[] @@ -51,5 +58,7 @@ include::common-issues/high-jvm-memory-pressure.asciidoc[] include::common-issues/red-yellow-cluster-status.asciidoc[] include::common-issues/rejected-requests.asciidoc[] include::common-issues/task-queue-backlog.asciidoc[] +include::common-issues/mapping-explosion.asciidoc[] include::common-issues/hotspotting.asciidoc[] include::common-issues/diagnose-unassigned-shards.asciidoc[] +