[Reporting/Docs] Organize troubleshooting page and add CSV content (#182202)

## Summary

Closes elastic/platform-docs-team#309

The troubleshooting page for Kibana Reporting
([here](https://www.elastic.co/guide/en/kibana/current/reporting-troubleshooting.html))
page only covered screenshot-based reports (PNG, PDF). This PR updates
the guide to include advice for CSV reports as well.

---------

Co-authored-by: lcawl <lcawley@elastic.co>
This commit is contained in:
Tim Sullivan 2024-06-24 14:53:57 -07:00 committed by GitHub
parent 4e86f5a6ad
commit 3c78092f3a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 260 additions and 142 deletions

View file

@ -171,3 +171,5 @@ NOTE: *Public URL* is available only when anonymous access is configured and you
include::automating-report-generation.asciidoc[]
include::reporting-troubleshooting.asciidoc[]
include::reporting-csv-troubleshooting.asciidoc[leveloffset=+1]
include::reporting-pdf-troubleshooting.asciidoc[leveloffset=+1]

View file

@ -0,0 +1,90 @@
[[reporting-troubleshooting-csv]]
== Troubleshooting CSV reports
++++
<titleabbrev>CSV</titleabbrev>
++++
The CSV export feature in Kibana makes queries to Elasticsearch and formats the results into CSV.
This feature offers a solution that attempts to provide the most benefit to the most use cases.
However, things could go wrong during export.
Elasticsearch can stop responding, repeated querying can take so long that authentication tokens can time
out, and the format of exported data can be too complex for spreadsheet applications to handle.
Such situations are outside of the control of Kibana.
If the use case becomes complex enough, it's recommended that you create scripts that query Elasticsearch directly, using a scripting language like Python and the public {es} APIs.
For more advice about common problems, refer to <<reporting-troubleshooting>>.
[NOTE]
============
It is recommended that you use CSV reports to export moderate amounts of data only.
The feature enables analysis of data in external tools, but it's not intended for bulk export or to backup {es} data.
If you need to export more than 250 MB of CSV, rather than increasing <<reporting-csv-settings,`xpack.reporting.csv.maxSizeBytes`>>, use
filters to create multiple smaller reports or extract the data you need directly from {es}.
The following deployment configurations may lead to failed report jobs or incomplete reports:
* Any shard needed for search is unavailable.
* Data is stored on slow storage tiers.
* Network latency between nodes is high.
* {ccs-cap} is used.
To export large amounts of data, use {es} APIs directly.
Check out the {ref}/point-in-time-api.html[Point in time API] or {ref}/sql-rest-format.html#_csv[SQL with CSV response data format].
============
[float]
[[reporting-troubleshooting-csv-configure-scan-api]]
=== Configuring CSV export to use the scroll API
The Kibana CSV export feature collects all of the data from Elasticsearch by using multiple requests to page
over all of the documents.
Internally, the feature uses the {ref}/point-in-time-api.html[Point in time API and
`search_after` parameters in the queries] to do so.
There are some limitations related to the point in time API:
1. Permissions to read data aliases alone will not work: the permissions are needed on the underlying indices or data streams.
2. In cases where data shards are unavailable or time out, the export will be empty rather than returning partial data.
Some users may benefit from using the {ref}/paginate-search-results.html#scroll-search-results[scroll API], an
alternative to paging through the data.
The behavior of this API does not have the limitations of point in time API, however it has its own limitations:
1. Search is limited to 500 shards at the very most.
2. In cases where the data shards are unavailable or time out, the export may return partial data.
If you prefer the internal implementation of CSV export to use the scroll API, you can configure this in
`kibana.yml`:
[source,yml]
-------------------------------------------
xpack.reporting.csv.scroll.strategy: scroll
-------------------------------------------
For more details about CSV export settings, go to <<reporting-csv-settings>>.
[float]
[[reporting-troubleshooting-csv-socket-hangup]]
=== Socket hangups
A "socket hangup" is a generic type of error meaning that a remote service (in this case Elasticsearch or a proxy in Cloud) closed the connection.
Kibana can't foresee when this might happen and can't force the remote service to keep the connection open.
To work around this situation, consider lowering the size of results that come back in each request or increase the amount of time the remote services will
allow to keep the request open.
For example:
[source,yml]
---------------------------------------
xpack.reporting.csv.scroll.size: 50
xpack.reporting.csv.scroll.duration: 2m
---------------------------------------
Such changes aren't guaranteed to solve the issue, but give the functionality a better
chance of working in this use case.
Unfortunately, lowering the scroll size will require more requests to Elasticsearch during export, which adds more time overhead, which could unintentionally create more instances of auth token expiration errors.
[float]
[[reporting-troubleshooting-csv-token-expired]]
=== Token expiration
To avoid token expirations, use a type of authentication that doesn't expire (such as Basic auth) or run the export using scripts that query Elasticsearch directly.
In a custom script, you have the ability to refresh the auth token as needed, such as once before each query.

View file

@ -0,0 +1,136 @@
[[reporting-troubleshooting-pdf]]
== Troubleshooting PDF and PNG reports
++++
<titleabbrev>PDF/PNG</titleabbrev>
++++
For the most reliable configuration of PDF/PNG {report-features}, consider installing {kib} using <<docker,Docker>> or using <<set-up-on-cloud,Elastic Cloud>>.
For more advice about common problems, refer to <<reporting-troubleshooting>>.
[float]
[[reporting-diagnostics]]
=== Reporting diagnostics
Reporting comes with a built-in utility to try to automatically find common issues.
When {kib} is running, navigate to the *Report Listing* page, and click *Run reporting diagnostics*.
This will open up a diagnostic tool that checks various parts of the {kib} deployment and comes up with any relevant recommendations.
If the diagnostic information doesn't reveal the problem, you can troubleshoot further by starting the Kibana server with an environment variable for revealing additional debugging logs.
Refer to <<reporting-troubleshooting-puppeteer-debug-logs>>.
[float]
[[reporting-troubleshooting-nss-dependency]]
=== Network security service libraries
You must install Network Security Service (NSS) libraries for {report-features} to work.
Reporting using the Chromium browser relies on these libraries.
Install the appropriate nss package for your distribution.
Refer to <<install-reporting-packages>>.
[float]
[[reporting-troubleshooting-sandbox-dependency]]
=== Chromium sandbox requirements
Chromium uses sandboxing techniques that are built on top of operating system primitives.
The Linux sandbox depends on user namespaces, which were introduced with the 3.8 Linux kernel.
However, many distributions don't have user namespaces enabled by default or they require the CAP_SYS_ADMIN capability.
If the sandbox is not explicitly disabled in Kibana, either based on operating system detection or with the `xpack.screenshotting.browser.chromium.disableSandbox` setting, Chrome will try to enable the sandbox.
If it fails due to operating system or permissions restrictions, Chrome will crash during initialization.
Elastic recommends that you research the feasibility of enabling unprivileged user namespaces before disabling the sandbox.
An exception is if you are running Kibana in Docker because the container runs in a user namespace with the built-in seccomp/bpf filters.
[float]
[[reporting-troubleshooting-text-incorrect]]
=== Text rendered incorrectly in generated reports
If a report label is rendered as an empty rectangle, no system fonts are available.
Install at least one font package on the system.
If the report is missing certain Chinese, Japanese or Korean characters, ensure that a system font with those characters is installed.
[float]
[[reporting-troubleshooting-missing-data]]
=== Missing data in PDF report of data table visualization
There is currently a known limitation with the data table visualization that only the first page of data rows, which are the only data
visible on the screen, are shown in PDF reports.
[float]
[[reporting-troubleshooting-pdf-connection-refused]]
=== Connection refused errors
If PDF or PNG reports are not working due to a "connection refused" or "unable to connect" type of error, ensure that the `kibana.yml`
file uses the setting of `server.host: 0.0.0.0`.
Also verify that no firewall rules or other routing rules prevent local services from accessing this address.
Find out more at <<set-reporting-server-host>>.
[float]
[[reporting-troubleshooting-file-permissions]]
=== File permissions
Ensure that the `headless_shell` binary located in your Kibana data directory is owned by the user who is running Kibana, that the
user has the execute permission, and if applicable, that the filesystem is mounted with the `exec` option.
[NOTE]
--
The Chromium binary is located in the Kibana installation directory as `data/headless_shell-OS_TYPE/headless_shell`.
The full path is logged the first time Kibana starts when verbose logging is enabled.
--
[float]
[[reporting-troubleshooting-puppeteer-debug-logs]]
=== Puppeteer debug logs
The Chromium browser that {kib} launches on the server is driven by a NodeJS library for Chromium called Puppeteer.
The Puppeteer library has its own command-line method to generate its own debug logs, which can sometimes be helpful, particularly to figure out if a problem is caused by Kibana or Chromium.
Learn more https://github.com/GoogleChrome/puppeteer/blob/v1.19.0/README.md#debugging-tips[debugging tips].
Using Puppeteer's debug method when launching Kibana would look like:
```
env DEBUG="puppeteer:*" ./bin/kibana
```
The internal DevTools protocol traffic will be logged via the `debug` module under the `puppeteer` namespace.
The Puppeteer logs are very verbose and could possibly contain sensitive information.
Handle the generated output with care.
[float]
[[reporting-troubleshooting-system-requirements]]
=== System requirements
In Elastic Cloud, the {kib} instances that most configurations provide by default is for 1GB of RAM for the instance.
That is enough for {kib} {report-features} when the visualization or dashboard is relatively simple, such as a single pie chart or a dashboard with a few visualizations.
However, certain visualization types incur more load than others.
For example, a TSVB panel has a lot of network requests to render.
If the {kib} instance doesn't have enough memory to run the report, the report fails with an error such as `Error: Page crashed!`.
In this case, try increasing the memory for the {kib} instance to 2GB.
[float]
[[reporting-troubleshooting-maps-ems]]
=== Unable to connect to Elastic Maps Service
https://www.elastic.co/elastic-maps-service[{ems} ({ems-init})] is a service that hosts tile layers and vector shapes of administrative boundaries.
If a report contains a map with a missing basemap layer or administrative boundary, the {kib} server does not have access to {ems-init}.
Refer to <<maps-connect-to-ems>> for information about how to connect your {kib} server to {ems-init}.
[float]
[[reporting-manual-chromium-install]]
=== Manually install the Chromium browser for Darwin
Chromium is not embedded into {kib} for the Darwin (Mac OS) architecture.
When running {kib} on Darwin, {report-features} will download Chromium into the proper area of the {kib} installation path the first time the server starts.
If the server does not have access to the internet, you must download the Chromium browser and install it into the {kib} installation path.
1. Download the Chromium zip file:
** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/901912/chrome-mac.zip[x64] systems
** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac_Arm/901913/chrome-mac.zip[ARM] systems
2. Copy the zip file into the holding area. Relative to the root directory of {kib}, the path is:
** `.chromium/x64` for x64 systems
** `.chromium/arm64` for ARM systems
When {kib} starts, it will automatically extract the browser from the zip file and is then ready for PNG and PDF reports.

View file

@ -1,73 +1,31 @@
[role="xpack"]
[[reporting-troubleshooting]]
== Reporting troubleshooting
== Troubleshooting {report-features}
++++
<titleabbrev>Troubleshooting</titleabbrev>
++++
Having trouble? Here are solutions to common problems you might encounter while using Reporting.
Kibana excels as a data visualization tool. The {report-features} exist to export data as simple reports, however Kibana is not a data export tool.
To export data at a large scale, there are better ways and better architectures for exporting data at scale from Elasticsearch.
* <<reporting-diagnostics>>
* <<reporting-troubleshooting-text-incorrect>>
* <<reporting-troubleshooting-missing-data>>
* <<reporting-troubleshooting-file-permissions>>
* <<reporting-troubleshooting-error-messages>>
* <<reporting-troubleshooting-puppeteer-debug-logs>>
* <<reporting-troubleshooting-system-requirements>>
* <<reporting-troubleshooting-maps-ems>>
* <<reporting-manual-chromium-install>>
[float]
[[reporting-diagnostics]]
=== Reporting diagnostics
Reporting comes with a built-in utility to try to automatically find common issues. When {kib} is running,
navigate to the Report Listing page, and click *Run reporting diagnostics*. This will open up a diagnostic tool
that checks various parts of the {kib} deployment and come up with any relevant recommendations.
If the diagnostic information doesn't reveal the problem, you can troubleshoot further by starting the Kibana
server with an environment variable for revealing additional debugging logs. Refer to
<<reporting-troubleshooting-puppeteer-debug-logs>>.
[float]
[[reporting-troubleshooting-text-incorrect]]
=== Text rendered incorrectly in generated reports
If a report label is rendered as an empty rectangle, no system fonts are available. Install at least one font package on the system.
If the report is missing certain Chinese, Japanese or Korean characters, ensure that a system font with those characters is installed.
[float]
[[reporting-troubleshooting-missing-data]]
=== Missing data in PDF report of data table visualization
There is currently a known limitation with the Data Table visualization that only the first page of data rows, which are the only data
visible on the screen, are shown in PDF reports.
[float]
[[reporting-troubleshooting-file-permissions]]
=== File permissions
Ensure that the `headless_shell` binary located in your Kibana data directory is owned by the user who is running Kibana, that the
user has the execute permission, and if applicable, that the filesystem is mounted with the `exec` option.
[NOTE]
--
The Chromium binary is located in the Kibana installation directory as `data/headless_shell-OS_TYPE/headless_shell`. The full path is logged
the first time Kibana starts when verbose logging is enabled.
--
If you have trouble creating simple reports, there are some general solutions to common problems you might encounter while using {report-features}.
For tips related to specific types of reports, refer to <<reporting-troubleshooting-csv>> and <<reporting-troubleshooting-pdf>>.
[float]
[[reporting-troubleshooting-error-messages]]
=== Error messages
Whenever possible, a Reporting error message tries to be as self-explanatory as possible. Here are some error messages you might encounter,
along with the solution.
There are some common solutions for error messages that you might encounter in {report-features}.
[float]
==== `StatusCodeError: [version_conflict_engine_exception]`
If you are running multiple instances of {kib} in a cluster, the instances share the work of executing report jobs to evenly distribute
the work load. Each instance searches the reporting index for "pending" jobs that the user has requested. It is possible for
multiple instances to find the same job in these searches. Only the instance that successfully updated the job status to
"processing" will actually execute the report job. The other instances that unsuccessfully tried to make the same update will log
something similar to this:
[[reporting-troubleshooting-version-conflict-exception]]
==== Version conflict engine exceptions
If you are running multiple instances of {kib} in a cluster, the instances share the work of running report jobs to evenly distribute the workload.
Each instance searches the reporting index for "pending" jobs that the user has requested.
It is possible for multiple instances to find the same job in these searches.
Only the instance that successfully updated the job status to "processing" will actually run the report job.
The other instances that unsuccessfully tried to make the same update will log something similar to this:
[source,text]
--------------------------------------------------------------------------------
@ -85,103 +43,34 @@ StatusCodeError: [version_conflict_engine_exception] [...]: version conflict, re
}
--------------------------------------------------------------------------------
These messages alone don't indicate a problem. They show normal events that happen in a healthy system.
These messages alone don't indicate a problem.
They show normal events that happen in a healthy system.
[float]
==== Max attempts reached
There are two primary causes of this error:
There are two primary causes for a "Max attempts reached" error:
* You're creating a PDF of a visualization or dashboard that spans a large amount of data and Kibana is hitting the `xpack.reporting.queue.timeout`
* Kibana is hosted behind a reverse-proxy, and the <<reporting-kibana-server-settings, Kibana server settings>> are not configured correctly
* Kibana is hosted behind a reverse-proxy, and the <<reporting-kibana-server-settings,Kibana server settings>> are not configured correctly
Create a Markdown visualization and then create a PDF report. If this succeeds, increase the `xpack.reporting.queue.timeout` setting. If the
PDF report fails with "Max attempts reached," check your <<reporting-kibana-server-settings, Kibana server settings>>.
[float]
[[reporting-troubleshooting-nss-dependency]]
==== You must install nss for Reporting to work
Reporting using the Chromium browser relies on the Network Security Service libraries (NSS). Install the appropriate nss package for your
distribution.
[float]
[[reporting-troubleshooting-sandbox-dependency]]
==== Unable to use Chromium sandbox
Chromium uses sandboxing techniques that are built on top of operating system primitives. The Linux sandbox depends on user namespaces,
which were introduced with the 3.8 Linux kernel. However, many distributions don't have user namespaces enabled by default, or they require
the CAP_SYS_ADMIN capability. If the sandbox is not explicitly disabled in Kibana, either based on OS detection or with the
`xpack.screenshotting.browser.chromium.disableSandbox` setting, Chrome will try to enable the sandbox. If it fails due to OS or permissions
restrictions, Chrome will crash during initialization.
Elastic recommends that you research the feasibility of enabling unprivileged user namespaces before disabling the sandbox. An exception
is if you are running Kibana in Docker because the container runs in a user namespace with the built-in seccomp/bpf filters.
Create a Markdown visualization and then create a PDF report.
If this succeeds, increase the `xpack.reporting.queue.timeout` setting.
If the
PDF report fails with "Max attempts reached," check your <<reporting-kibana-server-settings,Kibana server settings>>.
[float]
[[reporting-troubleshooting-verbose-logs]]
=== Verbose logs
{kib} server logs have a lot of useful information for troubleshooting and understanding how things work. If you're having any issues at
all, the full logs from Reporting will be the first place to look. In `kibana.yml`:
=== Verbose logging
{kib} server logs have a lot of useful information for troubleshooting and understanding how things work.
The full logs from {report-features} are a good place to look when you encounter problems.
In `kibana.yml`:
[source,yaml]
--------------------------------------------------------------------------------
logging.root.level: all
--------------------------------------------------------------------------------
For more information about logging, see <<logging-root-level,Kibana configuration settings>>.
[float]
[[reporting-troubleshooting-puppeteer-debug-logs]]
=== Puppeteer debug logs
The Chromium browser that {kib} launches on the server is driven by a NodeJS library for Chromium called Puppeteer. The Puppeteer library
has its own command-line method to generate its own debug logs, which can sometimes be helpful, particularly to figure out if a problem is
caused by Kibana or Chromium. See more at https://github.com/GoogleChrome/puppeteer/blob/v1.19.0/README.md#debugging-tips[debugging tips].
Using Puppeteer's debug method when launching Kibana would look like:
```
env DEBUG="puppeteer:*" ./bin/kibana
```
The internal DevTools protocol traffic will be logged via the `debug` module under the `puppeteer` namespace.
The Puppeteer logs are very verbose and could possibly contain sensitive information. Handle the generated output with care.
[float]
[[reporting-troubleshooting-system-requirements]]
=== System requirements
In Elastic Cloud, the {kib} instances that most configurations provide by default is for 1GB of RAM for the instance. That is enough for
{kib} Reporting when the visualization or dashboard is relatively simple, such as a single pie chart or a dashboard with
a few visualizations. However, certain visualization types incur more load than others. For example, a TSVB panel has a lot of network
requests to render.
If the {kib} instance doesn't have enough memory to run the report, the report fails with an error such as `Error: Page crashed!`
In this case, try increasing the memory for the {kib} instance to 2GB.
[float]
[[reporting-troubleshooting-maps-ems]]
=== Unable to connect to Elastic Maps Service
https://www.elastic.co/elastic-maps-service[{ems} ({ems-init})] is a service that hosts
tile layers and vector shapes of administrative boundaries.
If a report contains a map with a missing basemap layer or administrative boundary, the {kib} server does not have access to {ems-init}.
See <<maps-connect-to-ems>> for information on how to connect your {kib} server to {ems-init}.
[float]
[[reporting-manual-chromium-install]]
=== Manually install the Chromium browser for Darwin
Chromium is not embedded into {kib} for the Darwin (Mac OS) architecture. When
running {kib} on Darwin, Reporting will download Chromium into the proper area of
the {kib} installation path the first time the server starts. If the server
does not have access to the Internet, you must download the
Chromium browser and install it into the {kib} installation path.
1. Download the Chromium zip file:
** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/901912/chrome-mac.zip[x64] systems
** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac_Arm/901913/chrome-mac.zip[ARM] systems
2. Copy the zip file into the holding area. Relative to the root directory of {kib}, the path is:
** `.chromium/x64` for x64 systems
** `.chromium/arm64` for ARM systems
When {kib} starts, it will automatically extract the browser from the zip file, and is then ready for PNG and PDF reports.
For more information about logging, check out <<logging-root-level,Kibana configuration settings>>.