kibana/docs/user/reporting/reporting-troubleshooting.asciidoc

[role="xpack"]
[[reporting-troubleshooting]]
== Reporting troubleshooting

++++
<titleabbrev>Troubleshooting</titleabbrev>
++++

Having trouble? Here are solutions to common problems you might encounter while using Reporting.

* <<reporting-diagnostics>>
* <<reporting-troubleshooting-text-incorrect>>
* <<reporting-troubleshooting-missing-data>>
* <<reporting-troubleshooting-file-permissions>>
* <<reporting-troubleshooting-error-messages>>
* <<reporting-troubleshooting-puppeteer-debug-logs>>
* <<reporting-troubleshooting-system-requirements>>
* <<reporting-troubleshooting-maps-ems>>
* <<reporting-manual-chromium-install>>

[float]
[[reporting-diagnostics]]
=== Reporting diagnostics
Reporting comes with a built-in utility to try to automatically find common issues. When {kib} is running,
navigate to the Report Listing page, and click *Run reporting diagnostics*. This will open up a diagnostic tool
that checks various parts of the {kib} deployment and come up with any relevant recommendations.

If the diagnostic information doesn't reveal the problem, you can troubleshoot further by starting the Kibana
server with an environment variable for revealing additional debugging logs. Refer to
<<reporting-troubleshooting-puppeteer-debug-logs>>.

[float]
[[reporting-troubleshooting-text-incorrect]]
=== Text rendered incorrectly in generated reports

If a report label is rendered as an empty rectangle, no system fonts are available. Install at least one font package on the system.

If the report is missing certain Chinese, Japanese or Korean characters, ensure that a system font with those characters is installed.

[float]
[[reporting-troubleshooting-missing-data]]
=== Missing data in PDF report of data table visualization
There is currently a known limitation with the Data Table visualization that only the first page of data rows, which are the only data
visible on the screen, are shown in PDF reports.

[float]
[[reporting-troubleshooting-file-permissions]]
=== File permissions
Ensure that the `headless_shell` binary located in your Kibana data directory is owned by the user who is running Kibana, that the
user has the execute permission, and if applicable, that the filesystem is mounted with the `exec` option.

[NOTE]
--
The Chromium binary is located in the Kibana installation directory as `data/headless_shell-OS_TYPE/headless_shell`. The full path is logged
the first time Kibana starts when verbose logging is enabled.
--

[float]
[[reporting-troubleshooting-error-messages]]
=== Error messages
Whenever possible, a Reporting error message tries to be as self-explanatory as possible. Here are some error messages you might encounter,
along with the solution.

[float]
==== `StatusCodeError: [version_conflict_engine_exception]`
If you are running multiple instances of {kib} in a cluster, the instances share the work of executing report jobs to evenly distribute
the work load. Each instance searches the reporting index for "pending" jobs that the user has requested. It is possible for
multiple instances to find the same job in these searches. Only the instance that successfully updated the job status to
"processing" will actually execute the report job. The other instances that unsuccessfully tried to make the same update will log
something similar to this:

[source,text]
--------------------------------------------------------------------------------
StatusCodeError: [version_conflict_engine_exception] [...]: version conflict, required seqNo [6124], primary term [1]. current document has seqNo [6125] and primary term [1], with { ... }
  status: 409,
  displayName: 'Conflict',
  path: '/.reporting-...',
  body: {
    error: {
      type: 'version_conflict_engine_exception',
      reason: '[...]: version conflict, required seqNo [6124], primary term [1]. current document has seqNo [6125] and primary term [1]',
    },
  },
  statusCode: 409
}
--------------------------------------------------------------------------------

These messages alone don't indicate a problem. They show normal events that happen in a healthy system.

[float]
==== Max attempts reached
There are two primary causes of this error:

* You're creating a PDF of a visualization or dashboard that spans a large amount of data and Kibana is hitting the `xpack.reporting.queue.timeout`

* Kibana is hosted behind a reverse-proxy, and the <<reporting-kibana-server-settings, Kibana server settings>> are not configured correctly

Create a Markdown visualization and then create a PDF report. If this succeeds, increase the `xpack.reporting.queue.timeout` setting. If the
PDF report fails with "Max attempts reached," check your <<reporting-kibana-server-settings, Kibana server settings>>.

[float]
[[reporting-troubleshooting-nss-dependency]]
==== You must install nss for Reporting to work
Reporting using the Chromium browser relies on the Network Security Service libraries (NSS). Install the appropriate nss package for your
distribution.

[float]
[[reporting-troubleshooting-sandbox-dependency]]
==== Unable to use Chromium sandbox
Chromium uses sandboxing techniques that are built on top of operating system primitives. The Linux sandbox depends on user namespaces,
which were introduced with the 3.8 Linux kernel. However, many distributions don't have user namespaces enabled by default, or they require
the CAP_SYS_ADMIN capability. If the sandbox is not explicitly disabled in Kibana, either based on OS detection or with the
`xpack.screenshotting.browser.chromium.disableSandbox` setting, Chrome will try to enable the sandbox. If it fails due to OS or permissions
restrictions, Chrome will crash during initialization.

Elastic recommends that you research the feasibility of enabling unprivileged user namespaces before disabling the sandbox. An exception
is if you are running Kibana in Docker because the container runs in a user namespace with the built-in seccomp/bpf filters.

[float]
[[reporting-troubleshooting-verbose-logs]]
=== Verbose logs
{kib} server logs have a lot of useful information for troubleshooting and understanding how things work. If you're having any issues at
all, the full logs from Reporting will be the first place to look. In `kibana.yml`:

[source,yaml]
--------------------------------------------------------------------------------
logging.root.level: all
--------------------------------------------------------------------------------

For more information about logging, see <<logging-root-level,Kibana configuration settings>>.

[float]
[[reporting-troubleshooting-puppeteer-debug-logs]]
=== Puppeteer debug logs
The Chromium browser that {kib} launches on the server is driven by a NodeJS library for Chromium called Puppeteer. The Puppeteer library
has its own command-line method to generate its own debug logs, which can sometimes be helpful, particularly to figure out if a problem is
caused by Kibana or Chromium. See more at https://github.com/GoogleChrome/puppeteer/blob/v1.19.0/README.md#debugging-tips[debugging tips].

Using Puppeteer's debug method when launching Kibana would look like:
```
env DEBUG="puppeteer:*" ./bin/kibana
```
The internal DevTools protocol traffic will be logged via the `debug` module under the `puppeteer` namespace.


The Puppeteer logs are very verbose and could possibly contain sensitive information. Handle the generated output with care.

[float]
[[reporting-troubleshooting-system-requirements]]
=== System requirements
In Elastic Cloud, the {kib} instances that most configurations provide by default is for 1GB of RAM for the instance. That is enough for
{kib} Reporting when the visualization or dashboard is relatively simple, such as a single pie chart or a dashboard with
a few visualizations. However, certain visualization types incur more load than others. For example, a TSVB panel has a lot of network
requests to render.

If the {kib} instance doesn't have enough memory to run the report, the report fails with an error such as `Error: Page crashed!`
In this case, try increasing the memory for the {kib} instance to 2GB.

[float]
[[reporting-troubleshooting-maps-ems]]
=== Unable to connect to Elastic Maps Service

https://www.elastic.co/elastic-maps-service[{ems} ({ems-init})] is a service that hosts
tile layers and vector shapes of administrative boundaries.
If a report contains a map with a missing basemap layer or administrative boundary, the {kib} server does not have access to {ems-init}.
See <<maps-connect-to-ems>> for information on how to connect your {kib} server to {ems-init}.

[float]
[[reporting-manual-chromium-install]]
=== Manually install the Chromium browser for Darwin
Chromium is not embedded into {kib} for the Darwin (Mac OS) architecture. When
running {kib} on Darwin, Reporting will download Chromium into the proper area of
the {kib} installation path the first time the server starts. If the server
does not have access to the Internet, you must download the
Chromium browser and install it into the {kib} installation path.

1. Download the Chromium zip file:

** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/901912/chrome-mac.zip[x64] systems
** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac_Arm/901913/chrome-mac.zip[ARM] systems

2. Copy the zip file into the holding area. Relative to the root directory of {kib}, the path is:

** `.chromium/x64` for x64 systems
** `.chromium/arm64` for ARM systems

When {kib} starts, it will automatically extract the browser from the zip file, and is then ready for PNG and PDF reports.