[[troubleshooting]]
== Troubleshooting

++++
<titleabbrev>Troubleshooting</titleabbrev>
++++

This section describes common problems you might encounter with the APM app.
To add to this page, please consider opening a
https://github.com/elastic/kibana/pulls[pull request] with your proposed changes.

If your issue is potentially related to other components of the APM ecosystem,
don't forget to check our other troubleshooting guides or discussion forum:

* {apm-guide-ref}/troubleshoot-apm.html[APM Server troubleshooting]
* {apm-dotnet-ref}/troubleshooting.html[.NET agent troubleshooting]
* {apm-go-ref}/troubleshooting.html[Go agent troubleshooting]
* {apm-ios-ref}/troubleshooting.html[iOS agent troubleshooting]
* {apm-java-ref}/trouble-shooting.html[Java agent troubleshooting]
* {apm-node-ref}/troubleshooting.html[Node.js agent troubleshooting]
* {apm-php-ref}/troubleshooting.html[PHP agent troubleshooting]
* {apm-py-ref}/troubleshooting.html[Python agent troubleshooting]
* {apm-ruby-ref}/debugging.html[Ruby agent troubleshooting]
* {apm-rum-ref}/troubleshooting.html[RUM troubleshooting]
* https://discuss.elastic.co/c/apm[APM discussion forum].

[discrete]
[[troubleshooting-apm-app]]
== Troubleshoot common APM app problems

* <<no-apm-data-found>>
* <<troubleshooting-too-many-transactions>>
* <<troubleshooting-unknown-route>>
* <<troubleshooting-fields-unsearchable>>
* <<service-map-rum-connections>>

[float]
[[no-apm-data-found]]
=== No APM data found

This section can help with any of the following:

* Data isn't displaying in the APM app
* Data isn't displaying in the APM app after an upgrade
* You see a message like "No Services Found",
* You see errors like "Fielddata is disabled on text fields by default..."

These problems are likely to be caused by a missing index template or ingest pipeline.
By default, {fleet} sets up these and other APM assets when the APM integration is installed.
Try reinstalling the APM integration by navigating to
**Integrations** > **Elastic APM** > **Manage in Fleet** > **Settings** > **Reinstall Elastic APM**.

Because assets cannot be applied to indices retroactively,
after reinstalling the APM integration you must either wait for the index to rollover or force a rollover.
To force a rollover, use the {ref}/indices-rollover-index.html[rollover API] to target the relevant {apm-guide-ref}/apm-data-streams.html[APM data streams].

[float]
[[troubleshooting-too-many-transactions]]
=== Too many unique transaction names

Transaction names are defined in each APM agent; when an APM agent supports a framework,
it includes logic for naming the transactions that the framework creates.
In some cases though, like when using an APM agent's API to create custom transactions,
it is up to the user to define a pattern for transaction naming.
When transactions are named incorrectly, each unique URL can be associated with a unique transaction group—causing
an explosion in the number of transaction groups per service, and leading to inaccuracies in the APM app.

To fix a large number of unique transaction names,
you need to change how you are using the APM agent API to name your transactions.
To do this, ensure you are **not** naming based on parameters that can change.
For example, user ids, product ids, order numbers, query parameters, etc.,
should be stripped away, and commonality should be found between your unique URLs.

Let's look at an example from the RUM agent documentation. Here are a few URLs you might find on Elastic.co:

[source,yml]
----
// Blog Posts
https://www.elastic.co/blog/reflections-on-three-years-in-the-elastic-public-sector
https://www.elastic.co/blog/say-heya-to-the-elastic-search-awards
https://www.elastic.co/blog/and-the-winner-of-the-elasticon-2018-training-subscription-drawing-is

// Documentation
https://www.elastic.co/guide/en/elastic-stack/current/index.html
https://www.elastic.co/guide/en/apm/get-started/current/index.html
https://www.elastic.co/guide/en/infrastructure/guide/current/index.html
----

These URLs, like most, include unique names.
If we named transactions based on each unique URL, we'd end up with the problem described above—a
very large number of different transaction names.
Instead, we should strip away the unique information and group our transactions based on common information.
In this case, that means naming all blog transactions, `/blog`, and all documentation transactions, `/guide`.

If you feel like you'd be losing valuable information by following this naming convention, don't fret!
You can always add additional metadata to your transactions using {apm-guide-ref}/data-model-metadata.html#data-model-labels[labels] (indexed) or
{apm-guide-ref}/data-model-metadata.html#data-model-custom[custom context] (non-indexed).

After ensuring you've correctly named your transactions,
you might still see errors in the APM app related to transaction group limit reached:

`The number of transaction groups has been reached. Current APM server capacity for handling unique transaction groups has been reached. There are at least X transactions missing in this list. Please decrease the number of transaction groups in your service or increase the memory allocated to APM server.`

You will see this warning if an agent is creating too many transaction groups. This could indicate incorrect instrumentation which will have to be fixed in your application. Alternatively you can increase the memory of the APM server.

`Number of transaction groups exceed the allowed maximum(1,000) that are displayed. The maximum number of transaction groups displayed in Kibana has been reached. Try narrowing down results by using the query bar..`

You will see this warning if your results have more than `1000` unique transaction groups. Alternatively you can use the query bar to reduce the number of unique transaction groups in your results.

**More information**

While this can happen with any APM agent, it typically occurs with the RUM agent.
For more information on how to correctly set `transaction.name` in the RUM agent,
see {apm-rum-ref}/custom-transaction-name.html[custom initial page load transaction names].

The RUM agent can also set the `transaction.name` when observing for transaction events.
See {apm-rum-ref}/agent-api.html#observe[`apm.observe()`] for more information.

If your problem is occurring in a different APM agent, the tips above still apply.
See the relevant {apm-agents-ref}[Agent API documentation] to adjust how you're naming your transactions.

[float]
[[troubleshooting-unknown-route]]
=== Unknown route

The {apm-app-ref}/transactions.html[transaction overview] will only display helpful information
when the transactions in your services are named correctly.
If you're seeing "GET unknown route" or "unknown route" in the APM app,
it could be a sign that something isn't working as it should.

Elastic APM agents come with built-in support for popular frameworks out-of-the-box.
This means, among other things, that the APM agent will try to automatically name HTTP requests.
As an example, the Node.js agent uses the route that handled the request, while the Java agent uses the Servlet name.

"Unknown route" indicates that the APM agent can't determine what to name the request,
perhaps because the technology you're using isn't supported, the agent has been installed incorrectly,
or because something is happening to the request that the agent doesn't understand.

To resolve this, you'll need to head over to the relevant {apm-agents-ref}[APM agent documentation].
Specifically, view the agent's supported technologies page.
You can also use the agent's public API to manually set a name for the transaction.

[float]
[[troubleshooting-fields-unsearchable]]
=== Fields are not searchable

In Elasticsearch, index templates are used to define settings and mappings that determine how fields should be analyzed.
The recommended index templates for APM are installed by {fleet} when the Elastic APM integration is installed.
These templates, by default, enable and disable indexing on certain fields.

As an example, some APM agents store cookie values in `http.request.cookies`.
Since `http.request` has disabled dynamic indexing, and `http.request.cookies` is not declared in a custom mapping,
the values in `http.request.cookies` are not indexed and thus not searchable.

*Ensure an APM data view exists*
As a first step, you should ensure the correct data view exists.
In {kib}, go to *Stack Management* > *Data views*.
You should see the APM data view--the default is
`traces-apm*,apm-*,logs-apm*,apm-*,metrics-apm*,apm-*`.
If you don't, the data view doesn't exist.
To fix this, navigate to the APM app in {kib} and select *Add data*.
In the APM tutorial, click *Load Kibana objects* to create the APM data view.

If creating an APM data view doesn't solve the problem,
see <<no-apm-data-found>> for further troubleshooting.

*Ensure a field is searchable*
There are two things you can do to if you'd like to ensure a field is searchable:

1. Index your additional data as {apm-guide-ref}/data-model-metadata.html[labels] instead.
These are dynamic by default, which means they will be indexed and become searchable and aggregatable.

2. Create a custom mapping for the field.
// link will be added in a later PR.
// docs will be added in https://github.com/elastic/apm-server/pull/6940

[float]
[[service-map-rum-connections]]
=== Service Maps: no connection between client and server

If the service map is not showing an expected connection between the client and server,
it's likely because you haven't configured
{apm-rum-ref}/distributed-tracing-guide.html[`distributedTracingOrigins`].


This setting is necessary, for example, for cross-origin requests.
If you have a basic web application that provides data via an API on `localhost:4000`,
and serves HTML from `localhost:4001`, you'd need to set `distributedTracingOrigins: ['https://localhost:4000']`
to ensure the origin is monitored as a part of distributed tracing.
In other words, `distributedTracingOrigins` is consulted prior to the APM agent adding the
distributed tracing `traceparent` header to each request.
