kibana/docs/developer/getting-started/debugging.asciidoc

[[kibana-debugging]]
== Debugging {kib}

For information about how to debug unit tests, refer to <<debugging-unit-tests>>.

[discrete]
=== Server Code

`yarn debug` will start the server with Node's inspect flag. {kib}'s development mode will start three processes on ports `9229`, `9230`, and `9231`. Chrome's developer tools need to be configured to connect to all three connections. Add `localhost:<port>` for each {kib} process in Chrome's developer tools connection tab.

[discrete]
=== Instrumenting with Elastic APM

{kib} ships with the
https://github.com/elastic/apm-agent-nodejs[Elastic APM Node.js Agent]
built-in for debugging purposes.

With an application as varied and complex as Kibana has become, it's not practical or scalable to craft all possible performance measurements by hand ahead of time. As such, we need to rely on tooling to help us catch things we may otherwise have missed.

For example, say you implement a brand new feature, plugin or service but don't quite know how it will impact Kibana's performance as a whole. APM allows us to not only spot that something is slow, but also hints at why it might be performing slowly. For example, if a function is slow on specific types of inputs, we can see where the time is spent by viewing the trace for that function call in the APM UI.

image::images/apm_example_trace.png[]

The net of metrics captured by APM are both a wide and deep because the entire application is instrumented at runtime and we simply take a sample of these metrics. This means that we don't have to know what we need to measure ahead of time, we'll instead just get (most) of the data we're likely going to need by default.

This type of data can help us identify unknown bottlenecks, spot when a performance regression may have been introduced, and inform how the performance of Kibana is changing between releases. Using APM allows us to be proactive in getting ahead of potential performance regressions before they are released.

The default APM configuration is meant to be used by core {kib} developers
only, but it can easily be re-configured to your needs. In its default
configuration it’s disabled and will, once enabled, send APM data to a
centrally managed {es} cluster accessible only to Elastic
employees.

To change the location where data is sent, use the
https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#server-url[`serverUrl`]
APM config option. To activate the APM agent, use the
https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#active[`active`]
APM config option.

All config options can be set by
creating an appropriate config file under `config/apm.dev.js`. 

Example `config/apm.dev.js` file:

[source,js]
----
module.exports = {
  active: true,
};
----

APM
https://www.elastic.co/guide/en/apm/agent/rum-js/current/index.html[Real
User Monitoring agent] is not available in the {kib} distributables,
however the agent can be enabled by setting `ELASTIC_APM_ACTIVE` to
`true`. flags

....
ELASTIC_APM_ACTIVE=true yarn start
// activates both Node.js and RUM agent
....

Once the agent is active, it will trace all incoming HTTP requests to
{kib}, monitor for errors, and collect process-level metrics. The
collected data will be sent to the APM Server and is viewable in the APM
UI in {kib}.

[discrete]
=== Running Kibana with the APM Agent Locally

The easiest and recommended way of running Kibana with the APM agent locally is to use the solution provided by the https://github.com/elastic/apm-integration-testing[apm-integration-testing] repo. You’ll need https://www.docker.com/community-edition[Docker] and https://docs.docker.com/compose/install/[Docker Compose] to use the tool.

[discrete]
==== Quick start guide

. Clone the https://github.com/elastic/apm-integration-testing[elastic/apm-integration-testing] repo.
. Change into the apm-integration-testing repo: 
+
[source,bash]
----
cd apm-integration-testing
----

. Run {es} and the APM servers without running Kibana:
+
[source,bash]
----
./scripts/compose.py start master --no-kibana
----

. Change into the {kib} repo:
+
[source,bash]
----
cd ../kibana
----

. Change the elasticsearch credentials in your `kibana.yml` configuration file to match those needed by elasticsearch and the APM server (see the apm-integration-testing repo's https://github.com/elastic/apm-integration-testing#logging-in[README] for users provided to test different scenarios).
. Make sure that the APM agent is active and points to the local APM server by adding the following configuration settings to a config file under `config/apm.dev.js`:
+
Example `config/apm.dev.js` file:
+
[source,js]
----
module.exports = {
  active: true,
  serverUrl: 'http://127.0.0.1:8200', // supports `http://localhost:8200`
  centralConfig: false,
  breakdownMetrics: false,
  transactionSampleRate: 0.1,
  metricsInterval: '120s'
};
----

. Start Kibana with APM active using: 
+
[source,bash]
----
yarn start
----

. After Kibana starts up, navigate to the APM app, where you should see some transactions.

image::images/apm_ui_transactions.png[]

You can now continue doing what you want to in Kibana (e.g. install sample data sets, issue queries in dashboards, build new visualizations etc).
Once you're finished, you can stop Kibana normally, then stop the {es} and APM servers in the apm-integration-testing clone with the following script:

[source,bash]
----
./scripts/compose.py stop
----

=== Using {kib} server logs
{kib} Logs is a great way to see what's going on in your application and to debug performance issues. Navigating through a large number of generated logs can be overwhelming, and following are some techniques that you can use to optimize the process.

Start by defining a problem area that you are interested in. For example, you might be interested in seeing how a particular {kib} Plugin is performing, so no need to gather logs for all of {kib}. Or you might want to focus on a particular feature, such as requests from the {kib} server to the {es} server.
Depending on your needs, you can configure {kib} to generate logs for a specific feature.
[source,yml]
----
logging:
  appenders:
    file:
      type: file
      fileName: ./kibana.log
      layout:
        type: json

### gather all the Kibana logs into a file
logging.root:
    appenders: [file]
    level: all

### or gather a subset of the logs
logging.loggers:
  ### responses to an HTTP request
  - name: http.server.response
    level: debug
    appenders: [file]
  ### result of a query to the Elasticsearch server
  - name: elasticsearch.query
    level: debug
    appenders: [file]
  ### logs generated by my plugin
  - name: plugins.myPlugin
    level: debug
    appenders: [file]
----
WARNING: Kibana's `file` appender is configured to produce logs in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format. It's the only format that includes the meta information necessary for https://www.elastic.co/guide/en/apm/agent/nodejs/current/log-correlation.html[log correlation] out-of-the-box.

The next step is to define what https://www.elastic.co/observability[observability tools] are available. 
For a better experience, set up an https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[Observability integration] provided by Elastic to debug your application with the <<debugging-logs-apm-ui, APM UI.>>
To debug something quickly without setting up additional tooling, you can work with <<plain-kibana-logs, the plain {kib} logs.>>

[[debugging-logs-apm-ui]]
==== APM UI
*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.

To debug {kib} with the APM UI, you must set up the APM infrastructure. You can find instructions for the setup process
https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[on the Observability integrations page].

Once you set up the APM infrastructure, you can enable the APM agent and put {kib} under load to collect APM events. To analyze the collected metrics and logs, use the APM UI as demonstrated https://www.elastic.co/guide/en/kibana/master/transactions.html#transaction-trace-sample[in the docs].

[[plain-kibana-logs]]
==== Plain {kib} logs
*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.

Open {kib} Logs and search for an operation you are interested in.
For example, suppose you want to investigate the response times for queries to the `/api/telemetry/v2/clusters/_stats` {kib} endpoint.
Open Kibana Logs and search for the HTTP server response for the endpoint. It looks similar to the following (some fields are omitted for brevity).
[source,json]
----
{
  "message":"POST /api/telemetry/v2/clusters/_stats 200 1014ms - 43.2KB",
  "log":{"level":"DEBUG","logger":"http.server.response"},
  "trace":{"id":"9b99131a6f66587971ef085ef97dfd07"},
  "transaction":{"id":"d0c5bbf14f5febca"}
}
----
You are interested in the https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html#field-trace-id[trace.id] field, which is a unique identifier of a trace. The `trace.id` provides a way to group multiple events, like transactions, which belong together. You can search for `"trace":{"id":"9b99131a6f66587971ef085ef97dfd07"}` to get all the logs that belong to the same trace. This enables you to see how many {es} requests were triggered during the `9b99131a6f66587971ef085ef97dfd07` trace, what they looked like, what {es} endpoints were hit, and so on.