[[kibana-troubleshooting-kibana-server-logs]]
=== Using {kib} server logs
{kib} Logs is a great way to see what's going on in your application and to debug performance issues. Navigating through a large number of generated logs can be overwhelming, and following are some techniques that you can use to optimize the process.

Start by defining a problem area that you are interested in. For example, you might be interested in seeing how a particular {kib} Plugin is performing, so no need to gather logs for all of {kib}. Or you might want to focus on a particular feature, such as requests from the {kib} server to the {es} server.
Depending on your needs, you can configure {kib} to generate logs for a specific feature.
[source,yml]
----
logging:
  appenders:
    file:
      type: file
      fileName: ./kibana.log
      layout:
        type: json

### gather all the Kibana logs into a file
logging.root:
    appenders: [file]
    level: all

### or gather a subset of the logs
logging.loggers:
  ### responses to an HTTP request
  - name: http.server.response
    level: debug
    appenders: [file]
  ### result of a query to the Elasticsearch server
  - name: elasticsearch.query
    level: debug
    appenders: [file]
  ### logs generated by my plugin
  - name: plugins.myPlugin
    level: debug
    appenders: [file]
----
WARNING: Kibana's `file` appender is configured to produce logs in {ecs-ref}/ecs-reference.html[ECS JSON] format. It's the only format that includes the meta information necessary for {apm-node-ref}/log-correlation.html[log correlation] out-of-the-box.

The next step is to define what https://www.elastic.co/observability[observability tools] are available. 
For a better experience, set up an https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[Observability integration] provided by Elastic to debug your application with the <<debugging-logs-apm-ui, APM UI.>>
To debug something quickly without setting up additional tooling, you can work with <<plain-kibana-logs, the plain {kib} logs.>>

[[debugging-logs-apm-ui]]
==== APM UI
*Prerequisites* {kib} logs are configured to be in {ecs-ref}/ecs-reference.html[ECS JSON] format to include tracing identifiers.

To debug {kib} with the APM UI, you must set up the APM infrastructure. You can find instructions for the setup process
https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[on the Observability integrations page].

Once you set up the APM infrastructure, you can enable the APM agent and put {kib} under load to collect APM events. To analyze the collected metrics and logs, use the APM UI as demonstrated {kibana-ref}/transactions.html#transaction-trace-sample[in the docs].

[[plain-kibana-logs]]
==== Plain {kib} logs
*Prerequisites* {kib} logs are configured to be in {ecs-ref}/ecs-reference.html[ECS JSON] format to include tracing identifiers.

Open {kib} Logs and search for an operation you are interested in.
For example, suppose you want to investigate the response times for queries to the `/internal/telemetry/clusters/_stats` {kib} endpoint.
Open Kibana Logs and search for the HTTP server response for the endpoint. It looks similar to the following (some fields are omitted for brevity).
[source,json]
----
{
  "message":"POST /internal/telemetry/clusters/_stats 200 1014ms - 43.2KB",
  "log":{"level":"DEBUG","logger":"http.server.response"},
  "trace":{"id":"9b99131a6f66587971ef085ef97dfd07"},
  "transaction":{"id":"d0c5bbf14f5febca"}
}
----
You are interested in the https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html#field-trace-id[trace.id] field, which is a unique identifier of a trace. The `trace.id` provides a way to group multiple events, like transactions, which belong together. You can search for `"trace":{"id":"9b99131a6f66587971ef085ef97dfd07"}` to get all the logs that belong to the same trace. This enables you to see how many {es} requests were triggered during the `9b99131a6f66587971ef085ef97dfd07` trace, what they looked like, what {es} endpoints were hit, and so on.