[APM][docs][WIP] Update APM UI documentation (#28621)

* docs: initial APM UI updates

* docs: using-the-apm-ui updates

* more docs updates

* quick edits

* docs: apply feedback and fix screenshots

* docs: incorporate feedback, bold page names

* finishing touches and clean up

* docs: incorporate feedback from sarah

* docs: add feedback

* docs: incorporate feedback from gchaps

docs/apm/bottlenecks.asciidoc

@@ -2,25 +2,24 @@
 [[apm-bottlenecks]]
 == Visualizing Application Bottlenecks
 
-Elastic APM monitors transactions and errors in your application. A transaction
-can be a request to your server or a batch job or a custom transaction type.
-You can see information about response times, requests per minute, and status
-codes per endpoint. You can also dive into a specific request sample and get a
-complete waterfall view of what your application is spending its time on. For
-example, you might see that the bottlenecks are in database queries, cache
-calls, or external requests. This information enables you to easily compare and
-debug fast responses to slow responses.
+Elastic APM captures different types of information from within instrumented applications:
 
-//TBD: What are we wanting them to notice in the following screenshot?
-//For example, the APM UI shows that the following transaction is spending over 400 ms querying the orders database?
+* {apm-overview-ref-v}/transaction-spans.html[*Spans*] contain information about a specific code path that has been executed.
+They measure from the start to end of an activity,
+and they can have a parent/child relationship with other spans.
+* {apm-overview-ref-v}/transactions.html[*Transactions*] are a special kind of span that have extra metadata associated with them.
+You can think of transactions as the highest level of work you’re measuring within a service.
+As an example, a transaction could be a request to your server, a batch job, or a custom transaction type.
+* {apm-overview-ref-v}/errors.html[*Errors*] contain information about the original exception that occurred or about a log created when the exception occurred.
 
-[role="screenshot"]
-image::apm/images/apm-transaction.png[Example view of a transaction in the APM UI in Kibana]
+Each of these information types have a specific page associated with them in the APM UI.
+These various pages display the captured data in curated charts and tables that allow you to easily compare and debug your applications.
 
-For each incoming request and each application error, you can also see
-contextual information such as the request header, user information, system
-values, or custom data that you manually attached to the request.
+For example, you can see information about response times, requests per minute, and status codes per endpoint.
+You can even dive into a specific request sample and get a complete waterfall view of what your application is spending its time on.
+You might see that your bottlenecks are in database queries, cache calls, or external requests.
+For each incoming request and each application error,
+you can also see contextual information such as the request header, user information,
+system values, or custom data that you manually attached to the request.
 
-Having access to application-level insights with just a few clicks can
-drastically decrease the time you spend debugging errors, slow response times,
-and crashes.
+Having access to application-level insights with just a few clicks can drastically decrease the time you spend debugging errors, slow response times, and crashes.

docs/apm/errors.asciidoc

@@ -0,0 +1,46 @@
+[[errors]]
+=== Errors overview
+
+TIP: {apm-overview-ref-v}/errors.html[Errors] are defined as groups of exceptions with matching exception or log messages.
+
+The *Errors* overview provides a high-level view of the error message and culprit,
+the number of occurrences, and the most recent occurrence.
+Just like the transaction overview, you'll notice we group together like errors.
+This makes it very easy to quickly see which errors are affecting your services,
+and to take actions to rectify them.
+
+[role="screenshot"]
+image::apm/images/apm-errors-overview.png[Example view of the errors overview in the APM UI in Kibana]
+
+Selecting an error group ID or error message brings you to the *Error group*.
+
+[role="screenshot"]
+image::apm/images/apm-error-group.png[Example view of the error group page in the APM UI in Kibana]
+
+Here, you'll see the error message, culprit, and the number of occurrences over time.
+
+Further down, you'll see the Error occurrence table.
+This is where you can see the details of a sampled error within this group.
+The error shown is always the most recent to occur.
+
+Each error occurrence features a breakdown of the exception, including the stack trace from when the error occurred,
+and additional contextual information to help debug the issue.
+In some cases, you might also see a Transaction sample ID.
+This feature allows you to make a connection between the errors and transactions,
+by linking you to the specific transaction where the error occurred.
+This allows you to see the whole trace, including which services the request went through. 
+
+[float]
+[[errors-alerts-with-watcher]]
+==== Error reports with Watcher
+
+You can use the power of the alerting features with Watcher to get reports on error occurrences.
+The Watcher assistant, which is available on the errors overview, can help you set up a watch per service.
+
+Configure the watch with an occurrences threshold, time interval, and the desired actions, such as email or Slack notifications.
+With Watcher, your team can set up reports within minutes.
+
+Watches are managed separately in the dedicated Watcher UI available in Advanced Settings.
+
+[role="screenshot"]
+image::apm/images/apm-errors-watcher-assistant.png[Example view of the Watcher assistant for errors in APM UI in Kibana]

docs/apm/getting-started.asciidoc

@@ -2,21 +2,20 @@
 [[apm-getting-started]]
 == Getting Started
 
-If you have not already installed and configured Elastic APM, the
-**APM > Getting Started** page in {kib} provides information to help you
-complete the setup.
+If you have not already installed and configured Elastic APM,
+the *Setup Instructions* will get you started.
 
 [role="screenshot"]
-image::apm/images/apm-setup.jpg[Installation instructions on the APM page in Kibana]
+image::apm/images/apm-setup.png[Installation instructions on the APM page in Kibana]
 
-After you install the Elastic APM agent library in your application, the
-application automatically appears in the APM UI in {kib}. No further
-configuration is required.
+After you install the Elastic APM agent library in your application,
+the application automatically appears in the APM UI in {kib}.
+No further configuration is required.
 
-If you also use Elastic Stack for logging and server-level metrics, you can
-optionally import the APM dashboards that come with the APM Server. You can use
-these APM-specific visualizations to correlate APM data with other data sources.
-To get the dashboards, click the "Load Kibana objects" button at the bottom of the Getting Started guides for APM in Kibana.
+If you also use the Elastic Stack for logging and server-level metrics,
+you can import the APM dashboards that come with the APM Server.
+You can use these APM specific visualizations to correlate APM data with other data sources.
+To get the dashboards, click *Load Kibana objects* at the bottom of the Setup Instructions.
 
-For more setup information, see
-{apm-get-started-ref}/index.html[Getting Started with APM].
+[role="screenshot"]
+image::apm/images/apm-setup-dashboards.png[Install dashboards for APM in Kibana]

docs/apm/index.asciidoc

@@ -11,11 +11,16 @@ The **APM** page in {kib} is provided with the basic license. It
 enables developers to drill down into the performance data for their applications
 and quickly locate the performance bottlenecks.
 
-For more information about the components of Elastic APM, see
-{apm-get-started-ref}/overview.html[APM Overview].
+* <<apm-getting-started>>
+* <<apm-bottlenecks>>
+* <<apm-ui>>
 
+NOTE: For more information about the components of Elastic APM,
+see the {apm-get-started-ref}/overview.html[APM Overview].
 --
 
 include::getting-started.asciidoc[]
+
 include::bottlenecks.asciidoc[]
+
 include::using-the-apm-ui.asciidoc[]

docs/apm/metrics.asciidoc

@@ -0,0 +1,26 @@
+[[metrics]]
+=== Metrics overview
+
+The *Metrics* overview shows a combination of transaction, error, CPU, and memory data.
+
+If you're experiencing a problem with your service, you can use this page to attempt to find the underlying cause.
+For example, you might be able to correlate a high number of errors with a long transaction duration, high CPU usage, or a memory leak.   
+
+[role="screenshot"]
+image::apm/images/apm-metrics.png[Example view of the Metrics overview in APM UI in Kibana]
+
+[[machine-learning-integration]]
+=== Machine Learning integration
+
+The Machine Learning integration will initiate a new job predefined to calculate anomaly scores on transaction response times.
+The response time graph will show the expected bounds and annotate the graph when the anomaly score is 75 or above.
+
+[role="screenshot"]
+image::apm/images/apm-ml-integration.png[Example view of anomaly scores on response times in APM UI in Kibana]
+
+Jobs can be created per transaction type and based on the average response time.
+You can manage jobs in the *Machine Learning jobs management*.
+It might take some time for results to appear on the graph.
+
+Machine learning is a platinum feature. For a comparison of the Elastic license levels,
+see https://www.elastic.co/subscriptions[the subscription page]. 

docs/apm/query-bar.asciidoc

@@ -0,0 +1,31 @@
+[[query-bar]]
+=== Advanced queries
+
+The query bar is a powerful data query feature.
+Similar to the query bar in {kibana-ref}/discover.html[Discover],
+it enables you to pass advanced queries on your data to filter on particular pieces of information that you're interested in.
+It comes with a handy autocomplete that helps find the fields and even provides suggestions to the data they include.
+You can select the query bar and hit the down arrow on your keyboard to begin seeing recommendations.
+
+When querying, you're simply searching and selecting data from fields in Elasticsearch documents.
+It may be helpful to view some of your documents in {kibana-ref}/discover.html[Discover] to better understand how APM data is stored in Elasticsearch. 
+
+The query bar is available in the Services, Transactions, Errors, Metrics, and Traces views,
+and any input will persist as you move between pages.
+
+TIP: Interactions with the query bar change the URL of the page you're on.
+This means you can simply copy and paste the URL of your page to share a specific query or view with others.
+
+In the screenshot below, you can begin to see some of the transaction fields available for filtering on:  
+
+[role="screenshot"]
+image::apm/images/apm-query-bar.png[Example of the Kibana Query bar in APM UI in Kibana]
+
+==== Example queries
+
+* Exclude response times slower than 2000 ms: `transaction.duration.us > 2000000`
+* Filter by response status code: `context.response.status_code >= 400`
+* Filter by single user ID: `context.user.id : 12`
+* View _all_ transactions for an endpoint, instead of just a sample - `processor.event: "transaction" AND transaction.name: "<TRANSACTION_NAME_HERE>"`
+
+TIP: Read the {kibana-ref}/kuery-query.html[Kibana Query Language Enhancements] documentation to learn more about the capabilities of the {kib} query language.

docs/apm/services.asciidoc

@@ -0,0 +1,10 @@
+[[services]]
+=== Services overview
+
+The *Services* overview gives you quick insights into the health and general performance of each service.
+The <<set-time-filter,global time range filter>> in {kib} defines which services are available.
+
+You can add services by setting the `service.name` configuration in each of the {apm-agents-ref}[APM agents] you’re instrumenting.
+
+[role="screenshot"]
+image::apm/images/apm-services-overview.png[Example view of services table the APM UI in Kibana]

docs/apm/spans.asciidoc

@@ -0,0 +1,40 @@
+[[spans]]
+=== Span timeline
+
+TIP: A {apm-overview-ref-v}/transaction-spans.html[span] is defined as the duration of a single event.
+Spans are automatically captured by APM agents, and you can also define custom spans.
+Each span has a type and is defined by a different color in the timeline/waterfall visualization.
+
+The span timeline visualization is a bird's-eye view of what your application was doing while it was trying to respond to the request that came in.
+This makes it useful for visualizing where the selected transaction spent most of its time.
+
+[role="screenshot"]
+image::apm/images/apm-distributed-tracing.png[Example view of the distributed tracing in APM UI in Kibana]
+
+View a span in detail by clicking on it in the timeline waterfall.
+For example, in the below screenshot we've clicked on an SQL Select database query.
+The information displayed includes the actual SQL that was executed, how long it took,
+and the percentage of the trace's total time.
+You also get a stack trace, which shows the SQL query in your code.
+Finally, APM knows which files are your code and which are just modules or libraries that you've installed.
+These library frames will be minimized by default in order to show you the most relevant stack trace. 
+
+[role="screenshot"]
+image::apm/images/apm-span-detail.png[Example view of a span detail in the APM UI in Kibana]
+
+If your span timeline is colorful, it's indicative of a <<distributed-tracing,distributed trace>>.
+Services in a distributed trace are separated by color and listed in the order they occur.
+
+[role="screenshot"]
+image::apm/images/apm-services-trace.png[Example of distributed trace colors in the APM UI in Kibana]
+
+Don't forget, a distributed trace includes more than one transaction.
+When viewing these distributed traces in the timeline waterfall, you'll see this image:apm/images/transaction-icon.png[APM icon] icon,
+which indicates the next transaction in the trace.
+These transactions can be expanded and viewed in detail by clicking on them.
+
+After exploring these traces,
+you can return to the full trace by clicking *View full trace* in the upper right hand corner of the page.
+
+[role="screenshot"]
+image::apm/images/apm-view-full-trace.png[Example of distributed trace colors in the APM UI in Kibana]

docs/apm/traces.asciidoc

@@ -0,0 +1,36 @@
+[[traces]]
+=== Traces overview
+
+The *Traces* overview displays the entry transaction for all traces in your application.
+If you're using <<distributed-tracing>>, this view is key to finding the critical paths within your application.
+Transactions with the same name are grouped together and only shown once in this table.
+
+By default, transactions are sorted by _Impact_.
+Impact helps show the most used and slowest endpoints in your service - in other words,
+it's the collective amount of pain a specific endpoint is causing your users.
+If there's a particular endpoint you're worried about, you can click on it to view the <<transaction-details, transaction details>>.
+
+[role="screenshot"]
+image::apm/images/apm-traces.png[Example view of the Traces overview in APM UI in Kibana]
+
+[float]
+[[distributed-tracing]]
+==== Distributed tracing
+
+Elastic APM supports distributed tracing.
+Distributed tracing is a key feature of modern application performance monitoring as application architectures are shifting from monolithic to more distributed,
+service-based architectures.
+
+Distributed tracing allows APM users to automatically trace requests all the way through the service architecture,
+and visualize those traces in one single view in the APM UI.
+This is accomplished by tracing all of the requests, from the initial web request to your front-end service,
+to queries made to your back-end services.
+This makes finding possible bottlenecks throughout your application much easier and faster.
+
+By definition, a distributed trace includes more than one transaction.
+You can use the <<spans,span timeline visualization>> to view a waterfall display of all of the transactions from individual services that are connected in a trace.
+
+[role="screenshot"]
+image::apm/images/apm-distributed-tracing.png[Example view of the distributed tracing in APM UI in Kibana]
+
+TIP: Distributed tracing is supported by all APM agents and there’s no additional configuration needed.

docs/apm/transactions.asciidoc

@@ -0,0 +1,90 @@
+[[transactions]]
+=== Transaction overview
+
+TIP: A {apm-overview-ref-v}/transactions.html[transaction] describes an event captured by an Elastic APM agent instrumenting a service.
+The APM agents automatically collect performance metrics on HTTP requests, database queries, and much more.
+
+// Clicking *service* brings you to detail of transaction
+Selecting a <<services,*service*>> will display all associated *transactions*.
+The charts and table on this dashboard display the transaction duration, requests per minute, and a list of transactions for the selected service.
+
+[role="screenshot"]
+image::apm/images/apm-transactions-overview.png[Example view of transactions table in the APM UI in Kibana]
+
+*Transaction duration* shows the response times for this service and is broken down into average, 95th, and 99th percentile.
+If there's a weird spike that you'd like to investigate,
+you can simply zoom in on the graph - this will adjust the specific time range,
+and all of the data on the page will update accordingly.
+
+*Requests per minute* is divided into response codes: 2xx, 3xx, 4xx, etc.,
+and is useful for determining if you're serving more of one code than you typically do.
+Like in the Transaction duration graph, you can zoom in on anomalies to further investigate them.
+
+The table at the bottom is similar to the <<traces,traces>> overview and shows the name of each transaction occurring in the selected service.
+Transactions with the same name are grouped together and only shown once in this table.
+By default, transactions are sorted by _Impact_.
+Impact helps show the most used and slowest endpoints in your service - in other words,
+it's the collective amount of pain a specific endpoint is causing your users.
+If there's a particular endpoint you're worried about, you can click on it to view the <<transaction-details, transaction details>>.
+
+[IMPORTANT]
+====
+The transaction overview will only display helpful information when the transactions in your service are named correctly.
+
+Elastic APM Agents come with built-in support for popular frameworks out-of-the-box.
+However, if you only see one route in the Transaction overview page, or if you have transactions named "unknown route",
+it could be a symptom that the agent either wasn't installed correctly or doesn't support your framework. 
+
+For further details, including troubleshooting and custom implementation instructions,
+refer to the documentation for each {apm-agents-ref}[APM Agent] you've implemented.
+====
+
+[[transaction-details]]
+==== Transaction details
+
+Selecting a transaction group will bring you to the *transaction* details.
+Transaction details include a high-level overview of the transaction group duration,
+requests per minute, and transaction group duration distribution.
+It's important to note that all three of these graphs show data from every transaction within the selected transaction group. 
+
+[role="screenshot"]
+image::apm/images/apm-transaction-response-dist.png[Example view of response time distribution]
+
+A single sampled transaction is also displayed.
+This sampled transaction is based on your selection in the *Transactions duration distribution*.
+You can update the sampled transaction by selecting a new _bucket_ in the transactions duration distribution graph.
+The number of requests per bucket is displayed when hovering over the graph, and the selected bucket is highlighted to stand out.
+
+[role="screenshot"]
+image::apm/images/apm-transaction-duration-dist.png[Example view of transactions duration distribution graph]
+
+Let's look at an example.
+In the screenshot below,
+you'll notice most of our requests fall into buckets on the left side of the graph,
+with a long tail of smaller buckets to the right.
+This is a typical distribution, and indicates most of our requests were served quickly - awesome!
+It's the requests on the right, the ones taking longer than average, that we probably want to focus on.
+By clicking on these buckets,
+we're presented with a span timeline waterfall showing what a typical request in that bucket was doing.
+By investigating this timeline waterfall, we can hopefully see why it was slow and then implement a fix.
+
+[role="screenshot"]
+image::apm/images/apm-transaction-sample.png[Example view of transactions sample]
+
+NOTE: More information on timeline waterfalls is available in <<spans, spans>>.
+
+For a particular transaction sample, we can get even more information in the tabs:
+
+* *Timeline* - See the <<spans, Spans>> section for more information.
+* *Request* - The URL, headers, body, etc..
+* *Response* - The response.
+* *System* - The system hostname, architecture, platform, etc..
+* *Service* - The service/application runtime, agent, name, etc..
+* *Process* - The process id that served up the request.
+* *User* - This requires additional configuration, but allows you to see which user experienced the current transaction.
+This can be extremely useful if it's determined that specific users are getting slow requests.
+* *Tags* - Useful if you want to start correlating transactions with log files or metrics from Metricbeat.
+* *Custom* - You can configure your agent to add custom contextual information on transactions.
+
+TIP: All of this data is stored in documents in Elasticsearch.
+This means you can select "Actions - View sample document" to see the actual Elasticsearch document under the discover tab.

docs/apm/using-the-apm-ui.asciidoc

@@ -1,91 +1,32 @@
 [role="xpack"]
 [[apm-ui]]
-== Using the APM UI
+== Using APM
 
-The APM UI is designed to be as intuitive as possible, but you might come across certain terms or concepts that don’t feel native to you. Not to worry, that’s why we’ve created this guide to help you get the most out of the solution.
+APM is designed to be as intuitive as possible,
+but you might come across certain terms or concepts that don’t feel native to you.
+Not to worry, we've created this guide to help you get the most out of Elastic APM.
 
-* <<services>>
-* <<transactions>>
-* <<spans>>
-* <<errors>>
-* <<errors-alerts-with-watcher>>
-* <<machine-learning-integration>>
-* <<query-bar>>
+APM is available via the navigation sidebar in {Kib}.
 
-[[services]]
-=== Services
+* <<services,Services overview>>
+* <<traces,Traces overview>>
+* <<transactions,Transaction overview>>
+* <<spans,Span timeline visualization>>
+* <<errors,Debug errors>>
+* <<metrics,Metrics overview>>
+* <<machine-learning-integration,Machine learning integration>>
+* <<query-bar,Advanced queries>>
 
-You can add services by setting the `service.name` in the APM agent configuration per service you’re instrumenting. The global time range filter in Kibana defines which services are available. The Services overview gives you a quick insight into the health and general performance of each service.
+include::services.asciidoc[]
 
-[role="screenshot"]
-image::apm/images/apm-services-overview.png[Example view of services table the APM UI in Kibana]
+include::traces.asciidoc[]
 
-[[transactions]]
-=== Transactions
+include::transactions.asciidoc[]
 
-The APM agents automatically collect performance metrics on HTTP requests, database queries, and much more. Please refer to the documentation for each agent for more detail. Each `transaction.type` is displayed separately to make it easy for you to navigate to the transaction.
+include::spans.asciidoc[]
 
-[role="screenshot"]
-image::apm/images/apm-transactions-overview.png[Example view of transactions table in the APM UI in Kibana]
+include::errors.asciidoc[]
 
-[[spans]]
-=== Spans
+include::metrics.asciidoc[]
 
-A span is defined as the duration of a single event. Spans are automatically captured by the APM agents, but you can also define custom spans. Each span has a type and is defined by a different colour in the Timeline visualization (also known as the waterfall).
-
-[role="screenshot"]
-image::apm/images/apm-transaction-detail.png[Example view of transactions detail page in the APM UI in Kibana]
-
-You can view a span in detail by clicking it in the Timeline. This displays the recorded SQL database query or source code (in-app frames) around the event that was captured.
-
-[role="screenshot"]
-image::apm/images/apm-span-detail.png[Example view of a span detail in the APM UI in Kibana]
-
-[[errors]]
-=== Errors
-
-Errors are defined as groups of exceptions with matching exception or log messages. Each error group features a breakdown of the exception including the stackframes from when the error occurred and additional contextual information to help debug the issue.
-
-[role="screenshot"]
-image::apm/images/apm-errors-overview.png[Example view of the errors overview in the APM UI in Kibana]
-
-[role="screenshot"]
-image::apm/images/apm-error-group.png[Example view of the error group page in the APM UI in Kibana]
-
-[[errors-alerts-with-watcher]]
-=== Errors alerts with Watcher
-
-You can use the power of the alerting features with Watcher to get alerts on error occurrences. The Watcher assistant, which is available on the Errors overview page, can help you set up a watch per service.
-
-Configure the watch with occurrences threshold and time interval and set the desired actions, such as email or Slack notifications. Your team can be set up with alerts in minutes.
-
-Watches are managed separately in the dedicated Watcher UI available in Advanced Settings.
-
-[role="screenshot"]
-image::apm/images/apm-errors-watcher-assistant.png[Example view of the Watcher assistant for errors in APM UI in Kibana]
-
-[[machine-learning-integration]]
-=== Machine Learning integration
-
-The Machine Learning integration will initiate a new job predefined to calculate anomaly scores on transaction response times. The response time graph will show the expected bounds and annotate the graph when the anomaly score is 75 or above.
-
-[role="screenshot"]
-image::apm/images/apm-ml-integration.png[Example view of anomaly scores on response times in APM UI in Kibana]
-
-Jobs can be created per transaction type and based on the average response time. You can manage jobs in the Machine Learning jobs management page. It might take some time for results to appear on the graph.
-
-[[query-bar]]
-=== Query bar
-
-The query bar is a powerful data query feature. Similar to the query bar in {kibana-ref}/discover.html[Discover] it enables you to pass advanced queries on your data to filter on particular pieces of information that you're interested in. It comes with a handy autocomplete that helps find the fields and even provides suggestions to the data they include. The query bar is available on Services, Transaction and Errors views, and any input will persist as you move between them.
-
-[role="screenshot"]
-image::apm/images/apm-query-bar.png[Example of the Kibana Query bar in APM UI in Kibana]
-
-==== Example queries
-
-* Exclude response times faster than 2000 ms; `transaction.duration.us > 2000000`
-* Filter by response status code; `context.response.status_code >= 400`
-* Filter by single user ID; `context.user.id : 12`
-
-Read the {kibana-ref}/kuery-query.html[Kibana Query Language Enhancements] documentation to learn more about the capabilities in the Kibana query language.
+include::query-bar.asciidoc[]