diff --git a/docs/index-shared3.asciidoc b/docs/index-shared3.asciidoc index 3a9714dc3..c9ef27b7b 100644 --- a/docs/index-shared3.asciidoc +++ b/docs/index-shared3.asciidoc @@ -74,11 +74,6 @@ include::static/performance-checklist.asciidoc[] :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/monitoring.asciidoc include::static/monitoring.asciidoc[] -// Pipeline viewer - -:edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/pipeline-viewer.asciidoc -include::static/pipeline-viewer.asciidoc[] - // Monitoring APIs :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/monitoring-apis.asciidoc diff --git a/docs/static/images/monitoring-ui.png b/docs/static/images/monitoring-ui.png deleted file mode 100644 index e8c42b9ec..000000000 Binary files a/docs/static/images/monitoring-ui.png and /dev/null differ diff --git a/docs/static/images/nodestats.png b/docs/static/images/nodestats.png deleted file mode 100644 index 8aef35ff1..000000000 Binary files a/docs/static/images/nodestats.png and /dev/null differ diff --git a/docs/static/images/overviewstats.png b/docs/static/images/overviewstats.png deleted file mode 100644 index b796a5170..000000000 Binary files a/docs/static/images/overviewstats.png and /dev/null differ diff --git a/docs/static/images/pipeline-diagram.png b/docs/static/images/pipeline-diagram.png deleted file mode 100644 index 63f85b8f4..000000000 Binary files a/docs/static/images/pipeline-diagram.png and /dev/null differ diff --git a/docs/static/images/pipeline-filter-detail.png b/docs/static/images/pipeline-filter-detail.png deleted file mode 100644 index dc9d1cf22..000000000 Binary files a/docs/static/images/pipeline-filter-detail.png and /dev/null differ diff --git a/docs/static/images/pipeline-input-detail.png b/docs/static/images/pipeline-input-detail.png deleted file mode 100644 index 4f01346c3..000000000 Binary files a/docs/static/images/pipeline-input-detail.png and /dev/null differ diff --git a/docs/static/images/pipeline-output-detail.png b/docs/static/images/pipeline-output-detail.png deleted file mode 100644 index 83392eaab..000000000 Binary files a/docs/static/images/pipeline-output-detail.png and /dev/null differ diff --git a/docs/static/images/pipeline-viewer-overview.png b/docs/static/images/pipeline-viewer-overview.png deleted file mode 100644 index 815538f93..000000000 Binary files a/docs/static/images/pipeline-viewer-overview.png and /dev/null differ diff --git a/docs/static/monitoring.asciidoc b/docs/static/monitoring.asciidoc index d9ee9870d..6d70102cc 100644 --- a/docs/static/monitoring.asciidoc +++ b/docs/static/monitoring.asciidoc @@ -12,37 +12,11 @@ The metrics collected by Logstash include: runtime stats. * Hot threads. -You can use the <> in {xpack} to view -these metrics and gain insight into how your Logstash deployment is running. +You can use the basic <> providing by Logstash +to retrieve these metrics. These APIs are available by default without +requiring any extra configuration. -Or you can use the basic <> providing by Logstash -to retrieve these metrics. - -The <> in {xpack} offers additional -visibility into the behavior and performance of complex pipeline configurations. -It shows a graph representation of the overall pipeline topology, data flow, and -branching logic, overlayed with important metrics, like events per second, for -each plugin in the view. - -[role="xpack"] -[[logstash-monitoring-ui]] -=== Monitoring UI - -NOTE: Monitoring is an X-Pack feature under the Basic License and is therefore -*free to use*. To get started, consult the -{xpack-ref}/monitoring-logstash.html[Logstash monitoring documentation]. - -When running Logstash 5.2 or greater, you can use the -https://www.elastic.co/products/x-pack/monitoring[monitoring feature in X-Pack] -to gain deep visibility into metrics about your Logstash deployment. In the -overview dashboard, you can see all events received and sent by Logstash, plus -info about memory usage and uptime: - -image::static/images/overviewstats.png[Logstash monitoring overview dashboard in Kibana] - -Then you can drill down to see stats about a specific node: - -image::static/images/nodestats.png[Logstash monitoring node stats dashboard in Kibana] - -See the {xpack-ref}/monitoring-logstash.html[Logstash monitoring documentation] to learn -how to set up and use this feature. +ifdef::include-xpack[] +:edit_url!: +include::{xls-repo-dir}/monitoring/index.asciidoc[] +endif::include-xpack[] diff --git a/docs/static/pipeline-viewer.asciidoc b/docs/static/pipeline-viewer.asciidoc deleted file mode 100644 index cfe9a40e3..000000000 --- a/docs/static/pipeline-viewer.asciidoc +++ /dev/null @@ -1,130 +0,0 @@ -[role="xpack"] -[[logstash-pipeline-viewer]] -=== Pipeline Viewer UI - -NOTE: The Logstash pipeline viewer is an -https://www.elastic.co/products/x-pack[X-Pack] feature under the Basic License -and is therefore free to use. - -The pipeline viewer in {xpack} provides a simple way for you to visualize and -monitor the behavior of complex Logstash pipeline configurations. Within the -pipeline viewer, you can explore a directed acyclic graph (DAG) representation -of the overall pipeline topology, data flow, and branching logic. The diagram -is overlayed with important metrics, like events per second and time spent in -milliseconds, for each plugin in the view. - -The diagram includes visual indicators to draw your attention to potential -bottlenecks in the pipeline, making it easy for you to diagnose and fix -problems. - -[IMPORTANT] -========================================================================== -When you configure the stages in your Logstash pipeline, make sure you specify -semantic IDs. If you don't specify IDs, Logstash generates them for you. - -Using semantic IDs makes it easier to identify the configurations that are -causing bottlenecks. For example, you may have several grok filters running -in your pipeline. If you haven't specified semantic IDs, you won't be able -to tell at a glance which filters are slow. If you specify semantic IDs, -such as `apacheParsingGrok` and `cloudwatchGrok`, you'll know exactly which -grok filters are causing bottlenecks. - -========================================================================== - -Before using the pipeline viewer, you need to -{logstash-ref}/setup-xpack.html[set up {xpack}] and configure -{xpack-ref}/monitoring-logstash.html[Logstash monitoring]. - -[float] -==== What types of problems does the pipeline viewer show? - -The pipeline viewer highlights CPU% and event latency in cases where the values -are anomalous. The purpose of these highlights is to enable users to quickly -identify processing that is disproportionately slow. This may not necessarily -mean that anything is wrong with a given plugin, since some plugins are slower -than others due to the nature of the work they do. For instance, you may find -that a grok filter that uses a complicated regexp runs a lot slower than a -mutate filter that simply adds a field. The grok filter might be highlighted in -this case, though it may not be possible to further optimize its work. - -The exact formula used is a heuristic, and thus is subject to change. - -[float] -==== View the pipeline diagram - -To view the pipeline diagram: - -. In Logstash, start the Logstash pipeline that you want to monitor. -+ -Assuming that you've set up Logstash monitoring, Logstash will begin shipping -metrics to the monitoring cluster. - -. Navigate to the Monitoring tab in Kibana. -+ -You should see a Logstash section. -+ -image::static/images/monitoring-ui.png[Monitoring UI] - -. Click the *Pipelines* link under Logstash to see all the pipelines that are -being monitored. -+ -Each pipeline is identified by a pipeline ID (`main` by default). For each -pipeline, you'll see a list of all versions of the pipeline stats that were -captured during the specified time range. -+ -image::static/images/pipeline-viewer-overview.png[Pipeline Overview] -+ -The version information is auto-generated by Logstash. Each time you modify a -pipeline, Logstash generates a new version hash. Viewing different versions -of the pipeline stats allows you see how changes to the pipeline over time -affect throughput and other metrics. Note that Logstash stores multiple versions -of the pipeline stats; it does not store multiple versions of the pipeline -configurations themselves. - -. Click a pipeline version in the list to drill down and explore the pipeline -diagram. -+ -The diagram shows all the stages feeding data through the pipeline. It also shows -conditional logic. -+ -image::static/images/pipeline-diagram.png[Pipeline Diagram] -+ -The information displayed on each node varies depending on the plugin type. -+ -Here's an example of an *input* node: -+ -image::static/images/pipeline-input-detail.png[Input node] -+ -The *I* badge indicates that this is an input stage. The node shows: -+ --- -* input type - *stdin* -* user-supplied ID - *logfileRead* -* throughput expressed in events per second - *0.7 e/s* - -Here's an example of a *filter* node. - -image::static/images/pipeline-filter-detail.png[Filter node] - -The filter icon indicates that this is a filter stage. The node shows: - -* filter type - *sleep* -* user-supplied ID - *caSleep* -* worker usage expressed as the percentage of total execution time - *0%* -* performance - the number of milliseconds spent processing each event - *20.00 ms/e* -* throughput - the number of events sent per second - *0.0 e/s* - -Stats that are anomalously slow appear highlighted in the pipeline viewer. -This doesn't necessarily indicate a problem, but it highlights potential -bottle necks so that you can find them quickly. - -An *output* node shows the same information as a filter node, but it has an -*O* badge to indicate that it is an output stage: - -image::static/images/pipeline-output-detail.png[Output node] --- - -. Hover over a node in the diagram, and you'll see only the related nodes that -are ancestors or descendants of the current node. - -. Explore the diagram and look for performance anomalies.