Restructure monitoring docs to support new and legacy internal collectors (#11714)

* [Doc] added description of xpack.monitoring.collection.write_direct.enabled setting

* Added page to mark as deprecated the legacy internal collector and fixed all the `xpack.monitoring.*` references

* Included legacy collector file into monitoring overview

* Restructure monitoring docs

* Incorporate review comments

Co-authored-by: andsel <selva.andre@gmail.com>

Fixes #11787
This commit is contained in:
Karen Metts 2020-04-14 15:47:56 -04:00 committed by Karen Metts
parent ba7fd0cedc
commit dfe5cc5ee7
16 changed files with 507 additions and 136 deletions

View file

@ -187,14 +187,13 @@ include::static/deploying.asciidoc[]
:edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/performance-checklist.asciidoc
include::static/performance-checklist.asciidoc[]
// Monitoring
:edit_url!:
include::static/monitoring/monitoring.asciidoc[]
// X-Pack Monitoring
:edit_url!:
include::static/monitoring/monitoring-overview.asciidoc[]
// Monitoring
:edit_url!:
include::static/monitoring/monitoring.asciidoc[]
// Working with Plugins

View file

@ -153,7 +153,7 @@ Some example translations are shown here:
**Environment Variable**:: **Logstash Setting**
`PIPELINE_WORKERS`:: `pipeline.workers`
`LOG_LEVEL`:: `log.level`
`XPACK_MONITORING_ENABLED`:: `xpack.monitoring.enabled`
`MONITORING_ENABLED`:: `monitoring.enabled`
In general, any setting listed in the <<logstash-settings-file, settings
documentation>> can be configured with this technique.
@ -170,9 +170,9 @@ images:
[horizontal]
`http.host`:: `0.0.0.0`
`xpack.monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200`
`monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200`
NOTE: The setting `xpack.monitoring.elasticsearch.hosts` is not
NOTE: The setting `monitoring.elasticsearch.hosts` is not
defined in the `-oss` image.
These settings are defined in the default `logstash.yml`. They can be overridden

View file

@ -0,0 +1,49 @@
[float]
[role="xpack"]
[[logstash-monitoring-collectors-legacy]]
===== Collectors
Collectors, as their name implies, collect things. In monitoring for Logstash,
collectors are just <<pipeline,Inputs>> in the same way that ordinary Logstash
configurations provide inputs.
Like monitoring for {es}, each collector can create zero or more monitoring
documents. As it is currently implemented, each Logstash node runs two types of
collectors: one for node stats and one for pipeline stats.
[options="header"]
|=======================
| Collector | Data Types | Description
| Node Stats | `logstash_stats`
| Gathers details about the running node, such as memory utilization and CPU
usage (for example, `GET /_stats`).
This runs on every Logstash node with monitoring enabled. One common
failure is that Logstash directories are copied with their `path.data` directory
included (`./data` by default), which copies the persistent UUID of the Logstash
node along with it. As a result, it generally appears that one or more Logstash
nodes are failing to collect monitoring data, when in fact they are all really
misreporting as the _same_ Logstash node. Re-use `path.data` directories only
when upgrading Logstash, such that upgraded nodes replace the previous versions.
| Pipeline Stats | `logstash_state`
| Gathers details about the node's running pipelines, which powers the
Monitoring Pipeline UI.
|=======================
Per collection interval, which defaults to 10 seconds (`10s`), each collector is
run. The failure of an individual collector does not impact any other collector.
Each collector, as an ordinary Logstash input, creates a separate Logstash event
in its isolated monitoring pipeline. The Logstash output then sends the data.
The collection interval can be configured dynamically and you can also disable
data collection. For more information about the configuration options for the
collectors, see <<monitoring-settings-legacy>>.
WARNING: Unlike {es} and {kib} monitoring, there is no
`xpack.monitoring.collection.enabled` setting on Logstash. You must use the
`xpack.monitoring.enabled` setting to enable and disable data collection.
If gaps exist in the monitoring charts in {kib}, it is typically because either
a collector failed or the monitoring cluster did not receive the data (for
example, it was being restarted). In the event that a collector fails, a logged
error should exist on the node that attempted to perform the collection.

View file

@ -3,11 +3,11 @@
[[logstash-monitoring-collectors]]
===== Collectors
Collectors, as their name implies, collect things. In {monitoring} for Logstash,
Collectors, as their name implies, collect things. In monitoring for Logstash,
collectors are just <<pipeline,Inputs>> in the same way that ordinary Logstash
configurations provide inputs.
Like {monitoring} for {es}, each collector can create zero or more monitoring
Like monitoring for {es}, each collector can create zero or more monitoring
documents. As it is currently implemented, each Logstash node runs two types of
collectors: one for node stats and one for pipeline stats.
@ -18,7 +18,7 @@ collectors: one for node stats and one for pipeline stats.
| Gathers details about the running node, such as memory utilization and CPU
usage (for example, `GET /_stats`).
This runs on every Logstash node with {monitoring} enabled. One common
This runs on every Logstash node with monitoring enabled. One common
failure is that Logstash directories are copied with their `path.data` directory
included (`./data` by default), which copies the persistent UUID of the Logstash
node along with it. As a result, it generally appears that one or more Logstash
@ -39,9 +39,9 @@ The collection interval can be configured dynamically and you can also disable
data collection. For more information about the configuration options for the
collectors, see <<monitoring-settings>>.
WARNING: Unlike {monitoring} for {es} and {kib}, there is no
`xpack.monitoring.collection.enabled` setting on Logstash. You must use the
`xpack.monitoring.enabled` setting to enable and disable data collection.
WARNING: Unlike for {es} and {kib} monitoring, there is no
`monitoring.collection.enabled` setting on Logstash. You must use the
`monitoring.enabled` setting to enable and disable data collection.
If gaps exist in the monitoring charts in {kib}, it is typically because either
a collector failed or the monitoring cluster did not receive the data (for

View file

@ -0,0 +1,155 @@
[role="xpack"]
[[monitoring-internal-collection-legacy]]
=== Collect {ls} monitoring data using legacy internal collectors
++++
<titleabbrev>Legacy internal collection</titleabbrev>
++++
IMPORTANT: Use <<monitoring-internal-collection,internal collection>> or
<<monitoring-with-metricbeat, {metricbeat} collection>> instead of legacy
internal collection. If you are currently using legacy internal collection, you
should migrate to either Metricbeat collection or internal collection.
==== Migrating from legacy internal collection to new internal collection
Migrating from legacy internal collection to new
<<monitoring-internal-collection,Internal collection>> is straightforward.
** Drop the `xpack.` prefix from your configuration settings.
For example, `xpack.monitoring.enabled` is now `monitoring.enabled`. See
<<monitoring-settings, Monitoring Settings>> for the full list.
** Change the hosts settings and configuration options to point to the monitoring
cluster instead of the production cluster.
** If you dont have an Elasticsearch output plugin configured in the pipelines,
add the <<monitoring-cluster-uuid,`monitoring.cluster_uuid`>> setting to your
logstash.yml.
==== Components for legacy internal collection
Monitoring {ls} with legacy internal collection uses these components:
* <<logstash-monitoring-collectors-legacy,Collectors>>
* <<logstash-monitoring-output-legacy,Output>>
These pieces live outside of the default Logstash pipeline in a dedicated monitoring
pipeline. This configuration ensures that all data and processing has a minimal
impact on ordinary Logstash processing. Existing Logstash features, such as the
<<plugins-outputs-elasticsearch,`elasticsearch` output>>, can be reused to
benefit from its retry policies.
NOTE: The `elasticsearch` output that is used for monitoring {ls} is
configured exclusively through settings found in `logstash.yml`. It is not
configured by using anything from the Logstash configurations that might also be
using their own separate `elasticsearch` outputs.
The production {es} cluster should be configured to receive {ls} monitoring
data. This configuration enables the production {es} cluster to add metadata
(for example, its cluster UUID) to the Logstash monitoring data and then route
it to the monitoring clusters. For more information about typical monitoring
architectures, see {ref}/how-monitoring-works.html[How monitoring works] in the
{ref}[Elasticsearch Reference].
include::collectors-legacy.asciidoc[]
include::monitoring-output-legacy.asciidoc[]
[[configure-internal-collectors-legacy]]
==== Configure {ls} monitoring with internal collectors
++++
<titleabbrev>Configure internal collection</titleabbrev>
++++
To monitor Logstash nodes:
. Specify where to send monitoring data. This cluster is often referred to as
the _production cluster_. For examples of typical monitoring architectures, see
{ref}/how-monitoring-works.html[How monitoring works].
+
--
IMPORTANT: To visualize Logstash as part of the Elastic Stack (as shown in Step
6), send metrics to your _production_ cluster. Sending metrics to a dedicated
monitoring cluster will show the Logstash metrics under the _monitoring_ cluster.
--
. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the
production cluster. If that setting is `false`, the collection of monitoring data
is disabled in {es} and data is ignored from all other sources.
. Configure your Logstash nodes to send metrics by setting the
`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features}
are enabled, you also need to specify the credentials for the
{ref}/built-in-users.html[built-in `logstash_system` user]. For more information
about these settings, see <<monitoring-settings-legacy>>.
+
--
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1>
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "changeme"
--------------------------------------------------
<1> If SSL/TLS is enabled on the production cluster, you must
connect through HTTPS. As of v5.2.1, you can specify multiple
Elasticsearch hosts as an array as well as specifying a single
host as a string. If multiple URLs are specified, Logstash
can round-robin requests to these production nodes.
--
. If SSL/TLS is enabled on the production {es} cluster, specify the trusted
CA certificates that will be used to verify the identity of the nodes
in the cluster.
+
--
To add a CA certificate to a Logstash node's trusted certificates, you
can specify the location of the PEM encoded certificate with the
`certificate_authority` setting:
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt
--------------------------------------------------
Alternatively, you can configure trusted certificates using a truststore
(a Java Keystore file that contains the certificates):
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file
xpack.monitoring.elasticsearch.ssl.truststore.password: password
--------------------------------------------------
Also, optionally, you can set up client certificate using a keystore
(a Java Keystore file that contains the certificate):
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
xpack.monitoring.elasticsearch.ssl.keystore.password: password
--------------------------------------------------
Set sniffing to `true` to enable discovery of other nodes of the {es} cluster.
It defaults to `false`.
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.sniffing: false
--------------------------------------------------
--
. Restart your Logstash nodes.
. To verify your monitoring configuration, point your web browser at your {kib}
host, and select **Monitoring** from the side navigation. Metrics reported from
your Logstash nodes should be visible in the Logstash section. When security is
enabled, to view the monitoring dashboards you must log in to {kib} as a user
who has the `kibana_user` and `monitoring_user` roles.
+
image::images/monitoring-ui.png["Monitoring",link="monitoring/images/monitoring-ui.png"]
include::../settings/monitoring-settings-legacy.asciidoc[]

View file

@ -1,36 +1,15 @@
[role="xpack"]
[[monitoring-internal-collection]]
=== Collect {ls} monitoring data using internal collectors
=== Use internal collectors to send monitoring data
++++
<titleabbrev>Internal collection</titleabbrev>
++++
Using internal collectors for {ls} {monitoring} these components:
* <<logstash-monitoring-collectors,Collectors>>
* <<logstash-monitoring-output,Output>>
These pieces live outside of the default Logstash pipeline in a dedicated monitoring
pipeline. This configuration ensures that all data and processing has a minimal
impact on ordinary Logstash processing. Existing Logstash features, such as the
<<plugins-outputs-elasticsearch,`elasticsearch` output>>, can be reused to
benefit from its retry policies.
NOTE: The `elasticsearch` output that is used by {monitoring} for Logstash is
configured exclusively via settings found in `logstash.yml`. It is not
configured by using anything from the Logstash configurations that might also be
using their own separate `elasticsearch` outputs.
The {es} cluster that is configured for use with {monitoring} for Logstash is
expected to be the production cluster. This configuration enables the production
{es} cluster to add metadata (for example, its cluster UUID) to the Logstash
monitoring data then route it to the monitoring clusters. For more information
about typical monitoring architectures, see
{ref}/how-monitoring-works.html[How monitoring works] in the {ref}[Elasticsearch Reference].
include::collectors.asciidoc[]
include::monitoring-output.asciidoc[]
Internal collectors send {ls} monitoring data directly to your _monitoring_ cluster.
The benefit of internal collection is that you have fewer pieces of software to
install and maintain.
<<monitoring-with-metricbeat, {metricbeat} collection>> is available as an alternative.
[[configure-internal-collectors]]
==== Configure {ls} monitoring with internal collectors
@ -40,23 +19,16 @@ include::monitoring-output.asciidoc[]
To monitor Logstash nodes:
. Specify where to send monitoring data. This cluster is often referred to as
the _production cluster_. For examples of typical monitoring architectures, see
{ref}/how-monitoring-works.html[How monitoring works].
+
--
IMPORTANT: To visualize Logstash as part of the Elastic Stack (as shown in Step
6), send metrics to your _production_ cluster. Sending metrics to a dedicated
monitoring cluster will show the Logstash metrics under the _monitoring_ cluster.
. Specify the location of the _monitoring cluster_. For examples of typical
monitoring architectures, see {ref}/how-monitoring-works.html[How monitoring
works] in the {ref}[Elasticsearch Reference].
--
. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the
production cluster. If that setting is `false`, the collection of monitoring data
is disabled in {es} and data is ignored from all other sources.
. Verify that the `monitoring.collection.enabled` setting is `true` on the
monitoring cluster. If that setting is `false`, the collection of monitoring data
is disabled in {es}, and data is ignored from all other sources.
. Configure your Logstash nodes to send metrics by setting the
`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features}
`monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features}
are enabled, you also need to specify the credentials for the
{ref}/built-in-users.html[built-in `logstash_system` user]. For more
information about these settings, see <<monitoring-settings>>.
@ -64,21 +36,18 @@ information about these settings, see <<monitoring-settings>>.
--
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1>
xpack.monitoring.elasticsearch.username: "logstash_system" <2>
xpack.monitoring.elasticsearch.password: "changeme"
monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"]
monitoring.elasticsearch.username: "logstash_system"
monitoring.elasticsearch.password: "changeme"
--------------------------------------------------
<1> If SSL/TLS is enabled on the production cluster, you must
connect through HTTPS. As of v5.2.1, you can specify multiple
Elasticsearch hosts as an array as well as specifying a single
host as a string. If multiple URLs are specified, Logstash
can round-robin requests to these production nodes.
<2> If {security-features} are disabled on the production cluster, you can omit
these `username` and `password` settings.
If SSL/TLS is enabled on the monitoring cluster, you must connect through HTTPS.
You can specify a single host as a string, or multiple Elasticsearch hosts as an
array. If multiple URLs are specified, Logstash can round-robin requests to
these monitoring nodes.
--
. If SSL/TLS is enabled on the production {es} cluster, specify the trusted
. If SSL/TLS is enabled on the monitoring {es} cluster, specify the trusted
CA certificates that will be used to verify the identity of the nodes
in the cluster.
+
@ -89,7 +58,7 @@ can specify the location of the PEM encoded certificate with the
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt
monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt
--------------------------------------------------
Alternatively, you can configure trusted certificates using a truststore
@ -97,8 +66,8 @@ Alternatively, you can configure trusted certificates using a truststore
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file
xpack.monitoring.elasticsearch.ssl.truststore.password: password
monitoring.elasticsearch.ssl.truststore.path: /path/to/file
monitoring.elasticsearch.ssl.truststore.password: password
--------------------------------------------------
Also, optionally, you can set up client certificate using a keystore
@ -106,8 +75,8 @@ Also, optionally, you can set up client certificate using a keystore
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
xpack.monitoring.elasticsearch.ssl.keystore.password: password
monitoring.elasticsearch.ssl.keystore.path: /path/to/file
monitoring.elasticsearch.ssl.keystore.password: password
--------------------------------------------------
Set sniffing to `true` to enable discovery of other nodes of the {es} cluster.
@ -115,28 +84,46 @@ It defaults to `false`.
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.sniffing: false
monitoring.elasticsearch.sniffing: false
--------------------------------------------------
--
. Restart your Logstash nodes.
. To verify your {monitoring} configuration, point your web browser at your {kib}
. To verify your monitoring configuration, point your web browser at your {kib}
host, and select **Monitoring** from the side navigation. Metrics reported from
your Logstash nodes should be visible in the Logstash section. When security is
enabled, to view the monitoring dashboards you must log in to {kib} as a user
who has the `kibana_user` and `monitoring_user` roles.
enabled, you must log in to {kib} as a user who has the `kibana_user` and
`monitoring_user` roles.
+
image::images/monitoring-ui.png["Monitoring",link="monitoring/images/monitoring-ui.png"]
[float]
[[monitoring-upgraded-logstash]]
===== Re-enabling Logstash Monitoring After Upgrading
When upgrading from older versions of {xpack}, the built-in `logstash_system`
user is disabled for security reasons. To resume monitoring,
change the password and re-enable the logstash_system user.
include::../settings/monitoring-settings.asciidoc[]
[[internal-collector-components]]
==== How {ls} monitoring with internal collectors works
Monitoring {ls} with internal collectors uses these components:
* <<logstash-monitoring-collectors,Collectors>>
* <<logstash-monitoring-output,Output>>
These pieces live outside of the default Logstash pipeline in a dedicated
monitoring pipeline. This configuration ensures that all data and processing has
a minimal impact on ordinary Logstash processing. Existing Logstash features,
such as the <<plugins-outputs-elasticsearch,`elasticsearch` output>>, can be
reused to benefit from its retry policies.
NOTE: The `elasticsearch` output for Logstash monitoring is configured
exclusively through settings in `logstash.yml`.
The monitoring {es} cluster should be configured to receive {ls} monitoring
data directly from {ls}. For more information about typical monitoring
architectures, see {ref}/how-monitoring-works.html[How monitoring works] in the
{ref}[Elasticsearch Reference].
include::collectors.asciidoc[]
include::monitoring-output.asciidoc[]

View file

@ -6,16 +6,19 @@
<titleabbrev>{metricbeat} collection</titleabbrev>
++++
You can use {metricbeat} to collect data about {ls}
and ship it to the monitoring cluster, rather than routing it through the
production cluster as described in <<monitoring-internal-collection>>.
You can use {metricbeat} to collect data about {ls} and ship it to the
monitoring cluster. The benefit of Metricbeat collection is that the monitoring
agent remains active even if the {ls} instance does not.
<<monitoring-internal-collection,Internal collection>> is available as an
alternative.
//NOTE: The tagged regions are re-used in the Stack Overview.
To collect and ship monitoring data:
. <<disable-default,Disable default collection of monitoring metrics>>
. <<define-cluster__uuid,Specify optionally the target `cluster_uuid`>>
. <<define-cluster__uuid,Specify the target `cluster_uuid` (optional)>>
. <<configure-metricbeat,Install and configure {metricbeat} to collect monitoring data>>
[float]
@ -29,7 +32,7 @@ commented out:
[source,yaml]
----------------------------------
xpack.monitoring.enabled: false
monitoring.enabled: false
----------------------------------
Remove the `#` at the beginning of the line to enable the setting.

View file

@ -0,0 +1,47 @@
[float]
[role="xpack"]
[[logstash-monitoring-output-legacy]]
==== Output
Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is
to send events to outputs. In the case of monitoring for Logstash, the output
is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines,
the output is configured within the `logstash.yml` settings file via the
`xpack.monitoring.elasticsearch.*` settings.
Other than its unique manner of configuration, this `elasticsearch` output
behaves like all `elasticsearch` outputs, including its ability to pause data
collection when issues exist with the output.
IMPORTANT: It is critical that all Logstash nodes share the same setup.
Otherwise, monitoring data might be routed in different ways or to different places.
[float]
[[logstash-monitoring-default-legacy]]
===== Default Configuration
If a Logstash node does not explicitly define a monitoring output setting,
the following default configuration is used:
[source,yaml]
---------------------------------------------------
xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]
---------------------------------------------------
All data produced by monitoring for Logstash is indexed in the monitoring
cluster by using the `.monitoring-logstash` template, which is managed by the
{ref}/es-monitoring-exporters.html[exporters] within {es}.
If you are working with a cluster that has {security} enabled, extra steps are
necessary to properly configure Logstash. For more information, see
<<configuring-logstash>>.
IMPORTANT: When discussing security relative to the `elasticsearch` output, it
is critical to remember that all users are managed on the production cluster,
which is identified in the `xpack.monitoring.elasticsearch.hosts` setting.
This is particularly important to remember when you move from development
environments to production environments, where you often have dedicated
monitoring clusters.
For more information about the configuration options for the output, see
<<monitoring-settings-legacy>>.

View file

@ -4,10 +4,10 @@
==== Output
Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is
to send events to outputs. In the case of {monitoring} for Logstash, the output
to send events to outputs. In the case of Logstash monitoring, the output
is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines,
the output is configured within the `logstash.yml` settings file via the
`xpack.monitoring.elasticsearch.*` settings.
`monitoring.elasticsearch.*` settings.
Other than its unique manner of configuration, this `elasticsearch` output
behaves like all `elasticsearch` outputs, including its ability to pause data
@ -20,15 +20,15 @@ Otherwise, monitoring data might be routed in different ways or to different pla
[[logstash-monitoring-default]]
===== Default Configuration
If a Logstash node does not explicitly define an {monitoring} output setting,
If a Logstash node does not explicitly define a monitoring output setting,
the following default configuration is used:
[source,yaml]
---------------------------------------------------
xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]
monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]
---------------------------------------------------
All data produced by {monitoring} for Logstash is indexed in the monitoring
All data produced by Logstash monitoring is indexed in the monitoring
cluster by using the `.monitoring-logstash` template, which is managed by the
{ref}/es-monitoring-exporters.html[exporters] within {es}.
@ -36,12 +36,11 @@ If you are working with a cluster that has {security-features} enabled, extra
steps are necessary to properly configure Logstash. For more information, see
<<configuring-logstash>>.
IMPORTANT: When discussing security relative to the `elasticsearch` output, it
is critical to remember that all users are managed on the production cluster,
which is identified in the `xpack.monitoring.elasticsearch.hosts` setting.
This is particularly important to remember when you move from development
environments to production environments, where you often have dedicated
monitoring clusters.
IMPORTANT: When discussing security relative to the `elasticsearch` output,
remember that all users are managed on the production cluster, which is
identified in the `monitoring.elasticsearch.hosts` setting. This is particularly
important when you move from development environments to production
environments, where you often have dedicated monitoring clusters.
For more information about the configuration options for the output, see
<<monitoring-settings>>.

View file

@ -1,27 +1,40 @@
[role="xpack"]
[[configuring-logstash]]
== Monitoring {ls} with {xpack}
== Monitoring {ls}
Use the {stack} {monitor-features} to gain insight into the health of
{ls} instances running in your environment.
For an introduction to monitoring your Elastic stack, see
{ref}/monitor-elasticsearch-cluster.html[Monitoring a cluster].
{ref}/monitor-elasticsearch-cluster.html[Monitoring a cluster] in the
{ref}[Elasticsearch Reference].
[float]
[[configuring-logstash-xpack]]
=== Configuring {xpack} monitoring for {ls}
=== Configuring monitoring for {ls}
Make sure monitoring is enabled on your {es} cluster. Then configure *one* of
these methods to collect {ls} metrics:
* <<monitoring-with-metricbeat, {metricbeat} collection>>
* <<monitoring-internal-collection,Internal collection>>
* <<monitoring-with-metricbeat, {metricbeat} collection>>. Metricbeat collects
monitoring data from your {ls} instance and sends it directly to your monitoring
cluster. The benefit of Metricbeat collection is that the monitoring
agent remains active even if the {ls} instance does not.
* <<monitoring-internal-collection,Internal collection>>. Internal collectors send
monitoring data directly to your monitoring cluster. The benefit of internal
collection is that you have fewer pieces of software to install and maintain.
* <<monitoring-internal-collection-legacy,Legacy internal collection>>. Legacy
internal collectors send monitoring data to your production cluster.
This is no longer a preferred approach. Use
<<monitoring-internal-collection,internal collection>> or
<<monitoring-with-metricbeat, {metricbeat} collection>> instead.
include::monitoring-mb.asciidoc[]
include::monitoring-internal.asciidoc[]
include::monitoring-internal-legacy.asciidoc[]
include::monitoring-ui.asciidoc[]
include::pipeline-viewer.asciidoc[]
include::troubleshooting.asciidoc[]

View file

@ -19,7 +19,7 @@ requiring any extra configuration.
Alternatively, you can <<configuring-logstash,configure {monitoring}>> to send
data to a monitoring cluster.
NOTE: {monitoring} is a feature under the Basic License and is therefore
NOTE: Monitoring is a feature under the Basic License and is therefore
*free to use*.

View file

@ -1,6 +1,6 @@
[role="xpack"]
[[monitoring-troubleshooting]]
=== Troubleshooting {monitoring} in Logstash
=== Troubleshooting monitoring in Logstash
++++
<titleabbrev>Troubleshooting</titleabbrev>
++++

View file

@ -217,8 +217,8 @@ Then configure the user and password in the `logstash.yml` configuration file:
[source,yaml]
----------------------------------------------------------
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: t0p.s3cr3t
monitoring.elasticsearch.username: logstash_system
monitoring.elasticsearch.password: t0p.s3cr3t
----------------------------------------------------------
If you initially installed an older version of {xpack} and then upgraded, the

View file

@ -0,0 +1,106 @@
[role="xpack"]
[[monitoring-settings-legacy]]
==== Monitoring settings for legacy internal collection
++++
<titleabbrev>Monitoring Settings</titleabbrev>
++++
You can set the following `xpack.monitoring` settings in `logstash.yml` to
control how monitoring data is collected from your Logstash nodes. However, the
defaults work best in most circumstances. For more information about configuring
Logstash, see <<logstash-settings-file>>.
[[monitoring-general-settings-legacy]]
===== General monitoring settings
`xpack.monitoring.enabled`::
Monitoring is disabled by default. Set to `true` to enable {xpack} monitoring.
`xpack.monitoring.elasticsearch.hosts`::
The {es} instances that you want to ship your Logstash metrics to. This might be
the same {es} instance specified in the `outputs` section in your Logstash
configuration, or a different one. This is *not* the URL of your dedicated
monitoring cluster. Even if you are using a dedicated monitoring cluster, the
Logstash metrics must be routed through your production cluster. You can specify
a single host as a string, or specify multiple hosts as an array. Defaults to
`http://localhost:9200`.
NOTE: If your Elasticsearch cluster is configured with dedicated master-eliglble
nodes, Logstash metrics should _not_ be routed to these nodes, as doing so can
create resource contention and impact the stability of the Elasticsearch
cluster. Therefore, do not include such nodes in
`xpack.monitoring.elasticsearch.hosts`.
`xpack.monitoring.elasticsearch.username` and `xpack.monitoring.elasticsearch.password`::
If your {es} is protected with basic authentication, these settings provide the
username and password that the Logstash instance uses to authenticate for
shipping monitoring data.
[[monitoring-collection-settings-legacy]]
===== Monitoring collection settings
`xpack.monitoring.collection.interval`::
Controls how often data samples are collected and shipped on the Logstash side.
Defaults to `10s`. If you modify the collection interval, set the
`xpack.monitoring.min_interval_seconds` option in `kibana.yml` to the same value.
[[monitoring-ssl-settings-legacy]]
===== Monitoring TLS/SSL settings
You can configure the following Transport Layer Security (TLS) or
Secure Sockets Layer (SSL) settings. For more information, see
<<ls-monitoring-user>>.
`xpack.monitoring.elasticsearch.ssl.certificate_authority`::
Optional setting that enables you to specify a path to the `.pem` file for the
certificate authority for your {es} instance.
`xpack.monitoring.elasticsearch.ssl.truststore.path`::
Optional settings that provide the paths to the Java keystore (JKS) to validate
the servers certificate.
`xpack.monitoring.elasticsearch.ssl.truststore.password`::
Optional settings that provide the password to the truststore.
`xpack.monitoring.elasticsearch.ssl.keystore.path`::
Optional settings that provide the paths to the Java keystore (JKS) to validate
the clients certificate.
`xpack.monitoring.elasticsearch.ssl.keystore.password`::
Optional settings that provide the password to the keystore.
`xpack.monitoring.elasticsearch.ssl.verification_mode`::
Option to validate the servers certificate. Defaults to `certificate`. To
disable, set to `none`. Disabling this severely compromises security.
[[monitoring-additional-settings-legacy]]
===== Additional settings
`xpack.monitoring.elasticsearch.cloud_id`::
If you're using {es} in {ecloud}, you should specify the identifier here.
This setting is an alternative to `xpack.monitoring.elasticsearch.hosts`.
If `cloud_id` is configured, `xpack.monitoring.elasticsearch.hosts` should not be used.
The {es} instances that you want to ship your Logstash metrics to. This might be
the same {es} instance specified in the `outputs` section in your Logstash
configuration, or a different one.
`xpack.monitoring.elasticsearch.cloud_auth`::
If you're using {es} in {ecloud}, you can set your auth credentials here.
This setting is an alternative to both `xpack.monitoring.elasticsearch.username`
and `xpack.monitoring.elasticsearch.password`. If `cloud_auth` is configured,
those settings should not be used.

View file

@ -5,7 +5,7 @@
<titleabbrev>Monitoring Settings</titleabbrev>
++++
You can set the following `xpack.monitoring` settings in `logstash.yml` to
You can set the following `monitoring` settings in `logstash.yml` to
control how monitoring data is collected from your Logstash nodes. However, the
defaults work best in most circumstances. For more information about configuring
Logstash, see <<logstash-settings-file>>.
@ -14,27 +14,25 @@ Logstash, see <<logstash-settings-file>>.
[[monitoring-general-settings]]
===== General monitoring settings
`xpack.monitoring.enabled`::
`monitoring.enabled`::
Monitoring is disabled by default. Set to `true` to enable {xpack} monitoring.
Monitoring is disabled by default. Set to `true` to enable monitoring.
`xpack.monitoring.elasticsearch.hosts`::
`monitoring.elasticsearch.hosts`::
The {es} instances that you want to ship your Logstash metrics to. This might be
the same {es} instance specified in the `outputs` section in your Logstash
configuration, or a different one. This is *not* the URL of your dedicated
monitoring cluster. Even if you are using a dedicated monitoring cluster, the
Logstash metrics must be routed through your production cluster. You can specify
a single host as a string, or specify multiple hosts as an array. Defaults to
The {es} monitoring cluster that you want to ship your Logstash metrics to. This
might be the same {es} instance specified in the `outputs` section in your
Logstash configuration, or a dedicated monitoring cluster. You can specify a
single host as a string, or specify multiple hosts as an array. Defaults to
`http://localhost:9200`.
NOTE: If your Elasticsearch cluster is configured with dedicated master-eliglble
NOTE: If your Elasticsearch cluster is configured with dedicated master-eligible
nodes, Logstash metrics should _not_ be routed to these nodes, as doing so can
create resource contention and impact the stability of the Elasticsearch
cluster. Therefore, do not include such nodes in
`xpack.monitoring.elasticsearch.hosts`.
`monitoring.elasticsearch.hosts`.
`xpack.monitoring.elasticsearch.username` and `xpack.monitoring.elasticsearch.password`::
`monitoring.elasticsearch.username` and `monitoring.elasticsearch.password`::
If your {es} is protected with basic authentication, these settings provide the
username and password that the Logstash instance uses to authenticate for
@ -44,65 +42,80 @@ shipping monitoring data.
[[monitoring-collection-settings]]
===== Monitoring collection settings
`xpack.monitoring.collection.interval`::
`monitoring.collection.interval`::
Controls how often data samples are collected and shipped on the Logstash side.
Defaults to `10s`. If you modify the collection interval, set the
`xpack.monitoring.min_interval_seconds` option in `kibana.yml` to the same value.
`monitoring.min_interval_seconds` option in `kibana.yml` to the same value.
[[monitoring-cluster-uuid]]
`monitoring.cluster_uuid`::
The universally unique identifier (UUID) for the monitoring cluster.
By default, {ls} identifies and uses the `cluster uuid` value from each
elasticsearch output defined in the pipelines, and ignores this
setting.
+
If no `cluster_uuid` is discovered in elasticsearch outputs, then {ls}
uses this value to tag the data shipped to the monitoring cluster.
[[monitoring-ssl-settings]]
===== {monitoring} TLS/SSL settings
===== Monitoring TLS/SSL settings
You can configure the following Transport Layer Security (TLS) or
Secure Sockets Layer (SSL) settings. For more information, see
<<ls-monitoring-user>>.
`xpack.monitoring.elasticsearch.ssl.certificate_authority`::
`monitoring.elasticsearch.ssl.certificate_authority`::
Optional setting that enables you to specify a path to the `.pem` file for the
certificate authority for your {es} instance.
`xpack.monitoring.elasticsearch.ssl.truststore.path`::
`monitoring.elasticsearch.ssl.truststore.path`::
Optional settings that provide the paths to the Java keystore (JKS) to validate
the servers certificate.
`xpack.monitoring.elasticsearch.ssl.truststore.password`::
`monitoring.elasticsearch.ssl.truststore.password`::
Optional settings that provide the password to the truststore.
`xpack.monitoring.elasticsearch.ssl.keystore.path`::
`monitoring.elasticsearch.ssl.keystore.path`::
Optional settings that provide the paths to the Java keystore (JKS) to validate
the clients certificate.
`xpack.monitoring.elasticsearch.ssl.keystore.password`::
`monitoring.elasticsearch.ssl.keystore.password`::
Optional settings that provide the password to the keystore.
`xpack.monitoring.elasticsearch.ssl.verification_mode`::
`monitoring.elasticsearch.ssl.verification_mode`::
Option to validate the servers certificate. Defaults to `certificate`. To
disable, set to `none`. Disabling this severely compromises security.
`monitoring.elasticsearch.sniffing`::
Finds all {es} cluster nodes and adds them to the hosts list.
Defaults to `false`.
[[monitoring-additional-settings]]
===== Additional settings
`xpack.monitoring.elasticsearch.cloud_id`::
`monitoring.elasticsearch.cloud_id`::
If you're using {es} in {ecloud}, you should specify the identifier here.
This setting is an alternative to `xpack.monitoring.elasticsearch.hosts`.
If `cloud_id` is configured, `xpack.monitoring.elasticsearch.hosts` should not be used.
This setting is an alternative to `monitoring.elasticsearch.hosts`.
If `cloud_id` is configured, `monitoring.elasticsearch.hosts` should not be used.
The {es} instances that you want to ship your Logstash metrics to. This might be
the same {es} instance specified in the `outputs` section in your Logstash
configuration, or a different one.
`xpack.monitoring.elasticsearch.cloud_auth`::
`monitoring.elasticsearch.cloud_auth`::
If you're using {es} in {ecloud}, you can set your auth credentials here.
This setting is an alternative to both `xpack.monitoring.elasticsearch.username`
and `xpack.monitoring.elasticsearch.password`. If `cloud_auth` is configured,
This setting is an alternative to both `monitoring.elasticsearch.username`
and `monitoring.elasticsearch.password`. If `cloud_auth` is configured,
those settings should not be used.

View file

@ -11,7 +11,7 @@ Before upgrading Logstash:
While upgrading Logstash:
* If you use {monitoring}, you must re-use the data directory when you
* If you use monitoring, you must re-use the data directory when you
upgrade Logstash. Otherwise, the Logstash node is assigned a new persistent UUID
and becomes a new node in the monitoring data.
===========================================