[role="xpack"]
[[monitoring-internal-collection]]
=== Collect {ls} monitoring data using internal collectors
++++
<titleabbrev>Internal collection</titleabbrev>
++++

Using internal collectors for {ls} {monitoring} these components:

* <<logstash-monitoring-collectors,Collectors>>
* <<logstash-monitoring-output,Output>>

These pieces live outside of the default Logstash pipeline in a dedicated monitoring
pipeline. This configuration ensures that all data and processing has a minimal
impact on ordinary Logstash processing. Existing Logstash features, such as the
<<plugins-outputs-elasticsearch,`elasticsearch` output>>, can be reused to
benefit from its retry policies. 

NOTE: The `elasticsearch` output that is used by {monitoring} for Logstash is 
configured exclusively via settings found in `logstash.yml`. It is not 
configured by using anything from the Logstash configurations that might also be 
using their own separate `elasticsearch` outputs.

The {es} cluster that is configured for use with {monitoring} for Logstash is 
expected to be the production cluster. This configuration enables the production 
{es} cluster to add metadata (for example, its cluster UUID) to the Logstash
monitoring data then route it to the monitoring clusters. For more information 
about typical monitoring architectures, see 
{ref}/how-monitoring-works.html[How monitoring works] in the {ref}[Elasticsearch Reference]. 

include::collectors.asciidoc[]
include::monitoring-output.asciidoc[]


[[configure-internal-collectors]]
==== Configure {ls} monitoring with internal collectors
++++
<titleabbrev>Configure internal collection</titleabbrev>
++++

To monitor Logstash nodes:

. Specify where to send monitoring data. This cluster is often referred to as
the _production cluster_. For examples of typical monitoring architectures, see
{ref}/how-monitoring-works.html[How monitoring works].
+
--
IMPORTANT: To visualize Logstash as part of the Elastic Stack (as shown in Step
6), send metrics to your _production_ cluster. Sending metrics to a dedicated
monitoring cluster will show the Logstash metrics under the _monitoring_ cluster.

--

. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the
production cluster. If that setting is `false`, the collection of monitoring data
is disabled in {es} and data is ignored from all other sources.

. Configure your Logstash nodes to send metrics by setting the
`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features}
are enabled, you also need to specify the credentials for the
{ref}/built-in-users.html[built-in `logstash_system` user]. For more
information about these settings, see <<monitoring-settings>>.
+
--
[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1>
xpack.monitoring.elasticsearch.username: "logstash_system" <2>
xpack.monitoring.elasticsearch.password: "changeme"
--------------------------------------------------

<1> If SSL/TLS is enabled on the production cluster, you must
connect through HTTPS. As of v5.2.1, you can specify multiple
Elasticsearch hosts as an array as well as specifying a single
host as a string. If multiple URLs are specified, Logstash
can round-robin requests to these production nodes.
<2> If {security-features} are disabled on the production cluster, you can omit 
these `username` and `password` settings. 
--

. If SSL/TLS is enabled on the production {es} cluster, specify the trusted
CA certificates that will be used to verify the identity of the nodes
in the cluster.
+
--
To add a CA certificate to a Logstash node's trusted certificates, you
can specify the location of the PEM encoded certificate with the
`certificate_authority` setting:

[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt
--------------------------------------------------

Alternatively, you can configure trusted certificates using a truststore
(a Java Keystore file that contains the certificates):

[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file
xpack.monitoring.elasticsearch.ssl.truststore.password: password
--------------------------------------------------

Also, optionally, you can set up client certificate using a keystore
(a Java Keystore file that contains the certificate):

[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
xpack.monitoring.elasticsearch.ssl.keystore.password: password
--------------------------------------------------

Set sniffing to `true` to enable discovery of other nodes of the {es} cluster.
It defaults to `false`.

[source,yaml]
--------------------------------------------------
xpack.monitoring.elasticsearch.sniffing: false
--------------------------------------------------

--

. Restart your Logstash nodes.

. To verify your {monitoring} configuration, point your web browser at your {kib}
host, and select **Monitoring** from the side navigation. Metrics reported from
your Logstash nodes should be visible in the Logstash section. When security is
enabled, to view the monitoring dashboards you must log in to {kib} as a user
who has the `kibana_user` and `monitoring_user` roles.
+
image::images/monitoring-ui.png["Monitoring",link="monitoring/images/monitoring-ui.png"]

[float]
[[monitoring-upgraded-logstash]]
===== Re-enabling Logstash Monitoring After Upgrading

When upgrading from older versions of {xpack}, the built-in `logstash_system`
user is disabled for security reasons. To resume monitoring,
change the password and re-enable the logstash_system user.

include::../settings/monitoring-settings.asciidoc[]