mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
Karen Metts
parent
5dbe95bfae
commit
cbc6209b75
6 changed files with 0 additions and 354 deletions
49
docs/static/monitoring/collectors.asciidoc
vendored
49
docs/static/monitoring/collectors.asciidoc
vendored
|
|
@ -1,49 +0,0 @@
|
|||
[float]
|
||||
[role="xpack"]
|
||||
[[logstash-monitoring-collectors]]
|
||||
===== Collectors
|
||||
|
||||
Collectors, as their name implies, collect things. In monitoring for Logstash,
|
||||
collectors are just <<pipeline,Inputs>> in the same way that ordinary Logstash
|
||||
configurations provide inputs.
|
||||
|
||||
Like monitoring for {es}, each collector can create zero or more monitoring
|
||||
documents. As it is currently implemented, each Logstash node runs two types of
|
||||
collectors: one for node stats and one for pipeline stats.
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
| Collector | Data Types | Description
|
||||
| Node Stats | `logstash_stats`
|
||||
| Gathers details about the running node, such as memory utilization and CPU
|
||||
usage (for example, `GET /_stats`).
|
||||
|
||||
This runs on every Logstash node with monitoring enabled. One common
|
||||
failure is that Logstash directories are copied with their `path.data` directory
|
||||
included (`./data` by default), which copies the persistent UUID of the Logstash
|
||||
node along with it. As a result, it generally appears that one or more Logstash
|
||||
nodes are failing to collect monitoring data, when in fact they are all really
|
||||
misreporting as the _same_ Logstash node. Re-use `path.data` directories only
|
||||
when upgrading Logstash, such that upgraded nodes replace the previous versions.
|
||||
| Pipeline Stats | `logstash_state`
|
||||
| Gathers details about the node's running pipelines, which powers the
|
||||
Monitoring Pipeline UI.
|
||||
|=======================
|
||||
|
||||
Per collection interval, which defaults to 10 seconds (`10s`), each collector is
|
||||
run. The failure of an individual collector does not impact any other collector.
|
||||
Each collector, as an ordinary Logstash input, creates a separate Logstash event
|
||||
in its isolated monitoring pipeline. The Logstash output then sends the data.
|
||||
|
||||
The collection interval can be configured dynamically and you can also disable
|
||||
data collection. For more information about the configuration options for the
|
||||
collectors, see <<monitoring-settings>>.
|
||||
|
||||
WARNING: Unlike for {es} and {kib} monitoring, there is no
|
||||
`monitoring.collection.enabled` setting on Logstash. You must use the
|
||||
`monitoring.enabled` setting to enable and disable data collection.
|
||||
|
||||
If gaps exist in the monitoring charts in {kib}, it is typically because either
|
||||
a collector failed or the monitoring cluster did not receive the data (for
|
||||
example, it was being restarted). In the event that a collector fails, a logged
|
||||
error should exist on the node that attempted to perform the collection.
|
||||
127
docs/static/monitoring/monitoring-internal.asciidoc
vendored
127
docs/static/monitoring/monitoring-internal.asciidoc
vendored
|
|
@ -1,127 +0,0 @@
|
|||
[role="xpack"]
|
||||
[[monitoring-internal-collection]]
|
||||
=== Use internal collectors to send monitoring data (Experimental)
|
||||
experimental[]
|
||||
++++
|
||||
<titleabbrev>Internal collection (Experimental)</titleabbrev>
|
||||
++++
|
||||
|
||||
Internal collectors send {ls} monitoring data directly to your _monitoring_ cluster.
|
||||
<<monitoring-with-metricbeat, {metricbeat} collection>> is available as an alternative.
|
||||
|
||||
IMPORTANT: All Logstash nodes must share the same setup.
|
||||
Otherwise, monitoring data might be routed in different ways or to different places.
|
||||
|
||||
[[configure-internal-collectors]]
|
||||
==== Configure {ls} monitoring with internal collectors
|
||||
experimental[]
|
||||
++++
|
||||
<titleabbrev>Configure internal collection</titleabbrev>
|
||||
++++
|
||||
|
||||
To monitor Logstash nodes:
|
||||
|
||||
. Specify the location of the _monitoring cluster_. For examples of typical
|
||||
monitoring architectures, see {ref}/how-monitoring-works.html[How monitoring
|
||||
works] in the {ref}[Elasticsearch Reference].
|
||||
|
||||
. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the
|
||||
monitoring cluster. If that setting is `false`, the collection of monitoring data
|
||||
is disabled in {es}, and data is ignored from all other sources.
|
||||
|
||||
. Configure your Logstash nodes to send metrics by setting the
|
||||
`monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features}
|
||||
are enabled, you also need to specify the credentials for the
|
||||
{ref}/built-in-users.html[built-in `logstash_system` user]. For more
|
||||
information about these settings, see <<monitoring-settings>>.
|
||||
+
|
||||
--
|
||||
[source,yaml]
|
||||
--------------------------------------------------
|
||||
monitoring.elasticsearch.hosts: ["http://es-monitoring-node-1:9200", "http://es-monitoring-node-2:9200"]
|
||||
monitoring.elasticsearch.username: "logstash_system"
|
||||
monitoring.elasticsearch.password: "changeme"
|
||||
--------------------------------------------------
|
||||
|
||||
If SSL/TLS is enabled on the monitoring cluster, you must connect through HTTPS.
|
||||
You can specify a single host as a string, or multiple Elasticsearch hosts as an
|
||||
array. If multiple URLs are specified, Logstash can round-robin requests to
|
||||
these monitoring nodes.
|
||||
--
|
||||
|
||||
. If SSL/TLS is enabled on the monitoring {es} cluster, specify the trusted
|
||||
CA certificates that will be used to verify the identity of the nodes
|
||||
in the cluster.
|
||||
+
|
||||
--
|
||||
To add a CA certificate to a Logstash node's trusted certificates, you
|
||||
can specify the location of the PEM encoded certificate with the
|
||||
`certificate_authority` setting:
|
||||
|
||||
[source,yaml]
|
||||
--------------------------------------------------
|
||||
monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt
|
||||
--------------------------------------------------
|
||||
|
||||
Alternatively, you can configure trusted certificates using a truststore
|
||||
(a Java Keystore file that contains the certificates):
|
||||
|
||||
[source,yaml]
|
||||
--------------------------------------------------
|
||||
monitoring.elasticsearch.ssl.truststore.path: /path/to/file
|
||||
monitoring.elasticsearch.ssl.truststore.password: password
|
||||
--------------------------------------------------
|
||||
|
||||
Also, optionally, you can set up client certificate using a keystore
|
||||
(a Java Keystore file that contains the certificate):
|
||||
|
||||
[source,yaml]
|
||||
--------------------------------------------------
|
||||
monitoring.elasticsearch.ssl.keystore.path: /path/to/file
|
||||
monitoring.elasticsearch.ssl.keystore.password: password
|
||||
--------------------------------------------------
|
||||
|
||||
Set sniffing to `true` to enable discovery of other nodes of the {es} cluster.
|
||||
It defaults to `false`.
|
||||
|
||||
[source,yaml]
|
||||
--------------------------------------------------
|
||||
monitoring.elasticsearch.sniffing: false
|
||||
--------------------------------------------------
|
||||
|
||||
--
|
||||
|
||||
. Restart your Logstash nodes.
|
||||
|
||||
. To verify your monitoring configuration, point your web browser at your {kib}
|
||||
host, and select **Monitoring** from the side navigation. Metrics reported from
|
||||
your Logstash nodes should be visible in the Logstash section. When security is
|
||||
enabled, you must log in to {kib} as a user who has the `kibana_user` and
|
||||
`monitoring_user` roles.
|
||||
|
||||
include::../settings/monitoring-settings.asciidoc[]
|
||||
|
||||
|
||||
[[internal-collector-components]]
|
||||
==== How {ls} monitoring with internal collectors works
|
||||
|
||||
Monitoring {ls} with internal collectors uses these components:
|
||||
|
||||
* <<logstash-monitoring-collectors,Collectors>>
|
||||
* <<logstash-monitoring-output,Output>>
|
||||
|
||||
These pieces live outside of the default Logstash pipeline in a dedicated
|
||||
monitoring pipeline. This configuration ensures that all data and processing has
|
||||
a minimal impact on ordinary Logstash processing.
|
||||
|
||||
NOTE: The `elasticsearch` output for Logstash monitoring is configured
|
||||
exclusively through settings in `logstash.yml`.
|
||||
|
||||
The monitoring {es} cluster should be configured to receive {ls} monitoring
|
||||
data directly from {ls}. For more information about typical monitoring
|
||||
architectures, see {ref}/how-monitoring-works.html[How monitoring works] in the
|
||||
{ref}[Elasticsearch Reference].
|
||||
|
||||
|
||||
include::collectors.asciidoc[]
|
||||
include::monitoring-output.asciidoc[]
|
||||
|
|
@ -10,9 +10,6 @@ You can use {metricbeat} to collect data about {ls} and ship it to the
|
|||
monitoring cluster. The benefit of Metricbeat collection is that the monitoring
|
||||
agent remains active even if the {ls} instance does not.
|
||||
|
||||
<<monitoring-internal-collection,Internal collection>> is available as an
|
||||
alternative.
|
||||
|
||||
//NOTE: The tagged regions are re-used in the Stack Overview.
|
||||
|
||||
To collect and ship monitoring data:
|
||||
|
|
|
|||
|
|
@ -1,45 +0,0 @@
|
|||
[float]
|
||||
[role="xpack"]
|
||||
[[logstash-monitoring-output]]
|
||||
==== Output
|
||||
|
||||
Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is
|
||||
to send events to outputs. In the case of Logstash monitoring, the output
|
||||
is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines,
|
||||
the output is configured within the `logstash.yml` settings file via the
|
||||
`monitoring.elasticsearch.*` settings.
|
||||
|
||||
Other than its unique manner of configuration, this `elasticsearch` output
|
||||
behaves like all `elasticsearch` outputs, including its ability to pause data
|
||||
collection when issues exist with the output.
|
||||
|
||||
IMPORTANT: It is critical that all Logstash nodes share the same setup.
|
||||
Otherwise, monitoring data might be routed in different ways or to different places.
|
||||
|
||||
[float]
|
||||
[[logstash-monitoring-default]]
|
||||
===== Default Configuration
|
||||
|
||||
If a Logstash node does not explicitly define a monitoring output setting,
|
||||
the following default configuration is used:
|
||||
|
||||
[source,yaml]
|
||||
---------------------------------------------------
|
||||
monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]
|
||||
---------------------------------------------------
|
||||
|
||||
All data produced by Logstash monitoring is indexed in the monitoring
|
||||
cluster using the `.monitoring-logstash` template.
|
||||
|
||||
If you are working with a cluster that has {security-features} enabled, extra
|
||||
steps are necessary to properly configure Logstash. For more information, see
|
||||
<<configuring-logstash>>.
|
||||
|
||||
IMPORTANT: When discussing security relative to the `monitoring.elasticsearch`
|
||||
settings, remember that all users are managed on the monitoring cluster, which
|
||||
is identified in the `monitoring.elasticsearch.hosts` setting. This is
|
||||
particularly important when you move from development environments to production
|
||||
environments, where you often have dedicated monitoring clusters.
|
||||
|
||||
For more information about the configuration options for the output, see
|
||||
<<monitoring-settings>>.
|
||||
|
|
@ -21,14 +21,10 @@ monitoring data from your {ls} instance and sends it directly to your monitoring
|
|||
cluster. The benefit of Metricbeat collection is that the monitoring
|
||||
agent remains active even if the {ls} instance does not.
|
||||
|
||||
* <<monitoring-internal-collection,Internal collection (Experimental)>>.
|
||||
Internal collectors send monitoring data directly to your monitoring cluster.
|
||||
|
||||
* <<monitoring-internal-collection-legacy,Legacy internal collection>>. Legacy
|
||||
internal collectors send monitoring data to your production cluster.
|
||||
|
||||
include::monitoring-mb.asciidoc[]
|
||||
include::monitoring-internal.asciidoc[]
|
||||
include::monitoring-internal-legacy.asciidoc[]
|
||||
include::monitoring-ui.asciidoc[]
|
||||
include::pipeline-viewer.asciidoc[]
|
||||
|
|
|
|||
126
docs/static/settings/monitoring-settings.asciidoc
vendored
126
docs/static/settings/monitoring-settings.asciidoc
vendored
|
|
@ -1,126 +0,0 @@
|
|||
[role="xpack"]
|
||||
[[monitoring-settings]]
|
||||
==== Monitoring settings for internal collection
|
||||
experimental[]
|
||||
++++
|
||||
<titleabbrev>Monitoring Settings</titleabbrev>
|
||||
++++
|
||||
|
||||
You can set the following `monitoring` settings in `logstash.yml` to
|
||||
control how monitoring data is collected from your Logstash nodes. However, the
|
||||
defaults work best in most circumstances. For more information about configuring
|
||||
Logstash, see <<logstash-settings-file>>.
|
||||
|
||||
|
||||
[[monitoring-general-settings]]
|
||||
===== General monitoring settings
|
||||
|
||||
`monitoring.enabled`::
|
||||
|
||||
Monitoring is disabled by default. Set to `true` to enable monitoring.
|
||||
|
||||
`monitoring.elasticsearch.hosts`::
|
||||
|
||||
The {es} monitoring cluster that you want to ship your Logstash metrics to. This
|
||||
might be the same {es} instance specified in the `outputs` section in your
|
||||
Logstash configuration, or a dedicated monitoring cluster. You can specify a
|
||||
single host as a string, or specify multiple hosts as an array. Defaults to
|
||||
`http://localhost:9200`.
|
||||
|
||||
NOTE: If your Elasticsearch monitoring cluster is configured with dedicated
|
||||
master-eligible nodes, Logstash metrics should _not_ be routed to these nodes.
|
||||
Doing so can create resource contention and impact the stability of the
|
||||
Elasticsearch cluster. Therefore, do not include such nodes in
|
||||
`monitoring.elasticsearch.hosts`.
|
||||
|
||||
`monitoring.elasticsearch.username` and `monitoring.elasticsearch.password`::
|
||||
|
||||
If your {es} is protected with basic authentication, these settings provide the
|
||||
username and password that the Logstash instance uses to authenticate for
|
||||
shipping monitoring data.
|
||||
|
||||
`monitoring.elasticsearch.proxy`::
|
||||
|
||||
Optional setting that allows you to specify a proxy URL if Logstash needs to use a proxy
|
||||
to reach your Elasticsearch cluster.
|
||||
|
||||
[[monitoring-collection-settings]]
|
||||
===== Monitoring collection settings
|
||||
|
||||
`monitoring.collection.interval`::
|
||||
|
||||
Controls how often data samples are collected and shipped on the Logstash side.
|
||||
Defaults to `10s`. If you modify the collection interval, set the
|
||||
`monitoring.min_interval_seconds` option in `kibana.yml` to the same value.
|
||||
|
||||
[[monitoring-cluster-uuid]]
|
||||
`monitoring.cluster_uuid`::
|
||||
|
||||
The universally unique identifier (UUID) for the monitoring cluster.
|
||||
By default, {ls} identifies and uses the `cluster uuid` value from each
|
||||
elasticsearch output defined in the pipelines, and ignores this
|
||||
setting.
|
||||
+
|
||||
If no `cluster_uuid` is discovered in elasticsearch outputs, then {ls}
|
||||
uses this value to tag the data shipped to the monitoring cluster.
|
||||
|
||||
[[monitoring-ssl-settings]]
|
||||
===== Monitoring TLS/SSL settings
|
||||
|
||||
You can configure the following Transport Layer Security (TLS) or
|
||||
Secure Sockets Layer (SSL) settings. For more information, see
|
||||
<<ls-monitoring-user>>.
|
||||
|
||||
`monitoring.elasticsearch.ssl.certificate_authority`::
|
||||
|
||||
Optional setting that enables you to specify a path to the `.pem` file for the
|
||||
certificate authority for your {es} instance.
|
||||
|
||||
`monitoring.elasticsearch.ssl.truststore.path`::
|
||||
|
||||
Optional settings that provide the paths to the Java keystore (JKS) to validate
|
||||
the server’s certificate.
|
||||
|
||||
`monitoring.elasticsearch.ssl.truststore.password`::
|
||||
|
||||
Optional settings that provide the password to the truststore.
|
||||
|
||||
`monitoring.elasticsearch.ssl.keystore.path`::
|
||||
|
||||
Optional settings that provide the paths to the Java keystore (JKS) to validate
|
||||
the client’s certificate.
|
||||
|
||||
`monitoring.elasticsearch.ssl.keystore.password`::
|
||||
|
||||
Optional settings that provide the password to the keystore.
|
||||
|
||||
`monitoring.elasticsearch.ssl.verification_mode`::
|
||||
|
||||
Option to validate the server’s certificate. Defaults to `certificate`. To
|
||||
disable, set to `none`. Disabling this severely compromises security.
|
||||
|
||||
`monitoring.elasticsearch.sniffing`::
|
||||
|
||||
Finds all {es} cluster nodes and adds them to the hosts list.
|
||||
Defaults to `false`.
|
||||
|
||||
[[monitoring-additional-settings]]
|
||||
===== Additional settings
|
||||
|
||||
`monitoring.elasticsearch.cloud_id`::
|
||||
|
||||
If you're using {es} in {ecloud}, you should specify the identifier here.
|
||||
This setting is an alternative to `monitoring.elasticsearch.hosts`.
|
||||
If `cloud_id` is configured, `monitoring.elasticsearch.hosts` should not be used.
|
||||
The {es} instances that you want to ship your Logstash metrics to. This might be
|
||||
the same {es} instance specified in the `outputs` section in your Logstash
|
||||
configuration, or a different one.
|
||||
|
||||
`monitoring.elasticsearch.cloud_auth`::
|
||||
|
||||
If you're using {es} in {ecloud}, you can set your auth credentials here.
|
||||
This setting is an alternative to both `monitoring.elasticsearch.username`
|
||||
and `monitoring.elasticsearch.password`. If `cloud_auth` is configured,
|
||||
those settings should not be used.
|
||||
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue