mirror of
https://github.com/elastic/logstash.git
synced 2025-06-29 02:03:30 -04:00
948a0edf1a
* Logstash monitoring doc improvements. --------- Co-authored-by: Rob Bavey <rob.bavey@elastic.co> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
205 lines
6.9 KiB
Text
205 lines
6.9 KiB
Text
[role="xpack"]
|
|
[[monitoring-with-metricbeat]]
|
|
=== Collect {ls} monitoring data with {metricbeat}
|
|
[subs="attributes"]
|
|
++++
|
|
<titleabbrev>{metricbeat} collection</titleabbrev>
|
|
++++
|
|
|
|
You can use {metricbeat} to collect data about {ls} and ship it to the
|
|
monitoring cluster. The benefit of Metricbeat collection is that the monitoring
|
|
agent remains active even if the {ls} instance does not.
|
|
|
|
This step requires {ref}/configuring-metricbeat.html[{es} with {metricbeat} monitoring setup].
|
|
|
|
To collect and ship monitoring data:
|
|
|
|
. <<disable-default,Disable default collection of monitoring metrics>>
|
|
. <<define-cluster__uuid,Specify the target `cluster_uuid`>>
|
|
. <<configure-metricbeat,Install and configure {metricbeat} to collect monitoring data>>
|
|
|
|
Want to use {agent} instead? Refer to <<monitoring-with-elastic-agent>>.
|
|
|
|
[float]
|
|
[[disable-default]]
|
|
==== Disable default collection of {ls} monitoring metrics
|
|
|
|
--
|
|
Set the `monitoring.enabled` to `false` in logstash.yml to disable to default monitoring:
|
|
|
|
[source,yaml]
|
|
----------------------------------
|
|
monitoring.enabled: false
|
|
----------------------------------
|
|
|
|
--
|
|
|
|
[float]
|
|
[[define-cluster__uuid]]
|
|
==== Determine target Elasticsearch cluster
|
|
You will need to determine which Elasticsearch cluster that {ls} will bind metrics to in the Stack Monitoring UI by specifying the `cluster_uuid`.
|
|
When pipelines contain <<plugins-outputs-elasticsearch,{es} output plugins>>, the `cluster_uuid` is automatically calculated, and the metrics should be bound without any additional settings.
|
|
|
|
To override automatic values, or if your pipeline does not contain any <<plugins-outputs-elasticsearch,{es} output plugins>>, you can bind the metrics of {ls} to a specific cluster, by defining the target cluster in the `monitoring.cluster_uuid` setting.
|
|
in the configuration file (logstash.yml):
|
|
|
|
[source,yaml]
|
|
----------------------------------
|
|
monitoring.cluster_uuid: PRODUCTION_ES_CLUSTER_UUID
|
|
----------------------------------
|
|
Refer to {ref}/cluster-stats.html[{es} cluster stats page] to figure out how to get your cluster `cluster_uuid`.
|
|
|
|
[float]
|
|
[[configure-metricbeat]]
|
|
==== Install and configure {metricbeat}
|
|
|
|
. {metricbeat-ref}/metricbeat-installation-configuration.html[Install {metricbeat}] on the
|
|
same server as {ls}.
|
|
|
|
. Enable the `logstash-xpack` module in {metricbeat}. +
|
|
+
|
|
--
|
|
To enable the default configuration in the {metricbeat} `modules.d` directory,
|
|
run:
|
|
|
|
*deb or rpm:* +
|
|
|
|
["source","sh",subs="attributes"]
|
|
----
|
|
metricbeat modules enable logstash-xpack
|
|
----
|
|
|
|
*linux or mac:*
|
|
|
|
["source","sh",subs="attributes"]
|
|
----
|
|
./metricbeat modules enable logstash-xpack
|
|
----
|
|
|
|
*win:*
|
|
|
|
["source","sh",subs="attributes"]
|
|
----
|
|
PS > .{backslash}metricbeat.exe modules enable logstash-xpack
|
|
----
|
|
|
|
For more information, see
|
|
{metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and
|
|
{metricbeat-ref}/metricbeat-module-beat.html[beat module].
|
|
--
|
|
|
|
. Configure the `logstash-xpack` module in {metricbeat}. +
|
|
+
|
|
--
|
|
The `modules.d/logstash-xpack.yml` file contains these settings:
|
|
|
|
[source,yaml]
|
|
----------------------------------
|
|
- module: logstash
|
|
metricsets:
|
|
- node
|
|
- node_stats
|
|
period: 10s
|
|
hosts: ["localhost:9600"]
|
|
#username: "user"
|
|
#password: "secret"
|
|
xpack.enabled: true
|
|
----------------------------------
|
|
|
|
Set the `hosts`, `username`, and `password` to authenticate with {ls}.
|
|
For other module settings, it's recommended that you accept the
|
|
defaults.
|
|
|
|
By default, the module collects {ls} monitoring data from
|
|
`localhost:9600`.
|
|
|
|
To monitor multiple {ls} instances, specify a list of hosts, for example:
|
|
[source,yaml]
|
|
----------------------------------
|
|
hosts: ["http://localhost:9601","http://localhost:9602","http://localhost:9603"]
|
|
----------------------------------
|
|
|
|
**Elastic security.** The Elastic {security-features} are enabled by default.
|
|
You must provide a user ID and password so that {metricbeat} can collect metrics
|
|
successfully:
|
|
|
|
.. Create a user on the production cluster that has the
|
|
`remote_monitoring_collector` {ref}/built-in-roles.html[built-in role].
|
|
|
|
.. Add the `username` and `password` settings to the module configuration
|
|
file (`logstash-xpack.yml`).
|
|
--
|
|
|
|
. Optional: Disable the system module in the {metricbeat}.
|
|
+
|
|
--
|
|
By default, the {metricbeat-ref}/metricbeat-module-system.html[system module] is
|
|
enabled. The information it collects, however, is not shown on the
|
|
*Stack Monitoring* page in {kib}. Unless you want to use that information for
|
|
other purposes, run the following command:
|
|
|
|
["source","sh",subs="attributes,callouts"]
|
|
----------------------------------------------------------------------
|
|
metricbeat modules disable system
|
|
----------------------------------------------------------------------
|
|
--
|
|
|
|
. Identify where to send the monitoring data. +
|
|
+
|
|
--
|
|
TIP: In production environments, we strongly recommend using a separate cluster
|
|
(referred to as the _monitoring cluster_) to store the data. Using a separate
|
|
monitoring cluster prevents production cluster outages from impacting your
|
|
ability to access your monitoring data. It also prevents monitoring activities
|
|
from impacting the performance of your production cluster.
|
|
|
|
For example, specify the {es} output information in the {metricbeat}
|
|
configuration file (`metricbeat.yml`):
|
|
|
|
[source,yaml]
|
|
----------------------------------
|
|
output.elasticsearch:
|
|
# Array of hosts to connect to.
|
|
hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
|
|
|
|
# Optional protocol and basic auth credentials.
|
|
#protocol: "https"
|
|
#username: "elastic"
|
|
#password: "changeme"
|
|
----------------------------------
|
|
<1> In this example, the data is stored on a monitoring cluster with nodes
|
|
`es-mon-1` and `es-mon-2`.
|
|
|
|
If you configured the monitoring cluster to use encrypted communications, you
|
|
must access it via HTTPS. For example, use a `hosts` setting like
|
|
`https://es-mon-1:9200`.
|
|
|
|
IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the
|
|
cluster that stores the monitoring data must have at least one ingest node.
|
|
|
|
**Elastic security.** The Elastic {security-features} are enabled by default.
|
|
You must provide a user ID and password so that {metricbeat} can send metrics
|
|
successfully:
|
|
|
|
.. Create a user on the monitoring cluster that has the
|
|
`remote_monitoring_agent` {ref}/built-in-roles.html[built-in role].
|
|
Alternatively, use the `remote_monitoring_user`
|
|
{ref}/built-in-users.html[built-in user].
|
|
+
|
|
TIP: If you're using index lifecycle management, the remote monitoring user
|
|
requires additional privileges to create and read indices. For more
|
|
information, see `<<feature-roles>>`.
|
|
|
|
.. Add the `username` and `password` settings to the {es} output information in
|
|
the {metricbeat} configuration file.
|
|
|
|
For more information about these configuration options, see
|
|
{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output].
|
|
--
|
|
|
|
. {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}] to begin
|
|
collecting monitoring data.
|
|
|
|
. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].
|
|
|
|
Your monitoring setup is complete.
|