[role="xpack"] [[monitoring-with-metricbeat]] === Collect {ls} monitoring data with {metricbeat} [subs="attributes"] ++++ {metricbeat} collection ++++ You can use {metricbeat} to collect data about {ls} and ship it to the monitoring cluster. The benefit of Metricbeat collection is that the monitoring agent remains active even if the {ls} instance does not. This step requires {ref}/configuring-metricbeat.html[{es} with {metricbeat} monitoring setup]. To collect and ship monitoring data: . <> . <> . <> Want to use {agent} instead? Refer to <>. [float] [[disable-default]] ==== Disable default collection of {ls} monitoring metrics -- Set the `monitoring.enabled` to `false` in logstash.yml to disable to default monitoring: [source,yaml] ---------------------------------- monitoring.enabled: false ---------------------------------- -- [float] [[define-cluster__uuid]] ==== Determine target Elasticsearch cluster You will need to determine which Elasticsearch cluster that {ls} will bind metrics to in the Stack Monitoring UI by specifying the `cluster_uuid`. When pipelines contain <>, the `cluster_uuid` is automatically calculated, and the metrics should be bound without any additional settings. To override automatic values, or if your pipeline does not contain any <>, you can bind the metrics of {ls} to a specific cluster, by defining the target cluster in the `monitoring.cluster_uuid` setting. in the configuration file (logstash.yml): [source,yaml] ---------------------------------- monitoring.cluster_uuid: PRODUCTION_ES_CLUSTER_UUID ---------------------------------- Refer to {ref}/cluster-stats.html[{es} cluster stats page] to figure out how to get your cluster `cluster_uuid`. [float] [[configure-metricbeat]] ==== Install and configure {metricbeat} . {metricbeat-ref}/metricbeat-installation-configuration.html[Install {metricbeat}] on the same server as {ls}. . Enable the `logstash-xpack` module in {metricbeat}. + + -- To enable the default configuration in the {metricbeat} `modules.d` directory, run: *deb or rpm:* + ["source","sh",subs="attributes"] ---- metricbeat modules enable logstash-xpack ---- *linux or mac:* ["source","sh",subs="attributes"] ---- ./metricbeat modules enable logstash-xpack ---- *win:* ["source","sh",subs="attributes"] ---- PS > .{backslash}metricbeat.exe modules enable logstash-xpack ---- For more information, see {metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and {metricbeat-ref}/metricbeat-module-beat.html[beat module]. -- . Configure the `logstash-xpack` module in {metricbeat}. + + -- The `modules.d/logstash-xpack.yml` file contains these settings: [source,yaml] ---------------------------------- - module: logstash metricsets: - node - node_stats period: 10s hosts: ["localhost:9600"] #username: "user" #password: "secret" xpack.enabled: true ---------------------------------- Set the `hosts`, `username`, and `password` to authenticate with {ls}. For other module settings, it's recommended that you accept the defaults. By default, the module collects {ls} monitoring data from `localhost:9600`. To monitor multiple {ls} instances, specify a list of hosts, for example: [source,yaml] ---------------------------------- hosts: ["http://localhost:9601","http://localhost:9602","http://localhost:9603"] ---------------------------------- **Elastic security.** The Elastic {security-features} are enabled by default. You must provide a user ID and password so that {metricbeat} can collect metrics successfully: .. Create a user on the production cluster that has the `remote_monitoring_collector` {ref}/built-in-roles.html[built-in role]. .. Add the `username` and `password` settings to the module configuration file (`logstash-xpack.yml`). -- . Optional: Disable the system module in the {metricbeat}. + -- By default, the {metricbeat-ref}/metricbeat-module-system.html[system module] is enabled. The information it collects, however, is not shown on the *Stack Monitoring* page in {kib}. Unless you want to use that information for other purposes, run the following command: ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- metricbeat modules disable system ---------------------------------------------------------------------- -- . Identify where to send the monitoring data. + + -- TIP: In production environments, we strongly recommend using a separate cluster (referred to as the _monitoring cluster_) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. For example, specify the {es} output information in the {metricbeat} configuration file (`metricbeat.yml`): [source,yaml] ---------------------------------- output.elasticsearch: # Array of hosts to connect to. hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1> # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" ---------------------------------- <1> In this example, the data is stored on a monitoring cluster with nodes `es-mon-1` and `es-mon-2`. If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`. IMPORTANT: The {es} {monitor-features} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node. **Elastic security.** The Elastic {security-features} are enabled by default. You must provide a user ID and password so that {metricbeat} can send metrics successfully: .. Create a user on the monitoring cluster that has the `remote_monitoring_agent` {ref}/built-in-roles.html[built-in role]. Alternatively, use the `remote_monitoring_user` {ref}/built-in-users.html[built-in user]. + TIP: If you're using index lifecycle management, the remote monitoring user requires additional privileges to create and read indices. For more information, see `<>`. .. Add the `username` and `password` settings to the {es} output information in the {metricbeat} configuration file. For more information about these configuration options, see {metricbeat-ref}/elasticsearch-output.html[Configure the {es} output]. -- . {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}] to begin collecting monitoring data. . {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. Your monitoring setup is complete.