From cbc6209b75b3c241bd4726d0332362876bd59bcc Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Wed, 22 Apr 2020 09:01:29 -0400 Subject: [PATCH] Remove new internal collection Fixes #11823 --- docs/static/monitoring/collectors.asciidoc | 49 ------- .../monitoring/monitoring-internal.asciidoc | 127 ------------------ docs/static/monitoring/monitoring-mb.asciidoc | 3 - .../monitoring/monitoring-output.asciidoc | 45 ------- .../monitoring/monitoring-overview.asciidoc | 4 - .../settings/monitoring-settings.asciidoc | 126 ----------------- 6 files changed, 354 deletions(-) delete mode 100644 docs/static/monitoring/collectors.asciidoc delete mode 100644 docs/static/monitoring/monitoring-internal.asciidoc delete mode 100644 docs/static/monitoring/monitoring-output.asciidoc delete mode 100644 docs/static/settings/monitoring-settings.asciidoc diff --git a/docs/static/monitoring/collectors.asciidoc b/docs/static/monitoring/collectors.asciidoc deleted file mode 100644 index 8b09cf6a0..000000000 --- a/docs/static/monitoring/collectors.asciidoc +++ /dev/null @@ -1,49 +0,0 @@ -[float] -[role="xpack"] -[[logstash-monitoring-collectors]] -===== Collectors - -Collectors, as their name implies, collect things. In monitoring for Logstash, -collectors are just <> in the same way that ordinary Logstash -configurations provide inputs. - -Like monitoring for {es}, each collector can create zero or more monitoring -documents. As it is currently implemented, each Logstash node runs two types of -collectors: one for node stats and one for pipeline stats. - -[options="header"] -|======================= -| Collector | Data Types | Description -| Node Stats | `logstash_stats` -| Gathers details about the running node, such as memory utilization and CPU -usage (for example, `GET /_stats`). - -This runs on every Logstash node with monitoring enabled. One common -failure is that Logstash directories are copied with their `path.data` directory -included (`./data` by default), which copies the persistent UUID of the Logstash -node along with it. As a result, it generally appears that one or more Logstash -nodes are failing to collect monitoring data, when in fact they are all really -misreporting as the _same_ Logstash node. Re-use `path.data` directories only -when upgrading Logstash, such that upgraded nodes replace the previous versions. -| Pipeline Stats | `logstash_state` -| Gathers details about the node's running pipelines, which powers the -Monitoring Pipeline UI. -|======================= - -Per collection interval, which defaults to 10 seconds (`10s`), each collector is -run. The failure of an individual collector does not impact any other collector. -Each collector, as an ordinary Logstash input, creates a separate Logstash event -in its isolated monitoring pipeline. The Logstash output then sends the data. - -The collection interval can be configured dynamically and you can also disable -data collection. For more information about the configuration options for the -collectors, see <>. - -WARNING: Unlike for {es} and {kib} monitoring, there is no -`monitoring.collection.enabled` setting on Logstash. You must use the -`monitoring.enabled` setting to enable and disable data collection. - -If gaps exist in the monitoring charts in {kib}, it is typically because either -a collector failed or the monitoring cluster did not receive the data (for -example, it was being restarted). In the event that a collector fails, a logged -error should exist on the node that attempted to perform the collection. diff --git a/docs/static/monitoring/monitoring-internal.asciidoc b/docs/static/monitoring/monitoring-internal.asciidoc deleted file mode 100644 index f7bd95aae..000000000 --- a/docs/static/monitoring/monitoring-internal.asciidoc +++ /dev/null @@ -1,127 +0,0 @@ -[role="xpack"] -[[monitoring-internal-collection]] -=== Use internal collectors to send monitoring data (Experimental) -experimental[] -++++ -Internal collection (Experimental) -++++ - -Internal collectors send {ls} monitoring data directly to your _monitoring_ cluster. -<> is available as an alternative. - -IMPORTANT: All Logstash nodes must share the same setup. -Otherwise, monitoring data might be routed in different ways or to different places. - -[[configure-internal-collectors]] -==== Configure {ls} monitoring with internal collectors -experimental[] -++++ -Configure internal collection -++++ - -To monitor Logstash nodes: - -. Specify the location of the _monitoring cluster_. For examples of typical -monitoring architectures, see {ref}/how-monitoring-works.html[How monitoring -works] in the {ref}[Elasticsearch Reference]. - -. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the -monitoring cluster. If that setting is `false`, the collection of monitoring data -is disabled in {es}, and data is ignored from all other sources. - -. Configure your Logstash nodes to send metrics by setting the -`monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features} -are enabled, you also need to specify the credentials for the -{ref}/built-in-users.html[built-in `logstash_system` user]. For more -information about these settings, see <>. -+ --- -[source,yaml] --------------------------------------------------- -monitoring.elasticsearch.hosts: ["http://es-monitoring-node-1:9200", "http://es-monitoring-node-2:9200"] -monitoring.elasticsearch.username: "logstash_system" -monitoring.elasticsearch.password: "changeme" --------------------------------------------------- - -If SSL/TLS is enabled on the monitoring cluster, you must connect through HTTPS. -You can specify a single host as a string, or multiple Elasticsearch hosts as an -array. If multiple URLs are specified, Logstash can round-robin requests to -these monitoring nodes. --- - -. If SSL/TLS is enabled on the monitoring {es} cluster, specify the trusted -CA certificates that will be used to verify the identity of the nodes -in the cluster. -+ --- -To add a CA certificate to a Logstash node's trusted certificates, you -can specify the location of the PEM encoded certificate with the -`certificate_authority` setting: - -[source,yaml] --------------------------------------------------- -monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt --------------------------------------------------- - -Alternatively, you can configure trusted certificates using a truststore -(a Java Keystore file that contains the certificates): - -[source,yaml] --------------------------------------------------- -monitoring.elasticsearch.ssl.truststore.path: /path/to/file -monitoring.elasticsearch.ssl.truststore.password: password --------------------------------------------------- - -Also, optionally, you can set up client certificate using a keystore -(a Java Keystore file that contains the certificate): - -[source,yaml] --------------------------------------------------- -monitoring.elasticsearch.ssl.keystore.path: /path/to/file -monitoring.elasticsearch.ssl.keystore.password: password --------------------------------------------------- - -Set sniffing to `true` to enable discovery of other nodes of the {es} cluster. -It defaults to `false`. - -[source,yaml] --------------------------------------------------- -monitoring.elasticsearch.sniffing: false --------------------------------------------------- - --- - -. Restart your Logstash nodes. - -. To verify your monitoring configuration, point your web browser at your {kib} -host, and select **Monitoring** from the side navigation. Metrics reported from -your Logstash nodes should be visible in the Logstash section. When security is -enabled, you must log in to {kib} as a user who has the `kibana_user` and -`monitoring_user` roles. - -include::../settings/monitoring-settings.asciidoc[] - - -[[internal-collector-components]] -==== How {ls} monitoring with internal collectors works - -Monitoring {ls} with internal collectors uses these components: - -* <> -* <> - -These pieces live outside of the default Logstash pipeline in a dedicated -monitoring pipeline. This configuration ensures that all data and processing has -a minimal impact on ordinary Logstash processing. - -NOTE: The `elasticsearch` output for Logstash monitoring is configured -exclusively through settings in `logstash.yml`. - -The monitoring {es} cluster should be configured to receive {ls} monitoring -data directly from {ls}. For more information about typical monitoring -architectures, see {ref}/how-monitoring-works.html[How monitoring works] in the -{ref}[Elasticsearch Reference]. - - -include::collectors.asciidoc[] -include::monitoring-output.asciidoc[] diff --git a/docs/static/monitoring/monitoring-mb.asciidoc b/docs/static/monitoring/monitoring-mb.asciidoc index dec7f6925..a6bd60edc 100644 --- a/docs/static/monitoring/monitoring-mb.asciidoc +++ b/docs/static/monitoring/monitoring-mb.asciidoc @@ -10,9 +10,6 @@ You can use {metricbeat} to collect data about {ls} and ship it to the monitoring cluster. The benefit of Metricbeat collection is that the monitoring agent remains active even if the {ls} instance does not. -<> is available as an -alternative. - //NOTE: The tagged regions are re-used in the Stack Overview. To collect and ship monitoring data: diff --git a/docs/static/monitoring/monitoring-output.asciidoc b/docs/static/monitoring/monitoring-output.asciidoc deleted file mode 100644 index 811a9faec..000000000 --- a/docs/static/monitoring/monitoring-output.asciidoc +++ /dev/null @@ -1,45 +0,0 @@ -[float] -[role="xpack"] -[[logstash-monitoring-output]] -==== Output - -Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is -to send events to outputs. In the case of Logstash monitoring, the output -is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines, -the output is configured within the `logstash.yml` settings file via the -`monitoring.elasticsearch.*` settings. - -Other than its unique manner of configuration, this `elasticsearch` output -behaves like all `elasticsearch` outputs, including its ability to pause data -collection when issues exist with the output. - -IMPORTANT: It is critical that all Logstash nodes share the same setup. -Otherwise, monitoring data might be routed in different ways or to different places. - -[float] -[[logstash-monitoring-default]] -===== Default Configuration - -If a Logstash node does not explicitly define a monitoring output setting, -the following default configuration is used: - -[source,yaml] ---------------------------------------------------- -monitoring.elasticsearch.hosts: [ "http://localhost:9200" ] ---------------------------------------------------- - -All data produced by Logstash monitoring is indexed in the monitoring -cluster using the `.monitoring-logstash` template. - -If you are working with a cluster that has {security-features} enabled, extra -steps are necessary to properly configure Logstash. For more information, see -<>. - -IMPORTANT: When discussing security relative to the `monitoring.elasticsearch` -settings, remember that all users are managed on the monitoring cluster, which -is identified in the `monitoring.elasticsearch.hosts` setting. This is -particularly important when you move from development environments to production -environments, where you often have dedicated monitoring clusters. - -For more information about the configuration options for the output, see -<>. \ No newline at end of file diff --git a/docs/static/monitoring/monitoring-overview.asciidoc b/docs/static/monitoring/monitoring-overview.asciidoc index aa62cafed..dc7fb369c 100644 --- a/docs/static/monitoring/monitoring-overview.asciidoc +++ b/docs/static/monitoring/monitoring-overview.asciidoc @@ -21,14 +21,10 @@ monitoring data from your {ls} instance and sends it directly to your monitoring cluster. The benefit of Metricbeat collection is that the monitoring agent remains active even if the {ls} instance does not. -* <>. -Internal collectors send monitoring data directly to your monitoring cluster. - * <>. Legacy internal collectors send monitoring data to your production cluster. include::monitoring-mb.asciidoc[] -include::monitoring-internal.asciidoc[] include::monitoring-internal-legacy.asciidoc[] include::monitoring-ui.asciidoc[] include::pipeline-viewer.asciidoc[] diff --git a/docs/static/settings/monitoring-settings.asciidoc b/docs/static/settings/monitoring-settings.asciidoc deleted file mode 100644 index f0092a4e0..000000000 --- a/docs/static/settings/monitoring-settings.asciidoc +++ /dev/null @@ -1,126 +0,0 @@ -[role="xpack"] -[[monitoring-settings]] -==== Monitoring settings for internal collection -experimental[] -++++ -Monitoring Settings -++++ - -You can set the following `monitoring` settings in `logstash.yml` to -control how monitoring data is collected from your Logstash nodes. However, the -defaults work best in most circumstances. For more information about configuring -Logstash, see <>. - - -[[monitoring-general-settings]] -===== General monitoring settings - -`monitoring.enabled`:: - -Monitoring is disabled by default. Set to `true` to enable monitoring. - -`monitoring.elasticsearch.hosts`:: - -The {es} monitoring cluster that you want to ship your Logstash metrics to. This -might be the same {es} instance specified in the `outputs` section in your -Logstash configuration, or a dedicated monitoring cluster. You can specify a -single host as a string, or specify multiple hosts as an array. Defaults to -`http://localhost:9200`. - -NOTE: If your Elasticsearch monitoring cluster is configured with dedicated -master-eligible nodes, Logstash metrics should _not_ be routed to these nodes. -Doing so can create resource contention and impact the stability of the -Elasticsearch cluster. Therefore, do not include such nodes in -`monitoring.elasticsearch.hosts`. - -`monitoring.elasticsearch.username` and `monitoring.elasticsearch.password`:: - -If your {es} is protected with basic authentication, these settings provide the -username and password that the Logstash instance uses to authenticate for -shipping monitoring data. - -`monitoring.elasticsearch.proxy`:: - -Optional setting that allows you to specify a proxy URL if Logstash needs to use a proxy -to reach your Elasticsearch cluster. - -[[monitoring-collection-settings]] -===== Monitoring collection settings - -`monitoring.collection.interval`:: - -Controls how often data samples are collected and shipped on the Logstash side. -Defaults to `10s`. If you modify the collection interval, set the -`monitoring.min_interval_seconds` option in `kibana.yml` to the same value. - -[[monitoring-cluster-uuid]] -`monitoring.cluster_uuid`:: - -The universally unique identifier (UUID) for the monitoring cluster. -By default, {ls} identifies and uses the `cluster uuid` value from each -elasticsearch output defined in the pipelines, and ignores this -setting. -+ -If no `cluster_uuid` is discovered in elasticsearch outputs, then {ls} -uses this value to tag the data shipped to the monitoring cluster. - -[[monitoring-ssl-settings]] -===== Monitoring TLS/SSL settings - -You can configure the following Transport Layer Security (TLS) or -Secure Sockets Layer (SSL) settings. For more information, see -<>. - -`monitoring.elasticsearch.ssl.certificate_authority`:: - -Optional setting that enables you to specify a path to the `.pem` file for the -certificate authority for your {es} instance. - -`monitoring.elasticsearch.ssl.truststore.path`:: - -Optional settings that provide the paths to the Java keystore (JKS) to validate -the server’s certificate. - -`monitoring.elasticsearch.ssl.truststore.password`:: - -Optional settings that provide the password to the truststore. - -`monitoring.elasticsearch.ssl.keystore.path`:: - -Optional settings that provide the paths to the Java keystore (JKS) to validate -the client’s certificate. - -`monitoring.elasticsearch.ssl.keystore.password`:: - -Optional settings that provide the password to the keystore. - -`monitoring.elasticsearch.ssl.verification_mode`:: - -Option to validate the server’s certificate. Defaults to `certificate`. To -disable, set to `none`. Disabling this severely compromises security. - -`monitoring.elasticsearch.sniffing`:: - -Finds all {es} cluster nodes and adds them to the hosts list. -Defaults to `false`. - -[[monitoring-additional-settings]] -===== Additional settings - -`monitoring.elasticsearch.cloud_id`:: - -If you're using {es} in {ecloud}, you should specify the identifier here. -This setting is an alternative to `monitoring.elasticsearch.hosts`. -If `cloud_id` is configured, `monitoring.elasticsearch.hosts` should not be used. -The {es} instances that you want to ship your Logstash metrics to. This might be -the same {es} instance specified in the `outputs` section in your Logstash -configuration, or a different one. - -`monitoring.elasticsearch.cloud_auth`:: - -If you're using {es} in {ecloud}, you can set your auth credentials here. -This setting is an alternative to both `monitoring.elasticsearch.username` -and `monitoring.elasticsearch.password`. If `cloud_auth` is configured, -those settings should not be used. - -