diff --git a/x-pack/docs/en/management/centralized-pipelines.asciidoc b/x-pack/docs/en/management/centralized-pipelines.asciidoc index 01d856b9b..bae669459 100644 --- a/x-pack/docs/en/management/centralized-pipelines.asciidoc +++ b/x-pack/docs/en/management/centralized-pipelines.asciidoc @@ -3,7 +3,7 @@ === Centralized Pipeline Management The pipeline management feature centralizes the creation and -management of Logstash configuration pipelines. +management of Logstash configuration pipelines in {kib}. NOTE: Centralized pipeline management is an {xpack} feature that is not included with the basic license. If you want to try all of the features, you can start a @@ -13,23 +13,13 @@ information, see https://www.elastic.co/subscriptions and {stack-ov}/license-management.html[License Management]. -From within the pipeline -management UI in {kib}, you can control multiple Logstash instances. You can -add, edit, and delete pipeline configurations. On the Logstash side, you simply -need to enable configuration management and register Logstash to use the -centrally managed pipeline configurations. +You can control multiple Logstash instances from the pipeline management UI in +{kib}. You can add, edit, and delete pipeline configurations. On the Logstash +side, you simply need to enable configuration management and register Logstash +to use the centrally managed pipeline configurations. -The pipeline configurations, along with some metadata, are stored in -Elasticsearch. Any changes that you make to a pipeline definition in the UI are -picked up and loaded automatically by all Logstash instances registered to use -the pipeline. The changes are applied immediately; you do not have to restart -Logstash to pick up the changes, as long as Logstash is already registered to -use the pipeline. -NOTE: Centralized management is disabled until you configure and enable -{security}. - -==== Managing Pipelines +==== Manage pipelines Before using the pipeline management UI, you must: @@ -38,16 +28,16 @@ Before using the pipeline management UI, you must: the `logstash_admin` role as well as the `logstash_writer` role that you created when you <>. -To centrally manage Logstash pipelines: +To manage Logstash pipelines in {kib}: . Open {kib} in your browser and go to the Management tab. If you've set up configuration management correctly, you'll see an area for managing Logstash. -Click the *Pipelines* link. + image::management/images/centralized_config.png[] -. To add a new pipeline, click the *Add* button and specify values for the -following fields: +. Click the *Pipelines* link. + +. To add a new pipeline, click *Create pipeline* and specify values. + -- [horizontal] @@ -65,26 +55,55 @@ Pipeline:: The pipeline configuration. You can treat the editor in the pipeline management UI like any other editor. You don't have to worry about whitespace or indentation. -image::management/images/new_pipeline.png[] +Pipeline workers:: +The number of parallel workers used to run the filter and output stages of the pipeline. + +Pipeline batch size:: +The maximum number of events an individual worker thread collects before +executing filters and outputs. + +Pipeline batch delay:: +Time in milliseconds to wait for each event before sending an undersized +batch to pipeline workers. + +Queue type:: +The internal queueing model for event buffering. Options are *memory* for +in-memory queueing, or *persisted* for disk-based acknowledged queueing. + +Queue max bytes:: +The total capacity of the queue. + +Queue checkpoint writes:: +The maximum number of events written before a checkpoint is forced when +persistent queues are enabled. + -- -. Click *Save*. +===== Pipeline behavior -The pipeline runs on all Logstash instances that are registered to use the -pipeline. There is no validation done at the UI level. The UI will save the new -configuration, and Logstash will attempt to load it. You need to check the local -Logstash logs for configuration errors. If you're using the Logstash monitoring -feature in {xpack}, you can also navigate to the Monitoring tab to check the -status of your Logstash nodes. +* The pipeline configurations and metadata are stored in Elasticsearch. Any +changes that you make to a pipeline definition are picked up and loaded +automatically by all Logstash instances registered to use the pipeline. The +changes are applied immediately. If Logstash is registered to use the pipeline, +you do not have to restart Logstash to pick up the changes. -You can specify multiple pipeline configurations that run in parallel on the +* The pipeline runs on all Logstash instances that are registered to use the +pipeline. {kib} saves the new configuration, and Logstash will attempt to load +it. There is no validation done at the UI level. + +* You need to check the local Logstash logs for configuration errors. If you're +using the Logstash monitoring feature in {xpack}, use the Monitoring tab to +check the status of your Logstash nodes. + +* You can specify multiple pipeline configurations that run in parallel on the same Logstash node. -If you edit a pipeline configuration and save the changes, Logstash reloads +* If you edit and save a pipeline configuration, Logstash reloads the configuration in the background and continues processing events. -If you delete a pipeline (for example, `apache`) from the UI, Logstash will -attempt to stop the pipeline if it's running. Logstash will wait until all -events have been fully processed by the pipeline. Before deleting a pipeline, -make sure you understand your data sources because stopping a pipeline may +* If you try to delete a pipeline that is running (for example, `apache`) in {kib}, Logstash will +attempt to stop the pipeline. Logstash waits until all +events have been fully processed by the pipeline. Before you delete a pipeline, +make sure you understand your data sources. Stopping a pipeline may lead to data loss. +