mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
parent
eae1cc0d0a
commit
3f58461cf7
1 changed files with 53 additions and 34 deletions
|
@ -3,7 +3,7 @@
|
||||||
=== Centralized Pipeline Management
|
=== Centralized Pipeline Management
|
||||||
|
|
||||||
The pipeline management feature centralizes the creation and
|
The pipeline management feature centralizes the creation and
|
||||||
management of Logstash configuration pipelines.
|
management of Logstash configuration pipelines in {kib}.
|
||||||
|
|
||||||
NOTE: Centralized pipeline management is an {xpack} feature that is not included
|
NOTE: Centralized pipeline management is an {xpack} feature that is not included
|
||||||
with the basic license. If you want to try all of the features, you can start a
|
with the basic license. If you want to try all of the features, you can start a
|
||||||
|
@ -13,23 +13,13 @@ information, see https://www.elastic.co/subscriptions and
|
||||||
{stack-ov}/license-management.html[License
|
{stack-ov}/license-management.html[License
|
||||||
Management].
|
Management].
|
||||||
|
|
||||||
From within the pipeline
|
You can control multiple Logstash instances from the pipeline management UI in
|
||||||
management UI in {kib}, you can control multiple Logstash instances. You can
|
{kib}. You can add, edit, and delete pipeline configurations. On the Logstash
|
||||||
add, edit, and delete pipeline configurations. On the Logstash side, you simply
|
side, you simply need to enable configuration management and register Logstash
|
||||||
need to enable configuration management and register Logstash to use the
|
to use the centrally managed pipeline configurations.
|
||||||
centrally managed pipeline configurations.
|
|
||||||
|
|
||||||
The pipeline configurations, along with some metadata, are stored in
|
|
||||||
Elasticsearch. Any changes that you make to a pipeline definition in the UI are
|
|
||||||
picked up and loaded automatically by all Logstash instances registered to use
|
|
||||||
the pipeline. The changes are applied immediately; you do not have to restart
|
|
||||||
Logstash to pick up the changes, as long as Logstash is already registered to
|
|
||||||
use the pipeline.
|
|
||||||
|
|
||||||
NOTE: Centralized management is disabled until you configure and enable
|
==== Manage pipelines
|
||||||
{security}.
|
|
||||||
|
|
||||||
==== Managing Pipelines
|
|
||||||
|
|
||||||
Before using the pipeline management UI, you must:
|
Before using the pipeline management UI, you must:
|
||||||
|
|
||||||
|
@ -38,16 +28,16 @@ Before using the pipeline management UI, you must:
|
||||||
the `logstash_admin` role as well as the `logstash_writer` role that you created
|
the `logstash_admin` role as well as the `logstash_writer` role that you created
|
||||||
when you <<ls-security,configured Logstash to use basic authentication>>.
|
when you <<ls-security,configured Logstash to use basic authentication>>.
|
||||||
|
|
||||||
To centrally manage Logstash pipelines:
|
To manage Logstash pipelines in {kib}:
|
||||||
|
|
||||||
. Open {kib} in your browser and go to the Management tab. If you've set up
|
. Open {kib} in your browser and go to the Management tab. If you've set up
|
||||||
configuration management correctly, you'll see an area for managing Logstash.
|
configuration management correctly, you'll see an area for managing Logstash.
|
||||||
Click the *Pipelines* link.
|
|
||||||
+
|
+
|
||||||
image::management/images/centralized_config.png[]
|
image::management/images/centralized_config.png[]
|
||||||
|
|
||||||
. To add a new pipeline, click the *Add* button and specify values for the
|
. Click the *Pipelines* link.
|
||||||
following fields:
|
|
||||||
|
. To add a new pipeline, click *Create pipeline* and specify values.
|
||||||
+
|
+
|
||||||
--
|
--
|
||||||
[horizontal]
|
[horizontal]
|
||||||
|
@ -65,26 +55,55 @@ Pipeline::
|
||||||
The pipeline configuration. You can treat the editor in the pipeline management
|
The pipeline configuration. You can treat the editor in the pipeline management
|
||||||
UI like any other editor. You don't have to worry about whitespace or indentation.
|
UI like any other editor. You don't have to worry about whitespace or indentation.
|
||||||
|
|
||||||
image::management/images/new_pipeline.png[]
|
Pipeline workers::
|
||||||
|
The number of parallel workers used to run the filter and output stages of the pipeline.
|
||||||
|
|
||||||
|
Pipeline batch size::
|
||||||
|
The maximum number of events an individual worker thread collects before
|
||||||
|
executing filters and outputs.
|
||||||
|
|
||||||
|
Pipeline batch delay::
|
||||||
|
Time in milliseconds to wait for each event before sending an undersized
|
||||||
|
batch to pipeline workers.
|
||||||
|
|
||||||
|
Queue type::
|
||||||
|
The internal queueing model for event buffering. Options are *memory* for
|
||||||
|
in-memory queueing, or *persisted* for disk-based acknowledged queueing.
|
||||||
|
|
||||||
|
Queue max bytes::
|
||||||
|
The total capacity of the queue.
|
||||||
|
|
||||||
|
Queue checkpoint writes::
|
||||||
|
The maximum number of events written before a checkpoint is forced when
|
||||||
|
persistent queues are enabled.
|
||||||
|
|
||||||
--
|
--
|
||||||
|
|
||||||
. Click *Save*.
|
===== Pipeline behavior
|
||||||
|
|
||||||
The pipeline runs on all Logstash instances that are registered to use the
|
* The pipeline configurations and metadata are stored in Elasticsearch. Any
|
||||||
pipeline. There is no validation done at the UI level. The UI will save the new
|
changes that you make to a pipeline definition are picked up and loaded
|
||||||
configuration, and Logstash will attempt to load it. You need to check the local
|
automatically by all Logstash instances registered to use the pipeline. The
|
||||||
Logstash logs for configuration errors. If you're using the Logstash monitoring
|
changes are applied immediately. If Logstash is registered to use the pipeline,
|
||||||
feature in {xpack}, you can also navigate to the Monitoring tab to check the
|
you do not have to restart Logstash to pick up the changes.
|
||||||
status of your Logstash nodes.
|
|
||||||
|
|
||||||
You can specify multiple pipeline configurations that run in parallel on the
|
* The pipeline runs on all Logstash instances that are registered to use the
|
||||||
|
pipeline. {kib} saves the new configuration, and Logstash will attempt to load
|
||||||
|
it. There is no validation done at the UI level.
|
||||||
|
|
||||||
|
* You need to check the local Logstash logs for configuration errors. If you're
|
||||||
|
using the Logstash monitoring feature in {xpack}, use the Monitoring tab to
|
||||||
|
check the status of your Logstash nodes.
|
||||||
|
|
||||||
|
* You can specify multiple pipeline configurations that run in parallel on the
|
||||||
same Logstash node.
|
same Logstash node.
|
||||||
|
|
||||||
If you edit a pipeline configuration and save the changes, Logstash reloads
|
* If you edit and save a pipeline configuration, Logstash reloads
|
||||||
the configuration in the background and continues processing events.
|
the configuration in the background and continues processing events.
|
||||||
|
|
||||||
If you delete a pipeline (for example, `apache`) from the UI, Logstash will
|
* If you try to delete a pipeline that is running (for example, `apache`) in {kib}, Logstash will
|
||||||
attempt to stop the pipeline if it's running. Logstash will wait until all
|
attempt to stop the pipeline. Logstash waits until all
|
||||||
events have been fully processed by the pipeline. Before deleting a pipeline,
|
events have been fully processed by the pipeline. Before you delete a pipeline,
|
||||||
make sure you understand your data sources because stopping a pipeline may
|
make sure you understand your data sources. Stopping a pipeline may
|
||||||
lead to data loss.
|
lead to data loss.
|
||||||
|
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue