mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
parent
eae1cc0d0a
commit
3f58461cf7
1 changed files with 53 additions and 34 deletions
|
@ -3,7 +3,7 @@
|
|||
=== Centralized Pipeline Management
|
||||
|
||||
The pipeline management feature centralizes the creation and
|
||||
management of Logstash configuration pipelines.
|
||||
management of Logstash configuration pipelines in {kib}.
|
||||
|
||||
NOTE: Centralized pipeline management is an {xpack} feature that is not included
|
||||
with the basic license. If you want to try all of the features, you can start a
|
||||
|
@ -13,23 +13,13 @@ information, see https://www.elastic.co/subscriptions and
|
|||
{stack-ov}/license-management.html[License
|
||||
Management].
|
||||
|
||||
From within the pipeline
|
||||
management UI in {kib}, you can control multiple Logstash instances. You can
|
||||
add, edit, and delete pipeline configurations. On the Logstash side, you simply
|
||||
need to enable configuration management and register Logstash to use the
|
||||
centrally managed pipeline configurations.
|
||||
You can control multiple Logstash instances from the pipeline management UI in
|
||||
{kib}. You can add, edit, and delete pipeline configurations. On the Logstash
|
||||
side, you simply need to enable configuration management and register Logstash
|
||||
to use the centrally managed pipeline configurations.
|
||||
|
||||
The pipeline configurations, along with some metadata, are stored in
|
||||
Elasticsearch. Any changes that you make to a pipeline definition in the UI are
|
||||
picked up and loaded automatically by all Logstash instances registered to use
|
||||
the pipeline. The changes are applied immediately; you do not have to restart
|
||||
Logstash to pick up the changes, as long as Logstash is already registered to
|
||||
use the pipeline.
|
||||
|
||||
NOTE: Centralized management is disabled until you configure and enable
|
||||
{security}.
|
||||
|
||||
==== Managing Pipelines
|
||||
==== Manage pipelines
|
||||
|
||||
Before using the pipeline management UI, you must:
|
||||
|
||||
|
@ -38,16 +28,16 @@ Before using the pipeline management UI, you must:
|
|||
the `logstash_admin` role as well as the `logstash_writer` role that you created
|
||||
when you <<ls-security,configured Logstash to use basic authentication>>.
|
||||
|
||||
To centrally manage Logstash pipelines:
|
||||
To manage Logstash pipelines in {kib}:
|
||||
|
||||
. Open {kib} in your browser and go to the Management tab. If you've set up
|
||||
configuration management correctly, you'll see an area for managing Logstash.
|
||||
Click the *Pipelines* link.
|
||||
+
|
||||
image::management/images/centralized_config.png[]
|
||||
|
||||
. To add a new pipeline, click the *Add* button and specify values for the
|
||||
following fields:
|
||||
. Click the *Pipelines* link.
|
||||
|
||||
. To add a new pipeline, click *Create pipeline* and specify values.
|
||||
+
|
||||
--
|
||||
[horizontal]
|
||||
|
@ -65,26 +55,55 @@ Pipeline::
|
|||
The pipeline configuration. You can treat the editor in the pipeline management
|
||||
UI like any other editor. You don't have to worry about whitespace or indentation.
|
||||
|
||||
image::management/images/new_pipeline.png[]
|
||||
Pipeline workers::
|
||||
The number of parallel workers used to run the filter and output stages of the pipeline.
|
||||
|
||||
Pipeline batch size::
|
||||
The maximum number of events an individual worker thread collects before
|
||||
executing filters and outputs.
|
||||
|
||||
Pipeline batch delay::
|
||||
Time in milliseconds to wait for each event before sending an undersized
|
||||
batch to pipeline workers.
|
||||
|
||||
Queue type::
|
||||
The internal queueing model for event buffering. Options are *memory* for
|
||||
in-memory queueing, or *persisted* for disk-based acknowledged queueing.
|
||||
|
||||
Queue max bytes::
|
||||
The total capacity of the queue.
|
||||
|
||||
Queue checkpoint writes::
|
||||
The maximum number of events written before a checkpoint is forced when
|
||||
persistent queues are enabled.
|
||||
|
||||
--
|
||||
|
||||
. Click *Save*.
|
||||
===== Pipeline behavior
|
||||
|
||||
The pipeline runs on all Logstash instances that are registered to use the
|
||||
pipeline. There is no validation done at the UI level. The UI will save the new
|
||||
configuration, and Logstash will attempt to load it. You need to check the local
|
||||
Logstash logs for configuration errors. If you're using the Logstash monitoring
|
||||
feature in {xpack}, you can also navigate to the Monitoring tab to check the
|
||||
status of your Logstash nodes.
|
||||
* The pipeline configurations and metadata are stored in Elasticsearch. Any
|
||||
changes that you make to a pipeline definition are picked up and loaded
|
||||
automatically by all Logstash instances registered to use the pipeline. The
|
||||
changes are applied immediately. If Logstash is registered to use the pipeline,
|
||||
you do not have to restart Logstash to pick up the changes.
|
||||
|
||||
You can specify multiple pipeline configurations that run in parallel on the
|
||||
* The pipeline runs on all Logstash instances that are registered to use the
|
||||
pipeline. {kib} saves the new configuration, and Logstash will attempt to load
|
||||
it. There is no validation done at the UI level.
|
||||
|
||||
* You need to check the local Logstash logs for configuration errors. If you're
|
||||
using the Logstash monitoring feature in {xpack}, use the Monitoring tab to
|
||||
check the status of your Logstash nodes.
|
||||
|
||||
* You can specify multiple pipeline configurations that run in parallel on the
|
||||
same Logstash node.
|
||||
|
||||
If you edit a pipeline configuration and save the changes, Logstash reloads
|
||||
* If you edit and save a pipeline configuration, Logstash reloads
|
||||
the configuration in the background and continues processing events.
|
||||
|
||||
If you delete a pipeline (for example, `apache`) from the UI, Logstash will
|
||||
attempt to stop the pipeline if it's running. Logstash will wait until all
|
||||
events have been fully processed by the pipeline. Before deleting a pipeline,
|
||||
make sure you understand your data sources because stopping a pipeline may
|
||||
* If you try to delete a pipeline that is running (for example, `apache`) in {kib}, Logstash will
|
||||
attempt to stop the pipeline. Logstash waits until all
|
||||
events have been fully processed by the pipeline. Before you delete a pipeline,
|
||||
make sure you understand your data sources. Stopping a pipeline may
|
||||
lead to data loss.
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue