mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
0b18b79cce
Queue max bytes only applies to persistent queues Forwardports: #12334 to 7.x Co-authored-by: rsdrakh <33121848+rsdrakh@users.noreply.github.com>
115 lines
4.7 KiB
Text
115 lines
4.7 KiB
Text
[role="xpack"]
|
|
[[logstash-centralized-pipeline-management]]
|
|
=== Centralized Pipeline Management
|
|
|
|
The pipeline management feature centralizes the creation and
|
|
management of Logstash configuration pipelines in {kib}.
|
|
|
|
NOTE: Centralized pipeline management is an {xpack} feature that is not included
|
|
with the basic license. If you want to try all of the features, you can start a
|
|
30-day trial. At the end of the trial period, you can purchase a subscription to
|
|
keep using the full functionality of the {xpack} components. For more
|
|
information, see https://www.elastic.co/subscriptions and
|
|
{kibana-ref}/managing-licenses.html[License
|
|
Management].
|
|
|
|
You can control multiple Logstash instances from the pipeline management UI in
|
|
{kib}. You can add, edit, and delete pipeline configurations. On the Logstash
|
|
side, you simply need to enable configuration management and register Logstash
|
|
to use the centrally managed pipeline configurations.
|
|
|
|
IMPORTANT: After you configure {ls} to use centralized pipeline management, you can
|
|
no longer specify local pipeline configurations. The `pipelines.yml` file and
|
|
settings such as `path.config` and `config.string` are inactive when centralized
|
|
pipeline management is enabled.
|
|
|
|
==== Manage pipelines
|
|
|
|
Before using the pipeline management UI, you must:
|
|
|
|
* <<configuring-centralized-pipelines, Configure centralized pipeline management>>.
|
|
* If {kib} is protected with basic authentication, make sure your {kib} user has
|
|
the `logstash_admin` role as well as the `logstash_writer` role that you created
|
|
when you <<ls-security,configured Logstash to use basic authentication>>. Additionally,
|
|
in order to view (as read-only) non-centrally-managed pipelines in the pipeline management
|
|
UI, make sure your {kib} user has the `monitoring_user` role as well.
|
|
|
|
To manage Logstash pipelines in {kib}:
|
|
|
|
. Open {kib} in your browser and go to the Management tab. If you've set up
|
|
configuration management correctly, you'll see an area for managing Logstash.
|
|
+
|
|
image::static/management/images/centralized_config.png[]
|
|
|
|
. Click the *Pipelines* link.
|
|
|
|
. To add a new pipeline, click *Create pipeline* and specify values.
|
|
+
|
|
--
|
|
[horizontal]
|
|
Pipeline ID::
|
|
A name that uniquely identifies the pipeline. This is the ID that you used when
|
|
you
|
|
<<configuring-centralized-pipelines,configured centralized pipeline management>>
|
|
and specified a list of pipeline IDs in the `xpack.management.pipeline.id`
|
|
setting.
|
|
|
|
Description::
|
|
A description of the pipeline configuration. This information is for your use.
|
|
|
|
Pipeline::
|
|
The pipeline configuration. You can treat the editor in the pipeline management
|
|
UI like any other editor. You don't have to worry about whitespace or indentation.
|
|
|
|
Pipeline workers::
|
|
The number of parallel workers used to run the filter and output stages of the pipeline.
|
|
|
|
Pipeline batch size::
|
|
The maximum number of events an individual worker thread collects before
|
|
executing filters and outputs.
|
|
|
|
Pipeline batch delay::
|
|
Time in milliseconds to wait for each event before sending an undersized
|
|
batch to pipeline workers.
|
|
|
|
Queue type::
|
|
The internal queueing model for event buffering. Options are *memory* for
|
|
in-memory queueing, or *persisted* for disk-based acknowledged queueing.
|
|
|
|
Queue max bytes::
|
|
The total capacity of the queue when persistent queues are enabled.
|
|
|
|
Queue checkpoint writes::
|
|
The maximum number of events written before a checkpoint is forced when
|
|
persistent queues are enabled.
|
|
|
|
--
|
|
|
|
===== Pipeline behavior
|
|
|
|
* The pipeline configurations and metadata are stored in Elasticsearch. Any
|
|
changes that you make to a pipeline definition are picked up and loaded
|
|
automatically by all Logstash instances registered to use the pipeline. The
|
|
changes are applied immediately. If Logstash is registered to use the pipeline,
|
|
you do not have to restart Logstash to pick up the changes.
|
|
|
|
* The pipeline runs on all Logstash instances that are registered to use the
|
|
pipeline. {kib} saves the new configuration, and Logstash will attempt to load
|
|
it. There is no validation done at the UI level.
|
|
|
|
* You need to check the local Logstash logs for configuration errors. If you're
|
|
using the Logstash monitoring feature in {xpack}, use the Monitoring tab to
|
|
check the status of your Logstash nodes.
|
|
|
|
* You can specify multiple pipeline configurations that run in parallel on the
|
|
same Logstash node.
|
|
|
|
* If you edit and save a pipeline configuration, Logstash reloads
|
|
the configuration in the background and continues processing events.
|
|
|
|
* If you try to delete a pipeline that is running (for example, `apache`) in {kib}, Logstash will
|
|
attempt to stop the pipeline. Logstash waits until all
|
|
events have been fully processed by the pipeline. Before you delete a pipeline,
|
|
make sure you understand your data sources. Stopping a pipeline may
|
|
lead to data loss.
|
|
|