github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 14:47:19 -04:00
Code Issues Projects Releases Packages Wiki Activity

Add docs for persistent queues

Browse source 
Fixes #6344
This commit is contained in:
DeDe Morton 2016-11-30 21:32:09 -08:00
parent 9bd6a2b6a9
commit a6df53c149
8 changed files with 259 additions and 93 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/index.asciidoc
View file

@ -70,8 +70,11 @@ include::static/command-line-flags.asciidoc[]
pass::[<?edit_url https://github.com/elastic/logstash/edit/master/docs/static/logging.asciidoc ?>]
include::static/logging.asciidoc[]
pass::[<?edit_url https://github.com/elastic/logstash/edit/master/docs/static/stalled-shutdown.asciidoc ?>]
include::static/stalled-shutdown.asciidoc[]
pass::[<?edit_url https://github.com/elastic/logstash/edit/master/docs/static/persistent-queues.asciidoc ?>]
include::static/persistent-queues.asciidoc[]
pass::[<?edit_url https://github.com/elastic/logstash/edit/master/docs/static/shutdown.asciidoc ?>]
include::static/shutdown.asciidoc[]
// Breaking Changes
pass::[<?edit_url https://github.com/elastic/logstash/edit/master/docs/static/breaking-changes.asciidoc ?>]

18
docs/static/deploying.asciidoc vendored
View file

@ -1,5 +1,5 @@
[[deploying-and-scaling]]
=== Deploying and Scaling Logstash
== Deploying and Scaling Logstash
As your use case for Logstash evolves, the preferred architecture at a given scale will change. This section discusses
a range of Logstash architectures in increasing order of complexity, starting from a minimal installation and adding
@ -8,7 +8,7 @@ write to a large variety of {logstash}output-plugins.html[endpoints].
[float]
[[deploying-minimal-install]]
==== The Minimal Installation
=== The Minimal Installation
The minimal Logstash installation has one Logstash instance and one Elasticsearch instance. These instances are
directly connected. Logstash uses an {logstash}input-plugins.html[_input plugin_] to ingest data and an
@ -21,7 +21,7 @@ image::static/images/deploy_1.png[]
[float]
[[deploying-filter-threads]]
==== Using Filters
=== Using Filters
Log data is typically unstructured, often contains extraneous information that isn’t relevant to your use case, and
sometimes is missing relevant information that can be derived from the log contents. You can use a
@ -40,7 +40,7 @@ image::static/images/deploy_2.png[]
[float]
[[deploying-filebeat]]
==== Using Filebeat
=== Using Filebeat
https://www.elastic.co/guide/en/beats/filebeat/current/index.html[Filebeat] is a lightweight, resource-friendly tool
written in Go that collects logs from files on the server and forwards these logs to other machines for processing.
@ -55,7 +55,7 @@ image::static/images/deploy_3.png[]
[float]
[[deploying-larger-cluster]]
==== Scaling to a Larger Elasticsearch Cluster
=== Scaling to a Larger Elasticsearch Cluster
Typically, Logstash does not communicate with a single Elasticsearch node, but with a cluster that comprises several
nodes. By default, Logstash uses the HTTP protocol to move data into the cluster.
@ -90,12 +90,12 @@ image::static/images/deploy_4.png[]
[float]
[[deploying-message-queueing]]
==== Managing Throughput Spikes with Message Queueing
=== Managing Throughput Spikes with Message Queueing
When the data coming into a Logstash pipeline exceeds the Elasticsearch cluster's ability to ingest the data, you can
use a message broker as a buffer. By default, Logstash throttles incoming events when
indexer consumption rates fall below incoming data rates. Since this throttling can lead to events being buffered at
the data source, preventing backpressure with message brokers becomes an important part of managing your deployment.
the data source, preventing back pressure with message brokers becomes an important part of managing your deployment.
Adding a message broker to your Logstash deployment also provides a level of protection from data loss. When a Logstash
instance that has consumed data from the message broker fails, the data can be replayed from the message broker to an
@ -111,7 +111,7 @@ image::static/images/deploy_5.png[]
[float]
[[deploying-logstash-ha]]
==== Multiple Connections for Logstash High Availability
=== Multiple Connections for Logstash High Availability
To make your Logstash deployment more resilient to individual instance failures, you can set up a load balancer between
your data source machines and the Logstash cluster. The load balancer handles the individual connections to the
@ -131,7 +131,7 @@ by eliminating single points of failure.
[float]
[[deploying-scaling]]
==== Scaling Logstash
=== Scaling Logstash
A mature Logstash deployment typically has the following pipeline:

83
docs/static/life-of-an-event.asciidoc vendored
View file

@ -78,82 +78,17 @@ stacktrace messages into a single event.
For more information about the available codecs, see
<<codec-plugins,Codec Plugins>>.
[[fault-tolerance]]
=== Fault Tolerance
Logstash keeps all events in main memory during processing. Logstash responds to a SIGTERM by attempting to halt inputs and waiting for pending events to finish processing before shutting down. When the pipeline cannot be flushed due to a stuck output or filter, Logstash waits indefinitely. For example, when a pipeline sends output to a database that is unreachable by the Logstash instance, the instance waits indefinitely after receiving a SIGTERM.
To enable Logstash to detect these situations and terminate with a stalled pipeline, use the `--pipeline.unsafe_shutdown` flag.
WARNING: Unsafe shutdowns, force-kills of the Logstash process, or crashes of the Logstash process for any other reason result in data loss. Shut down Logstash safely whenever possible.
[[execution-model]]
==== Execution Model
=== Execution Model
The Logstash pipeline coordinates the execution of inputs, filters, and outputs. The following schematic sketches the data flow of a pipeline:
The Logstash event processing pipeline coordinates the execution of inputs,
filters, and outputs.
[source,js]
---------------------------------------------------
input threads | pipeline worker threads
---------------------------------------------------
Each input stage in the Logstash pipeline runs in its own thread. Inputs write events to a common Java https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/SynchronousQueue.html[SynchronousQueue]. This queue holds no events, instead transferring each pushed event to a free worker, blocking if all workers are busy. Each pipeline worker thread takes a batch of events off this queue, creating a buffer per worker, runs the batch of events through the configured filters, then runs the filtered events through any outputs. The size of the batch and number of pipeline worker threads are configurable (see <<tuning-logstash>>).
Pipelines in the current release of Logstash process filtering and output in the same thread. Prior to the 2.2 release of Logstash, filtering and output were distinct stages handled by distinct threads.
This change to the Logstash architecture improves performance and enables future persistence capabilities. The new pipeline substantially improves thread liveness, decreases resource usage, and increases throughput. The current Logstash pipeline is a micro-batching pipeline, which is inherently more efficient than a one-at-a-time approach. These efficiencies come in many places, two of the more prominent ones being a reduction in contention and a consequent improvement in thread liveness. These efficiencies are especially noticeable on many-core machines.
Each `input {}` statement in the Logstash configuration file runs in its own thread. Inputs write events to a common Java https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/SynchronousQueue.html[SynchronousQueue]. This queue holds no events, instead transferring each pushed event to a free worker, blocking if all workers are busy. Each pipeline worker thread takes a batch of events off this queue, creating a buffer per worker, runs the batch of events through the configured filters, then runs the filtered events through any outputs. The size of the batch and number of pipeline worker threads are configurable. The following pseudocode illustrates the process flow:
[source,ruby]
synchronous_queue = SynchronousQueue.new
inputs.each do |input|
Thread.new do
input.run(synchronous_queue)
end
end
num_pipeline_workers.times do
Thread.new do
while true
batch = take_batch(synchronous_queue, batch_size, batch_delay)
output_batch(filter_batch(batch))
end
end
end
wait_for_threads_to_terminate()
There are three configurable options in the pipeline, `--pipeline.workers`, `--pipeline.batch.size`, and `--pipeline.batch.delay`.
The `--pipeline.workers` or `-w` parameter determines how many threads to run for filter and output processing. If you find that events are backing up, or that the CPU is not saturated, consider increasing the value of this parameter to make better use of available processing power. Good results can even be found increasing this number past the number of available processors as these threads may spend significant time in an I/O wait state when writing to external systems. Legal values for this parameter are positive integers.
The `--pipeline.batch.size` or `-b` parameter defines the maximum number of events an individual worker thread collects before attempting to execute filters and outputs. Larger batch sizes are generally more efficient, but increase memory overhead. Some hardware configurations require you to increase JVM heap size by setting the `LS_HEAP_SIZE` variable to avoid performance degradation with this option. Values of this parameter in excess of the optimum range cause performance degradation due to frequent garbage collection or JVM crashes related to out-of-memory exceptions. Output plugins can process each batch as a logical unit. The Elasticsearch output, for example, issues https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[bulk requests] for each batch received. Tuning the `-b` parameter adjusts the size of bulk requests sent to Elasticsearch.
The `--pipeline.batch.delay` option rarely needs to be tuned. This option adjusts the latency of the Logstash pipeline. Pipeline batch delay is the maximum amount of time in milliseconds that Logstash waits for new messages after receiving an event in the current pipeline worker thread. After this time elapses, Logstash begins to execute filters and outputs.The maximum time that Logstash waits between receiving an event and processing that event in a filter is the product of the `pipeline_batch_delay` and `pipeline_batch_size` settings.
[float]
==== Notes on Pipeline Configuration and Performance
The total number of inflight events is determined by the product of the `pipeline_workers` and `pipeline_batch_size` parameters. This product is referred to as the _inflight count_. Keep the value of the inflight count in mind as you adjust the `pipeline_workers` and `pipeline_batch_size` parameters. Pipelines that intermittently receive large events at irregular intervals require sufficient memory to handle these spikes. Configure the `LS_HEAP_SIZE` option accordingly.
The Logstash defaults are chosen to provide fast, safe performance for most users. To increase performance, increase the number of pipeline workers or the batch size, taking into account the following suggestions:
Measure each change to make sure it increases, rather than decreases, performance.
Ensure that you leave enough memory available to cope with a sudden increase in event size. For example, an application that generates exceptions that are represented as large blobs of text.
The number of workers may be set higher than the number of CPU cores since outputs often spend idle time in I/O wait conditions.
Threads in Java have names and you can use the `jstack`, `top`, and the VisualVM graphical tools to figure out which resources a given thread uses.
On Linux platforms, Logstash labels all the threads it can with something descriptive. For example, inputs show up as `[base]<inputname`, filter/output workers show up as `[base]>workerN`, where N is an integer. Where possible, other threads are also labeled to help you identify their purpose.
[float]
==== Profiling the Heap
When tuning Logstash you may have to adjust the heap size. You can use the https://visualvm.java.net/[VisualVM] tool to profile the heap. The *Monitor* pane in particular is useful for checking whether your heap allocation is sufficient for the current workload. The screenshots below show sample *Monitor* panes. The first pane examines a Logstash instance configured with too many inflight events. The second pane examines a Logstash instance configured with an appropriate amount of inflight events. Note that the specific batch sizes used here are most likely not applicable to your specific workload, as the memory demands of Logstash vary in large part based on the type of messages you are sending.
image::static/images/pipeline_overload.png[]
image::static/images/pipeline_correct_load.png[]
In the first example we see that the CPU isn’t being used very efficiently. In fact, the JVM is often times having to stop the VM for “full GCs”. Full garbage collections are a common symptom of excessive memory pressure. This is visible in the spiky pattern on the CPU chart. In the more efficiently configured example, the GC graph pattern is more smooth, and the CPU is used in a more uniform manner. You can also see that there is ample headroom between the allocated heap size, and the maximum allowed, giving the JVM GC a lot of room to work with.
Examining the in-depth GC statistics with a tool similar to the excellent https://visualvm.java.net/plugins.html[VisualGC] plugin shows that the over-allocated VM spends very little time in the efficient Eden GC, compared to the time spent in the more resource-intensive Old Gen “Full” GCs.
NOTE: As long as the GC pattern is acceptable, heap sizes that occasionally increase to the maximum are acceptable. Such heap size spikes happen in response to a burst of large events passing through the pipeline. In general practice, maintain a gap between the used amount of heap memory and the maximum.
This document is not a comprehensive guide to JVM GC tuning. Read the official http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html[Oracle guide] for more information on the topic. We also recommend reading http://www.semicomplete.com/blog/geekery/debugging-java-performance.html[Debugging Java Performance].
By default, Logstash uses in-memory bounded queues between pipeline stages
(input → filter and filter → output) to buffer events. If Logstash terminates
unsafely, any events that are stored in memory will be lost. To prevent data
loss, you can enable Logstash to persist in-flight events to disk. See
<<persistent-queues>> for more information.

70
docs/static/performance-checklist.asciidoc vendored
View file

@ -1,13 +1,21 @@
[[performance-tuning]]
== Performance Tuning
This section includes the following information about tuning Logstash
performance:
* <<performance-troubleshooting>>
* <<tuning-logstash>>
[[performance-troubleshooting]]
=== Performance Troubleshooting Guide
== Performance Troubleshooting Guide
You can use this troubleshooting guide to quickly diagnose and resolve Logstash performance problems. Advanced knowledge of pipeline internals is not required to understand this guide. However, the https://www.elastic.co/guide/en/logstash/current/pipeline.html[pipeline documentation] is recommended reading if you want to go beyond this guide.
You can use this troubleshooting guide to quickly diagnose and resolve Logstash performance problems. Advanced knowledge of pipeline internals is not required to understand this guide. However, the <<pipeline,pipeline documentation>> is recommended reading if you want to go beyond this guide.
You may be tempted to jump ahead and change settings like `-w` as a first attempt to improve performance. In our experience, changing this setting makes it more difficult to troubleshoot performance problems because you increase the number of variables in play. Instead, make one change at a time and measure the results. Starting at the end of this list is a sure-fire way to create a confusing situation.
[float]
=== Performance Checklist
==== Performance Checklist
. *Check the performance of input sources and output destinations:*
+
@ -34,10 +42,62 @@ You may be tempted to jump ahead and change settings like `-w` as a first attemp
+
* Often times CPU utilization can go through the roof if the heap size is too low, resulting in the JVM constantly garbage collecting.
* A quick way to check for this issue is to double the heap size and see if performance improves. Do not increase the heap size past the amount of physical memory. Leave at least 1GB free for the OS and other processes.
* You can make more accurate measurements of the JVM heap by using either the `jmap` command line utility distributed with Java or by using VisualVM.
* You can make more accurate measurements of the JVM heap by using either the `jmap` command line utility distributed with Java or by using VisualVM. For more info, see <<profiling-the-heap>>.
. *Tune Logstash worker settings:*
+
* Begin by scaling up the number of pipeline workers by using the `-w` flag. This will increase the number of threads available for filters and outputs. It is safe to scale this up to a multiple of CPU cores, if need be, as the threads can become idle on I/O.
* Each output can only be active in a single pipeline worker thread by default. You can increase this by changing the `workers` setting in the configuration block for each output. Never make this value larger than the number of pipeline workers.
* You may also tune the output batch size. For many outputs, such as the Elasticsearch output, this setting will correspond to the size of I/O operations. In the case of the Elasticsearch output, this setting corresponds to the batch size.
[[tuning-logstash]]
=== Tuning and Profiling Logstash Performance
The Logstash defaults are chosen to provide fast, safe performance for most
users. However if you notice performance issues, you may need to modify
some of the defaults. Logstash provides the following configurable options
for tuning pipeline performance: `pipeline.workers`, `pipeline.batch.size`, and `pipeline.batch.delay`. For more information about setting these options, see <<logstash-settings-file>>.
Make sure you've read the <<performance-troubleshooting>> before modifying these options.
* The `pipeline.workers` setting determines how many threads to run for filter and output processing. If you find that events are backing up, or that the CPU is not saturated, consider increasing the value of this parameter to make better use of available processing power. Good results can even be found increasing this number past the number of available processors as these threads may spend significant time in an I/O wait state when writing to external systems. Legal values for this parameter are positive integers.
* The `pipeline.batch.size` setting defines the maximum number of events an individual worker thread collects before attempting to execute filters and outputs. Larger batch sizes are generally more efficient, but increase memory overhead. Some hardware configurations require you to increase JVM heap size by setting the `LS_HEAP_SIZE` variable to avoid performance degradation with this option. Values of this parameter in excess of the optimum range cause performance degradation due to frequent garbage collection or JVM crashes related to out-of-memory exceptions. Output plugins can process each batch as a logical unit. The Elasticsearch output, for example, issues https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[bulk requests] for each batch received. Tuning the `-b` parameter adjusts the size of bulk requests sent to Elasticsearch.
* The `pipeline.batch.delay` setting rarely needs to be tuned. This setting adjusts the latency of the Logstash pipeline. Pipeline batch delay is the maximum amount of time in milliseconds that Logstash waits for new messages after receiving an event in the current pipeline worker thread. After this time elapses, Logstash begins to execute filters and outputs.The maximum time that Logstash waits between receiving an event and processing that event in a filter is the product of the `pipeline.batch.delay` and `pipeline.batch.size` settings.
[float]
==== Notes on Pipeline Configuration and Performance
If you plan to modify the default pipeline settings, take into account the
following suggestions:
* The total number of inflight events is determined by the product of the `pipeline.workers` and `pipeline.batch.size` settings. This product is referred to as the _inflight count_. Keep the value of the inflight count in mind as you adjust the `pipeline.workers` and `pipeline.batch.size` settings. Pipelines that intermittently receive large events at irregular intervals require sufficient memory to handle these spikes. Configure the `LS_HEAP_SIZE` variable accordingly.
* Measure each change to make sure it increases, rather than decreases, performance.
* Ensure that you leave enough memory available to cope with a sudden increase in event size. For example, an application that generates exceptions that are represented as large blobs of text.
* The number of workers may be set higher than the number of CPU cores since outputs often spend idle time in I/O wait conditions.
* Threads in Java have names and you can use the `jstack`, `top`, and the VisualVM graphical tools to figure out which resources a given thread uses.
* On Linux platforms, Logstash labels all the threads it can with something descriptive. For example, inputs show up as `[base]<inputname`, filter/output workers show up as `[base]>workerN`, where N is an integer. Where possible, other threads are also labeled to help you identify their purpose.
[float]
[[profiling-the-heap]]
==== Profiling the Heap
When tuning Logstash you may have to adjust the heap size. You can use the https://visualvm.java.net/[VisualVM] tool to profile the heap. The *Monitor* pane in particular is useful for checking whether your heap allocation is sufficient for the current workload. The screenshots below show sample *Monitor* panes. The first pane examines a Logstash instance configured with too many inflight events. The second pane examines a Logstash instance configured with an appropriate amount of inflight events. Note that the specific batch sizes used here are most likely not applicable to your specific workload, as the memory demands of Logstash vary in large part based on the type of messages you are sending.
image::static/images/pipeline_overload.png[]
image::static/images/pipeline_correct_load.png[]
In the first example we see that the CPU isn’t being used very efficiently. In fact, the JVM is often times having to stop the VM for “full GCs”. Full garbage collections are a common symptom of excessive memory pressure. This is visible in the spiky pattern on the CPU chart. In the more efficiently configured example, the GC graph pattern is more smooth, and the CPU is used in a more uniform manner. You can also see that there is ample headroom between the allocated heap size, and the maximum allowed, giving the JVM GC a lot of room to work with.
Examining the in-depth GC statistics with a tool similar to the excellent https://visualvm.java.net/plugins.html[VisualGC] plugin shows that the over-allocated VM spends very little time in the efficient Eden GC, compared to the time spent in the more resource-intensive Old Gen “Full” GCs.
NOTE: As long as the GC pattern is acceptable, heap sizes that occasionally increase to the maximum are acceptable. Such heap size spikes happen in response to a burst of large events passing through the pipeline. In general practice, maintain a gap between the used amount of heap memory and the maximum.
This document is not a comprehensive guide to JVM GC tuning. Read the official http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html[Oracle guide] for more information on the topic. We also recommend reading http://www.semicomplete.com/blog/geekery/debugging-java-performance.html[Debugging Java Performance].

127
docs/static/persistent-queues.asciidoc vendored Normal file
View file

@ -0,0 +1,127 @@
[[persistent-queues]]
=== Persistent Queues
WARNING: This functionality is in beta and is subject to change. It should be deployed in production at your own risk.
By default, Logstash uses in-memory bounded queues between pipeline stages
(input → filter and filter → output) to buffer events. The size of these
in-memory queues is fixed and not configurable. If Logstash terminates unsafely,
either as the result of a software failure or the user forcing an unsafe
shutdown, it's possible to lose queued events.
To prevent event loss in these scenarios, you can configure Logstash to use
persisent queues. With persistent queues enabled, Logstash persists buffered
events to disk instead of storing them in memory.
Persistent queues are also useful for Logstash deployments that require high
throughput and resiliency. Instead of deploying and managing a message
broker, such as Redis, RabbitMQ, or Apache Kafka, to handle a mismatch in
cadence between the shipping stage and the relatively expensive processing
stage, you can enable persistent queues to buffer events on disk. The queue size
is variable and configurable, which means that you have more control over
managing situations that could result in back pressure to the source.
[[persistent-queues-advantages]]
==== Advantages of Persistent Queues
Using persistent queues:
* Provides protection from losing in-flight messages when the Logstash process is shut down or the machine is restarted.
* Handles the surge of events without having to use an external queueing mechanism like Redis or Kafka.
* Provides an at-least-once message delivery guarantee. If Logstash is restarted
while events are in-flight, Logstash will attempt to deliver messages stored
in the persistent queue until delivery succeeds at least once. In other words,
messages stored in the persistent queue may be duplicated, but not lost.
[[persistent-queues-limitations]]
==== Limitations of Persistent Queues
The current implementation of persistent queues has the following limitations:
* This version does not enable full end-to-end resiliency. Logstash only
acknowledges delivery of messages in the filter and output stages, and not all
the way back to the input or source.
* It does not handle permanent disk or machine failures. The data persisted to disk is not replicated, so it is still a single point of failure.
[[persistent-queues-architecture]]
==== How Persistent Queues Work
The persistent queue sits between the input and filter stages in the same
process:
input → persistent queue → filter + output
The input stage reads data from the configured data source and writes events to
the persistent queue for processing. As events pass through the pipeline,
Logstash pulls a batch of events from the persistent queue for processing them
in the filter and output stages. As events are processed, Logstash uses a
checkpoint file to track which events are successfully acknowledged (ACKed) as
processed by Logstash. An event is recorded as ACKed in the checkpoint file if
the event is successfully sent to the last output stage in the pipeline;
Logstash does not wait for the output to acknowledge delivery.
During a normal, controlled shutdown (*CTRL+C*), Logstash finishes
processing the current in-flight events (that is, the events being processed by
the filter and output stages, not the queued events), finalizes the ACKing
of these events, and then terminates the Logstash process. Upon restart,
Logstash uses the checkpoint file to pick up where it left off in the persistent
queue and processes the events in the backlog.
If Logstash crashes or experiences an uncontrolled shutdown, any in-flight
events are left as unACKed in the persistent queue. Upon restart, Logstash will
replay the events from its history, potentially leading to duplicate data being
written to the output.
[[configuring-persistent-queues]]
==== Configuring Persistent Queues
To configure persistent queues, you can specify the following options in the
Logstash settings file:
* `queue.type`: Specify `persisted` to enable persistent queues. By default, persistent queues are disabled (`queue.type: memory`).
* `path.queue`: The directory path where the data files will be stored. By default, the files are stored in `path.data/queue`.
* `queue.page_capacity`: The size of the page data file. The queue data consists of append-only data files separated into pages. The default size is 250mb.
* `queue.max_events`: The maximum number of unread events that are allowed in the queue. The default is 0 (unlimited).
* `queue.max_bytes`: The total capacity of the queue in number of bytes. The
default is 1024mb (1g). Make sure the capacity of your disk drive is greater
than the value you specify here. If both `queue.max_events` and
`queue.max_bytes` are specified, Logstash uses whichever criteria is reached
first.
You can also specify options that control when the checkpoint file gets updated (`queue.checkpoint.acks`, `queue.checkpoint.writes`, and
`queue.checkpoint.interval`). See <<logstash-settings-file>> for more
information about these options.
Example configuration:
[source, yaml]
queue.type: persisted
queue.max_bytes: 4g
[[backpressure-persistent-queues]]
==== Handling Back Pressure
Logstash has a built-in mechanism that adds back pressure to the data flow when
the queue is full. This mechanism helps Logstash control the rate of data flow
at the input stage without overwhelming downstream stages and outputs like
Elasticsearch.
You can control when back pressure happens by using the `queue.max_bytes`
setting to configure the capacity of the queue on disk. The following example
sets the total capacity of the queue to 8gb:
[source, yaml]
queue.type: persisted
queue.max_bytes: 8gb
With these settings specified, Logstash will buffer unACKed events on disk until
the size of the queue reaches 8gb. When the queue is full of unACKed events, and
the size limit has been reached, Logstash will no longer accept new events. This
process is called back pressure.
Each input handles back pressure independently. For example, when the
<<plugins-inputs-beats,beats>> input encounters back pressure, it no longer
accepts new connections and waits until the persistent queue has space to accept
more events. After the filter and output stages finish processing existing
events in the queue and ACKs them, Logstash automatically starts accepting new
events.

7
docs/static/setting-up-logstash.asciidoc vendored
View file

@ -8,8 +8,13 @@ This section includes additional information on how to set up and run Logstash,
* <<dir-layout>>
* <<config-setting-files>>
* <<running-logstash>>
* <<command-line-flags>>
* <<docker>>
* <<logstash-settings-file>>
* <<command-line-flags>>
* <<logging>>
* <<persistent-queues>>
* <<shutdown>>
[[dir-layout]]
=== Logstash Directory Layout

32
docs/static/settings-file.asciidoc vendored
View file

@ -102,6 +102,38 @@ The `logstash.yml` file includes the following settings:
in plaintext passwords appearing in your logs!
| `false`
| `queue.type`
| The internal queuing model to use for event buffering. Specify `memory` for legacy in-memory based queuing, or `persisted` for disk-based ACKed queueing (<<persistent-queues,persistent queues>>).
| `memory`
| `path.queue`
| The directory path where the data files will be stored when persistent queues are enabled (`queue.type: persisted`).
| `path.data/queue`
| `queue.page_capacity`
| The size of the page data files used when persistent queues are enabled (`queue.type: persisted`). The queue data consists of append-only data files separated into pages.
| 250mb
| `queue.max_events`
| The maximum number of unread events in the queue when persistent queues are enabled (`queue.type: persisted`).
| 0 (unlimited)
| `queue.max_bytes`
| The total capacity of the queue in number of bytes. Make sure the capacity of your disk drive is greater than the value you specify here. If both `queue.max_events` and `queue.max_bytes` are specified, Logstash uses whichever criteria is reached first.
| 1024mb (1g)
| `queue.checkpoint.acks`
| The maximum number of ACKed events before forcing a checkpoint when persistent queues are enabled (`queue.type: persisted`). Specify `queue.checkpoint.acks: 0` to set this value to unlimited.
|1024
| `queue.checkpoint.writes`
| The maximum number of written events before forcing a checkpoint when persistent queues are enabled (`queue.type: persisted`). Specify `queue.checkpoint.writes: 0` to set this value to unlimited.
| 1024
| `queue.checkpoint.interval`
| The interval in milliseconds when a checkpoint is forced on the head page when persistent queues are enabled (`queue.type: persisted`). Specify `queue.checkpoint.interval: 0` for no periodic checkpoint.
| 1000
| `http.host`
| The bind address for the metrics REST endpoint.
| `"127.0.0.1"`

8
docs/static/stalled-shutdown.asciidoc → docs/static/shutdown.asciidoc vendored
View file

@ -1,5 +1,5 @@
[[stalled-shutdown]]
=== Stalled Shutdown Detection
[[shutdown]]
=== Shutting Down Logstash
When you attempt to shut down a running Logstash instance, Logstash performs several steps before it can safely shut down. It must:
@ -23,6 +23,10 @@ worker threads.
To enable Logstash to forcibly terminate in the case of a stalled shutdown, use the `--pipeline.unsafe_shutdown` flag when
you start Logstash.
WARNING: Unsafe shutdowns, force-kills of the Logstash process, or crashes of the Logstash process for any other reason may result in data loss (unless you've
enabled Logstash to use <<persistent-queues,persistent queues>>). Shut down
Logstash safely whenever possible.
[[shutdown-stall-example]]
==== Stall Detection Example