mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
65 lines
2.5 KiB
Text
65 lines
2.5 KiB
Text
[[memory-queue]]
|
|
=== Memory queue
|
|
|
|
By default, Logstash uses in-memory bounded queues between pipeline stages (inputs → pipeline workers) to buffer events.
|
|
If Logstash experiences a temporary machine failure, the contents of the memory queue will be lost.
|
|
Temporary machine failures are scenarios where Logstash or its host machine are terminated abnormally, but are capable of being restarted.
|
|
|
|
[[mem-queue-benefits]]
|
|
==== Benefits of memory queues
|
|
|
|
The memory queue might be a good choice if you value throughput over data resiliency.
|
|
|
|
* Easier configuration
|
|
* Easier management and administration
|
|
* Faster throughput
|
|
|
|
[[mem-queue-limitations]]
|
|
==== Limitations of memory queues
|
|
|
|
* Can lose data in abnormal termination
|
|
* Don't do well handling sudden bursts of data, where extra capacity in needed for {ls} to catch up
|
|
|
|
TIP: Consider using <<persistent-queues,persistent queues>> to avoid these limitations.
|
|
|
|
[[sizing-mem-queue]]
|
|
==== Memory queue size
|
|
|
|
Memory queue size is not configured directly.
|
|
Instead, it depends on how you have Logstash tuned.
|
|
|
|
Its upper bound is defined by `pipeline.workers` (default: number of CPUs) times the `pipeline.batch.size` (default: 125) events.
|
|
This value, called the "inflight count," determines maximum number of events that can be held in each memory queue.
|
|
|
|
Doubling the number of workers OR doubling the batch size will effectively double the memory queue's capacity (and memory usage).
|
|
Doubling both will _quadruple_ the capacity (and usage).
|
|
|
|
IMPORTANT: Each pipeline has its own queue.
|
|
|
|
See <<tuning-logstash>> for more info on the effects of adjusting `pipeline.batch.size` and `pipeline.workers`.
|
|
|
|
If you need to absorb bursts of traffic, consider using <<persistent-queues,persistent queues>> instead.
|
|
Persistent queues are bound to allocated capacity on disk.
|
|
|
|
[[mq-settings]]
|
|
===== Settings that affect queue size
|
|
|
|
These values can be configured in `logstash.yml` and `pipelines.yml`.
|
|
|
|
pipeline.batch.size::
|
|
Number events to retrieve from inputs before sending to filters+workers
|
|
The default is 125.
|
|
|
|
pipelines.workers::
|
|
Number of workers that will, in parallel, execute the filters+outputs stage of the pipeline.
|
|
This value defaults to the number of the host's CPU cores.
|
|
|
|
[[backpressure-mem-queue]]
|
|
==== Back pressure
|
|
|
|
When the queue is full, Logstash puts back pressure on the inputs to stall data
|
|
flowing into Logstash.
|
|
This mechanism helps Logstash control the rate of data flow at the input stage
|
|
without overwhelming outputs like Elasticsearch.
|
|
|
|
Each input handles back pressure independently.
|