github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 14:47:19 -04:00
Code Issues Projects Releases Packages Wiki Activity

Doc: Expand PQ content for pipeline-pipeline (#13422)

Browse source 
Backports #13319 to 7.16
This commit is contained in:
Karen Metts 2021-11-18 20:57:42 -05:00 • committed by GitHub
parent 003dbb0c8b
commit 1f9f78274b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 26 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

22
docs/static/persistent-queues.asciidoc vendored
View file

@ -107,13 +107,33 @@ In a situation where Logstash is terminated or there is a hardware-level
failure, any data that has not been checkpointed, is lost.
See <<durability-persistent-queues>> to better understand the trade-offs.
[[pq-pline-pline]]
===== PQs and pipeline-to-pipeline communication
Persistent queues can play an important role in your <<pipeline-to-pipeline,pipeline-to-pipeline>> configuration.
[[uc-isolator]]
====== Use case: PQs and output isolator pattern
Here is a real world use case described by a Logstash user.
"_In our deployment, we use one pipeline per output, and each pipeline has a
large PQ. This configuration allows a single output to stall without blocking
the input (and thus all other outputs), until the operator can restore flow to
the stalled output and let the queue drain._"
"_Our real-time outputs must be low-latency, and our bulk outputs must be
consistent. We use PQs to protect against stalling the real-time outputs more so than to protect against data loss in the bulk outputs. (Although the protection is nice, too)._"
[[troubleshooting-pqs]]
==== Troubleshooting persistent queues
Symptoms of persistent queue problems include {ls} or one or more pipelines not starting successfully, accompanied by an error message similar to this one.
```
message=>"java.io.IOException: Page file size is too small to hold elements
message=>"java.io.IOException: Page file size is too small to hold elements"
```
This error indicates that the head page (the oldest in a directory and the one with lowest page id) has a size < 18 bytes, the size of a page header.

6
docs/static/pipeline-pipeline-config.asciidoc vendored
View file

@ -1,10 +1,14 @@
[[pipeline-to-pipeline]]
=== Pipeline-to-Pipeline Communication
=== Pipeline-to-pipeline communication
When using the multiple pipeline feature of Logstash, you may want to connect multiple pipelines within the same Logstash instance. This configuration can be useful to isolate the execution of these pipelines, as well as to help break-up the logic of complex pipelines. The `pipeline` input/output enables a number of advanced architectural patterns discussed later in this document.
If you need to set up communication _between_ Logstash instances, use either {logstash-ref}/ls-to-ls.html[Logstash-to-Logstash] communications, or an intermediary queue, such as Kafka or Redis.
TIP: Persistent queues (PQs) can help keep data moving through pipelines.
See <<pq-pline-pline>> to learn how PQs can enhance your
pipeline-to-pipeline communication strategy.
[[pipeline-to-pipeline-overview]]
==== Configuration overview