mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
Karen Metts
karen.metts
parent
9838f43b45
commit
e42257f664
3 changed files with 113 additions and 72 deletions
|
|
@ -209,6 +209,14 @@ include::{plugins-repo-dir}/plugins/outputs.asciidoc[]
|
|||
include::{plugins-repo-dir}/plugins/filters.asciidoc[]
|
||||
include::{plugins-repo-dir}/plugins/codecs.asciidoc[]
|
||||
|
||||
// FAQ and Troubleshooting
|
||||
|
||||
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/faq.asciidoc
|
||||
include::static/faq.asciidoc[]
|
||||
|
||||
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/troubleshooting.asciidoc
|
||||
include::static/troubleshooting.asciidoc[]
|
||||
|
||||
:edit_url:
|
||||
|
||||
// Contributing to Logstash
|
||||
|
|
@ -238,13 +246,6 @@ include::static/maintainer-guide.asciidoc[]
|
|||
:edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/submitting-a-plugin.asciidoc
|
||||
include::static/submitting-a-plugin.asciidoc[]
|
||||
|
||||
// FAQ and Troubleshooting
|
||||
|
||||
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/faq.asciidoc
|
||||
include::static/faq.asciidoc[]
|
||||
|
||||
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/troubleshooting.asciidoc
|
||||
include::static/troubleshooting.asciidoc[]
|
||||
|
||||
// Glossary of Terms
|
||||
|
||||
|
|
|
|||
98
docs/static/faq.asciidoc
vendored
98
docs/static/faq.asciidoc
vendored
|
|
@ -1,24 +1,73 @@
|
|||
[[faq]]
|
||||
== Frequently Asked Questions (FAQ)
|
||||
[[tips]]
|
||||
== Tips and Best Practices
|
||||
|
||||
We will be adding more questions and answers, so please check back soon.
|
||||
We will be adding more tips and best practices, so please check back soon.
|
||||
If you have something you would like to see added, please:
|
||||
|
||||
* create an issue at
|
||||
https://github.com/elastic/logstash/issues, or
|
||||
* create a pull request with your proposed changes at https://github.com/elastic/logstash.
|
||||
|
||||
// After merge, update PR link to link directly to this topic in GH
|
||||
|
||||
Also check out the https://discuss.elastic.co/c/logstash[Logstash discussion
|
||||
forum].
|
||||
|
||||
[float]
|
||||
[[faq-kafka]]
|
||||
=== Kafka
|
||||
|
||||
This section is a summary of the most common Kafka questions from the last few months.
|
||||
[[tip-cli]]
|
||||
=== Command line
|
||||
|
||||
[float]
|
||||
[[faq-kafka-settings]]
|
||||
[[tip-windows-cli]]
|
||||
==== Shell commands on Windows OS
|
||||
|
||||
Command line often show single quotes.
|
||||
On Windows systems, replace a single quote `'' with a double quote `"`.
|
||||
|
||||
*Example*
|
||||
|
||||
Instead of:
|
||||
|
||||
-----
|
||||
bin/logstash -e 'input { stdin { } } output { stdout {} }'
|
||||
-----
|
||||
|
||||
Use this format on Windows systems:
|
||||
|
||||
-----
|
||||
bin\logstash -e "input { stdin { } } output { stdout {} }"
|
||||
-----
|
||||
|
||||
[float]
|
||||
[[tip-pipelines]]
|
||||
=== Pipelines
|
||||
|
||||
[float]
|
||||
[[tip-pipeline-mgmt]]
|
||||
==== Pipeline management
|
||||
|
||||
You can manage pipelines in a {ls} instance using either local pipeline configurations or
|
||||
{logstash-ref}/configuring-centralized-pipelines.html[centralized pipeline management]
|
||||
in {kib}.
|
||||
|
||||
After you configure Logstash to use centralized pipeline management, you can
|
||||
no longer specify local pipeline configurations. The `pipelines.yml` file and
|
||||
settings such as `path.config` and `config.string` are inactive when centralized
|
||||
pipeline management is enabled.
|
||||
|
||||
[float]
|
||||
[[tip-kafka]]
|
||||
=== Kafka
|
||||
|
||||
[float]
|
||||
[[tip-kafka-settings]]
|
||||
==== Kafka settings
|
||||
|
||||
[float]
|
||||
[[faq-kafka-partitions]]
|
||||
===== How many partitions should be used per topic?
|
||||
[[tip-kafka-partitions]]
|
||||
===== Partitions per topic
|
||||
|
||||
"How many partitions should I use per topic?"
|
||||
|
||||
At least: Number of {ls} nodes multiplied by consumer threads per node.
|
||||
|
||||
|
|
@ -32,8 +81,10 @@ Err on the side of over-partitioning up to a total 1000
|
|||
partitions overall. Try not to exceed 1000 partitions.
|
||||
|
||||
[float]
|
||||
[[faq-kafka-threads]]
|
||||
===== How many consumer threads should I configure?
|
||||
[[tip-kafka-threads]]
|
||||
===== Consumer threads
|
||||
|
||||
"How many consumer threads should I configure?"
|
||||
|
||||
Lower values tend to be more efficient and have less memory overhead. Try a
|
||||
value of `1` then iterate your way up. The value should in general be lower than
|
||||
|
|
@ -41,24 +92,21 @@ the number of pipeline workers. Values larger than 4 rarely result in a
|
|||
performance improvement.
|
||||
|
||||
[float]
|
||||
[[faq-kafka-pq-persist]]
|
||||
[[tip-kafka-pq-persist]]
|
||||
==== Kafka input and persistent queue (PQ)
|
||||
[float]
|
||||
===== Does Kafka Input commit offsets only after the event has been safely persisted to the PQ?
|
||||
|
||||
No, we can’t make the guarantee. Offsets are committed to Kafka periodically. If
|
||||
writes to the PQ are slow/blocked, offsets for events that haven’t safely
|
||||
reached the PQ can be committed.
|
||||
|
||||
|
||||
[float]
|
||||
[[faq-kafka-offset-commit]]
|
||||
===== Does Kafa Input commit offsets only for events that have passed the pipeline fully?
|
||||
No, we can’t make the guarantee. Offsets are committed to Kafka periodically. If
|
||||
writes to the PQ are slow/blocked, offsets for events that haven’t safely
|
||||
[[tip-kafka-offset-commit]]
|
||||
===== Kafka offset commits
|
||||
|
||||
"Does Kafka Input commit offsets only after the event has been safely persisted to the PQ?"
|
||||
|
||||
"Does Kafa Input commit offsets only for events that have passed the pipeline fully?"
|
||||
|
||||
No, we can’t make that guarantee. Offsets are committed to Kafka periodically. If
|
||||
writes to the PQ are slow or blocked, offsets for events that haven’t safely
|
||||
reached the PQ can be committed.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
70
docs/static/troubleshooting.asciidoc
vendored
70
docs/static/troubleshooting.asciidoc
vendored
|
|
@ -1,7 +1,14 @@
|
|||
[[troubleshooting]]
|
||||
== Troubleshooting Common Problems
|
||||
|
||||
We will be adding more tips and solutions, so please check back soon.
|
||||
We will be adding more troubleshooting tips, so please check back soon. If you
|
||||
have something you would like to see added, please:
|
||||
|
||||
* create an issue at
|
||||
https://github.com/elastic/logstash/issues, or
|
||||
* create a pull request with your proposed changes at https://github.com/elastic/logstash.
|
||||
|
||||
// After merge, update PR link to link directly to this topic in GH
|
||||
|
||||
Also check out the https://discuss.elastic.co/c/logstash[Logstash discussion
|
||||
forum].
|
||||
|
|
@ -21,6 +28,17 @@ in certain plugins (the Netty network library in the TCP input, for example) cop
|
|||
executable files to the temp directory. This situation causes subsequent failures when
|
||||
`/tmp` is mounted `noexec`.
|
||||
|
||||
*Sample error*
|
||||
|
||||
-----
|
||||
[2018-03-25T12:23:01,149][ERROR][org.logstash.Logstash ]
|
||||
java.lang.IllegalStateException: org.jruby.exceptions.RaiseException:
|
||||
(LoadError) Could not load FFI Provider: (NotImplementedError) FFI not
|
||||
available: java.lang.UnsatisfiedLinkError: /tmp/jffi5534463206038012403.so:
|
||||
/tmp/jffi5534463206038012403.so: failed to map segment from shared object:
|
||||
Operation not permitted
|
||||
-----
|
||||
|
||||
*Possible solutions*
|
||||
|
||||
* Change setting to mount `/tmp` with `exec`.
|
||||
|
|
@ -36,11 +54,15 @@ executable files to the temp directory. This situation causes subsequent failure
|
|||
=== Error response code 429
|
||||
|
||||
A `429` message indicates that an application is busy handling other requests. For
|
||||
example, Elasticsearch throws a `429` code to notify Logstash (or other indexers)
|
||||
that the bulk failed because the ingest queue is full. Any documents that
|
||||
weren't processed should be retried.
|
||||
example, Elasticsearch sends a `429` code to notify Logstash (or other indexers)
|
||||
that the bulk failed because the ingest queue is full. Logstash will retry sending documents.
|
||||
|
||||
TBD: Does Logstash retry? Should the user take any action?
|
||||
*Possible actions*
|
||||
|
||||
Check {es} to see if it needs attention.
|
||||
|
||||
* {ref}/cluster-stats.html
|
||||
* {ref}/es-monitoring.html
|
||||
|
||||
*Sample error*
|
||||
|
||||
|
|
@ -56,10 +78,6 @@ pool size = 16, active threads = 16, queued tasks = 200, completed tasks =
|
|||
-----
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
[float]
|
||||
[[ts-performance]]
|
||||
== General performance tuning
|
||||
|
|
@ -76,15 +94,10 @@ For general performance tuning tips and guidelines, see <<performance-tuning>>.
|
|||
[[ts-kafka]]
|
||||
== Common Kafka support issues and solutions
|
||||
|
||||
This section contains a list of common Kafka issues from
|
||||
the last few months.
|
||||
|
||||
[float]
|
||||
[[ts-kafka-timeout]]
|
||||
=== Kafka session timeout issues (input side)
|
||||
|
||||
This is a common problem.
|
||||
|
||||
*Symptoms*
|
||||
|
||||
Throughput issues and duplicate event processing {ls} logs warnings:
|
||||
|
|
@ -167,7 +180,7 @@ evidence of stalling outputs, such as `ES output logging status 429`.
|
|||
|
||||
[float]
|
||||
[[ts-kafka-many-offset-commits]]
|
||||
=== Large number of offset commits (input side)
|
||||
=== Large number of offset commits (Kafka input side)
|
||||
|
||||
*Symptoms*
|
||||
|
||||
|
|
@ -231,30 +244,9 @@ https://github.com/logstash-plugins/logstash-input-kafka/issues/210
|
|||
[[ts-other]]
|
||||
== Other issues
|
||||
|
||||
[float]
|
||||
[[ts-cli]]
|
||||
=== Command line
|
||||
|
||||
[float]
|
||||
[[ts-windows-cli]]
|
||||
==== Shell commands on Windows OS
|
||||
|
||||
Command line often show single quotes.
|
||||
On Windows systems, replace a single quote `'' with a double quote `"`.
|
||||
|
||||
*Example*
|
||||
|
||||
Instead of:
|
||||
|
||||
-----
|
||||
bin/logstash -e 'input { stdin { } } output { stdout {} }'
|
||||
-----
|
||||
|
||||
Use this format on Windows systems:
|
||||
|
||||
-----
|
||||
bin/logstash -e "input { stdin { } } output { stdout {} }"
|
||||
-----
|
||||
Coming soon
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue