github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 14:47:19 -04:00
Code Issues Projects Releases Packages Wiki Activity

Incorporate review comments

Browse source 
Fixes #9943
This commit is contained in:
Karen Metts 2018-09-27 12:44:42 -04:00 • committed by karen.metts
parent 9838f43b45
commit e42257f664
3 changed files with 113 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

15
docs/index.asciidoc
View file

@ -209,6 +209,14 @@ include::{plugins-repo-dir}/plugins/outputs.asciidoc[]
include::{plugins-repo-dir}/plugins/filters.asciidoc[]
include::{plugins-repo-dir}/plugins/codecs.asciidoc[]
// FAQ and Troubleshooting
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/faq.asciidoc
include::static/faq.asciidoc[]
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/troubleshooting.asciidoc
include::static/troubleshooting.asciidoc[]
:edit_url:
// Contributing to Logstash
@ -238,13 +246,6 @@ include::static/maintainer-guide.asciidoc[]
:edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/submitting-a-plugin.asciidoc
include::static/submitting-a-plugin.asciidoc[]
// FAQ and Troubleshooting
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/faq.asciidoc
include::static/faq.asciidoc[]
// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/troubleshooting.asciidoc
include::static/troubleshooting.asciidoc[]
// Glossary of Terms

98
docs/static/faq.asciidoc vendored
View file

@ -1,24 +1,73 @@
[[faq]]
== Frequently Asked Questions (FAQ)
[[tips]]
== Tips and Best Practices
We will be adding more questions and answers, so please check back soon.
We will be adding more tips and best practices, so please check back soon.
If you have something you would like to see added, please:
* create an issue at
https://github.com/elastic/logstash/issues, or
* create a pull request with your proposed changes at https://github.com/elastic/logstash.
// After merge, update PR link to link directly to this topic in GH
Also check out the https://discuss.elastic.co/c/logstash[Logstash discussion
forum].
[float]
[[tip-cli]]
=== Command line
[float]
[[tip-windows-cli]]
==== Shell commands on Windows OS
Command line often show single quotes.
On Windows systems, replace a single quote `'' with a double quote `"`.
*Example*
Instead of:
-----
bin/logstash -e 'input { stdin { } } output { stdout {} }'
-----
Use this format on Windows systems:
-----
bin\logstash -e "input { stdin { } } output { stdout {} }"
-----
[float]
[[faq-kafka]]
[[tip-pipelines]]
=== Pipelines
[float]
[[tip-pipeline-mgmt]]
==== Pipeline management
You can manage pipelines in a {ls} instance using either local pipeline configurations or
{logstash-ref}/configuring-centralized-pipelines.html[centralized pipeline management]
in {kib}.
After you configure Logstash to use centralized pipeline management, you can
no longer specify local pipeline configurations. The `pipelines.yml` file and
settings such as `path.config` and `config.string` are inactive when centralized
pipeline management is enabled.
[float]
[[tip-kafka]]
=== Kafka
This section is a summary of the most common Kafka questions from the last few months.
[float]
[[faq-kafka-settings]]
[[tip-kafka-settings]]
==== Kafka settings
[float]
[[faq-kafka-partitions]]
===== How many partitions should be used per topic?
[[tip-kafka-partitions]]
===== Partitions per topic
"How many partitions should I use per topic?"
At least: Number of {ls} nodes multiplied by consumer threads per node.
@ -32,8 +81,10 @@ Err on the side of over-partitioning up to a total 1000
partitions overall. Try not to exceed 1000 partitions.
[float]
[[faq-kafka-threads]]
===== How many consumer threads should I configure?
[[tip-kafka-threads]]
===== Consumer threads
"How many consumer threads should I configure?"
Lower values tend to be more efficient and have less memory overhead. Try a
value of `1` then iterate your way up. The value should in general be lower than
@ -41,24 +92,21 @@ the number of pipeline workers. Values larger than 4 rarely result in a
performance improvement.
[float]
[[faq-kafka-pq-persist]]
[[tip-kafka-pq-persist]]
==== Kafka input and persistent queue (PQ)
[float]
===== Does Kafka Input commit offsets only after the event has been safely persisted to the PQ?
No, we can’t make the guarantee. Offsets are committed to Kafka periodically. If
writes to the PQ are slow/blocked, offsets for events that haven’t safely
[float]
[[tip-kafka-offset-commit]]
===== Kafka offset commits
"Does Kafka Input commit offsets only after the event has been safely persisted to the PQ?"
"Does Kafa Input commit offsets only for events that have passed the pipeline fully?"
No, we can’t make that guarantee. Offsets are committed to Kafka periodically. If
writes to the PQ are slow or blocked, offsets for events that haven’t safely
reached the PQ can be committed.
[float]
[[faq-kafka-offset-commit]]
===== Does Kafa Input commit offsets only for events that have passed the pipeline fully?
No, we can’t make the guarantee. Offsets are committed to Kafka periodically. If
writes to the PQ are slow/blocked, offsets for events that haven’t safely
reached the PQ can be committed.

72
docs/static/troubleshooting.asciidoc vendored
View file

@ -1,7 +1,14 @@
[[troubleshooting]]
== Troubleshooting Common Problems
We will be adding more tips and solutions, so please check back soon.
We will be adding more troubleshooting tips, so please check back soon. If you
have something you would like to see added, please:
* create an issue at
https://github.com/elastic/logstash/issues, or
* create a pull request with your proposed changes at https://github.com/elastic/logstash.
// After merge, update PR link to link directly to this topic in GH
Also check out the https://discuss.elastic.co/c/logstash[Logstash discussion
forum].
@ -21,6 +28,17 @@ in certain plugins (the Netty network library in the TCP input, for example) cop
executable files to the temp directory. This situation causes subsequent failures when
`/tmp` is mounted `noexec`.
*Sample error*
-----
[2018-03-25T12:23:01,149][ERROR][org.logstash.Logstash ]
java.lang.IllegalStateException: org.jruby.exceptions.RaiseException:
(LoadError) Could not load FFI Provider: (NotImplementedError) FFI not
available: java.lang.UnsatisfiedLinkError: /tmp/jffi5534463206038012403.so:
/tmp/jffi5534463206038012403.so: failed to map segment from shared object:
Operation not permitted
-----
*Possible solutions*
* Change setting to mount `/tmp` with `exec`.
@ -36,11 +54,15 @@ executable files to the temp directory. This situation causes subsequent failure
=== Error response code 429
A `429` message indicates that an application is busy handling other requests. For
example, Elasticsearch throws a `429` code to notify Logstash (or other indexers)
that the bulk failed because the ingest queue is full. Any documents that
weren't processed should be retried.
example, Elasticsearch sends a `429` code to notify Logstash (or other indexers)
that the bulk failed because the ingest queue is full. Logstash will retry sending documents.
TBD: Does Logstash retry? Should the user take any action?
*Possible actions*
Check {es} to see if it needs attention.
* {ref}/cluster-stats.html
* {ref}/es-monitoring.html
*Sample error*
@ -56,10 +78,6 @@ pool size = 16, active threads = 16, queued tasks = 200, completed tasks =
-----
[float]
[[ts-performance]]
== General performance tuning
@ -75,16 +93,11 @@ For general performance tuning tips and guidelines, see <<performance-tuning>>.
[float]
[[ts-kafka]]
== Common Kafka support issues and solutions
This section contains a list of common Kafka issues from
the last few months.
[float]
[[ts-kafka-timeout]]
=== Kafka session timeout issues (input side)
This is a common problem.
*Symptoms*
Throughput issues and duplicate event processing {ls} logs warnings:
@ -167,7 +180,7 @@ evidence of stalling outputs, such as `ES output logging status 429`.
[float]
[[ts-kafka-many-offset-commits]]
=== Large number of offset commits (input side)
=== Large number of offset commits (Kafka input side)
*Symptoms*
@ -231,30 +244,9 @@ https://github.com/logstash-plugins/logstash-input-kafka/issues/210
[[ts-other]]
== Other issues
[float]
[[ts-cli]]
=== Command line
[float]
[[ts-windows-cli]]
==== Shell commands on Windows OS
Command line often show single quotes.
On Windows systems, replace a single quote `'' with a double quote `"`.
*Example*
Instead of:
-----
bin/logstash -e 'input { stdin { } } output { stdout {} }'
-----
Use this format on Windows systems:
-----
bin/logstash -e "input { stdin { } } output { stdout {} }"
-----
Coming soon