diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 4acc54bc4..3d800bc30 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -209,6 +209,14 @@ include::{plugins-repo-dir}/plugins/outputs.asciidoc[] include::{plugins-repo-dir}/plugins/filters.asciidoc[] include::{plugins-repo-dir}/plugins/codecs.asciidoc[] +// FAQ and Troubleshooting + +// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/faq.asciidoc +include::static/faq.asciidoc[] + +// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/troubleshooting.asciidoc +include::static/troubleshooting.asciidoc[] + :edit_url: // Contributing to Logstash @@ -238,13 +246,6 @@ include::static/maintainer-guide.asciidoc[] :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/submitting-a-plugin.asciidoc include::static/submitting-a-plugin.asciidoc[] -// FAQ and Troubleshooting - -// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/faq.asciidoc -include::static/faq.asciidoc[] - -// :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/troubleshooting.asciidoc -include::static/troubleshooting.asciidoc[] // Glossary of Terms diff --git a/docs/static/faq.asciidoc b/docs/static/faq.asciidoc index 599b00ae5..dd97e5103 100644 --- a/docs/static/faq.asciidoc +++ b/docs/static/faq.asciidoc @@ -1,24 +1,73 @@ -[[faq]] -== Frequently Asked Questions (FAQ) +[[tips]] +== Tips and Best Practices -We will be adding more questions and answers, so please check back soon. +We will be adding more tips and best practices, so please check back soon. +If you have something you would like to see added, please: + +* create an issue at +https://github.com/elastic/logstash/issues, or +* create a pull request with your proposed changes at https://github.com/elastic/logstash. + +// After merge, update PR link to link directly to this topic in GH Also check out the https://discuss.elastic.co/c/logstash[Logstash discussion forum]. +[float] +[[tip-cli]] +=== Command line + +[float] +[[tip-windows-cli]] +==== Shell commands on Windows OS + +Command line often show single quotes. +On Windows systems, replace a single quote `'' with a double quote `"`. + +*Example* + +Instead of: + +----- +bin/logstash -e 'input { stdin { } } output { stdout {} }' +----- + +Use this format on Windows systems: + +----- +bin\logstash -e "input { stdin { } } output { stdout {} }" +----- + [float] -[[faq-kafka]] +[[tip-pipelines]] +=== Pipelines + +[float] +[[tip-pipeline-mgmt]] +==== Pipeline management + +You can manage pipelines in a {ls} instance using either local pipeline configurations or +{logstash-ref}/configuring-centralized-pipelines.html[centralized pipeline management] +in {kib}. + +After you configure Logstash to use centralized pipeline management, you can +no longer specify local pipeline configurations. The `pipelines.yml` file and +settings such as `path.config` and `config.string` are inactive when centralized +pipeline management is enabled. + +[float] +[[tip-kafka]] === Kafka -This section is a summary of the most common Kafka questions from the last few months. - [float] -[[faq-kafka-settings]] +[[tip-kafka-settings]] ==== Kafka settings [float] -[[faq-kafka-partitions]] -===== How many partitions should be used per topic? +[[tip-kafka-partitions]] +===== Partitions per topic + +"How many partitions should I use per topic?" At least: Number of {ls} nodes multiplied by consumer threads per node. @@ -32,8 +81,10 @@ Err on the side of over-partitioning up to a total 1000 partitions overall. Try not to exceed 1000 partitions. [float] -[[faq-kafka-threads]] -===== How many consumer threads should I configure? +[[tip-kafka-threads]] +===== Consumer threads + +"How many consumer threads should I configure?" Lower values tend to be more efficient and have less memory overhead. Try a value of `1` then iterate your way up. The value should in general be lower than @@ -41,24 +92,21 @@ the number of pipeline workers. Values larger than 4 rarely result in a performance improvement. [float] -[[faq-kafka-pq-persist]] +[[tip-kafka-pq-persist]] ==== Kafka input and persistent queue (PQ) -[float] -===== Does Kafka Input commit offsets only after the event has been safely persisted to the PQ? -No, we can’t make the guarantee. Offsets are committed to Kafka periodically. If -writes to the PQ are slow/blocked, offsets for events that haven’t safely +[float] +[[tip-kafka-offset-commit]] +===== Kafka offset commits + +"Does Kafka Input commit offsets only after the event has been safely persisted to the PQ?" + +"Does Kafa Input commit offsets only for events that have passed the pipeline fully?" + +No, we can’t make that guarantee. Offsets are committed to Kafka periodically. If +writes to the PQ are slow or blocked, offsets for events that haven’t safely reached the PQ can be committed. -[float] -[[faq-kafka-offset-commit]] -===== Does Kafa Input commit offsets only for events that have passed the pipeline fully? -No, we can’t make the guarantee. Offsets are committed to Kafka periodically. If -writes to the PQ are slow/blocked, offsets for events that haven’t safely -reached the PQ can be committed. - - - diff --git a/docs/static/troubleshooting.asciidoc b/docs/static/troubleshooting.asciidoc index 616915c27..2416122e3 100644 --- a/docs/static/troubleshooting.asciidoc +++ b/docs/static/troubleshooting.asciidoc @@ -1,7 +1,14 @@ [[troubleshooting]] == Troubleshooting Common Problems -We will be adding more tips and solutions, so please check back soon. +We will be adding more troubleshooting tips, so please check back soon. If you +have something you would like to see added, please: + +* create an issue at +https://github.com/elastic/logstash/issues, or +* create a pull request with your proposed changes at https://github.com/elastic/logstash. + +// After merge, update PR link to link directly to this topic in GH Also check out the https://discuss.elastic.co/c/logstash[Logstash discussion forum]. @@ -21,6 +28,17 @@ in certain plugins (the Netty network library in the TCP input, for example) cop executable files to the temp directory. This situation causes subsequent failures when `/tmp` is mounted `noexec`. +*Sample error* + +----- +[2018-03-25T12:23:01,149][ERROR][org.logstash.Logstash ] +java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: +(LoadError) Could not load FFI Provider: (NotImplementedError) FFI not +available: java.lang.UnsatisfiedLinkError: /tmp/jffi5534463206038012403.so: +/tmp/jffi5534463206038012403.so: failed to map segment from shared object: +Operation not permitted +----- + *Possible solutions* * Change setting to mount `/tmp` with `exec`. @@ -36,11 +54,15 @@ executable files to the temp directory. This situation causes subsequent failure === Error response code 429 A `429` message indicates that an application is busy handling other requests. For -example, Elasticsearch throws a `429` code to notify Logstash (or other indexers) -that the bulk failed because the ingest queue is full. Any documents that -weren't processed should be retried. +example, Elasticsearch sends a `429` code to notify Logstash (or other indexers) +that the bulk failed because the ingest queue is full. Logstash will retry sending documents. -TBD: Does Logstash retry? Should the user take any action? +*Possible actions* + +Check {es} to see if it needs attention. + +* {ref}/cluster-stats.html +* {ref}/es-monitoring.html *Sample error* @@ -56,10 +78,6 @@ pool size = 16, active threads = 16, queued tasks = 200, completed tasks = ----- - - - - [float] [[ts-performance]] == General performance tuning @@ -75,16 +93,11 @@ For general performance tuning tips and guidelines, see <>. [float] [[ts-kafka]] == Common Kafka support issues and solutions - -This section contains a list of common Kafka issues from -the last few months. - + [float] [[ts-kafka-timeout]] === Kafka session timeout issues (input side) -This is a common problem. - *Symptoms* Throughput issues and duplicate event processing {ls} logs warnings: @@ -167,7 +180,7 @@ evidence of stalling outputs, such as `ES output logging status 429`. [float] [[ts-kafka-many-offset-commits]] -=== Large number of offset commits (input side) +=== Large number of offset commits (Kafka input side) *Symptoms* @@ -231,30 +244,9 @@ https://github.com/logstash-plugins/logstash-input-kafka/issues/210 [[ts-other]] == Other issues -[float] -[[ts-cli]] -=== Command line - -[float] -[[ts-windows-cli]] -==== Shell commands on Windows OS - -Command line often show single quotes. -On Windows systems, replace a single quote `'' with a double quote `"`. - -*Example* - -Instead of: - ------ -bin/logstash -e 'input { stdin { } } output { stdout {} }' ------ - -Use this format on Windows systems: - ------ -bin/logstash -e "input { stdin { } } output { stdout {} }" ------ +Coming soon + +