Adds information on shutdown stalls, release notes, other housekeeping

docs/asciidoc/static/advanced-pipeline.asciidoc

@@ -545,3 +545,67 @@ Run an Elasticsearch query to find the same information in the Elasticsearch clu
 --------------------------------------------------------------------------------
 curl -XGET 'localhost:9200/logstash-2015.07.30/_search?q=agent=Mozilla'
 --------------------------------------------------------------------------------
+
+[[stalled-shutdown]]
+=== Stalled Shutdown Detection
+
+Shutting down a running Logstash instance involves the following steps:
+
+* Stop all input, filter and output plugins
+* Process all in-flight events
+* Terminate the Logstash process
+
+The following conditions affect the shutdown process:
+
+* An input plugin receiving data at a slow pace.
+* A slow filter, like a Ruby filter executing `sleep(10000)` or an Elasticsearch filter that is executing a very heavy 
+query.
+* A disconnected output plugin that is waiting to reconnect to flush in-flight events.
+
+These situations make the duration and success of the shutdown process unpredictable.
+
+Logstash has a stall detection mechanism that analyzes the behavior of the pipeline and plugins during shutdown.
+This mechanism produces periodic information about the count of inflight events in internal queues and a list of busy 
+worker threads.
+
+To enable Logstash to forcibly terminate in the case of a stalled shutdown, use the `--allow-unsafe-shutdown` flag when 
+you start Logstash.
+
+[[shutdown-stall-example]]
+==== Stall Detection Example
+
+In this example, slow filter execution prevents the pipeline from clean shutdown. By starting Logstash with the
+`--allow-unsafe-shutdown` flag, quitting with *Ctrl+C* results in an eventual shutdown that loses 20 events.
+
+========
+[source,shell]
+% bin/logstash -e 'input { generator { } } filter { ruby { code => "sleep 10000" } } \
+                     output { stdout { codec => dots } }' -w 1 --allow-unsafe-shutdown
+Default settings used: Filter workers: 1
+Logstash startup completed
+^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
+Received shutdown signal, but pipeline is still waiting for in-flight events
+to be processed. Sending another ^C will force quit Logstash, but this may cause
+data loss. {:level=>:warn}
+ {:level=>:warn, "INFLIGHT_EVENT_COUNT"=>{"input_to_filter"=>20, "total"=>20}, 
+ "STALLING_THREADS"=>
+ {["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>15, 
+ "name"=>"|filterworker.0", "current_call"=>"
+ (ruby filter code):1:in `sleep'"}]}}
+The shutdown process appears to be stalled due to busy or blocked plugins. Check 
+    the logs for more information. 
+{:level=>:error}
+ {:level=>:warn, "INFLIGHT_EVENT_COUNT"=>{"input_to_filter"=>20, "total"=>20}, 
+ "STALLING_THREADS"=>
+ {["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>15, 
+ "name"=>"|filterworker.0", "current_call"=>"
+ (ruby filter code):1:in `sleep'"}]}}
+ {:level=>:warn, "INFLIGHT_EVENT_COUNT"=>{"input_to_filter"=>20, "total"=>20}, 
+ "STALLING_THREADS"=>
+ {["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>15, 
+ "name"=>"|filterworker.0", "current_call"=>"
+ (ruby filter code):1:in `sleep'"}]}}
+Forcefully quitting logstash.. {:level=>:fatal}
+========
+
+When `--allow-unsafe-shutdown` isn't enabled, Logstash continues to run and produce these reports periodically.

docs/asciidoc/static/breaking-changes.asciidoc

@@ -5,7 +5,7 @@ Version 2.0 of Logstash has some changes that are incompatible with previous ver
 what you need to be aware of when migrating to this version.
 
 [float]
-=== Elasticsearch Output Default
+== Elasticsearch Output Default
 
 Starting with the 2.0 release of Logstash, the default Logstash output for Elasticsearch is HTTP. To use the `node` or
 `transport` protocols, download the https://www.elastic.co/guide/en/logstash/2.0/plugins-outputs-elasticsearch_java.html[Elasticsearch Java plugin]. The 
@@ -19,7 +19,7 @@ Be sure to specify the correct value for the `--version` option during installat
 `bin/plugin install --version 2.0.0 logstash-output-elasticsearch_java`
 
 [float]
-==== Configuration Changes
+=== Configuration Changes
 
 The Elasticsearch output plugin configuration has the following changes:
 
@@ -49,7 +49,9 @@ and update your configuration files accordingly.
 
 [float]
 === Metrics Filter Changes
-Prior implementations of the metrics filter plugin used dotted field names. Elasticsearch does not allow field names to have dots, beginning with version 2.0, so a change was made to use sub-fields instead of dots in this plugin. Please note that these changes make version 3.0.0 of the metrics filter plugin incompatible with previous releases.
+Prior implementations of the metrics filter plugin used dotted field names. Elasticsearch does not allow field names to 
+have dots, beginning with version 2.0, so a change was made to use sub-fields instead of dots in this plugin. Please note 
+that these changes make version 3.0.0 of the metrics filter plugin incompatible with previous releases.
 
 
 [float]

docs/asciidoc/static/contributing-to-logstash.asciidoc

@@ -35,6 +35,7 @@ Plugin Base class.
 
 Sample code for the new plugin shutdown APIs is https://github.com/logstash-plugins/logstash-input-example/blob/master/lib/logstash/inputs/example.rb[available].
 
+
 [float]
 === Extending Logstash core
 

docs/asciidoc/static/howtos-and-tutorials.asciidoc

@@ -1,16 +0,0 @@
-[[howtos-and-tutorials]]
-== Logstash HOWTOs and Tutorials
-Pretty self-explanatory, really
-
-=== Downloads and Releases
-* http://elasticsearch.org/#[Getting Started with Logstash]
-* http://elasticsearch.org/#[Configuration file overview]
-* http://elasticsearch.org/#[Command-line flags]
-* http://elasticsearch.org/#[The life of an event in Logstash]
-* http://elasticsearch.org/#[Using conditional logic]
-* http://elasticsearch.org/#[Glossary]
-* http://elasticsearch.org/#[referring to fields `[like][this]`]
-* http://elasticsearch.org/#[using the `%{fieldname}` syntax]
-* http://elasticsearch.org/#[Metrics from Logs]
-* http://elasticsearch.org/#[Using RabbitMQ]
-* http://elasticsearch.org/#[Contributing to Logstash]

docs/asciidoc/static/offline-plugins.asciidoc

@@ -10,6 +10,10 @@ server. This staging machine downloads and packages the files used for offline i
 See the <<private-rubygem,Private Gem Repositories>> section for information on setting up your own private 
 Rubygems server.
 
+Users who can work with a larger Logstash artifact size can use the *Logstash (All Plugins)* download link from the
+https://www.elastic.co/downloads/logstash[Logstash product page] to download Logstash bundled with the latest version of
+all available plugins. You can distribute this bundle to all nodes without further plugin staging.
+
 [float]
 === Building the Offline Package
 

docs/asciidoc/static/releasenotes.asciidoc

@@ -0,0 +1,61 @@
+[[releasenotes]]
+== Logstash 2.1 Release Notes
+
+[float]
+== General
+
+* {lsissue}2376[Issue 2376]: Added ability to install and upgrade Logstash plugins without requiring internet 
+connectivity. 
+* {lsissue}3576[Issue 3576]: Support alternate or private Ruby gems server to install and update plugins.
+* {lsissue}3451[Issue 3451]: Added ability to reliably shutdown Logstash when there is a stall in event processing. This 
+option can be enabled by passing `--allow-unsafe-shutdown` flag while starting Logstash. Please be aware that any in-
+flight events will be lost when shutdown happens.
+* {lsissue}4222[Issue 4222]: Fixed a memory leak which could be triggered when events having a date were serialized to 
+string.
+* Added JDBC input to default package.
+* {lsissue}3243[Issue 3243]: Adding `--debug` to `--configtest` now shows the configuration in blocks annotated by source 
+config file. Very useful when using multiple config files in a directory.
+* {lsissue}4130[Issue 4130]: Reset default worker threads to 1 when using non thread-safe filters like multiline.
+* Fixed file permissions for the `logrotate` configuration file.
+* {lsissue}3861[Issue 3861]: Changed the default heap size from 500MB to 1GB.
+* {lsissue}3645[Issue 3645]: Fixed config check option when starting Logstash through init scripts.
+
+[float]
+== Input Plugins
+
+[float]
+=== Twitter
+* https://github.com/logstash-plugins/logstash-input-twitter/issues/21[Issue 21]: Added an option to fetch data from the 
+sample Twitter streaming endpoint.
+* https://github.com/logstash-plugins/logstash-input-twitter/issues/22[Issue 22]: Added hashtags, symbols and 
+user_mentions as data for the non extended tweet event.
+* https://github.com/logstash-plugins/logstash-input-twitter/issues/20[Issue 20]: Added an option to filter per location 
+and language.
+* https://github.com/logstash-plugins/logstash-input-twitter/issues/11[Issue 11]: Added an option to stream data from a 
+list of users.
+
+[float]
+=== Beats
+* https://github.com/logstash-plugins/logstash-input-beats/issues/10[Issue 10]: Properly handle multiline events from 
+multiple sources, originating from Filebeat.
+
+[float]
+=== File
+* https://github.com/logstash-plugins/logstash-input-file/issues/44[Issue 44]: Properly handle multiline events from 
+multiple sources.
+
+[float]
+=== Eventlog
+* https://github.com/logstash-plugins/logstash-input-eventlog/issues/11[Issue 11]: Change the underlying library to 
+capture Event Logs from Windows more reliably. 
+
+[float]
+== Output
+
+[float]
+=== Elasticsearch
+* Improved the default template to use doc_values wherever possible.
+* Improved the default template to disable fielddata on analyzed string fields.
+* https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/260[Issue 260]: Added New setting: timeout. 
+This lets you control the behavior of a slow/stuck request to Elasticsearch that could be, for example, caused by network, 
+firewall, or load balancer issues.