logstash/docs/static/upgrading.asciidoc

[[upgrading-logstash]]
== Upgrading Logstash

[IMPORTANT]
===========================================
Before upgrading Logstash:

* Consult the <<breaking-changes,breaking changes>> docs.
* Test upgrades in a development environment before upgrading your production cluster.
===========================================

=== Upgrading Using Package Managers

This procedure uses <<package-repositories,package managers>> to upgrade Logstash.

1. Shut down your Logstash pipeline, including any inputs that send events to Logstash.
2. Using the directions in the _Package Repositories_ section, update your repository links to point to the 2.0 repositories
instead of the previous version.
3. Run the `apt-get upgrade logstash` or `yum update logstash` command as appropriate for your operating system.
4. Test your configuration file with the `logstash --configtest -f <configuration-file>` command. Configuration options for
some Logstash plugins have changed in the 2.0 release.
5. Restart your Logstash pipeline after updating your configuration file.

=== Upgrading Using a Direct Download

This procedure downloads the relevant Logstash binaries directly from Elastic.

1. Shut down your Logstash pipeline, including any inputs that send events to Logstash.
2. Download the https://www.elastic.co/downloads/logstash[Logstash installation file] that matches your host environment.
3. Unpack the installation file into your Logstash directory.
4. Test your configuration file with the `logstash --configtest -f <configuration-file>` command. Configuration options for
some Logstash plugins have changed in the 2.0 release.
5. Restart your Logstash pipeline after updating your configuration file.

=== Upgrading Logstash and Elasticsearch to 2.0

If you are using Elasticsearch as an output, and wish to upgrade to Elasticsearch 2.0, please be
aware of https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking-changes-2.0.html[breaking changes]
before you upgrade. In addition, the following steps needs to be performed after upgrading to Elasticsearch 2.0:

**Mapping changes:** Users may have custom template changes, so by default a Logstash upgrade will
leave the template as is. Even if you don't have a custom template, Logstash will not overwrite an existing
template by default.

There is one known issue (removal of https://www.elastic.co/guide/en/elasticsearch/reference/1.4/mapping-object-type.html#_path_3[path]) with using GeoIP filter that needs a manual update to the template.

Note: If you have custom template changes, please make sure to save it and merge any changes. You can
get the existing template by running:

[source,shell]
curl -XGET localhost:9200/_template/logstash


Add the following option to your Logstash config:

[source,json]
output {
	elasticsearch {
		template_overwrite => true
	}
}

Restart Logstash.

**Dots in fields:** Elasticsearch 2.0 does not allow field names to contain the `.` character.
Further details about this change https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots[here]. Some plugins already have been updated to compensate
for this breaking change, including logstash-filter-metrics and logstash-filter-elapsed.
These plugin updates are available for Logstash 2.0. To upgrade to the latest version of these
plugins, the command is:

[source,shell]
bin/plugin update <plugin_name>

**Multiline Filter:** If you are using the Multiline Filter in your configuration and upgrade to Logstash 2.0,
you will get an error. Make sure to explicitly set the number of filter workers (`-w`) to `1`. You can set the number
of workers by passing a command line flag such as:

[source,shell]
bin/logstash `-w 1`

[[upgrading-logstash-2.2]]
=== Upgrading Logstash to 2.2

Logstash 2.2 re-architected the pipeline stages to provide more performance and help future enhancements in resiliency.
The new pipeline introduced micro-batches, processing groups of events at a time. The default batch size is
125 per worker. Also, the filter and output stages are executed in the same thread, but still, as different stages.
The CLI flag `--pipeline-workers` or `-w` control the number of execution threads, which is set by default to number of cores.

**Considerations for Elasticsearch Output**
The default batch size of the pipeline is 125 events per worker. This will by default also be the bulk size
used for the elasticsearch output. The Elasticsearch output's `flush_size` now acts only as a maximum bulk
size (still defaulting to 500). For example, if your pipeline batch size is 3000 events, Elasticsearch
Output will send 500 events at a time, in 6 separate bulk requests. In other words, for Elasticsearch output,
bulk request size is chunked based on `flush_size` and `--pipeline-batch-size`. If `flush_size` is set greater
than `--pipeline-batch-size`, it is ignored and `--pipeline-batch-size` will be used.

The default number of output workers in Logstash 2.2 is now equal to the number of pipeline workers (`-w`)
unless overridden in the Logstash config file. This can be problematic for some users as the
extra workers may consume extra resources like file handles, especially in the case of the Elasticsearch
output. Users with more than one Elasticsearch host may want to override the `workers` setting
for the Elasticsearch output in their Logstash config to constrain that number to a low value, between 1 to 4.

**Performance Tuning in 2.2**
Since both filters and output workers are on the same thread, this could lead to threads being idle in I/O wait state.
Thus, in 2.2, you can safely set `-w` to a number which is a multiple of the number of cores on your machine.
A common way to tune performance is keep increasing the `-w` beyond the # of cores until performance no longer
improves. A note of caution - make sure you also keep heapsize in mind, because the number of in-flight events
are `#workers * batch_size * average_event size`. More in-flight events could add to memory pressure, eventually
leading to Out of Memory errors. You can change the heapsize in Logstash by setting `LS_HEAP_SIZE`