mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 22:57:16 -04:00
309 lines
14 KiB
Text
309 lines
14 KiB
Text
[[breaking-changes]]
|
||
== Breaking changes
|
||
|
||
This section discusses the changes that you need to be aware of when migrating your application to Logstash 5.0 from the previous major release of Logstash (2.x).
|
||
|
||
[float]
|
||
=== Changes in Logstash Core
|
||
|
||
These changes can impact any instance of Logstash and are plugin agnostic, but only if you are using the features that are impacted.
|
||
|
||
[float]
|
||
==== Application Settings
|
||
|
||
[IMPORTANT]
|
||
Logstash 5.0 introduces a new way to <<logstash-settings-file, configure application settings>> for Logstash through a
|
||
`logstash.yml` file.
|
||
|
||
This file is typically located in `${LS_HOME}/config`, or `/etc/logstash` when installed via packages. Logstash will not be
|
||
able to start without this file, so please make sure to pass in `--path.settings` if you are starting Logstash manually
|
||
after installing it via a package (RPM, DEB).
|
||
|
||
[source,bash]
|
||
----------------------------------
|
||
bin/logstash --path.settings /path/to/logstash.yml
|
||
----------------------------------
|
||
|
||
[float]
|
||
==== URL Changes for DEB/RPM Packages
|
||
|
||
The previous `packages.elastic.co` URL has been altered to `artifacts.elastic.co`.
|
||
Ensure you update your repository files before running the upgrade process, or
|
||
your operating system may not see the new packages.
|
||
|
||
[float]
|
||
==== Release Packages
|
||
|
||
When Logstash 5.0 is installed via DEB or RPM packages, it now uses `/usr/share/logstash` to
|
||
install binaries. Previously it used to install in `/opt/logstash` directory. This change was done to make the user experience consistent with other products in the Elastic Stack.
|
||
|
||
[cols="3", options="header"]
|
||
|===
|
||
| |DEB |RPM
|
||
|Logstash 2.x
|
||
|`/opt/logstash`
|
||
|`/opt/logstash`
|
||
|Logstash 5.0
|
||
|`/usr/share/logstash`
|
||
|`/usr/share/logstash`
|
||
|===
|
||
|
||
A complete directory layout is described in <<dir-layout>>. This will likely impact any scripts that you may have written
|
||
to support installing or manipulating Logstash, such as via Puppet.
|
||
|
||
[float]
|
||
==== Default Logging Level
|
||
|
||
The default log severity level changed to `INFO` instead of `WARN` to match Elasticsearch. Existing logs
|
||
(in core and plugins) were too noisy at the `INFO` level, so we auditted our log messages and switched some of them to
|
||
`DEBUG` level.
|
||
|
||
You can use the new `logstash.yml` file to configure the `log.level` setting or continue to pass the new
|
||
`--log.level` command line flag.
|
||
|
||
[source,bash]
|
||
----------------------------------
|
||
bin/logstash --log.level warn
|
||
----------------------------------
|
||
|
||
[float]
|
||
==== Plugin Manager Renamed
|
||
|
||
`bin/plugin` has been renamed to `bin/logstash-plugin`. This occurred in Logstash 2.3 and it was mainly prevent `PATH` being
|
||
polluted when other components of the Elastic Stack are installed on the same machine. Also, this provides a foundation
|
||
for future change which will allow Elastic Stack packs to be installed via this script.
|
||
|
||
Logstash 5.0 also adds a `remove` option, which is an alias for the now-deprecated `uninstall` option.
|
||
|
||
As with earlier releases, the updated script allows both online and offline plugin installation. For example, to install a
|
||
plugin named “my-plugin”, it’s as simple as running:
|
||
|
||
[source,bash]
|
||
----------------------------------
|
||
bin/logstash-plugin install my-plugin
|
||
----------------------------------
|
||
|
||
Similar to the package changes, this is likely to impact and scripts that have been written to follow Logstash
|
||
installations.
|
||
|
||
Like earlier releases of Logstash, most plugins are bundled directly with Logstash, so no additional action is required
|
||
while upgrading from earlier Logstash releases. However, if you are attempting to install a non-bundled plugin, then make
|
||
sure that it supports Logstash 5.0 before upgrading!
|
||
|
||
[float]
|
||
==== Logstash with All Plugins Download
|
||
|
||
The Logstash All Plugins download option has been removed. For users previously using this option as a convenience for
|
||
offline plugin management purposes (air-gapped environments), please see the <<offline-plugins>> documentation page.
|
||
|
||
There were 17 plugins removed from 5.0 default bundle. These plugins can still be installed manually for use:
|
||
|
||
* logstash-codec-oldlogstashjson
|
||
* logstash-input-eventlog
|
||
* logstash-input-log4j
|
||
* logstash-input-zeromq
|
||
* logstash-filter-anonymize
|
||
* logstash-filter-checksum
|
||
* logstash-filter-multiline
|
||
* logstash-output-email
|
||
* logstash-output-exec
|
||
* logstash-output-ganglia
|
||
* logstash-output-gelf
|
||
* logstash-output-hipchat
|
||
* logstash-output-juggernaut
|
||
* logstash-output-lumberjack
|
||
* logstash-output-nagios_nsca
|
||
* logstash-output-opentsdb
|
||
* logstash-output-zeromq
|
||
|
||
[float]
|
||
==== Command Line Interface
|
||
|
||
Some CLI Options changed in Logstash 5.0. If you were using the “long form” of the <<command-line-flags,options>>,
|
||
then this will impact the way that you launch Logstash. They were changed to match the `logstash.yml` format used to
|
||
simplify future setup, as well as behave in the same way as other products in the Elastic Stack. For example, here’s two
|
||
before-and-after examples. In Logstash 2.x, you may have run something:
|
||
|
||
[source,bash]
|
||
----------------------------------
|
||
bin/logstash --config my.conf --pipeline-workers 8 <1>
|
||
bin/logstash -f my.conf -w 8 <2>
|
||
----------------------------------
|
||
<1> Long form options `config` and `pipeline-workers` are used here.
|
||
<2> Short form options `f` and `w` (aliases for the former` are used here.
|
||
|
||
But, in Logstash 5.0, this becomes:
|
||
|
||
[source,bash]
|
||
----------------------------------
|
||
bin/logstash --path.config my.conf --pipeline.workers 8 <1>
|
||
bin/logstash -f my.conf -w 8 <2>
|
||
----------------------------------
|
||
<1> Long form options are changed to reflect the new options.
|
||
<2> Short form options are unchanged.
|
||
|
||
NOTE: None of the short form options have changed!
|
||
|
||
[float]
|
||
==== RSpec testing script
|
||
|
||
The `rspec` script is no longer bundled with Logstash release artifacts. This script has been used previously to
|
||
run unit tests for validating users Logstash configuration. While this was useful to some users, this mechanism assumed that Logstash users
|
||
were familiar with the RSpec framework, which is a Ruby testing framework.
|
||
|
||
|
||
[float]
|
||
=== Breaking Changes in Plugins
|
||
|
||
[float]
|
||
==== Elasticsearch Output `workers` Setting Removed
|
||
|
||
Starting with Logstash 5.0, the `workers` setting in the Elasticsearch output
|
||
plugin is no longer supported. Pipelines that specify this setting will no
|
||
longer start up. You need to specify the `pipeline.workers` setting at the
|
||
pipeline level instead. For more information about setting
|
||
`pipeline.workers`, see <<logstash-settings-file>>.
|
||
|
||
[float]
|
||
==== Elasticsearch Output Index Template
|
||
|
||
The index template for Elasticsearch 5.0 has been changed to reflect
|
||
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_mapping_changes.html[Elasticsearch's mapping changes]. Most
|
||
importantly, the subfield for string multi-fields has changed from `.raw` to `.keyword` to match Elasticsearch's default
|
||
behavior. The impact of this change to various user groups is detailed below:
|
||
|
||
** New Logstash 5.0 and Elasticsearch 5.0 users: Multi-fields (often called sub-fields) use `.keyword` from the
|
||
outset. In Kibana, you can use `my_field.keyword` to perform aggregations against text-based fields, in the same way that it
|
||
used to be `my_field.raw`.
|
||
** Existing users with custom templates: Using a custom template means that you control the template completely, and our
|
||
template changes do not impact you.
|
||
** Existing users with default template: Logstash does not force you to upgrade templates if one already exists. If you
|
||
intend to move to the new template and want to use `.keyword`, you will most likely want to reindex existing data so that it
|
||
also uses the `.keyword` field, unless you are able to transition from `.raw` to `.keyword`. Elasticsearch's
|
||
{ref}docs-reindex.html[reindexing API] can help move your data from using `.raw` subfields to `.keyword`, thereby avoiding any
|
||
transition time. You _can_ use a custom template to get both `.raw` and `.keyword` so that you can wait until all `.raw` data
|
||
has stopped existing before transitioning to only using `.keyword`; this will waste some storage space and memory, but it does
|
||
help users to avoid having to relearn operations.
|
||
|
||
[float]
|
||
==== Plugin Versions
|
||
|
||
Logstash is unique amongst the Elastic Stack with respect to its plugins. Unlike Elasticsearch and Kibana, which both
|
||
require plugins to be targeted to a specific release, Logstash’s plugin ecosystem provides more flexibility so that it can
|
||
support outside ecosystems _within the same release_. Unfortunately,
|
||
that flexibility can cause issues when handling upgrades.
|
||
|
||
Non-standard plugins must always be checked for compatibility, but some bundled plugins are upgraded in order to remain
|
||
compatible with the tools or frameworks that they use for communication. For example, the
|
||
<<plugins-inputs-kafka, Kafka Input>> and <<plugins-outputs-kafka, Kafka Output>> plugins serve as a primary example of
|
||
such compatibilty changes. The latest version of the Kafka plugins is only compatible with Kafka 0.10, but as the
|
||
compatibility matrices show: earlier plugin versions are required for earlier versions of Kafka (e.g., Kafka 0.9).
|
||
|
||
Automatic upgrades generally lead to improved features and support, but network layer changes like those above may make part
|
||
of your architecture incompatible. You should always test your Logstash configurations in a test environment before
|
||
deploying to production, which would catch these kinds of issues. If you do face such an issue, then you should also check
|
||
the specific plugin’s page to see how to get a compatible, older plugin version if necessary.
|
||
|
||
For example, if you upgrade to Logstash 5.0, but you want to run against Kafka 0.9, then you need to remove the
|
||
bundled plugin(s) that only work with Kafka 0.10 and replace them:
|
||
|
||
[source,bash]
|
||
----------------------------------
|
||
bin/logstash-plugin remove logstash-input-kafka
|
||
bin/logstash-plugin remove logstash-output-kafka
|
||
bin/logstash-plugin install --version 4.0.0 logstash-input-kafka
|
||
bin/logstash-plugin install --version 4.0.1 logstash-output-kafka
|
||
----------------------------------
|
||
|
||
The version numbers were found by checking the compatibility matrix for the individual plugins.
|
||
|
||
[float]
|
||
==== Kafka Input Configuration Changes
|
||
|
||
As described in the section <<plugin-versions, above>>, Kafka plugin has been updated to bring in new consumer features.
|
||
In addition, to the plugin being incompatible with 0.8.x version of Kafka broker, _most_ of the config options have
|
||
been changed to match the new consumer configurations from the Kafka Java consumer. Here's a list of important config options that have changed:
|
||
|
||
* `topic_id` is renamed to `topics` and accepts an array of topics to consume from.
|
||
* `zk_connect` has been dropped; you should use `bootstrap_servers`. There is no need for the consumer to go through ZooKeeper
|
||
* `consumer_id` is renamed to `client_id`
|
||
|
||
We recommend users of the Kafka plugin to check the documentation for the latest <<plugins-inputs-kafka, config options>>.
|
||
|
||
[float]
|
||
==== File Input
|
||
|
||
The <<plugins-inputs-file, File Input>> `SinceDB` file is now saved at `<path.data>/plugins/inputs/file` location,
|
||
where `path.data` is the path defined in the new `logstash.yml` file.
|
||
|
||
[cols="2", options="header"]
|
||
|===
|
||
| |Default `sincedb_path`
|
||
|Logstash 2.x
|
||
|`$HOME/.sincedb*`
|
||
|Logstash 5.0
|
||
|`<path.data>/plugins/inputs/file`
|
||
|===
|
||
|
||
If you have manually specified `sincedb_path` as part of the configuration, this change will not affect you.
|
||
If you are moving from Logstash 2.x to Logstash 5.0, and you would like to use the existing SinceDB file,
|
||
then it must be copied over to `path.data` manually to use the save state (or the path needs to be changed to point to it).
|
||
|
||
[float]
|
||
==== GeoIP Filter
|
||
|
||
The GeoIP filter has been updated to use MaxMind's GeoIP2 database. Previous GeoIP version is now considered legacy
|
||
by MaxMind. As a result of this, `.dat` version files are no longer supported, and only `.mmdb` format is supported.
|
||
The new database will not include ASN data in the basic free database file.
|
||
|
||
Previously, when the filter encountered an IP address for which there were no results in the database, the event
|
||
would just pass through the filter without modification. It will now add a `_geoip_lookup_failure` tag to the
|
||
event which will allow for some subsequent stage of the pipeline to identify those events and perform some other
|
||
operation. To simply get the same behavior as the earlier versions, just add a filter conditional on that tag
|
||
which then drops the tag from the event.
|
||
|
||
[float]
|
||
=== Ruby Filter and Custom Plugin Developers
|
||
|
||
With the migration to the new <<event-api>>, we have changed how you can access internal data compared to previous release.
|
||
The `event` object no longer returns a reference to the data. Instead, it returns a copy. This might change how you perform
|
||
manipulation of your data, especially when working with nested hashes. When working with nested hashes, it’s recommended that
|
||
you use the <<logstash-config-field-references, `field reference` syntax>> instead of using multiple square brackets.
|
||
|
||
As part of this change, Logstash has introduced new Getter/Setter APIs for accessing information in the `event` object.
|
||
|
||
**Examples:**
|
||
|
||
Prior to Logstash 5.0, you may have used Ruby filters like so:
|
||
|
||
[source, js]
|
||
----------------------------------
|
||
filter {
|
||
ruby {
|
||
code => "event['name'] = 'Logstash'"
|
||
}
|
||
ruby {
|
||
code => "event['product']['version'] = event['major'] + '.' + event['minor']"
|
||
}
|
||
}
|
||
----------------------------------
|
||
|
||
The above syntax, which uses the `event` object as a reference, is no longer supported in
|
||
Logstash 5.0. Fortunately, the change to make it work is very simple:
|
||
|
||
[source, js]
|
||
----------------------------------
|
||
filter {
|
||
ruby {
|
||
code => "event.set('name', 'Logstash')"
|
||
}
|
||
ruby {
|
||
code => "event.set('[product][version]', event.get('major') + '.' + event.get('minor'))"
|
||
}
|
||
}
|
||
----------------------------------
|
||
|
||
NOTE: Moving from the old syntax to the new syntax, it can be easy to miss that `['product']['version']` became
|
||
`'[product][version]'`. The quotes moved from inside of the square brackets to outside of the square brackets!
|
||
|
||
The <<event-api>> documentation describes the available syntax in great detail.
|