github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-22 13:47:21 -04:00
Code Issues Projects Releases Packages Wiki Activity
logstash/CHANGELOG
Jordan Sissel 08922cbc99 - update changelog
2013-08-26 23:19:46 -07:00

885 lines
45 KiB
Text
Raw Permalink Blame History

1.2.0 (???)
# general
- The logstash json schema has changed. (LOGSTASH-675)
For prior logstash users, you will be impacted one of several ways:
* You should check your elasticsearch templates and update them accordingly.
* If you want to reindex old data from elasticsearch with the new schema,
you should be able to do this with the elasticsearch input. Just make
sure you set 'codec => oldlogstashjson' in your elasticsearch input.
- The old logstash web ui has been replaced by Kibana 3. Kibana is a far
superior search and analytics interface.
- New feature: conditionals! You can now make "if this, then ..." decisions
in your filters or outputs. See the docs here:
https://logstash.net/docs/latest/configuration#conditionals
- A new syntax exists for referencing fields (LOGSTASH-1153). This replaces
the prior and undocumented syntax for field access (was 'foo.bar' and is
now '[foo][bar]'). Learn more about this here:
https://logstash.net/docs/latest/configuration#fieldreferences
- ElasticSearch version 0.90.3 is included. (#486, Gang Chen)
- The elasticsearch plugin now uses the bulk index api which should result
in lower cpu usage as well as higher performance than the previous
logstash version.
- Many deprecated features have been removed. If your config caused
deprecation warnings on startup in logstash v1.1.13, there is a good
chance that these deprecated settings are now absent.
- 'type' is no longer a required setting on inputs.
- New plugin type: codec. Used to implement decoding of events for inputs and
encoding of events for outputs. Codecs allow us to separate transport (like
tcp, redis, rabbitmq) from serialization (gzip text, json, msgpack, etc).
- Improved error messages that try to be helpful. If you see bad or confusing
error messages, it is a bug, so let us know! (Patch by Nick Ethier)
## inputs
- bugfix: gelf: work around gelf parser errors (#476, patch by Chris McCoy)
- broken: the twitter input is disabled because the twitter stream v1 api is
no longer supported and I couldn't find a replacement library that works
under JRuby.
- new: sqlite input (#484, patch by Evan Livingston)
- improvement: snmptrap: new 'yamlmibdir' setting for specifying an external
source for MIB definitions. (#477, patch by Dick Davies)
- improvement: stomp: vhost support (#490, patch by Matt Dainty)
- new: unix: unix socket input (#496, patch by Nikolay Bryskin)
- new: wmi: for querying wmi (windows). (#497, patch by Philip Seidel)
- improvement: sqs: new id_field and md5_field settings (LOGSTASH-1118, Louis
Zuckerman)
## filters
- feature: grok: 'singles' now defaults to true.
- bugfix: grep: allow repeating a field in the hash config (LOGSTASH-919)
- feature: specify timezone in date filter (#470, patch by Philippe Weber)
- feature: grok setting 'overwrite' now lets you overwrite fields instead
of appending to them.
- feature: the useragent filter now defaults to writing results to the top
level of the event instead of "ua"
- feature: grok now defaults 'singles' to true, meaning captured fields are
stored as single values in most cases instead of the old behavior of being
captured as an array of values.
- new: json_encoder filter (Ralph Meijer)
- new: cipher: gives you many options for encrypting fields (#493, patch by
saez0pub)
- feature: kv: new settings include_fields and exclude_fields. (patch by
Piavlo)
- feature: geoip: new 'target' setting for where to write geoip results.
(#491, patch by Richard Pijnenburg)
- feature: dns: now accepts custom nameservers to query (#495, patch by
Nikolay Bryskin)
- bugfix: ruby: multiple ruby filter instances now work (#501, patch by
Nikolay Bryskin)
## outputs
- improvement: stomp: vhost support (Patch by Matt Dainty)
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
- bugfix: emails: restored initial really useful documentation
- improvement: emails: allow @message, @source, @... in match (LOGSTASH-826,
LOGSTASH-823)
- feature: mongodb: replica sets are supported (#389, patch by Mathias Gug)
- new: s3: New plugin to write to amazon S3 (#439, patch by Mattia Peterle)
1.1.13 (May 28, 2013)
## general
- fixed bug in static file serving for logstash web (LOGSTASH-1067)
## outputs
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
1.1.12 (May 7, 2013)
## filters
- bugfix: useragent filter now works correctly with the built-in regexes.yaml
- bugfix: mail output with smtp now works again
1.1.11 (May 7, 2013)
## general
- This release is primarily a bugfix release for bugs introduced by the
previous release.
- Support for Rubinius and MRI exists once again.
## inputs
- bugfix: lumberjack now respects field data again (lumberjack --field foo=bar)
- bugfix: rabbitmq was broken by the previous release (LOGSTASH-1003,
LOGSTASH-1038; Patch by Jason Koppe)
- bugfix: relp: allow multiple client socket connections to RELP input
(LOGSTASH-707, LOGSTASH-736, LOGSTASH-921)
## filters
- bugfix: geoip was broken by the previous release (LOGSTASH-1013)
- feature: sleep now accepts an 'every' setting which causes it to
sleep every N events. Example; sleep every 10 events: every => 10.
- feature: grok now permits dashes and dots in captures, such as
%{WORD:foo-bar}.
- bugfix: useragent filter now ships with a default regexes.yaml file
that is used by default unless you otherwise specify (LOGSTASH-1051)
- bugfix: add_field now correctly sets top-level fields like @message
- bugfix: mutate 'replace' now sets a field regardless of whether or not
it exists.
- feature: new mutate 'update' setting to change a field's value but
only if that field exists already.
## outputs
- feature: irc output now supports 'secure' setting to use ssl (LOGSTASH-139)
- feature: nagios_nsca has new setting 'message_format'
- bugfix: fix graphite plugin broken in 1.1.10 (LOGSTASH-968)
- bugfix: elasticsearch_http was broken in 1.1.10 (LOGSTASH-1004)
- bugfix: rabbitmq was broken by the previous release (LOGSTASH-1003,
LOGSTASH-1038; Patch by Jason Koppe)
- feature: hipchat 'notify' setting now called 'trigger_notify' (#467, patch
by Richard Pijnenburg)
1.1.10 (April 16, 2013)
## general
- On linux, all threads will set their process names so you can identify
threads in tools like top(1).
- Java 5 is no longer supported (You must use Java 6 or newer).
- Windows line terminators (CRLF) are now accepted in config files.
- All AWS-related plugins now have the same configuration options:
region, access_key_id, secret_access_key, use_ssl, and
aws_credentials_file. Affected plugins: cloudwatch output,
sns output, sqs output, sqs input. (LOGSTASH-805)
- Lots of documentation fixes (James Turnbull, et al)
- The amqp plugins are now named 'rabbitmq' because it *only* works
with rabbitmq. The old 'amqp' name should still work, but it will
be removed soon while 'rabbitmq' will stay. (Patches by Michael Zaccari)
- New flag '--configtest' to test config and exit. (Patch by Darren Patterson)
- Improved error feedback logstash gives to you as a user.
## inputs
- new: elasticsearch: this input allows you to stream search results from
elasticsearch; it uses the Scroll API.
- new: websocket. Currently supports acting as a websocket client.
- new: snmptrap, to receive SNMP traps (patch by Paul Czar)
- new: varnishlog input to read from the Varnish Cache server's shared memory
log (LOGSTASH-978, #422; Louis Zuckerman)
- new: graphite input. Supports the plain text carbon tcp protocol.
- new: imap input. Read mail!
- feature: twitter: supports http proxying now (#276, patch by Richard
Pijnenburg)
- feature: loggly: supports http proxying now (#276, patch by Richard
Pijnenburg)
- feature: tcp: ssl now supported! (#318, patch by Matthew Richardson)
- feature: redis: now supports 'batch_count' option for doing bulk fetches
from redis lists. Requires Redis 2.6.0 or higher. (#320, patch by Piavlo)
- feature: irc: will use ssl if you set 'secure' (#393, patch by Tomas Doran)
- bugfix: log4j: respect add_fields (LOGSTASH-904, #358)
- bugfix: eventlog: input should now work
- bugfix: irc: passwords now work (#412, Nick Ethier)
## filters
- new: useragent: parses user agent strings in to structured data based on
BrowserScope data (#347, patch by Dan Everton)
- new: sleep: sleeps a given amount of time before passing the event.
Useful for rate limiting or replay simulation.
- new: ruby: experimental ruby plugin that lets you call custom ruby code
on every event.
- new: translate: for mapping values (#335, patch by Paul Czar)
- new: clone: creates a copy of the event.
- feature: grok: Adds tag_on_failure setting so you can prevent grok from
tagging events on failure. (#328, patch by Neil Prosser)
- deprecated: grok: deprecated the --grok-patterns-path flag (LOGSTASH-803)
- feature: date: nested field access is allowed now
- feature: csv, xml, kv, json, geoip: new common settings!
(LOGSTASH-756, #310, #311, #312, #383, #396; patches by Richard Pijnenburg)
source - what field the text comes from
target - where to store the parse result.
- feature: csv: new setting: columns - labels for each column parsed.
- bugfix: geoip: The built-in geoip database should work now (#326, patch
by Vincent Batts)
- bugfix: kv filter now respects add_tag, etc (LOGSTASH-935)
## outputs
- new: hipchat output (#428, Cameron Stokes)
- bugfix: mongo would fail to load bson_java support (LOGSTASH-849)
- bugfix: tags support to gelf output. Returns tags as _tags field
(LOGSTASH-880, patch by James Turnbull)
- bugfix: elasticsearch: Fix a race condition. (#340, patch by Raymond Feng)
- improvement: http: now supports a custom 'message' format for building your
own http bodies from an event. (#319, patch by Christian S)
- bugfix: Fix opentsdb output (LOGSTASH-689, #317; patch by Emmet Murphy)
- improvement: http output now supports a custom message format with
the 'message' setting (Patch by Christian Schröder)
- graphite output now lets you ship the whole (or part) of an event's fields
to graphite as metric updates. (#350, patch by Piavlo)
- email output now correctly defaults to not using authentication
(LOGSTASH-559, #365; patch by Stian Mathiassen)
- bugfix: file output now works correctly on fifos
- bugfix: irc passwords now work (#412, Nick Ethier)
- improvement: redis output now supports congestion detection. If
it appears nothing is consuming from redis, the output will stall
until that problem is resolved. This helps prevent a dead reader
from letting redis fill up memory. (Piavlo)
- feature: boundary: New 'auto' setting. (#413, Alden Jole)
1.1.9 (January 10, 2013)
## inputs
- bugfix: all inputs: fix bug where some @source values were not valid urls
## filters
- bugfix: mutate: skip missing fields in 'convert' (#244, patch by Ralph Meijer)
## outputs
- improvement: gelf: new tunable 'ignore_metadata' flag to set which fields
to ignore if ship_metadata is set. (#244, patch by Ralph Meijer)
- improvement: gelf: make short_message's field name tunable (#244, patch by
Ralph Meijer)
1.1.8 (January 10, 2013)
## general
- patched another work around for JRUBY-6970 (LOGSTASH-801)
## inputs
- bugfix: tcp: 'Address in use' errors now report the host/port involved.
(LOGSTASH-831)
- bugfix: zeromq: fix bug where an invalid url could be given as a source
(LOGSTASH-821, #306)
## outputs
- bugfix: elasticsearch_river: it now resolves evaluates %{} variables in
index and index_type settings. (LOGSTASH-819)
1.1.7 (January 3, 2013)
## inputs
- fix bug where @source_host was set to 'false' in many cases.
## outputs
- improvement: redis: shuffle_hosts is now enabled by default
1.1.6 (January 2, 2013)
## Overview of this release:
- new inputs: drupal_dblog.
- new filters: anonymize, metrics.
- new outputs: syslog, cloudwatch.
- new 'charset' setting for all inputs. This should resolve all known encoding
problems. The default charset is UTF-8.
- grok now captures (?<somename>...) regexp into 'somename' field
- Elasticsearch 0.20.2 is included. This means you are required to upgrade
your elasticsearch cluster to 0.20.2. If you wish to continue using an old
version of elasticsearch, you should use the elasticsearch_http plugin
instead of the elasticsearch one.
## general
- fixed internal dependency versioning on 'addressable' gem (LOGSTASH-694)
- fixed another case of 'watchdog timeout' (LOGSTASH-701)
- plugin flags are now deprecated. The grok filter (--grok-pattern-path) was
the only plugin to make use of this.
- the grok filter has improved documentation
- lots of documentation fixes (James Turnbull, Louis Zuckerman)
- lots of testing improvements (Philippe Weber, Laust Rud Jacobsen)
- all 'name' settings have been deprecated in favor of more descriptive
settings (LOGSTASH-755)
- JRuby upgraded to 1.7.1
- removed use of bundler
- Fixed timestamp parsing in MRI (patch by Rene Lengwinat)
## inputs
- All inputs now have a 'charset' setting to help you inform logstash of the
text encoding of the input. This is useful if you have Shift_JIS or CP1251
encoded log files. This should help resolve the many UTF-8 bugs that were
reported recently. The default charset is UTF-8.
- new: drupal_dblog: read events from a DBLog-enabled Drupal. (#251, Patch by
theduke)
- bugfix: zeromq: 'topology' is now a required setting
- bugfix: lumberjack: client connection closing is now handled properly.
(Patch by Nick Ethier)
- misc: lumberjack: jls-lumberjack gem updated to 0.0.7
- bugfix: stomp: fix startup problems causing early termination (#226
- bugfix: tcp: the 'source host' for events is now the client ip:port that
sent it, instead of the listen address that received it. (LOGSTASH-796)
- improvement: tcp: the default data_timeout is now -1 (never timeout).
This change was made because read timeouts were causing data loss, and
logstash should avoid losing events by default.
- improvement: amqp: the 'name' setting is now called 'queue' (#274)
- improvement: eventlog: the 'name' setting is now called 'logfile' (#274)
- bugfix: log4j: fix stacktrace reading (#253, patch by Alex Arutyunyants)
## filters
- new: anonymize: supports many hash mechanisms (murmur3, sha1, md5, etc) as
well as IP address anonymization (#280, #261; patches by Richard Pijnenburg
and Avishai Ish-Shalom)
- new: metrics: allows you to aggregate metrics from events and emit them
periodically. Think of this like 'statsd' but implemented as a logstash
filter instead of an external service.
- feature: date: now accepts 'match' as a setting. Use of this is preferable
to the old syntax. Where you previously had 'date { somefield =>
"somepattern" }' you should now do: 'date { match => [ "somefield",
"somepattern" ] }'. (#248, LOGSTASH-734, Patch by Louis Zuckerman)
- feature: grok: now accepts (?<foo>...) named captures. This lets you
compose a pattern in the grok config without needing to define it in a
patterns file. Example: (?<hostport>%{HOST}:%{POSINT}) to capture 'hostport'
- improvement: grok: allow '$' in JAVACLASS pattern (#241, patch by Corry
Haines)
- improvement: grok: can now match against number types. Example, if you're
sending a json format event with { "status": 403 } you can now grok that
field. The number is represented as a string "403" before pattern matching.
- bugfix: date: Fix a bug that would crash the pipeline if no date pattern
matched. (LOGSTASH-705)
- feature: kv: Adds field_split, value_split, prefix, and container
settings. (#225, patch by Alex Wheeler)
- bugfix: mutate: rename on a nonexistant field now does nothing as expected.
(LOGSTASH-757)
- bugfix: grok: don't tag an event with _grokparsefailure if it's already so
(#248, patch by Greg Brockman)
- feature: mutate: new settings - split, join, strip. "split" splits a field
into an array. "join" merges an array into a string. "strip" strips leading and
trailing whitespace. (Patch by Avishai Ish-Shalom)
## outputs
- new: syslog output supporting both RFC3164 and RFC5424 (#180, patch by
Rui Alves)
- new: cloudwatch output to emit metrics and other events to Amazon CloudWatch.
(LOGSTASH-461, patch by Louis Zuckerman)
- feature: stdout: added 'message' setting for configuring the output message
format. The default is same behavior as before this feature.
- feature: http: added 'format' option to select 'json' or form-encoded
request body to send with each request.
- feature: http: added 'content_Type' option set the Content-Type header.
This defaults to "application/json" if the 'format' is 'json'. Will default
to 'application/x-www-form-urlencoded' if the 'format' is 'form'
- bugfix: zeromq: 'topology' is now a required setting
- feature: mongodb: new setting 'isodate' that, when true, stores the
@timestamp field as a mongodb date instead of a string. (#224, patch by
Kevin Amorin)
- improvement: gelf: Allow full_message gelf property to be overridden (#245,
patch by Sébastien Masset)
- misc: lumberjack: jls-lumberjack gem updated to 0.0.6
- feature: nagios: New 'nagios_level' setting to let you change the level
of the passive check result sent to nagios. (#298, Patch by James Turnbull)
- feature: elasticsearch, elasticsearch_http, elasticsearch_river: new setting
'document_id' for explicitly setting the document id in each write to
elasticsearch. This is useful for overwriting existing documents.
- improvement: elasticsearch_river: 'name' is now 'queue' (#274)
- improvement: amqp: 'name' is now 'exchange' (#274)
- bugfix: the websocket output works again (supports RFC6455)
1.1.5 (November 10, 2012)
## Overview of this release:
* New inputs: zenoss, gemfire
* New outputs: lumberjack, gemfire
* Many UTF-8 crashing bugs were resolved
## general
- new runner command 'rspec' - lets you run rspec tests from the jar
This means you should now be able to write external tests that execute your
logstash configs and verify functionality.
- "file not found" errors related to paths that had "jar:" prefixes should
now work. (Fixes LOGSTASH-649, LOGSTASH-642, LOGSTASH-655)
- several plugins received UTF-8-related fixes (file, lumberjack, etc)
File bugs if you see any UTF-8 related crashes.
- 'json_event' format inputs will now respect 'tags' (#239, patch by
Tim Laszlo)
- logstash no longer uses nor recommends bundler (see 'gembag.rb'). The
Gemfile will be purged in the near future.
- amqp plugins are now marked 'unsupported' as there is no active maintainer
nor is there source of active support in the community. If you're interested
in maintainership, please email the mailling list or contact Jordan!
## inputs
- irc: now stores irc nick
- new: zenoss (#232, patch by Chet Luther)
- new: gemfire (#235, patch by Andrea Campi)
- bugfix: udp: skip close() call if we're already closed (#238, patch by kcrayon)
## filters
- bugfix: fix for zeromq filter initializer (#237, patch by Tom Howe)
## outputs
- new: lumberjack output (patch by Nick Ethier)
- new: gemfire output (#234, patch by Andrea Campi)
- improved: nagios_ncsa (patch by Tomas Doran)
- improved: elasticsearch: permit setting 'host' even if embedded. Also set the
host default to 'localhost' when using embedded. These fixes should help resolve
issues new users have when their distros surprisingly block multicast by
default.
- improved: elasticsearch: failed index attempts will be retried
- improved: irc: new 'password' setting (#283, patch by theduke)
1.1.4 (October 28, 2012)
## Overview of this release:
- bug fixes mostly
## filters
- date: Fix crashing on date filter failures. Wrote test to cover this case.
(LOGSTASH-641)
- grok: Improve QUOTEDSTRING pattern to avoid some more 'watchdog timeout' problems
## outputs
- nagios_nsca: Allow check status to be set from the event (#228, patch by
Tomas Doran)
- elasticsearch_http: Fix OpenSSL::X509::StoreError (LOGSTASH-642)
1.1.3 (October 22, 2012)
- rebuilt 1.1.2 for java 5 and 6
1.1.2 (October 22, 2012)
## Overview of this release:
* New input plugins: lumberjack, sqs, relp
* New output plugins: exec, sqs
* New filter plugins: kv, geoip, urldecode, alter
* file input supports backfill via 'start_position'
* filter watchdog timer set to 10 seconds (was 2 seconds)
## general
- Stopped using 'Gemfile' for dependencies, the logstash.gemspec has returned.
(Patch by Grant Rogers)
- New 'logstash-event.gemspec' for generating logstash events in your own
ruby programs (Patch by Garry Shutler)
- Wildcard config files are now sorted properly (agent -f
/etc/logstash/*.conf)
- The old '-vvv' setting ruby's internal $DEBUG is now gone. It was causing
too much confusion for users due to noise.
- Improved 'logstash event' creation speed by 3.5x
- Now uses JRuby 1.7.0
- Now ships with Elasticsearch 0.19.10
## inputs
- bugfix: redis: [LOGSTASH-526] fix bug with password passing
- new: lumberjack: for use with the lumberjack log shipper
(https://github.com/jordansissel/lumberjack)
- new: sqs: Amazon SQS input (Patch by Sean Laurent, #211)
- new: relp: RELP (rsyslog) plugin (Patch by Mike Worth, #177)
- file input: sincedb path is now automatically generated if not specified.
This helps work around a problem where two file inputs don't specify a
sincedb_path would clobber eachother (LOGSTASH-554)
- file input: no longer crashes if HOME is not set in env (LOGSTASH-458)
- log4j input: now supports MDC 'event properties' which are stored as fields
in the logstash event. (#216, #179. Patches by Charles Robertson and Jurjan
Woltman)
- pipe input: should work now.
## filters
- new: kv: useful for parsing log formats taht use 'foo=bar baz=fizz' and
similar key-value-like things.
- new: urldecode: a filter for urldecoding fields in your event. (Patch by
Joey Imbasciano, LOGSTASH-612)
- new: geoip: query a local geoip database for location information (Patch by
Avishai Ish-Shalom, #208)
- improvement: zeromq: an empty reply is now considered as a 'cancel this
event' operation (LOGSTASH-574)
- bugfix: mutate: fix bug in uppercase and lowercase feature that would
prevent it from actually doing the uppercasing/lowercasing.
- improvement: mutate: do the 'remove' action last (LOGSTASH-543)
- feature: grok: new 'singles' config option which, when true, stores
single-value fields simply as a single value rather than as an array, like
[value]. (LOGSTASH-185)
- grok patterns: the URIPARAM pattern now includes pipe '|' as a valid
character. (Patch by Chris Mague)
- grok patterns: improve haproxy log patterns (Patch by Kevin Nuckolls)
- grok patterns: include 'FATAL' as a valid LOGLEVEL match
(patch by Corry Haines)
- grok patterns: 'ZONE' is no longer captured by name in the HTTPDATE pattern
- new: alter: adds some conditional field modification as well as a
'coalesce' feature which sets the value of a field to the first non-null
value given in a list. (Patch by Francesco Salbaroli)
- improvement: date: add TAI64N support
- improvement: date: subsecond precision on UNIX timestamps is retained on
conversion (#213, Patch by Ralph Meijer)
- improvement: date: Add locale setting; useful for day/month name parsing.
(#100, Patch by Christian Schröder)
## outputs
- new: exec: run arbitrary commands based on an event.
- new: sqs: Amazon SQS output (Patch by Sean Laurent, #211)
- bugfix: redis: [LOGSTASH-526] fix bug with password passing
- improvement: redis: [LOGSTASH-573] retry on failure even in batch-mode. This
also fixes a prior bug where an exception in batch mode would cause logstash
to crash. (Patch by Alex Dean)
- improvement: riemann: metric and ttl values in riemann_event now support
sprintf %{foo} values. (pull #174)
- improvement: stdout: new 'dots' debug_format value emits one dot per event
useful for tracking event rates.
- gelf output: correct severity level mappings (patch by Jason Koppe)
- xmpp output: users and rooms are separate config settings now (patch by
Parker DeBardelaben)
- improvement: redis: 'host' setting now accepts a list of hosts for failover
of writes should the current host go down. (#222, patch by Corry Haines)
1.1.1 (July 14, 2012)
## Overview of this release:
* New input plugins: generator, heroku, pipe, ganglia, irc
* New output plugins: juggernaut, metricscatcher, nagios_ncsa, pipe,
opentsdb, pagerduty, irc
* New filter plugins: zeromq, environment, xml, csv, syslog_pri
* Fixes for gelf output
* Support for more than 1 filter worker (agent argument "-w")
## IMPORTANT CHANGES FOR UPGRADES FROM 1.1.0
- zeromq input and output rewritten
The previous zeromq support was an MVP. It has now been rewritten into
something more flexible. The configuration options have changed entirely.
While this is still listed as `experimental`, we don't predict any more
configuration syntax changes. The next release will bump this to beta.
- unix_timestamp
Previously, several plugins did not work as expected on MRI due to the
usage of the JRuby-only Jodatime library. We now have a contributed fix
for a slower parser on MRI/CRuby!
- elasticsearch version is now 0.19.8
This means your elasticsearch cluster must be running 0.19.x for
compatibility reasons.
- grok pattern %{POSINT} used to match '0' -- now it does not. If you want
to match non-negative integers, there is now a %{NONNEGINT} pattern.
- bug in file input fixed that led to an extra leading slash in @source_path.
Previously, file input would have @source = 'file://host//var/log/foo' and
@source_path = '//var/log/foo'; now @source = 'file://host/var/log/foo'
and @source_path = '/var/log/foo'. [LOGSTASH-501]
- file input now rejects relative paths. [LOGSTASH-503]
- event sprintf can now look inside structured field data. %{foo.bar} will
look in the event field "foo" (if it is a hash) for "bar". To preserve
compatibility, we first look for a top-level key that matches exactly
(so %{foo.bar} will first look for a field named "foo.bar", then look for
"bar" under "foo").
## general
- NOTE: gemspec removed; deploying logstash as a gem hasn't been supported
for a while.
- feature: logstash sub-commands "irb" and "pry" for an interactive debug
console, useful to debug jruby when running from the monolithic jar
- misc: newer cabin gem for logging
- misc: initial support for reporting internal metrics (currently outputs
to INFO log; eventually will be an internal event type)
- misc: added a "thread watchdog" to detect hanging filter workers, and
crash logstash w/an informational message
- misc: jar is built with jruby 1.6.7.2
- misc: better shutdown behavior when there are no inputs/plugins running
- feature: logstash web now uses relative URLs; useful if you want to
reverseproxy with a path other than "/"
## inputs
- bugfix: stdin: exit plugin gracefully on EOF
- feature: [LOGSTASH-410] - inputs can now be duplicated with the
'threads' parameter (where supported)
- bugfix: [LOGSTASH-490] - include cacert.pem in jar for twitter input
- feature: [LOGSTASH-139] - support for IRC
## filters
- feature: all filters support 'remove_tag' (remove tags on success)
- feature: all filters support 'exclude_tags' (inverse of 'tags')
- bugfix: [LOGSTASH-300] - bump grok pattern replace limit to 1000,
fixes "deep recursion pattern compilation" problems
- bugfix: [LOGSTASH-375] - fix bug in grep: don't drop when field is nil
and negate is true
- bugfix: [LOGSTASH-386] - fix some grok patterns for haproxy
- bugfix: [LOGSTASH-446] - fix grok %{QUOTEDSTRING} pattern, should fix
some grok filter hangs
- bugfix: some enhancements to grok pattern %{COMBINEDAPACHELOG}
- bugfix: grok: %{URIPATH} and %{URIPARAM} enhancements
- feature: grok: add %{UUID} pattern
- bugfix: grok: better error message when expanding unknown %{pattern}
- feature: mutate: now supports a 'gsub' operation for applying a regexp
substitution on event fields
## outputs
- bugfix: [LOGSTASH-351] - fix file input on windows
- feature: [LOGSTASH-356] - make file output flush intervals configurable
- feature: [LOGSTASH-392] - add 'field' attribute to restrict which fields
get sent to an output
- feature: [LOGSTASH-374] - add gzip support to file output
- bugfix: elastic search river now respects exchange_type and queue_name
- bugfix: ganglia plugin now respects metric_type
- bugfix: GELF output facility fixes; now defaults to 'logstash-gelf'
- feature: [LOGSTASH-139] - support for IRC
- bugfix: es_river: check river status after creation to verify status
- feature: es: allow setting node_name
- feature: redis: output batching for list mode
1.1.0.1 (January 30, 2012)
## Overview of this release:
* date filter bugfix: [LOGSTASH-438] - update joda-time to properly
handle leap days
1.1.0 (January 30, 2012)
## Overview of this release:
* New input plugins: zeromq, gelf
* New filter plugins: mutate, dns, json
* New output plugins: zeromq, file
* The logstash agent now runs also in MRI 1.9.2 and above
This is a large release due to the longevity of the 1.1.0 betas.
We don't like long releases and will try to avoid this in the future.
## IMPORTANT CHANGES FOR UPGRADES FROM 1.0.x
- grok filter: named_captures_only now defaults to true
This means simple patterns %{NUMBER} without any other name will
now not be included in the field set. You can revert to the old
behavior by setting 'named_captures_only => false' in your grok
filter config.
- grok filter: now uses Ruby's regular expression engine
The previous engine was PCRE. It is now Oniguruma (Ruby). Their
syntaxes are quite similar, but it is something to be aware of.
- elasticsearch library upgraded to 0.18.7
This means you will need to upgrade your elasticsearch servers,
if any, to the this version: 0.18.7
- AMQP parameters and usage have changed for the better. You might
find that your old (1.0.x) AMQP logstash configs do not work.
If so, please consult the documentation for that plugin to find
the new names of the parameters.
## general
- feature: [LOGSTASH-158] - MRI-1.9 compatible (except for some
plugins/functions which will throw a compatibility exception) This means
you can use most of the logstash agent under standard ruby.
- feature: [LOGSTASH-118] - logstash version output (--version or -V for
agent)
- feature: all plugins now have a 'plugin status' indicating the expectation
of stability, successful deployment, and rate of code change. If you
use an unstable plugin, you will now see a warning message on startup.
- bugfix: AMQP overhaul (input & output), please see docs for updated
config parameters.
- bugfix: [LOGSTASH-162,177,196] make sure plugin-contained global actions
happen serially across all plugins (with a mutex)
- bugfix: [LOGSTASH-286] - logstash agent should not truncate logfile on
startup
- misc: [LOGSTASH-160] - now use gnu make instead of rake.
- misc: now using cabin library for all internal logging
- test: use minitest
- upgrade: now using jruby in 1.9 mode
## inputs
- feature: zeromq input. Requires you have libzmq installed on your system.
- feature, bugfix: [LOGSTASH-40,65,234,296]: much smarter file watching for
file inputs. now supports globs, keeps state between runs, can handle
truncate, log rotation, etc. no more inotify is required, either (file
input now works on all platforms)
- feature: [LOGSTASH-172,201] - syslog input accepts ISO8601 timestamps
- feature: [LOGSTASH-159] - TCP input lets you configure what identifies
an input stream to the multiline filter (unique per host, or connection)
- feature: [LOGSTASH-168] - add new GELF input plugin
- bugfix: [LOGSTASH-8,233] - fix stomp input
- bugfix: [LOGSTASH-136,142] - file input should behave better with log rotations
- bugfix: [LOGSTASH-249] - Input syslog force facility type to be an integer
- bugfix: [LOGSTASH-317] - fix file input not to crash when a file
is unreadable
## filters
- feature: [LOGSTASH-66,150]: libgrok re-written in pure ruby (no more
FFI / external libgrok.so dependency!)
- feature: [LOGSTASH-292,316] - Filters should run on all events if no condition
is applied (type, etc).
- feature: [LOGSTASH-292,316] - Filters can now act on specific tags (or sets
of tags).
- bugfix: [LOGSTASH-285] - for grok, add 'keep_empty_captures' setting to
allow dropping of empty captures. This is true by default.
- feature: [LOGSTASH-219] - support parsing unix epoch times
- feature: [LOGSTASH-207] - new filter to parse a field as json merging it
into the event.
- feature: [LOGSTASH-267,254] - add DNS filter for doing forward or
reverse DNS on an event field
- feature: [LOGSTASH-57] - add mutate filter to help with manipulating
event field content and type
## outputs
- feature: zeromq output. Requires you have libzmq installed on your system.
- feature: new file output plugin
- bugfix: [LOGSTASH-307] embedded elasticsearch now acts as a full ES server;
previously embedded was only accessible from within the logstash process.
- bugfix: [LOGSTASH-302] - logstash's log level (-v, -vv flags) now control
the log output from the elasticsearch client via log4j.
- bugfix: many gelf output enhancements and bugfixes
- feature: [LOGSTASH-281] - add https support to loggly output
- bugfix: [LOGSTASH-167] - limit number of in-flight requests to the
elasticsearch node to avoid creating too many threads (one thread per
pending write request)
- bugfix: [LOGSTASH-181] - output/statsd: set sender properly
- bugfix: [LOGSTASH-173] - GELF output can throw an exception during gelf notify
- bugfix: [LOGSTASH-182] - grep filter should act on all events if no type is
specified.
- bugfix: [LOGSTASH-309] - file output can now write to named pipes (fifo)
1.0.17 (Aug 12, 2011)
- Bugs fixed
- [LOGSTASH-147] - grok filter incorrectly adding fields when a match failed
- [LOGSTASH-151] - Fix bug in routing keys on AMQP
- [LOGSTASH-156] - amqp issue with 1.0.16?
- Improvement
- [LOGSTASH-148] - AMQP input should allow queue name to be specified separately from exchange name
- [LOGSTASH-157] - Plugin doc generator should make regexp config names more readable
- New Feature
- [LOGSTASH-153] - syslog input: make timestamp an optional field
- [LOGSTASH-154] - Make error reporting show up in the web UI
1.0.16 (Aug 18, 2011)
- Fix elasticsearch client problem with 1.0.15 - jruby-elasticsearch gem
version required is now 0.0.10 (to work with elasticsearch 0.17.6)
1.0.15 (Aug 18, 2011)
- IMPORTANT: Upgraded to ElasticSearch 0.17.6 - this brings a number of bug
fixes including an OOM error caused during high index rates in some
conditions.
NOTE: You *must* use same main version of elasticsearch as logstash does,
so if you are still using elasticsearch server 0.16.x - you need to upgrade
your server before the elasticsearch output will work. If you are using
the 'embedded' elasticsearch feature of logstash, you do not need to make
any changes.
- feature: tcp input and output plugins can now operate in either client
(connect) or server (listen) modes.
- feature: new output plugin "statsd" which lets you increment or record
timings from your logs to a statsd agent
- feature: new redis 'pattern_channel' input support for PSUBSCRIBE
- feature: new output plugin "graphite" for taking metrics from events and
shipping them off to your graphite/carbon server.
- feature: new output plugin "ganglia" for shipping metrics to ganglia
gmond server.
- feature: new output plugin "xmpp" for shipping events over jabber/xmpp
- feature: new input plugin "xmpp" for receiving events over jabber/xmpp
- feature: amqp input now supports routing keys.
https://logstash.jira.com/browse/LOGSTASH-122
- feature: amqp output now supports setting routing key dynamically.
https://logstash.jira.com/browse/LOGSTASH-122
- feature: amqp input/output both now support SSL.
https://logstash.jira.com/browse/LOGSTASH-131
- feature: new input plugin "exec" for taking events from executed commands
like shell scripts or other tools.
- feature: new filter plugin "split" for splitting one event into multiple.
It was written primarily for the new "exec" input to allow you to split
the output of a single command run by line into multiple events.
- misc: upgraded jar releases to use JRuby 1.6.3
- bugfix: syslog input shouldn't crash anymore on weird network behaviors
like portscanning, etc.
https://logstash.jira.com/browse/LOGSTASH-130
1.0.14 (Jul 1, 2011)
- feature: new output plugin "loggly" which lets you ship logs to loggly.com
- feature: new output plugin "zabbix" - similar to the nagios output, but
works with the Zabbix monitoring system. Contributed by Johan at
Mach Technology.
- feature: New agent '-e' flag which lets you specify a config in a string.
If you specify no 'input' plugins, default is stdin { type => stdin }
If you specify no 'output' plugins, default is stdout { debug => true }
This is intended to be used for hacking with or debugging filters, but
you can specify an entire config here if you choose.
- feature: Agent '-f' flag now supports directories and globs. If you specify
a directory, all files in that directory will be loaded as a single config.
If you specify a glob, all files matching that glob will be loaded as a
single config.
- feature: gelf output now allows you to override the 'sender'. This defaults
to the source host originating the event, but can be set to anything now.
It supports dynamic values, so you can use fields from your event as the
sender. Contributed by John Vincent
Issue: https://github.com/logstash/logstash/pull/30
- feature: added new feature to libgrok that allows you to define patterns
in-line, like "%{FOO=\d+}" defines 'FOO' match \d+ and captures as such.
To use this new feature, you must upgrade libgrok to at least 1.20110630
Issue: https://logstash.jira.com/browse/LOGSTASH-94
- feature: grok filter now supports 'break_on_match' defaulting to true
(this was the original behavior). If you set it to false, it will attempt
to match all patterns and create new fields as normal. If left default
(true), it will break after the first successful match.
- feature: grok filter now supports parsing any field. You can do either of
these: grok { match => [ "fieldname", "pattern" ] }
or this: grok { fieldname => "pattern" }
The older 'pattern' attribute still means the same thing, and is equivalent
to this: grok { match => [ "@message", "pattern" ] }
Issue: https://logstash.jira.com/browse/LOGSTASH-101
- feature: elasticsearch - when embedded is true, you can now set the
'embedded_http_port' to configure which port the embedded elasticsearch
server listens on. This is only valid for the embedded elasticsearch
configuration. https://logstash.jira.com/browse/LOGSTASH-117
- bugfix: amqp input now reconnects properly when the amqp broker restarts.
- bugfix: Fix bug in gelf output when a fields were not arrays but numbers.
Issue: https://logstash.jira.com/browse/LOGSTASH-113
- bugfix: Fix a bug in syslog udp input due to misfeatures in Ruby's URI
class. https://logstash.jira.com/browse/LOGSTASH-115
- misc: jquery and jquery ui now ship with logstash; previously they were
loaded externally
- testing: fixed some bugs in the elasticsearch test itself, all green now.
- testing: fixed logstash-test to now run properly
1.0.12 (Jun 9, 2011)
- misc: clean up some excess debugging output
- feature: for tcp input, allow 'data_timeout => -1' to mean "never time out"
1.0.11 (Jun 9, 2011)
- deprecated: The redis 'name' and 'queue' options for both input and output
are now deprecated. They will be removed in a future version.
- feature: The redis input and output now supports both lists and channels.
- feature: Refactor runner to allow you to run multiple things in a single
process. You can end each instance with '--' flag. For example, to run one
agent and one web instance:
% java -jar logstash-blah.jar agent -f myconfig -- web
- feature: Add 'embedded' option to the elasticsearch output:
elasticsearch { embedded => true }
Default is false. If true, logstash will run an elasticsearch server
in the same process as logstash. This is really useful if you are just
starting out or only need one one elasticsearch server.
- feature: Added a logstash web backend feature for elasticsearch that tells
logstash to use the 'local' (in process) elasticsearch:
--backend elasticsearch:///?local
- feature: Added 'named_captures_only' option to grok filter. This will have
logstash only keep the captures you give names to - for example %{NUMBER}
won't be kept, but %{NUMBER:bytes} will be.
- feature: Add 'bind_host' option to elasticsearch output. This lets you choose the
address ElasticSearch client uses to bind to - useful if you have a
multihomed server.
- feature: The mongodb output now supports authentication
- bugfix: Fix bug in GELF output that caused the gelf short_message to be set as an
array if it came from a grok value. The short_message field should only
now be a string properly.
- bugfix: Fix bug in grep filter that would drop/cancel events if you had
more than one event type flowing through filters and didn't have a grep
filter defined for each type.
- misc: Updated gem dependencies (tests still pass)
- misc: With the above two points, you can now run a single logstash process
that includes elasticsearch server, logstash agent, and logstash web.
1.0.10 (May 23, 2011)
- Fix tcp input bug (LOGSTASH-88) that would drop connections.
- Grok patterns_dir (filter config) and --grok-patterns-dir (cmdline opt)
are now working.
- GELF output now properly sends extra fields from the log event (prefixed
with a "_") and sets timestamp to seconds-since-epoch (millisecond
precision and time zone information is lost, but this is the format GELF
asks for).
- Inputs support specifying the format of input data (see "format" and
"message_format" input config parameters).
- Grok filter no longer incorrectly tags _grokparsefailure when more than
one grok filter is enabled (for multiple types) or when an event has
no grok configuration for it's type.
- Fix bug where an invalid HTTP Referer: would break grok parsing of the
log line (used to expect %{URI}). Since Referer: is not sanitized in
the HTTP layer, we cannot assume it will be a well formed %{URI}.
1.0.9 (May 18, 2011)
- Fix crash bug caused by refactoring that left 'break' calls in code
that no longer used loops.
1.0.8 (May 17, 2011)
- Remove beanstalk support because the library (beanstalk-client) is GPL3. I
am not a lawyer, but I'm not waiting around to have someone complain about
license incompatibilities.
- fix bug in jar build
1.0.7 (May 16, 2011)
- logstash 'web' now allows you to specify the elasticsearch clustername;
--backend elasticsearch://[host[:port]]/[clustername]
- GELF output now supports dynamic strings for level and facility
https://logstash.jira.com/browse/LOGSTASH-83
- 'amqp' output supports persistent messages over AMQP, now. Tunable.
https://logstash.jira.com/browse/LOGSTASH-81
- Redis input and output are now supported. (Contributed by dokipen)
- Add shutdown processing. Shutdown starts when all inputs finish (like
stdin) The sequence progresses using the same pipeline as the
inputs/filters/outputs, so all in-flight events should finish getting
processed before the final shutdown event makes it's way to the outputs.
- Add retries to unhandled input exceptions (LOGSTASH-84)
1.0.6 (May 11, 2011)
* Remove 'sigar' from monolithic jar packaging. This removes a boatload of
unnecessary warning messages on startup whenever you use elasticsearch
output or logstash-web.
Issue: https://logstash.jira.com/browse/LOGSTASH-79
1.0.5 (May 10, 2011)
* fix queues when durable is set to true
1.0.4 (May 9, 2011)
* Fix bugs in syslog input
1.0.2 (May 8, 2011)
* Fix default-value handling for configs when the validation type is
'password'
1.0.1 (May 7, 2011)
* Fix password auth for amqp and stomp (Reported by Luke Macken)
* Fix default elasticsearch target for logstash-web (Reported by Donald Gordon)
1.0.0 (May 6, 2011)
* First major release.
Reference in a new issue View git blame Copy permalink