mirror of
https://github.com/elastic/logstash.git
synced 2025-04-20 20:57:27 -04:00
1572 lines
82 KiB
Text
1572 lines
82 KiB
Text
1.5.0-rc3
|
|
# general
|
|
- Added back the ability to install plugin gems built locally on top of Logstash. This will
|
|
help plugin developers iterate and test locally without having to publish plugins (#2779)
|
|
- Fixed performance regressions from 1.4.2 especially for configurations which have
|
|
conditionals in filter and output. Throughput numbers are either inline with 1.4.2
|
|
or improved for certain configurations (#2870)
|
|
- Fixed issue in core which was causing Logstash to not shutdown properly (#2796)
|
|
- Added ability to add extra JVM options while running LS. You can use the LS_JAVA_OPTS
|
|
environment variable to add to the default JVM options set out of the box. You could also
|
|
completely overwrite all the default options if you wish by setting JAVA_OPTS before
|
|
starting Logstash (#2942)
|
|
- Fixed a regression from 1.4.2 where removing a tag in filter fails if the input event is
|
|
JSON formatted (#2261)
|
|
- Fixed issue where setting workers > 1 would trigger messages like
|
|
"You are using a deprecated config setting ..." (#2865)
|
|
- Deprecated elasticsearch_http output plugin: All functionality is ported to
|
|
logstash-output-elasticsearch plugin using http protocol (#1757). If you try to use
|
|
the elasticsearch_http plugin, it will log a deprecated notice now.
|
|
|
|
# input
|
|
- File: When shutting down Logstash with file input, it would log a "permissions denied"
|
|
message. We fixed the underlying sinceDB issue while writing to a directory with no
|
|
permissions (#2964, #2935, #2882, file-input#16)
|
|
|
|
# filter
|
|
- Multiline: Fixed an issue where Logstash would crash while processing JSON formatted
|
|
events on Java 8 (#10)
|
|
- Mutate: Fixed issue where you can safely delete/rename fields which can have nil
|
|
values (#2977)
|
|
|
|
# output
|
|
- Deprecate the usage of index_type configuration. Added document_type to be consistent
|
|
with document_id (#102)
|
|
- Added warning when used with config embedded => true. Starting an embedded Elasticsearch
|
|
node is only recommended while prototyping. This should never be used in
|
|
production setting (#99)
|
|
|
|
1.5.0.rc2
|
|
# general
|
|
- Fixed an issue with packaging our release artifacts. Release artifacts were missing jar files
|
|
which caused Logstash to not start with Elasticsearch output while using the node or transport
|
|
client. (#2780)
|
|
|
|
# input
|
|
- Lumberjack: Fixed Logstash crashing because it was using old jls-lumberjack version (#7)
|
|
|
|
1.5.0.rc1
|
|
# general
|
|
- You can now use LS_HOME/patterns directory to add generic patterns for those that may not be
|
|
associated with a particular plugin. Patterns in this dir will be loaded by default (#2225)
|
|
- We now check if the config file is correctly encoded. Otherwise we show a verbose error message
|
|
to convert the failing config file(s) to UTF-8 (#LOGSTASH-1103)
|
|
- Fixed bug in pipeline to gracefully teardown output workers when num workers > 1 (#2180)
|
|
- Fixed nologin path in release debian packages (#2283)
|
|
- Resolved issue where Logstash was crashing for users still using exclude_tags in their output
|
|
configuration (#2323)
|
|
- Updated and releases a new version of the logstash-event gem with latest changes in 1.5.0 RC1
|
|
- Windows: Significantly improved the initial user experience with Windows platform (#2504, #1426).
|
|
Fixed many issues related to File input. Added support for using the plugin
|
|
framework (installing, upgrading, removing)
|
|
- With the release of Kibana 4, we have removed the `bin/logstash web` command and any reference to
|
|
Kibana from Logstash (#2661)
|
|
|
|
# input
|
|
- Added new CouchDB input plugin to fetch data from CouchDB. Using the _changes API, data can be kept
|
|
in sync with any output like Elasticsearch by using this input
|
|
- File: Fixed a number of issues on Windows platform. These include:
|
|
- Resolving file locking issues which was causing log files to not rotate (#1557, #1389)
|
|
- Added support for using SinceDB to record multiple files' last read information (#1902)
|
|
- Fixed encoding issues which applies to many inputs (#2507)
|
|
- Resolved Logstash skipping lines when moving between files which are being followed (#1902)
|
|
- Kafka: Added support for whitelisting and blacklisting topics in the input.
|
|
- EventLog: For Windows, this input gracefully shutsdown if there is a timeout while receiving events
|
|
This also prevents Logstash from being stuck (#1672)
|
|
- Heartbeat: We created a new input plugin for generating heartbeat messages at periodic intervals.
|
|
Use this to monitor Logstash -- you can measure the latency of the pipeline using these heartbeat events,
|
|
and also check for availability
|
|
- S3:
|
|
- Added IAM roles support so you can securely read and write events from S3 without providing your
|
|
AWS credentials (#1575).
|
|
- Added support for using temporary credentials obtained from AWS STS (#1946)
|
|
- Lumberjack: Resolved issue where unrelated events were getting merged into a single event while using
|
|
this input with with the multiline codec (#2016)
|
|
|
|
# filter
|
|
- Mutate:
|
|
- Resolved issue where convert option was creating an extra field in the event (#2268)
|
|
- Fixed issue where mutate with non-existent field was throwing an error (#2379)
|
|
- Multiline: Handled cases where we unintentionally deduplicated lines, such
|
|
as repeated lines in xml messages (#3)
|
|
|
|
# output
|
|
- Elasticsearch:
|
|
- Added support to be more resilient to transient errors in Elasticsearch. Previously, partial
|
|
failures from the bulk indexing functionality were not handled properly. With this fix, we added the ability
|
|
to capture failed requests from Elasticsearch and retry them. Error codes like 429 (too many requests) will
|
|
now be retried by default for 3 times. The number of retries and the interval between consecutive retries
|
|
can be configured (#1631)
|
|
- Logstash does not create a "message.raw" by default whic is usually not_analyzed; this
|
|
helps save disk space (#11)
|
|
- Added sniffing config to be able to list machines in the cluster while using the transport client (#22)
|
|
- S3:
|
|
- Fixed a critical problem in the S3 Output plugin when using the size_file option. This could cause
|
|
data loss and data corruption of old logs ()
|
|
- Added IAM roles support so you can securely read and write events from S3 without providing your AWS
|
|
credentials (#1575)
|
|
- Added support for using temporary credentials obtained from AWS STS (#1946)
|
|
- Fixed a bug when the tags were not set in the plain text format (#1626)
|
|
|
|
# codec
|
|
- Added new Elasticsearch bulk codec which can be used to read data formatted in the Elasticsearch
|
|
Bulk API (multiline json) format. For example, this codec can be used in combination with RabbitMQ input
|
|
to mirror the functionality of the RabbitMQ Elasticsearch river
|
|
- Cloudfront: Added support for handling Amazon CloudFront events
|
|
- Avro: We added a new codec for data serialization (#1566)
|
|
|
|
1.5.0.beta1
|
|
# general
|
|
- Performance improvements: Logstash 1.5.0 is much faster -- we have improved the throughput
|
|
of grok filter in some cases by 100%. In our benchmark testing, using only grok filter and
|
|
ingesting apache logs, throughput increased from 34K eps to 50K eps.
|
|
JSON serialization/deserialization are now implemented using JrJackson library which
|
|
improved performance significantly. Ingesting JSON events 1.3KB in size measured a throughput
|
|
increase from 16Keps to 30K eps. With events 45KB in size, throughput increased from
|
|
850 eps to 3.5K eps
|
|
- Allow spaces in field references like [hello world] (#1513)
|
|
- Add Plugin manager functionality to Logstash which allows to install, delete and
|
|
update Logstash plugins
|
|
- Remove ability to run multiple subcommands from bin/logstash like
|
|
bin/logstash agent -f something.conf -- web (#1747)
|
|
- Fixed Logstash crashing on converting from ASCII to UTF-8. This was caused by charset
|
|
conversion issues in input codec (LOGSTASH-1789)
|
|
- Allow storing 'metadata' to an event which is not sent/encoded on output. This eliminates
|
|
the need for intermediate fields for example, while using date filter. (#1834)
|
|
- Accept file and http uri in -f command line option for specifying config files (#1873)
|
|
- Filters that generated events (multiline, clone, split, metrics) now propagate those events
|
|
correctly to future conditionals (#1431)
|
|
- Bump Kibana version to 3.1.2
|
|
- Fixed file descriptor leaks when using HTTP. The fix prevents Logstash from stalling, and
|
|
in some cases crashing from out-of-memory errors (#1604, LOGSTASH-892)
|
|
|
|
# input
|
|
- Lumberjack: fixed Logstash crashes with Java Out Of Memory because of TCP
|
|
thread leaks (#LOGSTASH-2168)
|
|
- TCP: fixed connection threads leak (#1509)
|
|
- Stdin: prevent overwrite of host field if already present in Event (#1668)
|
|
- S3: AWS credentials can be specfied through environment variables (#1619)
|
|
- Kafka: merge @joekiller's plugin to Logstash to get events from Kafka (#1472)
|
|
- RabbitMQ: fixed march_hare client uses incorrect connection url (LOGSTASH-2276)
|
|
- RabbitMQ: use Bunny 1.5.0+ (#1894)
|
|
- Twitter: added improvements, robustness, fixes. full_tweet option now works, we handle
|
|
Twitter rate limiting errors (#1471)
|
|
- TCP: fixed input host field also contains source port (LOGSTASH-1849)
|
|
- Syslog: if input does not match syslog format, add tag _grokparsefailure_sysloginputplugin
|
|
which can be used to debug (#1593)
|
|
|
|
# filter
|
|
- Mutate: gsub evaluates variables like %{format} in the replacement text (#1529)
|
|
- Grok: "break_on_match => false" option now works correctly (#1547)
|
|
- Grok: allow user@hostname in commonapache log pattern (#1500 #1736)
|
|
- Grok: use optimized ruby-grok library which improves throughput in some cases by 50% (#1657)
|
|
- Date: fixed match defaults to 1970-01-01 when none of the formats matches and
|
|
UNIX format is present in the list (#1236, LOGSTASH-1597)
|
|
- Mutate: fixed confusing error message for invalid type conversion (#1656, LOGSTASH-2003)
|
|
- Date: support parsing almost-ISO8601 patterns like 2001-11-06 20:45:45.123-0000 (without a T)
|
|
which does not match %{TIMESTAMP_ISO8601}
|
|
- KV: allows dynamic include/exclude keys. For example, if an event has a key field and the user
|
|
wants to parse out a value using the kv filter, the user should be able to
|
|
include_keys: [ "%{key}" ]
|
|
- DNS: fixed add_tag adds tags even if filter was unsuccessful (#1785)
|
|
- XML: fixed UndefinedConversionError with UTF-8 encoding (LOGSTASH-2246)
|
|
- Mutate: fixed nested field notation for convert option like
|
|
'convert => [ "[a][0]", "float" ]' (#1401)
|
|
|
|
# output
|
|
- RabbitMQ: fixed crash while running Logstash for longer periods, typically when there's no
|
|
traffic on the logstash<->rabbitmq socket (LOGSTASH-1886)
|
|
- Statsd: fixed issue of converting very small float numbers to scientific notation
|
|
like 9.3e-05 (#1670)
|
|
- Fixed undefined method error when conditional on an output (#LOGSTASH-2288)
|
|
- Elasticsearch output: Added support for multiple hosts in configuration and enhanced stability
|
|
- Elasticsearch output: Logstash will not create a message.raw field by default now. Message field
|
|
is not_analyzed by Elasticsearch and adding a multi-field was essentially doubling the disk space
|
|
required, with no benefit
|
|
- Elasticsearch output: We have improved the security of the Elasticsearch output, input, and filter by
|
|
adding authentication and transport encryption support. In http protocol you can configure SSL/TLS to
|
|
enable encryption and HTTP basic authentication to provide a username and password while making
|
|
requests (#1453)
|
|
- Kafka: merge @joekiller's plugin into Logstash to produce events to Kafka (#1472)
|
|
- File: Added enhancements and validations for destination path. Absolute path cannot start with a dynamic
|
|
string like /%{myfield}/, /test-%{myfield}/
|
|
|
|
1.4.2 (June 24, 2014)
|
|
# general
|
|
- fixed path issues when invoking bin/logstash outside its home directory
|
|
|
|
# input
|
|
- bugfix: generator: fixed stdin option support
|
|
- bugfix: file: fixed debian 7 path issue
|
|
|
|
# codecs
|
|
- improvement: stdin/tcp: automatically select json_line and line codecs with the tcp and stdin streaming imputs
|
|
- improvement: collectd: add support for NaN values
|
|
|
|
# outputs
|
|
- improvement: nagios_nsca: fix external command invocation to avoid shell escaping
|
|
|
|
1.4.1 (May 6, 2014)
|
|
# General
|
|
- bumped Elasticsearch to 1.1.1 and Kibana to 3.0.1
|
|
- improved specs & testing (Colin Surprenant), packaging (Richard Pijnenburg) & doc (James Turnbull)
|
|
- better $JAVA_HOME handling (Marc Chadwick)
|
|
- fixed bin/plugin target dir for when installing out from form logstash home (lr1980)
|
|
- fixed Accessors reset bug in Event#overwrite that was causing the infamous
|
|
"undefined method `tv_sec'" bug with the multiline filter (Colin Surprenant)
|
|
- fixed agent stalling when also using web option (Colin Surprenant)
|
|
- fixed accessing array-indexed event fields (Jonathan Van Eenwyk)
|
|
- new sysv init style scripts based on pleaserun (Richard Pijnenburg)
|
|
- better handling of invalid command line parameters (LOGSTASH-2024, Colin Surprenant)
|
|
- fixed running from a path containing spaces (LOGSTASH-1983, Colin Surprenant)
|
|
|
|
# inputs
|
|
- improvement: rabbitmq: upgraded Bunny gem to 1.1.8, fixes a threading leak and improves
|
|
latency (Michael Klishin)
|
|
- improvement: twitter: added "full_tweet" option (Jordan Sissel)
|
|
- improvement: generator: fixed the example doc (LOGSTASH-2093, Jason Kendall)
|
|
- improvement: imap: option to disable certificate validation (Sverre Bakke)
|
|
|
|
# codecs
|
|
- new: collectd: better performance & error handling than collectd input (Aaron Mildenstein)
|
|
- improvement: graphite: removed unused charset option (Colin Surprenant)
|
|
- improvement: json_spooler: is now deprecated (Colin Surprenant)
|
|
- improvement: proper charset support in all codecs (Colin Surprenant)
|
|
|
|
# filters
|
|
- bugfix: date: on_success actions only when date parsing actually succeed (Philippe Weber)
|
|
- bugfix: multiline: "undefined method `tv_sec'" fix (Colin Surprenant)
|
|
- bugfix: multiline: fix for "undefined method `[]' for nil:NilClass" (#1258, Colin Surprenant)
|
|
- improvement: date: fix specs for non "en" locale (Olivier Le Moal)
|
|
- improvement: grok: better pattern for RFC-5424 syslog format (Guillaume Espanel)
|
|
- improvement: grok: refactored the LOGLEVEL pattern (Lorenzo González)
|
|
- improvement: grok: fix example doc (LOGSTASH-2093, Jason Kendall)
|
|
- improvement: metrics: document .pXX metric (Juarez Bochi)
|
|
|
|
# outputs
|
|
- improvement: rabbitmq: upgraded Bunny gem to 1.1.8, fixes a threading leak and improves
|
|
latency (Michael Klishin)
|
|
- improvement: elasticsearch: start embedded server before creating a client to fix discovery
|
|
problems "waited for 30s ..." (Jordan Sissel)
|
|
- improvement: elasticsearch: have embedded ES use "bind_host" option for "network.host"
|
|
ES config (Jordan Sissel)
|
|
|
|
1.4.0 (March 20, 2014)
|
|
# General
|
|
- We've included some upgrade-specific release notes with more details about
|
|
the tarball changes and contrib packaging here:
|
|
http://logstash.net/docs/1.4.0/release-notes
|
|
- Ships with Kibana 3.0.0
|
|
- Much faster field reference implementation (Colin Surprenant)
|
|
- Fix a bug in character encoding which would cause inputs using non-UTF-8
|
|
codecs to accidentally skip re-encoding the text to UTF-8. This should
|
|
solve a great number of UTF-8-related bugs. (Colin Surprenant)
|
|
- Fixes missing gem for logstash web which was broken in 1.4.0 beta1
|
|
(LOGSTASH-1918, Jordan Sissel)
|
|
- Fix 'help' output being emitted twice when --help is invoked.
|
|
(LOGSTASH-1952, #1168)
|
|
- Logstash now supports deletes! See outputs section below.
|
|
- Update template to fit ES 1.0 API changes (untergeek)
|
|
- Lots of Makefile, gem and build improvements courtesy of untergeek, Faye
|
|
Salwin, mrsolo, ronnocol, electrical, et al
|
|
- Add `env` command so you can run arbitrary commands with the logstash
|
|
environment setup (jordansissel)
|
|
- Bug fixes (lots). Did I mention bug fixes? (Thanks, community!)
|
|
- Elasticsearch 1.0 libraries are now included. See the Elasticsearch
|
|
release notes for details: http://www.elasticsearch.org/downloads/1-0-0/
|
|
- Kibana 3 milestone 5 is included as the 'web' process.
|
|
- An empty --pluginpath directory is now accepted (#917, Richard Pijnenburg)
|
|
- Piles of documentation improvements! A brand new introductory tutorial is
|
|
included, and many of the popular plugins have had their docs greatly
|
|
improved. This effort was lead by Kurt Hurtado with assists by James
|
|
Turnbull, Aaron Mildenstein, Brad Fritz, and others.
|
|
- Testing was another focus of this release. We added many more tests
|
|
to help us prevent regressions and verify expected behavior. Helping with
|
|
this effort was Richard Pijnenburg, Jordan Sissel, and others.
|
|
- The 'debug' setting was removed from most plugins. Prior to this,
|
|
most plugins advertised the availability of this setting but actually
|
|
did not use it (#996, Jordan Sissel).
|
|
- bugfix: --pluginpath now lets you load codecs. (#1077, Sergey Zhemzhitsky)
|
|
|
|
# inputs
|
|
- bugfix: collectd: Improve handling of 'NaN' values (#1015, Pieter Lexis)
|
|
- bugfix: snmptrap: Fixes exception when not specifying yamlmibdir (#950, Andres Koetsier)
|
|
- improvement: Add Multi-threaded workers and queues to UDP input (johnarnold + untergeek)
|
|
- improvement: log4j: port now defaults to 4560, the default log4j
|
|
SocketAppender port. (#757, davux)
|
|
- bugfix: rabbitmq: auto_delete and exclusive now default to 'false'.
|
|
The previous version's defaults caused data loss on logstash restarts.
|
|
Further, these settings are recommended by the RabbitMQ folks. (#864,
|
|
Michael Klishin)
|
|
This change breaks past default behavior, so just be aware. (Michael
|
|
Klishin)
|
|
- bugfix: collectd: fix some type calculation bugs (#905, Pieter Lexis)
|
|
- improvement: collectd: Now supports decryption and signature verification
|
|
(#905, Pieter Lexis)
|
|
- improvement: wmi: now supports remote hosts (#918, Richard Pijnenburg)
|
|
- bugfix: elasticsearch: Long scrollids now work correctly (#935, Jonathan
|
|
Van Eenwyk)
|
|
- bugfix: tcp: the 'host' field is correctly set now if you are using the
|
|
json codec and include a 'host' field in your events (#937, Jordan Sissel)
|
|
- bugfix: file: the 'host' field is correctly set now if you are using the
|
|
json codec and include a 'host' field in your events (#949, Piotr
|
|
Popieluch)
|
|
- bugfix: udp: the 'host' field is correctly set now if you are using the
|
|
json codec and include a 'host' field in your events (#965, Devin
|
|
Christensen)
|
|
- bugfix: syslog: fix regression (#986, Joshua Bussdieker)
|
|
|
|
# codecs
|
|
- improvement: netflow: You can now specify your own netflow field
|
|
definitions using the 'definitions' setting. See the netflow codec
|
|
docs for examples on how to do this. (#808, Matt Dainty)
|
|
|
|
# filters
|
|
- bugfix: clone: Correctly clone events with numeric field values.
|
|
(LOGSTASH-1225, #1158, Darren Holloway)
|
|
- bugfix: zeromq: Add `timeout` and `retries` settings for retrying on
|
|
request failures. Also adds `add_tag_on_timeout` so you can act on retry
|
|
failures. (logstash-contrib#23, Michael Hart)
|
|
- new: fingerprint: Checksum, anonymize, generate UUIDs, etc! A generalized
|
|
solution to replace the following filters: uuid, checksum, and anonymize.
|
|
(#907, Richard Pijnenburg)
|
|
- new: throttle: Allows you to tag or add fields to events that occur with a
|
|
given frequency. One use case is to have logstash email you only once if an
|
|
event occurs at least 3 times in 60 seconds. (#940, Mike Pilone) -
|
|
- improvement: translate: A new 'refresh_interval' setting lets you tell
|
|
logstash to periodically try reloading the 'dictionary_path' file
|
|
without requiring a restart. (#975, Kurt Hurtado)
|
|
- improvement: geoip: Now safe to use with multiple filter workers and
|
|
(#990, #997, LOGSTASH-1842; Avleen Vig, Jordan Sissel)
|
|
- improvement: metrics: Now safe to use with multiple filter workers (#993,
|
|
Bernd Ahlers)
|
|
- bugfix: date: Fix regression that caused times to be local time instead of
|
|
the intended timezone of UTC. (#1010, Jordan Sissel)
|
|
- bugfix: geoip: Fix encoding of fields created by geoip lookups
|
|
(LOGSTASH-1354, LOGSTASH-1372, LOGSTASH-1853, #1054, #1058; Jordan Sissel,
|
|
Nick Ethier)
|
|
|
|
# outputs
|
|
- bugfix: elasticsearch: flush any buffered events on logstash shutdown
|
|
(#1175)
|
|
- feature: riemann: Automatically map event fields to rieman event fields
|
|
(logstash-contrib#15, Byron Pezan)
|
|
- bugfix: lumberjack: fix off-by-one errors causing writes to another
|
|
logstash agent to block indefinitely
|
|
- bugfix: elasticsearch: Fix NameError Socket crash on startup
|
|
(LOGSTASH-1974, #1167)
|
|
- improvement: Added `action` awesomeness to elasticsearch output (#1105, jordansissel)
|
|
- improvement: Implement `protocol => http` in elasticsearch output (#1105, jordansissel)
|
|
- bugfix: fix broken pipe output to allow EBADF instead of EPIPE,
|
|
allowing pipe command to be restarted (#974, Paweł Puterla)
|
|
- improvement: Adding dns resolution to lumberjack output (#1048, Nathan Burns )
|
|
- improvement: added pre- and post-messages to the IRC output (#1111, Lance O'Connor)
|
|
- bugfix: pipe: fix handling of command failures (#1023, #1034, LOGSTASH-1860; ronnocol, Jordan Sissel)
|
|
- improvement: lumberjack: now supports codecs (#1048, LOGSTASH-1680; Nathan Burns)
|
|
|
|
1.3.3 (January 17, 2014)
|
|
# general
|
|
- bugfix: Fix SSL cert load problem on plugins using aws-sdk: S3, SNS, etc.
|
|
(LOGSTASH-1778, LOGSTASH-1787, LOGSTASH-1784, #924; Adam Peck)
|
|
- bugfix: Fix library load problems for aws-sdk (LOGSTASH-1718, #923; Jordan
|
|
Sissel)
|
|
- bugfix: Fix regression introduced in 1.3.2 while trying to improve time
|
|
parsing performance. (LOGSTASH-1732, LOGSTASH-1738, #913; Jordan Sissel)
|
|
- bugfix: rabbitmq: honour the passive option when creating queues.
|
|
(LOGSTASH-1461, Tim Potter)
|
|
|
|
# codecs
|
|
- bugfix: json_lines, json: Fix bug causing invalid json to be incorrectly
|
|
handled with respect to encoding (#920, LOGSTASH-1595; Jordan Sissel)
|
|
|
|
1.3.2 (December 23, 2013)
|
|
# upgrade notes
|
|
- Users of logstash 1.3.0 or 1.3.1 should set 'template_overwrite => true' in
|
|
your elasticsearch (or elasticsearch_http) outputs before upgrading to this
|
|
version to ensure you receive the fixed index template.
|
|
|
|
# general
|
|
- web: don't crash if an invalid http request was sent
|
|
(#878, LOGSTASH-704; Jordan Sissel)
|
|
- Ships with Elasticsearch 0.90.9
|
|
- logstash will now try to make sure the @timestamp field is of the
|
|
correct format.
|
|
- Fix a bug in 1.3.1/1.3.0's elasticsearch index template causing phrase
|
|
searching to not work. Added tests to ensure search behavior works as
|
|
expected with this template. (Aaron Mildenstein, Jordan Sissel)
|
|
- Update README.md to be consistent with Makefile use of JRuby 1.7.8
|
|
- Time parsing in things like the json codec (and other similar parts of
|
|
logstash) are *much* faster now. This fixes a speed regression that was
|
|
introduced in logstash 1.2.0.
|
|
|
|
# filters
|
|
- improvement: date: roughly 20% faster (Jordan Sissel)
|
|
|
|
# outputs
|
|
- new: csv: write csv format to files output. (Matt Gray)
|
|
(This output will become a codec usable with file output in the next
|
|
major version!)
|
|
|
|
1.3.1 (December 11, 2013)
|
|
# general
|
|
- Fix path to the built-in elasticsearch index template
|
|
|
|
1.3.0 (December 11, 2013)
|
|
# general
|
|
- oops: The --help flag now reports help again, instead of barfing an "I need
|
|
help" exception (LOGSTASH-1436, LOGSTASH-1392; Jordan Sissel)
|
|
- Resolved encoding errors caused by environmental configurations, such as
|
|
'InvalidByteSequenceError ... on US-ASCII' (LOGSTASH-1595, #842;
|
|
Jordan Sissel)
|
|
- Fix bug causing "no such file to load -- base64" (LOGSTASH-1310,
|
|
LOGSTASH-1519, LOGSTASH-1325, LOGSTASH-1522, #834; Jordan Sissel)
|
|
- Elasticsearch version 0.90.7
|
|
- Bug fixes galore!
|
|
|
|
## inputs
|
|
- new: collectd: receive metrics from collectd's network protocol
|
|
(#785, Aaron Mildenstein)
|
|
- bugfix: gelf: handle chunked gelf message properly (#718, Thomas De Smedt)
|
|
- bugfix: s3: fix bug in region endpoint setting (#740, Andrea Ascari)
|
|
- bugfix: pipe: restart the command when it finishes (#754, Jonathan Van
|
|
Eenwyk)
|
|
- bugfix: redis: if redis fails, reconnect. (#767, LOGSTASH-1475; Jordan Sissel)
|
|
- feature: imap: add 'content_type' setting for multipart messages and
|
|
choosing the part that becomes the event message. (#784, Brad Fritz)
|
|
- bugfix: zeromq: don't override the 'host' field if the event already
|
|
has one. (Jordan Sissel)
|
|
- bugfix: ganglia: fix regressions; plugin should work again (LOGSTASH-1655,
|
|
#818; Jordan Sissel)
|
|
- bugfix: Fix missing library in sqs input (#775, LOGSTASH-1294; Toby
|
|
Collier)
|
|
|
|
## filters
|
|
- new: unique: removes duplicate values from a given field in an event.
|
|
(#676, Adam Tucker)
|
|
- new: elapsed: time duration between two tagged events. (#713, Andrea Forni)
|
|
- new: i18n: currently supports 'transliterate' which does best-effort
|
|
conversion of text to "plain" letters. Like 'ó' to 'o'. (#671,
|
|
Juarez Bochi)
|
|
- bugfix: restore filter flushing thread (LOGSTASH-1284, #689; Bernd Ahlers)
|
|
- new: elasticsearch: query elasticsearch and update your event based on the
|
|
results. (#707, Jonathan Van Eenwyk)
|
|
- new: sumnumbers: finds all numbers in a message and sums them (#752, Avleen
|
|
Vig)
|
|
- feature: geoip: new field 'location' is GeoJSON derived from the lon/lat
|
|
coordinates for use with elasticsearch, kibana, and anything else that
|
|
understands GeoJSON (#763, Aaron Mildenstein)
|
|
- new: punct: Removes all text except punctuation and stores it in another
|
|
field. Useful for as a means for fingerprinting events. (#813, Guixing Bai)
|
|
- feature: metrics: Make percentiles configurable. Also make rates (1, 5,
|
|
15-minute) optional. (#817, Juarez Bochi)
|
|
|
|
## codecs
|
|
- new: compressed_spooler: batches events and sends/receives them in
|
|
compressed form. Useful over high latency links or with transports
|
|
with higher-than-desired transmission costs. (Avleen Vig)
|
|
- new: fluent: receive data serialized using the Fluent::Logger for easier
|
|
migration away from fluentd or for folks who simply like the logger
|
|
library (#759, Jordan Sissel)
|
|
- new: edn: encode and decode the EDN serialization format. Commonly used
|
|
in Clojure. For more details, see: https://github.com/edn-format/edn
|
|
(#778, Lee Hinman)
|
|
- bugfix: oldlogstashjson: Fix encoding to work correctly. (#788, #795;
|
|
Brad Fritz)
|
|
- bugfix: oldlogstashjson: Fallback to plain text on invalid JSON
|
|
(LOGSTASH-1534, #850; Jordan Sissel)
|
|
|
|
## outputs
|
|
- feature: elasticsearch and elasticsearch_http now will apply a default
|
|
index mapping template (included) which has the settings recommended by
|
|
Elasticsearch for Logstash specifically.
|
|
Configuration options allow disabling this feature and providing a path
|
|
to your own template. (#826, #839; Aaron Mildenstein)
|
|
- feature: elasticsearch_http: optional 'user' and 'password' settings to
|
|
make use of http authentication (LOGSTASH-902, #684; Ian Neubert)
|
|
- new: google_bigquery: upload logs to bigquery for analysis later (Rodrigo
|
|
De Castro)
|
|
- bugfix: datadog_metrics: fix validation bug (#789, Ian Paredes)
|
|
- feature: elasticsearch: new 'transport' setting letting you tell logstash
|
|
to act as a cluster node (default, prior behavior) or as a 'transport
|
|
client'. With the new 'transport' mode, your firewall rules may be simpler
|
|
(unicast, one direction) and transport clients do not show up in your
|
|
cluster node list. (LOGSTASH-102, #841; Jordan Sissel)
|
|
- feature: elasticsearch: new 'bind_port setting for 'node' protocol which
|
|
lets you chose the local port to bind on (#841, Jordan Sissel)
|
|
- bugfix: Fix missing library in sqs input (#775, LOGSTASH-1294; Toby
|
|
Collier)
|
|
|
|
1.2.2 (October 22, 2013)
|
|
# general
|
|
- new 'worker' setting for outputs. This helps improve throughput on
|
|
request-oriented outputs such as redis, rabbitmq, elasticsearch,
|
|
elasticsearch_http, etc. Workers run in separate threads each handling
|
|
events as they come in. This allows you to linearly scale up outputs across
|
|
cores or as blocking-io permits.
|
|
- grok performance is up 600%
|
|
- lots of bug fixes
|
|
- bugfixes to conditionals (#682, Matt Dainty)
|
|
- rabbitmq now replaces the old deprecated amqp plugins. amqp plugins are
|
|
removed.
|
|
- inputs will now do their best to handle text which is encoded differently
|
|
than the charset you have specified (LOGSTASH-1443, Jordan Sissel)
|
|
|
|
## inputs
|
|
- bugfix: udp: respects teardown requests via SIGINT, etc (LOGSTASH-1290,
|
|
Jordan Sissel)
|
|
- bugfix: rabbitmq: disable automatic connection recovery (LOGSTASH-1350,
|
|
#641, #642; Michael Klishin)
|
|
- bugfix: twitter: works again (#640, Bernd Ahlers)
|
|
- compatibility: Restored the old 'format' setting behavior. It is still
|
|
deprecated, but was accidentally removed in 1.2.0. It will be removed
|
|
later, but is restored as part of our backwards-compat promise (Jordan
|
|
Sissel)
|
|
- bugfix: s3: fix LOGSTASH-1321 and LOGSTASH-1319 (Richard Pijnenburg)
|
|
- bugfix: log4j: fix typo (Jordan Sissel)
|
|
- bugfix: rabbitmq: disable automatic connection recover because logstash
|
|
will handle it (LOGSTASH-1350, Michael Klishin)
|
|
- bugfix: heroku: works again (LOGSTASH-1347, #643; Bernd Ahlers)
|
|
- bugfix: tcp: improve detection of closed connections to reduce lost events
|
|
(Jordan Sissel)
|
|
- bugfix: elasticsearch: now works correctly (#670, Richard Pijnenburg)
|
|
- improvement: elasticsearch: make size and scroll time configurable (#670,
|
|
Richard Pijnenburg)
|
|
- improvement: elasticsearch: tunable search type (#670, Richard Pijnenburg)
|
|
- compatibility: restore 'format' setting which was accidentally removed in
|
|
1.2.0. This feature is still deprecated, but it has been restored
|
|
temporarily as part of our backwards compatibility promise. (#706, Jordan
|
|
Sissel)
|
|
- bugfix: syslog: fix socket leakage (#704, Bernd Ahlers)
|
|
- improvement: all aws-related plugins: Add proxy_uri setting (#714, Malthe
|
|
Borch)
|
|
- bugfix: unix: fix variable name crash (#720, Nikolay Bryskin)
|
|
|
|
## codecs
|
|
- new: graphite: parse graphite formated events (Nick Ethier)
|
|
- new: json_lines: parse streams that are lines of json objects (#731, Nick
|
|
Ethier)
|
|
- bugfix: multiline: time is now correctly in UTC. (Jordan Sissel)
|
|
- bugfix: oldlogstashjson: improved conversion of old logstash json to the
|
|
new schema (#654, Jordan Sissel)
|
|
- bugfix: oldlogstashjson: fix typo breaking encoding (#665, Tom Howe)
|
|
- bugfix: json: now assumes json delimited by newline character
|
|
(LOGSTASH-1332, #710; Nick Ethier)
|
|
- improvements: netflow: new target and versions settings (#686, Matt Dainty)
|
|
|
|
## filters
|
|
- performance: grok: 6.3x performance improvement (#681, Jordan Sissel)
|
|
- bugfix: geoip: empty values (nil, empty string) are not put into the event
|
|
anymore. (Jordan Sissel)
|
|
- bugfix: geoip: allow using Maxmind's ASN database (LOGSTASH-1394, #694;
|
|
Bernd Ahlers)
|
|
- improvement: kv: target will now overwrite any existing fields, including
|
|
the source (Jordan Sissel).
|
|
- improvement: Kv: 'prefix' setting now respects sprintf (LOGSTASH-913,
|
|
#647; Richard Pijnenburg)
|
|
- checksum: sha128 was not a valid digest, removed from list
|
|
- feature: metrics: added clear_interval and flush_interval parameters for
|
|
setting flush rates and when to clear metrics (#545)
|
|
- new: collate: group events by time and/or count into a single event. (#609,
|
|
Neway Liu)
|
|
- feature: date: now supports a 'target' field for writing the timestamp into
|
|
a field other than @timestamp. (#625, Jonathan Van Eenwyk)
|
|
- bugfix: riemann: event tagging works again (#631, Marc Fournier)
|
|
- improvement: grok: IPV6 pattern (#623, Matt Dainty)
|
|
- improvement: metrics: add clear_interval and flush_interval settings (#545,
|
|
Juarez Bochi)
|
|
- improvement: useragent: include operating system details (#656, Philip
|
|
Kubat)
|
|
- improvement: csv: new quote_char setting (#725, Alex Markham)
|
|
|
|
## outputs
|
|
- feature: all outputs have a 'worker' setting now that allows you to
|
|
perform more work at the same time. This is useful for plugins like
|
|
elasticsearch_http, redis, etc, which can bottleneck on waiting for
|
|
requests to complete but would otherwise be happy processing more
|
|
simultaneous requests. (#708, Jordan Sissel)
|
|
- bugfix: elasticsearch: requests are now synchronous. This avoid overloading
|
|
the client and server with unlimited in-flight requests. (#688, Jordan
|
|
Sissel)
|
|
- bugfix: elasticsearch_http: fix bug when sending multibyte utf-8 events
|
|
(LOGSTASH-1328, #678, #679, #695; Steve Merrill, Christian Winther,
|
|
NickEthier, Jordan Sissel)
|
|
- performance: elasticsearch_http: http client library uses TCP_NODELAY now
|
|
which dramatically improves performance. (#696, Jordan Sissel)
|
|
- feature: elasticsearch_http now supports a 'replication' setting to
|
|
allow you to choose how you wait for the response. THe default is 'sync'
|
|
which waits for all replica shards to be written. If you set it to 'async'
|
|
then all index requests will respond once only the primary shards have been
|
|
written and the replica shards will be written later. This can improve
|
|
throughput. (#700, Nick Ethier, Jordan Sissel)
|
|
- bugfix: elasticsearch: the default port range is now 9300-9305; the older
|
|
range up to 9400 was unnecessary and could cause problems for the
|
|
elasticsearch cluster in some cases.
|
|
- improvement: aws-based outputs (e.g. cloudwatch) now support proxy uri.
|
|
- bugfix: rabbitmq: disable automatic connection recovery (LOGSTASH-1350)
|
|
(#642)
|
|
- bugfix: riemann: fixed tagging of riemann events (#631)
|
|
- bugfix: s3: fix LOGSTASH-1321 and LOGSTASH-1319 (#636, #645; Richard
|
|
Pijnenburg)
|
|
- bugfix: mongodb: Fix mongodb auth (LOGSTASH-1371, #659; bitsofinfo)
|
|
- bugfix: datadog: Fix time conversion (LOGSTASH-1427, #690; Bernd Ahlers)
|
|
- bugfix: statsd: Permit plain floating point values correctly in the
|
|
config. Example: sample_rate => 0.5 (LOGSTASH-1441, #705; Jordan Sissel)
|
|
- bugfix: syslog: Fix timestamp date formation. 'timestamp' setting is now
|
|
deprecated and the format of the time depends on your rfc selection.
|
|
(LOGSTASH-1423, #692, #739; Jordan Sissel, Bernd Ahlers)
|
|
|
|
## patterns
|
|
- improvement: added IPV6 suppot to IP pattern (#623)
|
|
|
|
1.2.1 (September 7, 2013)
|
|
# general
|
|
- This is primarily a bugfix/stability release based on feedback from 1.2.0
|
|
- web: kibana's default dashboard now works with the new logstash 1.2 schema.
|
|
- docs: updated the tutorials to work in logstash 1.2.x
|
|
- agent: Restored the --configtest flag (unintentionally removed from 1.2.0)
|
|
- deprecation: Using deprecated plugin settings can now advise you on a
|
|
corrective path to take. One example is the 'type' setting on filters and
|
|
outputs will now advise you to use conditionals and give an example.
|
|
- conditionals: The "not in" operator is now supported.
|
|
|
|
## inputs
|
|
- bugfix: pipe: reopen the pipe and retry on any error. (#619, Jonathan Van
|
|
Eenwyk)
|
|
- bugfix: syslog: 'message' field no longer appears as an array.
|
|
- bugfix: rabbitmq: can now bind the queue to the exchange (#624, #628,
|
|
LOGSTASH-1300, patches by Jonathan Tron and Jonathan Van Eenwyk)
|
|
|
|
## codecs
|
|
- compatibility: json: if data given is not valid as json will now be used as
|
|
the "message" of an event . This restores the older behavior when using
|
|
1.1.13's "format => json" feature on inputs. (LOGSTASH-1299)
|
|
- new: netflow: process netflow data (#580, patches by Nikolay Bryskin and
|
|
Matt Dainty)
|
|
|
|
## filters
|
|
- bugfix: multiline: the multiline filter returns! It was unintentionally
|
|
removed from the previous (1.2.0) release.
|
|
- bugfix: json_encode: fix a syntax error in the code. (LOGSTASH-1296)
|
|
- feature: kv: now captures duplicate field names as a list, so 'foo=bar
|
|
foo=baz' becomes the field 'foo' with value ['bar', 'baz'] (an array).
|
|
(#622, patch by Matt Dainty)
|
|
|
|
## outputs
|
|
- new: google_cloud_storage: archive logs to Google Cloud Storage (#572,
|
|
Rodrigo De Castro)
|
|
- bugfix: fixed bug with 'tags' and 'exclude_tags' on outputs that would
|
|
crash if the event had no tags. (LOGSTASH-1286)
|
|
|
|
1.2.0 (September 3, 2013)
|
|
# general
|
|
- The logstash json schema has changed. (LOGSTASH-675)
|
|
For prior logstash users, you will be impacted one of several ways:
|
|
* You should check your elasticsearch templates and update them accordingly.
|
|
* If you want to reindex old data from elasticsearch with the new schema,
|
|
you should be able to do this with the elasticsearch input. Just make
|
|
sure you set 'codec => oldlogstashjson' in your elasticsearch input.
|
|
- The old logstash web ui has been replaced by Kibana 3. Kibana is a far
|
|
superior search and analytics interface.
|
|
- New feature: conditionals! You can now make "if this, then ..." decisions
|
|
in your filters or outputs. See the docs here:
|
|
http://logstash.net/docs/latest/configuration#conditionals
|
|
- A new syntax exists for referencing fields (LOGSTASH-1153). This replaces
|
|
the prior and undocumented syntax for field access (was 'foo.bar' and is
|
|
now '[foo][bar]'). Learn more about this here:
|
|
http://logstash.net/docs/latest/configuration#fieldreferences
|
|
- A saner hash syntax in the logstash config is now supported. It uses the
|
|
perl/ruby hash-rocket syntax: { "key" => "value", ... } (LOGSTASH-728)
|
|
- ElasticSearch version 0.90.3 is included. (#486, Gang Chen)
|
|
- The elasticsearch plugin now uses the bulk index api which should result
|
|
in lower cpu usage as well as higher performance than the previous
|
|
logstash version.
|
|
- Many deprecated features have been removed. If your config caused
|
|
deprecation warnings on startup in logstash v1.1.13, there is a good
|
|
chance that these deprecated settings are now absent.
|
|
- 'type' is no longer a required setting on inputs.
|
|
- New plugin type: codec. Used to implement decoding of events for inputs and
|
|
encoding of events for outputs. Codecs allow us to separate transport (like
|
|
tcp, redis, rabbitmq) from serialization (gzip text, json, msgpack, etc).
|
|
- Improved error messages that try to be helpful. If you see bad or confusing
|
|
error messages, it is a bug, so let us know! (Patch by Nick Ethier)
|
|
- The old 'plugin status' concept has been replaced by 'milestones'
|
|
(LOGSTASH-1137)
|
|
- SIGHUP should cause logstash to reopen it's logfile if you are using the
|
|
--log flag
|
|
|
|
## inputs
|
|
- new: s3: reads files from s3 (#537, patch by Mathieu Guillaume)
|
|
- feature: imap: now marks emails as read (#542, Raffael Schmid)
|
|
- feature: imap: lets you delete read email (#591, Jonathan Van Eenwyk)
|
|
- feature: rabbitmq: now well-supported again (patches by Michael Klishin)
|
|
- bugfix: gelf: work around gelf parser errors (#476, patch by Chris McCoy)
|
|
- broken: the twitter input is disabled because the twitter stream v1 api is
|
|
no longer supported and I couldn't find a replacement library that works
|
|
under JRuby.
|
|
- new: sqlite input (#484, patch by Evan Livingston)
|
|
- improvement: snmptrap: new 'yamlmibdir' setting for specifying an external
|
|
source for MIB definitions. (#477, patch by Dick Davies)
|
|
- improvement: stomp: vhost support (#490, patch by Matt Dainty)
|
|
- new: unix: unix socket input (#496, patch by Nikolay Bryskin)
|
|
- new: wmi: for querying wmi (windows). (#497, patch by Philip Seidel)
|
|
- improvement: sqs: new id_field and md5_field settings (LOGSTASH-1118, Louis
|
|
Zuckerman)
|
|
|
|
## filters
|
|
- feature: grok: 'singles' now defaults to true.
|
|
- bugfix: grep: allow repeating a field in the hash config (LOGSTASH-919)
|
|
- feature: specify timezone in date filter (#470, patch by Philippe Weber)
|
|
- feature: grok setting 'overwrite' now lets you overwrite fields instead
|
|
of appending to them.
|
|
- feature: the useragent filter now defaults to writing results to the top
|
|
level of the event instead of "ua"
|
|
- feature: grok now defaults 'singles' to true, meaning captured fields are
|
|
stored as single values in most cases instead of the old behavior of being
|
|
captured as an array of values.
|
|
- new: json_encoder filter (#554, patch by Ralph Meijer)
|
|
- new: cipher: gives you many options for encrypting fields (#493, patch by
|
|
saez0pub)
|
|
- feature: kv: new settings include_fields and exclude_fields. (patch by
|
|
Piavlo)
|
|
- feature: geoip: new 'target' setting for where to write geoip results.
|
|
(#491, patch by Richard Pijnenburg)
|
|
- feature: dns: now accepts custom nameservers to query (#495, patch by
|
|
Nikolay Bryskin)
|
|
- feature: dns: now accepts a timeout setting (#507, patch by Jay Luker)
|
|
- bugfix: ruby: multiple ruby filter instances now work (#501, patch by
|
|
Nikolay Bryskin)
|
|
- feature: uuid: new filter to add a uuid to each event (#531, Tomas Doran)
|
|
- feature: useragent: added 'prefix' setting to prefix field names created
|
|
by this filter. (#524, patch by Jay Luker)
|
|
- bugfix: mutate: strip works now (#590, Jonathan Van Eenwyk)
|
|
- new: extractnumbers: extract all numbers from a message (#579, patch by
|
|
Pablo Barrera)
|
|
|
|
## outputs
|
|
- new: jira: create jira tickets from an event (#536, patch by Martin Cleaver)
|
|
- feature: rabbitmq: now well-supported again (patches by Michael Klishin)
|
|
- improvement: stomp: vhost support (Patch by Matt Dainty)
|
|
- feature: elasticsearch: now uses the bulk index api and supports
|
|
a tunable bulk flushing size.
|
|
- feature: elasticsearch_http: will now flush when idle instead of always
|
|
waiting for a full buffer. This helps in slow-sender situations such
|
|
as testing by hand.
|
|
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
|
|
- bugfix: email: restored initial really useful documentation
|
|
- improvement: emails: allow @message, @source, @... in match (LOGSTASH-826,
|
|
LOGSTASH-823)
|
|
- feature: email: can now set Reply-To (#540, Tim Meighen)
|
|
- feature: mongodb: replica sets are supported (#389, patch by Mathias Gug)
|
|
- new: s3: New plugin to write to amazon S3 (#439, patch by Mattia Peterle)
|
|
- feature: statsd: now supports 'set' metrics (#513, patch by David Warden)
|
|
- feature: sqs: now supports batching (#522, patch by AaronTheApe)
|
|
- feature: ganglia: add slope and group settings (#583, patch by divanikus)
|
|
|
|
1.1.13 (May 28, 2013)
|
|
## general
|
|
- fixed bug in static file serving for logstash web (LOGSTASH-1067)
|
|
|
|
## outputs
|
|
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
|
|
|
|
1.1.12 (May 7, 2013)
|
|
## filters
|
|
- bugfix: useragent filter now works correctly with the built-in regexes.yaml
|
|
- bugfix: mail output with smtp now works again
|
|
|
|
1.1.11 (May 7, 2013)
|
|
## general
|
|
- This release is primarily a bugfix release for bugs introduced by the
|
|
previous release.
|
|
- Support for Rubinius and MRI exists once again.
|
|
|
|
## inputs
|
|
- bugfix: lumberjack now respects field data again (lumberjack --field foo=bar)
|
|
- bugfix: rabbitmq was broken by the previous release (LOGSTASH-1003,
|
|
LOGSTASH-1038; Patch by Jason Koppe)
|
|
- bugfix: relp: allow multiple client socket connections to RELP input
|
|
(LOGSTASH-707, LOGSTASH-736, LOGSTASH-921)
|
|
|
|
## filters
|
|
- bugfix: geoip was broken by the previous release (LOGSTASH-1013)
|
|
- feature: sleep now accepts an 'every' setting which causes it to
|
|
sleep every N events. Example; sleep every 10 events: every => 10.
|
|
- feature: grok now permits dashes and dots in captures, such as
|
|
%{WORD:foo-bar}.
|
|
- bugfix: useragent filter now ships with a default regexes.yaml file
|
|
that is used by default unless you otherwise specify (LOGSTASH-1051)
|
|
- bugfix: add_field now correctly sets top-level fields like @message
|
|
- bugfix: mutate 'replace' now sets a field regardless of whether or not
|
|
it exists.
|
|
- feature: new mutate 'update' setting to change a field's value but
|
|
only if that field exists already.
|
|
|
|
## outputs
|
|
- feature: irc output now supports 'secure' setting to use ssl (LOGSTASH-139)
|
|
- feature: nagios_nsca has new setting 'message_format'
|
|
- bugfix: fix graphite plugin broken in 1.1.10 (LOGSTASH-968)
|
|
- bugfix: elasticsearch_http was broken in 1.1.10 (LOGSTASH-1004)
|
|
- bugfix: rabbitmq was broken by the previous release (LOGSTASH-1003,
|
|
LOGSTASH-1038; Patch by Jason Koppe)
|
|
- feature: hipchat 'notify' setting now called 'trigger_notify' (#467, patch
|
|
by Richard Pijnenburg)
|
|
|
|
1.1.10 (April 16, 2013)
|
|
## general
|
|
- On linux, all threads will set their process names so you can identify
|
|
threads in tools like top(1).
|
|
- Java 5 is no longer supported (You must use Java 6 or newer).
|
|
- Windows line terminators (CRLF) are now accepted in config files.
|
|
- All AWS-related plugins now have the same configuration options:
|
|
region, access_key_id, secret_access_key, use_ssl, and
|
|
aws_credentials_file. Affected plugins: cloudwatch output,
|
|
sns output, sqs output, sqs input. (LOGSTASH-805)
|
|
- Lots of documentation fixes (James Turnbull, et al)
|
|
- The amqp plugins are now named 'rabbitmq' because it *only* works
|
|
with rabbitmq. The old 'amqp' name should still work, but it will
|
|
be removed soon while 'rabbitmq' will stay. (Patches by Michael Zaccari)
|
|
- New flag '--configtest' to test config and exit. (Patch by Darren Patterson)
|
|
- Improved error feedback logstash gives to you as a user.
|
|
|
|
## inputs
|
|
- new: elasticsearch: this input allows you to stream search results from
|
|
elasticsearch; it uses the Scroll API.
|
|
- new: websocket. Currently supports acting as a websocket client.
|
|
- new: snmptrap, to receive SNMP traps (patch by Paul Czar)
|
|
- new: varnishlog input to read from the Varnish Cache server's shared memory
|
|
log (LOGSTASH-978, #422; Louis Zuckerman)
|
|
- new: graphite input. Supports the plain text carbon tcp protocol.
|
|
- new: imap input. Read mail!
|
|
- feature: twitter: supports http proxying now (#276, patch by Richard
|
|
Pijnenburg)
|
|
- feature: loggly: supports http proxying now (#276, patch by Richard
|
|
Pijnenburg)
|
|
- feature: tcp: ssl now supported! (#318, patch by Matthew Richardson)
|
|
- feature: redis: now supports 'batch_count' option for doing bulk fetches
|
|
from redis lists. Requires Redis 2.6.0 or higher. (#320, patch by Piavlo)
|
|
- feature: irc: will use ssl if you set 'secure' (#393, patch by Tomas Doran)
|
|
- bugfix: log4j: respect add_fields (LOGSTASH-904, #358)
|
|
- bugfix: eventlog: input should now work
|
|
- bugfix: irc: passwords now work (#412, Nick Ethier)
|
|
|
|
## filters
|
|
- new: useragent: parses user agent strings in to structured data based on
|
|
BrowserScope data (#347, patch by Dan Everton)
|
|
- new: sleep: sleeps a given amount of time before passing the event.
|
|
Useful for rate limiting or replay simulation.
|
|
- new: ruby: experimental ruby plugin that lets you call custom ruby code
|
|
on every event.
|
|
- new: translate: for mapping values (#335, patch by Paul Czar)
|
|
- new: clone: creates a copy of the event.
|
|
- feature: grok: Adds tag_on_failure setting so you can prevent grok from
|
|
tagging events on failure. (#328, patch by Neil Prosser)
|
|
- deprecated: grok: deprecated the --grok-patterns-path flag (LOGSTASH-803)
|
|
- feature: date: nested field access is allowed now
|
|
- feature: csv, xml, kv, json, geoip: new common settings!
|
|
(LOGSTASH-756, #310, #311, #312, #383, #396; patches by Richard Pijnenburg)
|
|
source - what field the text comes from
|
|
target - where to store the parse result.
|
|
- feature: csv: new setting: columns - labels for each column parsed.
|
|
- bugfix: geoip: The built-in geoip database should work now (#326, patch
|
|
by Vincent Batts)
|
|
- bugfix: kv filter now respects add_tag, etc (LOGSTASH-935)
|
|
|
|
## outputs
|
|
- new: hipchat output (#428, Cameron Stokes)
|
|
- bugfix: mongo would fail to load bson_java support (LOGSTASH-849)
|
|
- bugfix: tags support to gelf output. Returns tags as _tags field
|
|
(LOGSTASH-880, patch by James Turnbull)
|
|
- bugfix: elasticsearch: Fix a race condition. (#340, patch by Raymond Feng)
|
|
- improvement: http: now supports a custom 'message' format for building your
|
|
own http bodies from an event. (#319, patch by Christian S)
|
|
- bugfix: Fix opentsdb output (LOGSTASH-689, #317; patch by Emmet Murphy)
|
|
- improvement: http output now supports a custom message format with
|
|
the 'message' setting (Patch by Christian Schröder)
|
|
- graphite output now lets you ship the whole (or part) of an event's fields
|
|
to graphite as metric updates. (#350, patch by Piavlo)
|
|
- email output now correctly defaults to not using authentication
|
|
(LOGSTASH-559, #365; patch by Stian Mathiassen)
|
|
- bugfix: file output now works correctly on fifos
|
|
- bugfix: irc passwords now work (#412, Nick Ethier)
|
|
- improvement: redis output now supports congestion detection. If
|
|
it appears nothing is consuming from redis, the output will stall
|
|
until that problem is resolved. This helps prevent a dead reader
|
|
from letting redis fill up memory. (Piavlo)
|
|
- feature: boundary: New 'auto' setting. (#413, Alden Jole)
|
|
|
|
1.1.9 (January 10, 2013)
|
|
## inputs
|
|
- bugfix: all inputs: fix bug where some @source values were not valid urls
|
|
|
|
## filters
|
|
- bugfix: mutate: skip missing fields in 'convert' (#244, patch by Ralph Meijer)
|
|
|
|
## outputs
|
|
- improvement: gelf: new tunable 'ignore_metadata' flag to set which fields
|
|
to ignore if ship_metadata is set. (#244, patch by Ralph Meijer)
|
|
- improvement: gelf: make short_message's field name tunable (#244, patch by
|
|
Ralph Meijer)
|
|
|
|
1.1.8 (January 10, 2013)
|
|
## general
|
|
- patched another work around for JRUBY-6970 (LOGSTASH-801)
|
|
|
|
## inputs
|
|
- bugfix: tcp: 'Address in use' errors now report the host/port involved.
|
|
(LOGSTASH-831)
|
|
- bugfix: zeromq: fix bug where an invalid url could be given as a source
|
|
(LOGSTASH-821, #306)
|
|
|
|
## outputs
|
|
- bugfix: elasticsearch_river: it now resolves evaluates %{} variables in
|
|
index and index_type settings. (LOGSTASH-819)
|
|
|
|
1.1.7 (January 3, 2013)
|
|
## inputs
|
|
- fix bug where @source_host was set to 'false' in many cases.
|
|
|
|
## outputs
|
|
- improvement: redis: shuffle_hosts is now enabled by default
|
|
|
|
1.1.6 (January 2, 2013)
|
|
## Overview of this release:
|
|
- new inputs: drupal_dblog.
|
|
- new filters: anonymize, metrics.
|
|
- new outputs: syslog, cloudwatch.
|
|
- new 'charset' setting for all inputs. This should resolve all known encoding
|
|
problems. The default charset is UTF-8.
|
|
- grok now captures (?<somename>...) regexp into 'somename' field
|
|
- Elasticsearch 0.20.2 is included. This means you are required to upgrade
|
|
your elasticsearch cluster to 0.20.2. If you wish to continue using an old
|
|
version of elasticsearch, you should use the elasticsearch_http plugin
|
|
instead of the elasticsearch one.
|
|
|
|
## general
|
|
- fixed internal dependency versioning on 'addressable' gem (LOGSTASH-694)
|
|
- fixed another case of 'watchdog timeout' (LOGSTASH-701)
|
|
- plugin flags are now deprecated. The grok filter (--grok-pattern-path) was
|
|
the only plugin to make use of this.
|
|
- the grok filter has improved documentation
|
|
- lots of documentation fixes (James Turnbull, Louis Zuckerman)
|
|
- lots of testing improvements (Philippe Weber, Laust Rud Jacobsen)
|
|
- all 'name' settings have been deprecated in favor of more descriptive
|
|
settings (LOGSTASH-755)
|
|
- JRuby upgraded to 1.7.1
|
|
- removed use of bundler
|
|
- Fixed timestamp parsing in MRI (patch by Rene Lengwinat)
|
|
|
|
## inputs
|
|
- All inputs now have a 'charset' setting to help you inform logstash of the
|
|
text encoding of the input. This is useful if you have Shift_JIS or CP1251
|
|
encoded log files. This should help resolve the many UTF-8 bugs that were
|
|
reported recently. The default charset is UTF-8.
|
|
- new: drupal_dblog: read events from a DBLog-enabled Drupal. (#251, Patch by
|
|
theduke)
|
|
- bugfix: zeromq: 'topology' is now a required setting
|
|
- bugfix: lumberjack: client connection closing is now handled properly.
|
|
(Patch by Nick Ethier)
|
|
- misc: lumberjack: jls-lumberjack gem updated to 0.0.7
|
|
- bugfix: stomp: fix startup problems causing early termination (#226
|
|
- bugfix: tcp: the 'source host' for events is now the client ip:port that
|
|
sent it, instead of the listen address that received it. (LOGSTASH-796)
|
|
- improvement: tcp: the default data_timeout is now -1 (never timeout).
|
|
This change was made because read timeouts were causing data loss, and
|
|
logstash should avoid losing events by default.
|
|
- improvement: amqp: the 'name' setting is now called 'queue' (#274)
|
|
- improvement: eventlog: the 'name' setting is now called 'logfile' (#274)
|
|
- bugfix: log4j: fix stacktrace reading (#253, patch by Alex Arutyunyants)
|
|
|
|
## filters
|
|
- new: anonymize: supports many hash mechanisms (murmur3, sha1, md5, etc) as
|
|
well as IP address anonymization (#280, #261; patches by Richard Pijnenburg
|
|
and Avishai Ish-Shalom)
|
|
- new: metrics: allows you to aggregate metrics from events and emit them
|
|
periodically. Think of this like 'statsd' but implemented as a logstash
|
|
filter instead of an external service.
|
|
- feature: date: now accepts 'match' as a setting. Use of this is preferable
|
|
to the old syntax. Where you previously had 'date { somefield =>
|
|
"somepattern" }' you should now do: 'date { match => [ "somefield",
|
|
"somepattern" ] }'. (#248, LOGSTASH-734, Patch by Louis Zuckerman)
|
|
- feature: grok: now accepts (?<foo>...) named captures. This lets you
|
|
compose a pattern in the grok config without needing to define it in a
|
|
patterns file. Example: (?<hostport>%{HOST}:%{POSINT}) to capture 'hostport'
|
|
- improvement: grok: allow '$' in JAVACLASS pattern (#241, patch by Corry
|
|
Haines)
|
|
- improvement: grok: can now match against number types. Example, if you're
|
|
sending a json format event with { "status": 403 } you can now grok that
|
|
field. The number is represented as a string "403" before pattern matching.
|
|
- bugfix: date: Fix a bug that would crash the pipeline if no date pattern
|
|
matched. (LOGSTASH-705)
|
|
- feature: kv: Adds field_split, value_split, prefix, and container
|
|
settings. (#225, patch by Alex Wheeler)
|
|
- bugfix: mutate: rename on a nonexistant field now does nothing as expected.
|
|
(LOGSTASH-757)
|
|
- bugfix: grok: don't tag an event with _grokparsefailure if it's already so
|
|
(#248, patch by Greg Brockman)
|
|
- feature: mutate: new settings - split, join, strip. "split" splits a field
|
|
into an array. "join" merges an array into a string. "strip" strips leading and
|
|
trailing whitespace. (Patch by Avishai Ish-Shalom)
|
|
|
|
## outputs
|
|
- new: syslog output supporting both RFC3164 and RFC5424 (#180, patch by
|
|
Rui Alves)
|
|
- new: cloudwatch output to emit metrics and other events to Amazon CloudWatch.
|
|
(LOGSTASH-461, patch by Louis Zuckerman)
|
|
- feature: stdout: added 'message' setting for configuring the output message
|
|
format. The default is same behavior as before this feature.
|
|
- feature: http: added 'format' option to select 'json' or form-encoded
|
|
request body to send with each request.
|
|
- feature: http: added 'content_Type' option set the Content-Type header.
|
|
This defaults to "application/json" if the 'format' is 'json'. Will default
|
|
to 'application/x-www-form-urlencoded' if the 'format' is 'form'
|
|
- bugfix: zeromq: 'topology' is now a required setting
|
|
- feature: mongodb: new setting 'isodate' that, when true, stores the
|
|
@timestamp field as a mongodb date instead of a string. (#224, patch by
|
|
Kevin Amorin)
|
|
- improvement: gelf: Allow full_message gelf property to be overridden (#245,
|
|
patch by Sébastien Masset)
|
|
- misc: lumberjack: jls-lumberjack gem updated to 0.0.6
|
|
- feature: nagios: New 'nagios_level' setting to let you change the level
|
|
of the passive check result sent to nagios. (#298, Patch by James Turnbull)
|
|
- feature: elasticsearch, elasticsearch_http, elasticsearch_river: new setting
|
|
'document_id' for explicitly setting the document id in each write to
|
|
elasticsearch. This is useful for overwriting existing documents.
|
|
- improvement: elasticsearch_river: 'name' is now 'queue' (#274)
|
|
- improvement: amqp: 'name' is now 'exchange' (#274)
|
|
- bugfix: the websocket output works again (supports RFC6455)
|
|
|
|
1.1.5 (November 10, 2012)
|
|
## Overview of this release:
|
|
* New inputs: zenoss, gemfire
|
|
* New outputs: lumberjack, gemfire
|
|
* Many UTF-8 crashing bugs were resolved
|
|
|
|
## general
|
|
- new runner command 'rspec' - lets you run rspec tests from the jar
|
|
This means you should now be able to write external tests that execute your
|
|
logstash configs and verify functionality.
|
|
- "file not found" errors related to paths that had "jar:" prefixes should
|
|
now work. (Fixes LOGSTASH-649, LOGSTASH-642, LOGSTASH-655)
|
|
- several plugins received UTF-8-related fixes (file, lumberjack, etc)
|
|
File bugs if you see any UTF-8 related crashes.
|
|
- 'json_event' format inputs will now respect 'tags' (#239, patch by
|
|
Tim Laszlo)
|
|
- logstash no longer uses nor recommends bundler (see 'gembag.rb'). The
|
|
Gemfile will be purged in the near future.
|
|
- amqp plugins are now marked 'unsupported' as there is no active maintainer
|
|
nor is there source of active support in the community. If you're interested
|
|
in maintainership, please email the mailling list or contact Jordan!
|
|
|
|
## inputs
|
|
- irc: now stores irc nick
|
|
- new: zenoss (#232, patch by Chet Luther)
|
|
- new: gemfire (#235, patch by Andrea Campi)
|
|
- bugfix: udp: skip close() call if we're already closed (#238, patch by kcrayon)
|
|
|
|
## filters
|
|
- bugfix: fix for zeromq filter initializer (#237, patch by Tom Howe)
|
|
|
|
## outputs
|
|
- new: lumberjack output (patch by Nick Ethier)
|
|
- new: gemfire output (#234, patch by Andrea Campi)
|
|
- improved: nagios_ncsa (patch by Tomas Doran)
|
|
- improved: elasticsearch: permit setting 'host' even if embedded. Also set the
|
|
host default to 'localhost' when using embedded. These fixes should help resolve
|
|
issues new users have when their distros surprisingly block multicast by
|
|
default.
|
|
- improved: elasticsearch: failed index attempts will be retried
|
|
- improved: irc: new 'password' setting (#283, patch by theduke)
|
|
|
|
1.1.4 (October 28, 2012)
|
|
## Overview of this release:
|
|
- bug fixes mostly
|
|
|
|
## filters
|
|
- date: Fix crashing on date filter failures. Wrote test to cover this case.
|
|
(LOGSTASH-641)
|
|
- grok: Improve QUOTEDSTRING pattern to avoid some more 'watchdog timeout' problems
|
|
|
|
## outputs
|
|
- nagios_nsca: Allow check status to be set from the event (#228, patch by
|
|
Tomas Doran)
|
|
- elasticsearch_http: Fix OpenSSL::X509::StoreError (LOGSTASH-642)
|
|
|
|
1.1.3 (October 22, 2012)
|
|
- rebuilt 1.1.2 for java 5 and 6
|
|
|
|
1.1.2 (October 22, 2012)
|
|
## Overview of this release:
|
|
* New input plugins: lumberjack, sqs, relp
|
|
* New output plugins: exec, sqs
|
|
* New filter plugins: kv, geoip, urldecode, alter
|
|
* file input supports backfill via 'start_position'
|
|
* filter watchdog timer set to 10 seconds (was 2 seconds)
|
|
|
|
## general
|
|
- Stopped using 'Gemfile' for dependencies, the logstash.gemspec has returned.
|
|
(Patch by Grant Rogers)
|
|
- New 'logstash-event.gemspec' for generating logstash events in your own
|
|
ruby programs (Patch by Garry Shutler)
|
|
- Wildcard config files are now sorted properly (agent -f
|
|
/etc/logstash/*.conf)
|
|
- The old '-vvv' setting ruby's internal $DEBUG is now gone. It was causing
|
|
too much confusion for users due to noise.
|
|
- Improved 'logstash event' creation speed by 3.5x
|
|
- Now uses JRuby 1.7.0
|
|
- Now ships with Elasticsearch 0.19.10
|
|
|
|
## inputs
|
|
- bugfix: redis: [LOGSTASH-526] fix bug with password passing
|
|
- new: lumberjack: for use with the lumberjack log shipper
|
|
(https://github.com/jordansissel/lumberjack)
|
|
- new: sqs: Amazon SQS input (Patch by Sean Laurent, #211)
|
|
- new: relp: RELP (rsyslog) plugin (Patch by Mike Worth, #177)
|
|
- file input: sincedb path is now automatically generated if not specified.
|
|
This helps work around a problem where two file inputs don't specify a
|
|
sincedb_path would clobber eachother (LOGSTASH-554)
|
|
- file input: no longer crashes if HOME is not set in env (LOGSTASH-458)
|
|
- log4j input: now supports MDC 'event properties' which are stored as fields
|
|
in the logstash event. (#216, #179. Patches by Charles Robertson and Jurjan
|
|
Woltman)
|
|
- pipe input: should work now.
|
|
|
|
## filters
|
|
- new: kv: useful for parsing log formats taht use 'foo=bar baz=fizz' and
|
|
similar key-value-like things.
|
|
- new: urldecode: a filter for urldecoding fields in your event. (Patch by
|
|
Joey Imbasciano, LOGSTASH-612)
|
|
- new: geoip: query a local geoip database for location information (Patch by
|
|
Avishai Ish-Shalom, #208)
|
|
- improvement: zeromq: an empty reply is now considered as a 'cancel this
|
|
event' operation (LOGSTASH-574)
|
|
- bugfix: mutate: fix bug in uppercase and lowercase feature that would
|
|
prevent it from actually doing the uppercasing/lowercasing.
|
|
- improvement: mutate: do the 'remove' action last (LOGSTASH-543)
|
|
- feature: grok: new 'singles' config option which, when true, stores
|
|
single-value fields simply as a single value rather than as an array, like
|
|
[value]. (LOGSTASH-185)
|
|
- grok patterns: the URIPARAM pattern now includes pipe '|' as a valid
|
|
character. (Patch by Chris Mague)
|
|
- grok patterns: improve haproxy log patterns (Patch by Kevin Nuckolls)
|
|
- grok patterns: include 'FATAL' as a valid LOGLEVEL match
|
|
(patch by Corry Haines)
|
|
- grok patterns: 'ZONE' is no longer captured by name in the HTTPDATE pattern
|
|
- new: alter: adds some conditional field modification as well as a
|
|
'coalesce' feature which sets the value of a field to the first non-null
|
|
value given in a list. (Patch by Francesco Salbaroli)
|
|
- improvement: date: add TAI64N support
|
|
- improvement: date: subsecond precision on UNIX timestamps is retained on
|
|
conversion (#213, Patch by Ralph Meijer)
|
|
- improvement: date: Add locale setting; useful for day/month name parsing.
|
|
(#100, Patch by Christian Schröder)
|
|
|
|
## outputs
|
|
- new: exec: run arbitrary commands based on an event.
|
|
- new: sqs: Amazon SQS output (Patch by Sean Laurent, #211)
|
|
- bugfix: redis: [LOGSTASH-526] fix bug with password passing
|
|
- improvement: redis: [LOGSTASH-573] retry on failure even in batch-mode. This
|
|
also fixes a prior bug where an exception in batch mode would cause logstash
|
|
to crash. (Patch by Alex Dean)
|
|
- improvement: riemann: metric and ttl values in riemann_event now support
|
|
sprintf %{foo} values. (pull #174)
|
|
- improvement: stdout: new 'dots' debug_format value emits one dot per event
|
|
useful for tracking event rates.
|
|
- gelf output: correct severity level mappings (patch by Jason Koppe)
|
|
- xmpp output: users and rooms are separate config settings now (patch by
|
|
Parker DeBardelaben)
|
|
- improvement: redis: 'host' setting now accepts a list of hosts for failover
|
|
of writes should the current host go down. (#222, patch by Corry Haines)
|
|
|
|
1.1.1 (July 14, 2012)
|
|
## Overview of this release:
|
|
* New input plugins: generator, heroku, pipe, ganglia, irc
|
|
* New output plugins: juggernaut, metricscatcher, nagios_ncsa, pipe,
|
|
opentsdb, pagerduty, irc
|
|
* New filter plugins: zeromq, environment, xml, csv, syslog_pri
|
|
* Fixes for gelf output
|
|
* Support for more than 1 filter worker (agent argument "-w")
|
|
|
|
## IMPORTANT CHANGES FOR UPGRADES FROM 1.1.0
|
|
- zeromq input and output rewritten
|
|
The previous zeromq support was an MVP. It has now been rewritten into
|
|
something more flexible. The configuration options have changed entirely.
|
|
While this is still listed as `experimental`, we don't predict any more
|
|
configuration syntax changes. The next release will bump this to beta.
|
|
- unix_timestamp
|
|
Previously, several plugins did not work as expected on MRI due to the
|
|
usage of the JRuby-only Jodatime library. We now have a contributed fix
|
|
for a slower parser on MRI/CRuby!
|
|
- elasticsearch version is now 0.19.8
|
|
This means your elasticsearch cluster must be running 0.19.x for
|
|
compatibility reasons.
|
|
- grok pattern %{POSINT} used to match '0' -- now it does not. If you want
|
|
to match non-negative integers, there is now a %{NONNEGINT} pattern.
|
|
- bug in file input fixed that led to an extra leading slash in @source_path.
|
|
Previously, file input would have @source = 'file://host//var/log/foo' and
|
|
@source_path = '//var/log/foo'; now @source = 'file://host/var/log/foo'
|
|
and @source_path = '/var/log/foo'. [LOGSTASH-501]
|
|
- file input now rejects relative paths. [LOGSTASH-503]
|
|
- event sprintf can now look inside structured field data. %{foo.bar} will
|
|
look in the event field "foo" (if it is a hash) for "bar". To preserve
|
|
compatibility, we first look for a top-level key that matches exactly
|
|
(so %{foo.bar} will first look for a field named "foo.bar", then look for
|
|
"bar" under "foo").
|
|
|
|
## general
|
|
- NOTE: gemspec removed; deploying logstash as a gem hasn't been supported
|
|
for a while.
|
|
- feature: logstash sub-commands "irb" and "pry" for an interactive debug
|
|
console, useful to debug jruby when running from the monolithic jar
|
|
- misc: newer cabin gem for logging
|
|
- misc: initial support for reporting internal metrics (currently outputs
|
|
to INFO log; eventually will be an internal event type)
|
|
- misc: added a "thread watchdog" to detect hanging filter workers, and
|
|
crash logstash w/an informational message
|
|
- misc: jar is built with jruby 1.6.7.2
|
|
- misc: better shutdown behavior when there are no inputs/plugins running
|
|
- feature: logstash web now uses relative URLs; useful if you want to
|
|
reverseproxy with a path other than "/"
|
|
|
|
## inputs
|
|
- bugfix: stdin: exit plugin gracefully on EOF
|
|
- feature: [LOGSTASH-410] - inputs can now be duplicated with the
|
|
'threads' parameter (where supported)
|
|
- bugfix: [LOGSTASH-490] - include cacert.pem in jar for twitter input
|
|
- feature: [LOGSTASH-139] - support for IRC
|
|
|
|
## filters
|
|
- feature: all filters support 'remove_tag' (remove tags on success)
|
|
- feature: all filters support 'exclude_tags' (inverse of 'tags')
|
|
- bugfix: [LOGSTASH-300] - bump grok pattern replace limit to 1000,
|
|
fixes "deep recursion pattern compilation" problems
|
|
- bugfix: [LOGSTASH-375] - fix bug in grep: don't drop when field is nil
|
|
and negate is true
|
|
- bugfix: [LOGSTASH-386] - fix some grok patterns for haproxy
|
|
- bugfix: [LOGSTASH-446] - fix grok %{QUOTEDSTRING} pattern, should fix
|
|
some grok filter hangs
|
|
- bugfix: some enhancements to grok pattern %{COMBINEDAPACHELOG}
|
|
- bugfix: grok: %{URIPATH} and %{URIPARAM} enhancements
|
|
- feature: grok: add %{UUID} pattern
|
|
- bugfix: grok: better error message when expanding unknown %{pattern}
|
|
- feature: mutate: now supports a 'gsub' operation for applying a regexp
|
|
substitution on event fields
|
|
|
|
## outputs
|
|
- bugfix: [LOGSTASH-351] - fix file input on windows
|
|
- feature: [LOGSTASH-356] - make file output flush intervals configurable
|
|
- feature: [LOGSTASH-392] - add 'field' attribute to restrict which fields
|
|
get sent to an output
|
|
- feature: [LOGSTASH-374] - add gzip support to file output
|
|
- bugfix: elastic search river now respects exchange_type and queue_name
|
|
- bugfix: ganglia plugin now respects metric_type
|
|
- bugfix: GELF output facility fixes; now defaults to 'logstash-gelf'
|
|
- feature: [LOGSTASH-139] - support for IRC
|
|
- bugfix: es_river: check river status after creation to verify status
|
|
- feature: es: allow setting node_name
|
|
- feature: redis: output batching for list mode
|
|
|
|
1.1.0.1 (January 30, 2012)
|
|
## Overview of this release:
|
|
* date filter bugfix: [LOGSTASH-438] - update joda-time to properly
|
|
handle leap days
|
|
|
|
1.1.0 (January 30, 2012)
|
|
## Overview of this release:
|
|
* New input plugins: zeromq, gelf
|
|
* New filter plugins: mutate, dns, json
|
|
* New output plugins: zeromq, file
|
|
* The logstash agent now runs also in MRI 1.9.2 and above
|
|
|
|
This is a large release due to the longevity of the 1.1.0 betas.
|
|
We don't like long releases and will try to avoid this in the future.
|
|
|
|
## IMPORTANT CHANGES FOR UPGRADES FROM 1.0.x
|
|
- grok filter: named_captures_only now defaults to true
|
|
This means simple patterns %{NUMBER} without any other name will
|
|
now not be included in the field set. You can revert to the old
|
|
behavior by setting 'named_captures_only => false' in your grok
|
|
filter config.
|
|
- grok filter: now uses Ruby's regular expression engine
|
|
The previous engine was PCRE. It is now Oniguruma (Ruby). Their
|
|
syntaxes are quite similar, but it is something to be aware of.
|
|
- elasticsearch library upgraded to 0.18.7
|
|
This means you will need to upgrade your elasticsearch servers,
|
|
if any, to the this version: 0.18.7
|
|
- AMQP parameters and usage have changed for the better. You might
|
|
find that your old (1.0.x) AMQP logstash configs do not work.
|
|
If so, please consult the documentation for that plugin to find
|
|
the new names of the parameters.
|
|
|
|
## general
|
|
- feature: [LOGSTASH-158] - MRI-1.9 compatible (except for some
|
|
plugins/functions which will throw a compatibility exception) This means
|
|
you can use most of the logstash agent under standard ruby.
|
|
- feature: [LOGSTASH-118] - logstash version output (--version or -V for
|
|
agent)
|
|
- feature: all plugins now have a 'plugin status' indicating the expectation
|
|
of stability, successful deployment, and rate of code change. If you
|
|
use an unstable plugin, you will now see a warning message on startup.
|
|
- bugfix: AMQP overhaul (input & output), please see docs for updated
|
|
config parameters.
|
|
- bugfix: [LOGSTASH-162,177,196] make sure plugin-contained global actions
|
|
happen serially across all plugins (with a mutex)
|
|
- bugfix: [LOGSTASH-286] - logstash agent should not truncate logfile on
|
|
startup
|
|
- misc: [LOGSTASH-160] - now use gnu make instead of rake.
|
|
- misc: now using cabin library for all internal logging
|
|
- test: use minitest
|
|
- upgrade: now using jruby in 1.9 mode
|
|
|
|
## inputs
|
|
- feature: zeromq input. Requires you have libzmq installed on your system.
|
|
- feature, bugfix: [LOGSTASH-40,65,234,296]: much smarter file watching for
|
|
file inputs. now supports globs, keeps state between runs, can handle
|
|
truncate, log rotation, etc. no more inotify is required, either (file
|
|
input now works on all platforms)
|
|
- feature: [LOGSTASH-172,201] - syslog input accepts ISO8601 timestamps
|
|
- feature: [LOGSTASH-159] - TCP input lets you configure what identifies
|
|
an input stream to the multiline filter (unique per host, or connection)
|
|
- feature: [LOGSTASH-168] - add new GELF input plugin
|
|
- bugfix: [LOGSTASH-8,233] - fix stomp input
|
|
- bugfix: [LOGSTASH-136,142] - file input should behave better with log rotations
|
|
- bugfix: [LOGSTASH-249] - Input syslog force facility type to be an integer
|
|
- bugfix: [LOGSTASH-317] - fix file input not to crash when a file
|
|
is unreadable
|
|
|
|
## filters
|
|
- feature: [LOGSTASH-66,150]: libgrok re-written in pure ruby (no more
|
|
FFI / external libgrok.so dependency!)
|
|
- feature: [LOGSTASH-292,316] - Filters should run on all events if no condition
|
|
is applied (type, etc).
|
|
- feature: [LOGSTASH-292,316] - Filters can now act on specific tags (or sets
|
|
of tags).
|
|
- bugfix: [LOGSTASH-285] - for grok, add 'keep_empty_captures' setting to
|
|
allow dropping of empty captures. This is true by default.
|
|
- feature: [LOGSTASH-219] - support parsing unix epoch times
|
|
- feature: [LOGSTASH-207] - new filter to parse a field as json merging it
|
|
into the event.
|
|
- feature: [LOGSTASH-267,254] - add DNS filter for doing forward or
|
|
reverse DNS on an event field
|
|
- feature: [LOGSTASH-57] - add mutate filter to help with manipulating
|
|
event field content and type
|
|
|
|
## outputs
|
|
- feature: zeromq output. Requires you have libzmq installed on your system.
|
|
- feature: new file output plugin
|
|
- bugfix: [LOGSTASH-307] embedded elasticsearch now acts as a full ES server;
|
|
previously embedded was only accessible from within the logstash process.
|
|
- bugfix: [LOGSTASH-302] - logstash's log level (-v, -vv flags) now control
|
|
the log output from the elasticsearch client via log4j.
|
|
- bugfix: many gelf output enhancements and bugfixes
|
|
- feature: [LOGSTASH-281] - add https support to loggly output
|
|
- bugfix: [LOGSTASH-167] - limit number of in-flight requests to the
|
|
elasticsearch node to avoid creating too many threads (one thread per
|
|
pending write request)
|
|
- bugfix: [LOGSTASH-181] - output/statsd: set sender properly
|
|
- bugfix: [LOGSTASH-173] - GELF output can throw an exception during gelf notify
|
|
- bugfix: [LOGSTASH-182] - grep filter should act on all events if no type is
|
|
specified.
|
|
- bugfix: [LOGSTASH-309] - file output can now write to named pipes (fifo)
|
|
|
|
|
|
1.0.17 (Aug 12, 2011)
|
|
- Bugs fixed
|
|
- [LOGSTASH-147] - grok filter incorrectly adding fields when a match failed
|
|
- [LOGSTASH-151] - Fix bug in routing keys on AMQP
|
|
- [LOGSTASH-156] - amqp issue with 1.0.16?
|
|
|
|
- Improvement
|
|
- [LOGSTASH-148] - AMQP input should allow queue name to be specified separately from exchange name
|
|
- [LOGSTASH-157] - Plugin doc generator should make regexp config names more readable
|
|
|
|
- New Feature
|
|
- [LOGSTASH-153] - syslog input: make timestamp an optional field
|
|
- [LOGSTASH-154] - Make error reporting show up in the web UI
|
|
|
|
1.0.16 (Aug 18, 2011)
|
|
- Fix elasticsearch client problem with 1.0.15 - jruby-elasticsearch gem
|
|
version required is now 0.0.10 (to work with elasticsearch 0.17.6)
|
|
|
|
1.0.15 (Aug 18, 2011)
|
|
- IMPORTANT: Upgraded to ElasticSearch 0.17.6 - this brings a number of bug
|
|
fixes including an OOM error caused during high index rates in some
|
|
conditions.
|
|
NOTE: You *must* use same main version of elasticsearch as logstash does,
|
|
so if you are still using elasticsearch server 0.16.x - you need to upgrade
|
|
your server before the elasticsearch output will work. If you are using
|
|
the 'embedded' elasticsearch feature of logstash, you do not need to make
|
|
any changes.
|
|
- feature: tcp input and output plugins can now operate in either client
|
|
(connect) or server (listen) modes.
|
|
- feature: new output plugin "statsd" which lets you increment or record
|
|
timings from your logs to a statsd agent
|
|
- feature: new redis 'pattern_channel' input support for PSUBSCRIBE
|
|
- feature: new output plugin "graphite" for taking metrics from events and
|
|
shipping them off to your graphite/carbon server.
|
|
- feature: new output plugin "ganglia" for shipping metrics to ganglia
|
|
gmond server.
|
|
- feature: new output plugin "xmpp" for shipping events over jabber/xmpp
|
|
- feature: new input plugin "xmpp" for receiving events over jabber/xmpp
|
|
- feature: amqp input now supports routing keys.
|
|
https://logstash.jira.com/browse/LOGSTASH-122
|
|
- feature: amqp output now supports setting routing key dynamically.
|
|
https://logstash.jira.com/browse/LOGSTASH-122
|
|
- feature: amqp input/output both now support SSL.
|
|
https://logstash.jira.com/browse/LOGSTASH-131
|
|
- feature: new input plugin "exec" for taking events from executed commands
|
|
like shell scripts or other tools.
|
|
- feature: new filter plugin "split" for splitting one event into multiple.
|
|
It was written primarily for the new "exec" input to allow you to split
|
|
the output of a single command run by line into multiple events.
|
|
- misc: upgraded jar releases to use JRuby 1.6.3
|
|
- bugfix: syslog input shouldn't crash anymore on weird network behaviors
|
|
like portscanning, etc.
|
|
https://logstash.jira.com/browse/LOGSTASH-130
|
|
|
|
1.0.14 (Jul 1, 2011)
|
|
- feature: new output plugin "loggly" which lets you ship logs to loggly.com
|
|
- feature: new output plugin "zabbix" - similar to the nagios output, but
|
|
works with the Zabbix monitoring system. Contributed by Johan at
|
|
Mach Technology.
|
|
- feature: New agent '-e' flag which lets you specify a config in a string.
|
|
If you specify no 'input' plugins, default is stdin { type => stdin }
|
|
If you specify no 'output' plugins, default is stdout { debug => true }
|
|
This is intended to be used for hacking with or debugging filters, but
|
|
you can specify an entire config here if you choose.
|
|
- feature: Agent '-f' flag now supports directories and globs. If you specify
|
|
a directory, all files in that directory will be loaded as a single config.
|
|
If you specify a glob, all files matching that glob will be loaded as a
|
|
single config.
|
|
- feature: gelf output now allows you to override the 'sender'. This defaults
|
|
to the source host originating the event, but can be set to anything now.
|
|
It supports dynamic values, so you can use fields from your event as the
|
|
sender. Contributed by John Vincent
|
|
Issue: https://github.com/logstash/logstash/pull/30
|
|
- feature: added new feature to libgrok that allows you to define patterns
|
|
in-line, like "%{FOO=\d+}" defines 'FOO' match \d+ and captures as such.
|
|
To use this new feature, you must upgrade libgrok to at least 1.20110630
|
|
Issue: https://logstash.jira.com/browse/LOGSTASH-94
|
|
- feature: grok filter now supports 'break_on_match' defaulting to true
|
|
(this was the original behavior). If you set it to false, it will attempt
|
|
to match all patterns and create new fields as normal. If left default
|
|
(true), it will break after the first successful match.
|
|
- feature: grok filter now supports parsing any field. You can do either of
|
|
these: grok { match => [ "fieldname", "pattern" ] }
|
|
or this: grok { fieldname => "pattern" }
|
|
The older 'pattern' attribute still means the same thing, and is equivalent
|
|
to this: grok { match => [ "@message", "pattern" ] }
|
|
Issue: https://logstash.jira.com/browse/LOGSTASH-101
|
|
- feature: elasticsearch - when embedded is true, you can now set the
|
|
'embedded_http_port' to configure which port the embedded elasticsearch
|
|
server listens on. This is only valid for the embedded elasticsearch
|
|
configuration. https://logstash.jira.com/browse/LOGSTASH-117
|
|
- bugfix: amqp input now reconnects properly when the amqp broker restarts.
|
|
- bugfix: Fix bug in gelf output when a fields were not arrays but numbers.
|
|
Issue: https://logstash.jira.com/browse/LOGSTASH-113
|
|
- bugfix: Fix a bug in syslog udp input due to misfeatures in Ruby's URI
|
|
class. https://logstash.jira.com/browse/LOGSTASH-115
|
|
- misc: jquery and jquery ui now ship with logstash; previously they were
|
|
loaded externally
|
|
- testing: fixed some bugs in the elasticsearch test itself, all green now.
|
|
- testing: fixed logstash-test to now run properly
|
|
|
|
1.0.12 (Jun 9, 2011)
|
|
- misc: clean up some excess debugging output
|
|
- feature: for tcp input, allow 'data_timeout => -1' to mean "never time out"
|
|
|
|
1.0.11 (Jun 9, 2011)
|
|
- deprecated: The redis 'name' and 'queue' options for both input and output
|
|
are now deprecated. They will be removed in a future version.
|
|
- feature: The redis input and output now supports both lists and channels.
|
|
- feature: Refactor runner to allow you to run multiple things in a single
|
|
process. You can end each instance with '--' flag. For example, to run one
|
|
agent and one web instance:
|
|
% java -jar logstash-blah.jar agent -f myconfig -- web
|
|
- feature: Add 'embedded' option to the elasticsearch output:
|
|
elasticsearch { embedded => true }
|
|
Default is false. If true, logstash will run an elasticsearch server
|
|
in the same process as logstash. This is really useful if you are just
|
|
starting out or only need one one elasticsearch server.
|
|
- feature: Added a logstash web backend feature for elasticsearch that tells
|
|
logstash to use the 'local' (in process) elasticsearch:
|
|
--backend elasticsearch:///?local
|
|
- feature: Added 'named_captures_only' option to grok filter. This will have
|
|
logstash only keep the captures you give names to - for example %{NUMBER}
|
|
won't be kept, but %{NUMBER:bytes} will be.
|
|
- feature: Add 'bind_host' option to elasticsearch output. This lets you choose the
|
|
address ElasticSearch client uses to bind to - useful if you have a
|
|
multihomed server.
|
|
- feature: The mongodb output now supports authentication
|
|
- bugfix: Fix bug in GELF output that caused the gelf short_message to be set as an
|
|
array if it came from a grok value. The short_message field should only
|
|
now be a string properly.
|
|
- bugfix: Fix bug in grep filter that would drop/cancel events if you had
|
|
more than one event type flowing through filters and didn't have a grep
|
|
filter defined for each type.
|
|
- misc: Updated gem dependencies (tests still pass)
|
|
- misc: With the above two points, you can now run a single logstash process
|
|
that includes elasticsearch server, logstash agent, and logstash web.
|
|
|
|
1.0.10 (May 23, 2011)
|
|
- Fix tcp input bug (LOGSTASH-88) that would drop connections.
|
|
- Grok patterns_dir (filter config) and --grok-patterns-dir (cmdline opt)
|
|
are now working.
|
|
- GELF output now properly sends extra fields from the log event (prefixed
|
|
with a "_") and sets timestamp to seconds-since-epoch (millisecond
|
|
precision and time zone information is lost, but this is the format GELF
|
|
asks for).
|
|
- Inputs support specifying the format of input data (see "format" and
|
|
"message_format" input config parameters).
|
|
- Grok filter no longer incorrectly tags _grokparsefailure when more than
|
|
one grok filter is enabled (for multiple types) or when an event has
|
|
no grok configuration for it's type.
|
|
- Fix bug where an invalid HTTP Referer: would break grok parsing of the
|
|
log line (used to expect %{URI}). Since Referer: is not sanitized in
|
|
the HTTP layer, we cannot assume it will be a well formed %{URI}.
|
|
|
|
1.0.9 (May 18, 2011)
|
|
- Fix crash bug caused by refactoring that left 'break' calls in code
|
|
that no longer used loops.
|
|
|
|
1.0.8 (May 17, 2011)
|
|
- Remove beanstalk support because the library (beanstalk-client) is GPL3. I
|
|
am not a lawyer, but I'm not waiting around to have someone complain about
|
|
license incompatibilities.
|
|
- fix bug in jar build
|
|
|
|
1.0.7 (May 16, 2011)
|
|
- logstash 'web' now allows you to specify the elasticsearch clustername;
|
|
--backend elasticsearch://[host[:port]]/[clustername]
|
|
- GELF output now supports dynamic strings for level and facility
|
|
https://logstash.jira.com/browse/LOGSTASH-83
|
|
- 'amqp' output supports persistent messages over AMQP, now. Tunable.
|
|
https://logstash.jira.com/browse/LOGSTASH-81
|
|
- Redis input and output are now supported. (Contributed by dokipen)
|
|
- Add shutdown processing. Shutdown starts when all inputs finish (like
|
|
stdin) The sequence progresses using the same pipeline as the
|
|
inputs/filters/outputs, so all in-flight events should finish getting
|
|
processed before the final shutdown event makes it's way to the outputs.
|
|
- Add retries to unhandled input exceptions (LOGSTASH-84)
|
|
|
|
1.0.6 (May 11, 2011)
|
|
* Remove 'sigar' from monolithic jar packaging. This removes a boatload of
|
|
unnecessary warning messages on startup whenever you use elasticsearch
|
|
output or logstash-web.
|
|
Issue: https://logstash.jira.com/browse/LOGSTASH-79
|
|
|
|
1.0.5 (May 10, 2011)
|
|
* fix queues when durable is set to true
|
|
|
|
1.0.4 (May 9, 2011)
|
|
* Fix bugs in syslog input
|
|
|
|
1.0.2 (May 8, 2011)
|
|
* Fix default-value handling for configs when the validation type is
|
|
'password'
|
|
|
|
1.0.1 (May 7, 2011)
|
|
* Fix password auth for amqp and stomp (Reported by Luke Macken)
|
|
* Fix default elasticsearch target for logstash-web (Reported by Donald Gordon)
|
|
|
|
1.0.0 (May 6, 2011)
|
|
* First major release.
|