This makes @metadata basically a way to store data along with an event
that is *NOT* included when serialized to an output.
Use cases:
- For elasticsearch output, set the index, type, document_id, routing
key, etc with metadata and you won't be burdened by storing a filed
named 'index' in your document!
- For elasticsearch input, we can set @metadata fields for the
index/type/document_id instead of polluting the event data itself.
- No need for "short-lived fields" such as timestamps. For example, a
common pattern is to use grok to capture a timestamp text and give that
to the date filter and finally use mutate to remove that captured text
field.
- Provide a kind of scratch space for events that are not part of the
event data.
Fixes#1834Fixes#1836
update jruby jar to 1.7.15
pinned i18n dependency and cosmetic cleanups
new generated Gemfile lock
refactor gem bundling using stadalone setup & use standard Gemfile with gemspec
update to jruby 1.7.16 and remove jar-dependencies, bouncy-castle-java, jruby-openssl from gemspec
get ready for future plugins mechanism
- Move helper functions in own modules and extend Rspec
- Refactor files into correct naming and paths
- Modify files to use new spec_helper and helpers
- Pin rspec to 2.14.x
Fixes#1758
The filter should only modify the event's fields and tags if and only if
all resolves/reverses succeed. So we clone the event, modify the new
copy and return it if all operations succeed. Otherwise the original
event is not modified.
For performance reasons we could reverse the clone logic: clone the
event, modify the original event and, it case of failure, return the
backup.
Note: this changes the dns filter behaviour towards add_tag
Fixes#1795
Addresses #1747.
This removes the argument list iteration and spawning of multiple
tasks.
It's still possible to specify aditional arguments but now they're
ignored.
PR: #1752
Previously the loading of the logstash gemspec was limited to the plugin manager
After more tests we need the gemspec to be loaded for any action with the plugins
Moving it to the environment module makes it available for more generic loading
Make sure plugin manager uses new function for loading gemspec
- Use better installer method to avoid documentation installation
With the previous method it would install the documentation which didn't exist.
- Create Fake gemspec info for logstash to help dependency management for plugins
Previously we would create a real gem and install it to aid in dependency management for plugins and logstash
Now we create a fake gemspec on the fly with the real version of logstash avoiding having to build it every time.
This will be used by the filter flush compiler
Add generation of a flush lambda for each filter.
This allows filters to flush and have any generated events proceed
downward through the config as you would expect, respecting any
branches or future plugins.
Die on IOError which occurs when reading from a closed STDIN
Make filter_flusher invoke the new (and correct, I hope!) way to flush.
- On shutdown, we will also flush all filters.
- The flusher thread will terminate if we are shutting down.
Clarify the comment
Fix comment generation in the code to avoid newlines.
Add 'max_age' setting to multiline for flushing.
This setting chooses how long (in seconds) an event is considered to be
fresh before it will be automatically flushed.
This is useful for:
* slow log sources to get the 'last event' flushed,
* transaction-id-style events that have no obvious "end" event and also
are mixed among other-id events in the same stream.
Also:
- Make filters have no teardown by default.
- Remove 'enable_flush' since it is not needed anymore; flush is always
enabled.
refactor flush
new spool filter and specs, mainly for testing flushing
turn off unrelated test error for now
fix the flush logic, fix the flush compiles code to not include output section
synchronize cross-thread access to @pending
refactor for performance and readability
synchronize cross-thread access to @spool
unused code
input:udp removed boggus ShutdownSignal handling, morphed loop do into while true, cosmetic reformat
use transcient events and not exceptions for in-flow pipeline signaling
inline flushing into filterworker
removed now unnecessary flushing thread safety
fix conditionals bug for new events generated by filters & specs
spec for issue #793
performance tweeks
simplify filter handling of events and new_events
this removes unecessary duplication when treating the original event as
a special case (different from new_events generated by a filter).
Also, since @filter_func only outputs non-cancelled events, some checks
were also removed.
Move multiple filter specs to a filter_chains file
append events generated by a filter using unshift instead of insert
closes#793, closes#1429, closes#1431, closes#1548
This allows sprintf to correctly print floats up to 15 decimal
places, which should be enough for most. Without this, very big
or very small floats will be converted to scientific notation.
closes#1670
Since ae867bc64, the march_hare version we use raises an error when the
network link is broken while the socket is idle. This case must be
handled by logstash to initiate the reconnection process.
