Commit graph

766 commits

Author SHA1 Message Date
Richard Pijnenburg
58f6bad61f An other missing one
Fixes #1758
2014-09-30 16:06:47 +00:00
Richard Pijnenburg
46a6e78aea Add missing tests
Fixes #1758
2014-09-30 16:06:47 +00:00
Richard Pijnenburg
51b3bf46c4 [RSPEC] Refactoring step 1
- Move helper functions in own modules and extend Rspec
- Refactor files into correct naming and paths
- Modify files to use new spec_helper and helpers
- Pin rspec to 2.14.x

Fixes #1758
2014-09-30 16:06:46 +00:00
wiibaa
7395512761 filter/xml fix for LOGSTASH-2246: extract non-ascii content with xpath
Fixes #1803
2014-09-29 23:59:58 +00:00
Joao Duarte
96c5e32f1f Fix add_tag behaviour in dns filter
The filter should only modify the event's fields and tags if and only if
all resolves/reverses succeed. So we clone the event, modify the new
copy and return it if all operations succeed. Otherwise the original
event is not modified.

For performance reasons we could reverse the clone logic: clone the
event, modify the original event and, it case of failure, return the
backup.

Note: this changes the dns filter behaviour towards add_tag

Fixes #1795
2014-09-29 22:52:28 +00:00
Colin Surprenant
8c133c3e45 add ruby filter spec
Fixes #1772
2014-09-27 05:44:23 +00:00
Colin Surprenant
e4ac936f0c honor common to_json method signature
Fixes #1772
2014-09-27 05:44:23 +00:00
Colin Surprenant
bf11118709 stringify all symbols in twitter hash
Fixes #1450
2014-09-27 05:40:20 +00:00
Joao Duarte
a536eefad2 spec the absense and wrong commands
PR: #1752
2014-09-27 04:57:33 +00:00
Joao Duarte
b0091b242c Remove the ability to run multiple commands
Addresses #1747.
This removes the argument list iteration and spawning of multiple
tasks.
It's still possible to specify aditional arguments but now they're
ignored.

PR: #1752
2014-09-27 04:57:21 +00:00
wiibaa
01e9f37591 test and fix for LOGSTASH-2288 2014-09-23 07:21:05 +02:00
Tray Torrance
bf953f0f90 Fixes finterprint filter handling of timestamp field
Fixes #1572
2014-09-08 11:57:36 +01:00
Jordan Sissel
4211522de1 Add recurse method for doing breadth-first traversal of the AST
This will be used by the filter flush compiler

Add generation of a flush lambda for each filter.

This allows filters to flush and have any generated events proceed
downward through the config as you would expect, respecting any
branches or future plugins.

Die on IOError which occurs when reading from a closed STDIN

Make filter_flusher invoke the new (and correct, I hope!) way to flush.

- On shutdown, we will also flush all filters.
- The flusher thread will terminate if we are shutting down.

Clarify the comment

Fix comment generation in the code to avoid newlines.

Add 'max_age' setting to multiline for flushing.

This setting chooses how long (in seconds) an event is considered to be
fresh before it will be automatically flushed.

This is useful for:
* slow log sources to get the 'last event' flushed,
* transaction-id-style events that have no obvious "end" event and also
  are mixed among other-id events in the same stream.

Also:
- Make filters have no teardown by default.
- Remove 'enable_flush' since it is not needed anymore; flush is always
  enabled.

refactor flush

new spool filter and specs, mainly for testing flushing

turn off unrelated test error for now

fix the flush logic, fix the flush compiles code to not include output section

synchronize cross-thread  access to @pending

refactor for performance and readability

synchronize cross-thread access to @spool

unused code

input:udp removed boggus ShutdownSignal handling, morphed loop do into while true, cosmetic reformat

use transcient events and not exceptions for in-flow pipeline signaling

inline flushing into  filterworker

removed now unnecessary flushing thread safety

fix conditionals bug for new events generated by filters & specs

spec for issue #793

performance tweeks

simplify filter handling of events and new_events

this removes unecessary duplication when treating the original event as
a special case (different from new_events generated by a filter).
Also, since @filter_func only outputs non-cancelled events, some checks
were also removed.

Move multiple filter specs to a filter_chains file

append events generated by a filter using unshift instead of insert

closes #793, closes #1429, closes #1431, closes #1548
2014-09-05 18:45:22 -04:00
Suyog Rao
01f5c753fb Merge remote-tracking branch 'upstream/master' 2014-09-05 12:18:15 -07:00
Suyog Rao
cff7c37603 Dont add kafka metadata by default
Closes #1691
2014-09-05 12:15:54 -07:00
Joao Duarte
39976a3c5c Prevents event.sprintf from converting floats to sci notation
This allows sprintf to correctly print floats up to 15 decimal
places, which should be enough for most. Without this, very big
or very small floats will be converted to scientific notation.

closes #1670
2014-09-05 17:07:14 +01:00
Suyog Rao
4683e51e4b Merge @joekiller's Kafka plugin to logstash-core
Add documentation and tests
Use scala version 2.9.2 and Kafka 0.8.1.1
Closes #1472
2014-09-03 15:00:04 -07:00
Paul Fletcher-Hill
73e52b2465 Dynamic include_keys and exclude_keys for kv filter
Closes #1618
2014-09-02 18:37:47 -07:00
wiibaa
3efce866c9 filter/date fix locale config
Closes #1664
2014-08-29 17:20:54 -07:00
Wiibaa
18049e138d Support almost-ISO8601 patterns in date filter ISO8601 config 2014-08-29 12:20:41 -07:00
wiibaa
8351fbd401 filter/mutate raise configError in register
Closes #1656
2014-08-27 11:30:49 -07:00
Suyog Rao
a764ed956e Added more tests without the need for live syslog input
Closes #1593
2014-08-26 14:38:37 -07:00
Simon Mulser
bdd69065a4 add unique tag for grok parser in module input 2014-08-04 00:21:48 +02:00
Jordan Sissel
4c86e3342f config grammar: Allow spaces in [field references][like this]
Tests included.

Note about grammar compilation:
  I had to use treetop 1.4.15 (1.5.3 generates ruby code that fails to
  parse, haven't spent any time debugging)

Fixes #1513
2014-08-01 17:35:48 +00:00
wiibaa
51949602f2 fix failing test 2014-07-30 10:55:04 -07:00
Wiibaa
95abfdebe8 reject invalid UNIX timestamp 2014-07-30 10:55:04 -07:00
Suyog Rao
0d987cac9e Add tests for issue to reproduce behavior
Fix break_on_match issue while evaluating multiple patterns

Closes #1547

Make better exception message

Add more specs for inputs with arrays

Closes #1547
2014-07-23 13:38:06 -07:00
Suyog Rao
6e3d377bba Evaluate dynamic fields in event when used with gsub
Closes #1529
2014-07-22 16:43:50 -07:00
Ehtesh Choudhury
4fa053dff8 Use the newer { "fieldname" => "pattern" } for grok filters.
Replace the older grok style:

    { match => [ "fieldname", "pattern" ] }

With the newer one:

    { match => { "fieldname" => "pattern" } }

Also see https://github.com/elasticsearch/logstash/issues/1499
for more examples.

Closes #1516
2014-07-18 16:50:12 -07:00
Philippe Weber
281e59a7ed consolidate java grok patterns 2014-07-15 14:08:37 -07:00
Jordan Sissel
ea350aaf23 inputs/base: Copy the codec's charset when switching.
Includes tests
2014-07-14 14:32:15 +00:00
Jordan Sissel
bf518f797b Add tests to cover codec autoswitch for stdin + tcp inputs
json->json_lines & plain->line

Tests for #1392
Closes #1394
2014-07-14 14:29:12 +00:00
Colin Surprenant
7b6ab95124 fix connection threads tracking leak
better mutex name, loop with while true

event per connection & thread cleanups specs

remove leftover

closes #1509
2014-07-11 16:02:23 -04:00
Suyog Rao
674c3812f5 Verify method is getting called 2014-06-30 11:18:02 -07:00
Suyog Rao
de99afb4f8 Squash extra lines 2014-06-30 11:18:02 -07:00
Suyog Rao
393a961805 Fix for #1451
Line codec was using an undefined variable which casued it to throw and exception.

Fixed and added tests for line codes

Closes #1451
2014-06-30 11:18:02 -07:00
Colin Surprenant
e03b67dc7d replace json parsers with JrJackson and Oj
refactored timestamps with new Timestamp class
closes #1434
2014-06-12 17:53:30 -04:00
Philippe Weber
ccb453d557 initial spec for pipe input
removed logging

fixed encoding header

closes 1385
2014-05-22 15:55:24 -04:00
Colin Surprenant
ee3e684feb fix stdin message handling & specs
closes #1386
2014-05-22 14:23:04 -04:00
Aaron Mildenstein
ed2cdbdf22 Add NaN support back in to collectd codec
* Add spec tests for NaN support
* Catch warning on NaN with nan_handling set to warn
* Make lambdas ivars and initialize so they can reference other ivars
* Fixed reference to point to ivar
* Streamline ivar declarations in one method
* Improve tests to cover more cases
* Make lambdas not be ivars
* Change idiomatic calling of init_lambdas to reflect lack of return value

closes 1363
2014-05-14 14:32:21 -05:00
Colin Surprenant
08a079fe6c exlude export_cypher tag in specs 2014-05-06 16:20:17 -04:00
Aaron Mildenstein
2c0c62969c Of course, after I added HeaderError to the rescue, the test started to fail :)
Fixed the test to instead count for lines which made it through.
2014-04-25 15:40:43 -05:00
Aaron Mildenstein
6edeee5931 Added a test for a part with an incorrect header length
This covers the `raise(HeaderError) if length > payload.length` segment in the collectd codec.
2014-04-25 10:55:54 -05:00
Aaron Mildenstein
25033496cd Latest info:
* Hash + Closures. Now replacing case/when statements with closures and hashes to improve performance
* Raise exceptions now instead of using `break`
* Zipped lines 163-164 per @colinsurprenant
* In line 280, this change is a regression.  The authfile is tiny, so this should still be okay, and it's only read once.  Tests failed without this change.
* Use `has_key?` instead of `include?()` in multiple places.
* In the spec/test, I fixed a typo that was bugging me.

With this commit, all tests pass!

```
$ bin/logstash rspec -fd spec/codecs/collectd.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :elasticsearch=>true, :broken=>true}

LogStash::Codecs::Collectd
  None
    should parse a normal packet
  Sign
    should parse a correctly signed packet
    should not parse an incorrectly signed packet
  Encrypt
    should parse an encrypted packet
    should not parse unencrypted packets when encrypt is configured

Finished in 2.1 seconds
5 examples, 0 failures
```
2014-04-24 17:39:31 -05:00
Aaron Mildenstein
1a02dbfdd8 Merge branch 'master' into collectd-untergeek
* master: (208 commits)
  Fix for LOGSTASH-2093
  Fix for LOGSTASH-2129
  Bump Elasticsearch to 1.1.1 and Kibana to 3.0.1
  Permit relative path or $PATH searches for the java command.
  Add LS_LOG_DIR option
  Improve package acceptance tests
  Fix minor issues with acceptance test
  Fix minor bug in init script found after acceptance test
  Add gemfile
  Initial commit of package acceptance testing
  Typo
  Copy-paste typo fix
  Make embedded => true respect `bind_host` setting.
  Start ES embedded server before creating a client.
  fix and regression tests for github issue #1258
  fix for undefined method + for nil:NilClass for invalid paramters
  reenabled multiline filter specs, refactored concurent streams generator, cleaups
  flush filters at end of test input
  flush support in multiline filter
  Adding new sysv init style scripts based on pleaserun
  ...
2014-04-22 13:42:11 -05:00
Jordan Sissel
53356f03cb Merge pull request #1244 from wiibaa/date-success
filter/date: apply on_success actions only when date parsing succeed
2014-04-16 08:42:53 -07:00
Jordan Sissel
64fbfacc86 Merge pull request #1063 from jbochi/metrics_percentile_fix
Fix percentiles on metrics filter
2014-04-16 08:20:36 -07:00
Richard Pijnenburg
3b13af4efd Merge pull request #1248 from olivierlemoal/master
Fix spec failures on non english systems. Solves #1247
2014-04-16 13:45:59 +01:00
Colin Surprenant
5c6ecf48dc fix and regression tests for github issue #1258 2014-04-11 15:43:52 -04:00
Colin Surprenant
94a1c41d03 reenabled multiline filter specs, refactored concurent streams generator, cleaups 2014-04-11 01:37:11 -04:00