Commit graph

1123 commits

Author SHA1 Message Date
Dan Hermann
7c9ce4c285 count unused space in page files towards current PQ size
Fixes #10052
2018-10-30 16:35:38 +00:00
Dan Hermann
0adb99c399 Handle equality checks on list and map types
Fixes #10074
2018-10-25 15:04:54 +00:00
Joao Duarte
935e7cd8ae make java execution beta
Fixes #10063
2018-10-24 09:19:35 +00:00
Guy Boertje
3fdee027d9
Add SC and EB input TLS support for the Logstash ArcSight module (#10056)
* Add SC and EB input TLS support for the Logstash ArcSight module
I added the ssl for the smart connector (tcp) and ssl and sasl for the
event broker.
Needs verification on a current stack.

* So this is the actual extent of changes (I thought it was to simple)
Fixed the docs omission
Fixed the load path issue
Fixed the ERB binding context problem
Added some basic happy path tests

* remove module settings in logstash.yml

* Overwrite my logstash.yml with the content on master

* Add comments to yaml fixtures
2018-10-23 09:12:12 +01:00
Dan Hermann
96ae3fc158 Handle equality comparison in Java execution where one or more fields is null
Fixes #9843.

Fixes #10039
2018-10-04 13:17:07 +00:00
Dan Hermann
6c71b50747 move gem-dependent test to rubyTests target
Fixes #10026
2018-09-27 14:49:57 +00:00
Guy Boertje
0e1b630502 Fix for failing spec in os_spec.rb
Fixes #10022

Fixes #10024
2018-09-26 17:45:22 +00:00
Colin Surprenant
1329e67174
run Agent specs using both memory and persisted queue (#10009)
* useless throws declaration

* run all specs using both memory and PQ
2018-09-25 12:33:51 -04:00
Guy Boertje
cdc1015629
Make cgroups more robust and provide the override similar to ES (#9999)
* Make cgroups more robust and provide the override similar to ES
and Kibana

This should go out as soon as possible and backport to 5.6

* refactor as per comments

Fixes #6797
Fixes #6824 
Fixes #7928
Fixes #8792
Fixes #9907
2018-09-22 12:13:32 +01:00
Mårten Svantesson
1a4bdd604f Fixes MissingConverterException when receiving data with the rabbitmq input plugin (#9984)
* logstash-plugins/logstash-input-rabbitmq#112 Fixes most cases of MissingConverterException when receiving data with the rabbitmq input plugin.

This is adding support for Byte, Short and (most importantly) Date. But to decrease verbosity I use the functionality in fallbackConvert to decode subclasses of the given classes.
See method ValueReader.readFieldValue in java package amqp-client for which types can appear over AMQP (except that LongString is changed to String by March Hare).

* logstash-plugins/logstash-input-rabbitmq#112 Fixes most cases of MissingConverterException when receiving data with the rabbitmq input plugin.

Be verbose instead of using functionality in fallbackConvert for Byte, Short, Integer and Long.
2018-09-19 10:49:02 +01:00
Colin Surprenant
069a42d35e
move queue opening bits from constructor to open_queue method (#9985) 2018-09-12 18:34:54 -04:00
Donal Diamond
234add03dc make worker thread names visbile to OS, fixes #9968
Fixes #9973
2018-09-12 01:02:20 +00:00
Colin Surprenant
5a49102b89
avoid race condition when initializing events and pipelines metrics (#9950) 2018-09-04 13:37:40 -04:00
Ry Biesemeyer
796eb3e4b3 extract kibana and elasticsearch client ssl config
shared helper does not ignore ssl options when user fails to supply
the `${MODULE}.var.${TARGET}.ssl.enabled=true` directive.

Fixes: https://github.com/elastic/logstash/issues/9664

Fixes #9945
2018-09-04 16:50:25 +00:00
Dan Hermann
62273f7f56 increase timeout for long-running PQ tests
Fixes #9926
2018-08-21 14:17:58 +00:00
Joao Duarte
59d7f86416 disable testWithRealSecondJvm
Fixes #9917
2018-08-17 17:02:38 +00:00
Joao Duarte
ea467150ad disable queueStableUnderStress tests
Fixes #9919
2018-08-17 17:02:17 +00:00
Joao Duarte
103677d0ce
disable reachMaxUnread 2018-08-16 16:08:01 +01:00
Joao Duarte
0ebad1932c add missing import
Fixes #9909
2018-08-16 13:11:12 +00:00
Joao Duarte
97515d6ee6 disable frequently failing test concurrentWritesTest
Fixes #9909
2018-08-16 13:11:12 +00:00
Joao Duarte
32027a44a9 disable spec/logstash/agent_spec.rb:409
Fixes #9908
2018-08-16 12:53:17 +00:00
Joao Duarte
9816af6c40 support more than 3 identifiers in cloud id
Fixes #9882
2018-08-09 13:14:26 +00:00
Joao Duarte
9f243d83f4 support port customization in cloud id
Fixes #9877
2018-08-08 10:47:42 +00:00
Ry Biesemeyer
d6f7f037f3
BREAKING: FieldReference - strict mode cutover (#9543)
* noop: swap expected/actual in tests so failures read correctly

* field_reference: fixes crash and adds many edge-case test cases

Some pathological inputs can cause `FieldReference#parse(CharSequence)` to
throw an `ArrayIndexOutOfBounds` exception, which hard-crashes the entire
Logstash process because this Java-based exception is not caught by the Ruby
plugins.

Instead, proactively throw a new runtime exception indicating that we
encountered illegal syntax, and adjust the ruby event wrapper to re-throw
a ruby RuntimeException that can be handled in the normal way.

Additionally, this commit adds specs for the current behaviour of a wide
variety of illegal-syntax inputs, to ensure we don't accidentally change
the current behaviour.

* field_reference: break legacy tokenizer out, lay groundwork for strict-mode

Extracts the bulk of the current `FieldReference#parse(CharSequence)` method
out into its own `LegacyTokenizer` class verbatim to simplify subsequent
review as we add a strict-mode tokenizer.

* field_reference: add strict-mode and compat-mode

Adds a strict-mode FieldReference tokenizer, which throws an illegal syntax
exception when it encounters illegal syntax.

Adds a compat-mode FieldReference tokenizer, which _warns_ when it encounters
an input that would produce a different output under the strict-mode parser,
but otherwise behaves the same as the legacy-mode parser

Enables users to opt-in to the strict-mode parsing, or to opt-out of the
warnings using a command line flag or a settings-file directive

* field_reference (BREAKING): remove LEGACY and COMPAT for 7.0
2018-07-23 10:42:48 -07:00
Ry Biesemeyer
db35ced2a7 hooks: fire pipeline_stopped when pipeline is stopped
Fixes #9627
2018-07-05 20:57:02 +00:00
Colin Surprenant
ac72778c52 backport a JRuby 9.1.16.0 stdlib resolv.rb patch in current 9.1.13.0 JRuby version
use Gem::Version to compare versions and improve comments

include JRuby license info
2018-07-04 13:30:12 -04:00
Josh Soref
c6cd247ec3 Multiple spelling corrections (#9782)
* spelling: actually
* spelling: already
* spelling: concurrency
* spelling: constructor
* spelling: destinations
* spelling: different
* spelling: elasticsearch
* spelling: encoding
* spelling: error
* spelling: explicitly
* spelling: failings
* spelling: falsey
* spelling: guarantees
* spelling: having
* spelling: implementation
* spelling: logstash
* spelling: module
* spelling: multiple
* spelling: omitted
* spelling: overridden
* spelling: pipeline
* spelling: raspberry
* spelling: receive
* spelling: recommended
* spelling: registered
* spelling: registering
* spelling: shutdown
* spelling: signal
* spelling: specified
* spelling: successful
* spelling: successfully
* spelling: valuefied
* spelling: vector
2018-07-04 10:41:10 +01:00
Ralph
947036682f #9724: Change usage note for --quiet
Fixes #9781
2018-06-28 16:39:58 +00:00
Andrei Hurynovich
c2a10417c4 Fix: Logstash fails to run if data.path is a symlink
Fixes #9706
2018-06-20 12:13:06 +00:00
Dan Hermann
4228d0b5e8 check free space on specified path rather than root folder
Fixes #9767
2018-06-19 18:24:46 +00:00
Armin
8dc46bece1 #9747 Correctly compare cases
Fixes #9760
2018-06-19 15:22:39 +00:00
Armin
c431aba536 #9708: Correctly handle non unicode event keys in serialization
Fixes #9764
2018-06-19 14:55:10 +00:00
Armin
cccd044c92 JAVAFICATION: Move more of the pipelines to Java
Fixes #9765
2018-06-19 12:02:11 +00:00
Dan Hermann
fbb167751e Correctly calculate millisecond duration.
Fixes #9744.

Fixes #9753
2018-06-18 17:37:37 +00:00
Armin
99b5f87f86 #9747 Fix Java Execution compilation breaking for datasets with more than 255 parents
Fixes #9755
2018-06-18 16:57:34 +00:00
Armin
c56d62f4ae JAVAFICATION: Cache filter callsite and dedup some method names
Fixes #9752
2018-06-18 13:27:26 +00:00
Armin
52e4f49177 JAVAFICATION: Move Ruby output calls to caching callsites
Fixes #9751
2018-06-16 14:50:27 +00:00
Armin
7bf2d129d0 MINOR: Move more dynamic Ruby calls to typed Java calls
Fixes #9743
2018-06-15 14:29:35 +00:00
Armin
d34f64cb66 MINOR: Move some dynamic ruby calls to typed calls, clean up some duplicate constants
Fixes #9742
2018-06-15 06:44:32 +00:00
Armin
f8860884fa JRUBY: install custom JRuby tar.gz
Fixes #9731
2018-06-14 12:28:39 +00:00
Armin
5c6e3e71d7 JAVAFICATION: Cleanup OutputDelegatorExt
Fixes #9740
2018-06-14 12:28:16 +00:00
Armin
30e055b1bf JAVAFICATION: Move more of the Java pipeline to Java
Fixes #9723
2018-06-12 13:50:15 +00:00
Armin
6d0230aa4f PQ Repair Utility
Fixes #9710
2018-06-12 08:28:06 +00:00
Dan Hermann
cc6ee1aeeb port output_delegator_spec to Java
Fixes #9717
2018-06-08 13:06:12 +00:00
Armin
76bb37166f TESTS: Fix RSpec CLI invocation
Fixes #9673
2018-06-07 12:39:11 +00:00
Armin
3c30459c80 JAVAFICATION: Move more of the pipeline code to Java
* More pipeline code pulled to Java superclass
* Stronger typing for PQ write client code

Fixes #9697
2018-06-07 09:59:13 +00:00
Tamara Braun
1cc5358892 Moved ConvergeResult to Java
Fixes #9699
2018-06-07 09:18:14 +00:00
Armin
060a8e3bd2 JAVAFICATION: Further dry up ruby and java pipeline and move more of their logic to Java
Fixes #9687
2018-06-01 04:39:44 +00:00
Andrew Cholakian
5541e70dda Use logger not @logger in source_loader
This makes it consistent with the rest of the class.

I noticed we have some test failures due to @logger not existing. Probably an issue
from when the logger was javafied. This is causing test failures like: https://logstash-ci.elastic.co/job/elastic+logstash+6.x+multijob-unix-compatibility/os=oraclelinux/140/consoleFull

Will open a separate issue about the missing @logger var

```
An exception happened when converging configuration {:exception=>NoMethodError, :message=>"undefined method `error' for nil:NilClass", :backtrace=>["/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/config/source_loader.rb:55:in `block in fetch'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/config/source_loader.rb:54:in `fetch'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/agent.rb:133:in `converge_state_and_update'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/agent.rb:84:in `execute'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/spec/support/shared_contexts.rb:26:in `block in (root)'", "org/jruby/RubyBasicObject.java:1728:in `instance_exec'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:350:in `run'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:509:in `block in run_owned_hooks_for'", "org/jruby/RubyArray.java:1734:in `each'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:508:in `run_owned_hooks_for'", "/var/
```

Fixes #9691
2018-05-31 12:56:37 +00:00
Armin
0aa798f350 JAVAFICATION: Move QueueFactory to Java
Fixes #9693
2018-05-31 11:32:13 +00:00