Commit graph

1202 commits

Author SHA1 Message Date
Ry Biesemeyer
937200d70d agent webserver: synchronise control
if we send `Agent#stop_webserver` while `Agent#start_webserver` is
executing, we risk the `@webserver_thread` being nil and getting a
`NoMethodError`.

Fixes #10393
2019-02-06 07:47:59 +00:00
João Duarte
8d19e6cef9
Cleanup dependencies (#10171)
* cleanup Gemfile.template dependencies
* cleanup logstash-core gem dependencies
2019-02-05 16:38:49 +00:00
Ry Biesemeyer
9e886a9682
tests: support more-precise clocks in Java 11 (#10381) 2019-02-05 08:08:49 -08:00
Ry Biesemeyer
583ec6b625
Java 11 support (#10279)
* bump jruby to 9.2

* don't rely on logstash-base docker image

* work around webmock ruby 2.5 support

* ensure data folder exists in docker

* change fixnum and bignum to integer

* FileUtils.rmdir to rm_rf

this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.

* bump jruby to 9.2.5.0 and fix test

* make rake default task since prepare pack needs it

* Resolve compiler warnings (#10247)

There are 3 types of compiler warnings that are either resolved or suppressed:

1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
   our codebase to the now "raw" type trigger warnings. In most cases we cannot
   actually resolve the issue, since the JRuby-provided methods for creating
   `RubyArray`s still return the raw type, so these have been suppressed.

2. Deprecations:
   - `RubyString#intern19()` -> `RubyString#intern()`
   - `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
   - `NativeException`: remove import & reference directly; suppress usage
     warnings
   - `RaiseException()`: migrate to equivalent non-deprecated methods wherever
     possible; in some cases where we are using this in conjunction with the
     also-deprecated `NativeException` to preserve java stacktraces, there
     seems to be no non-deprecated path forward, so these cases have been
     suppressed.

3. Redundant Casts
   - Resolved

* JRuby 9.2 bundler shenanigans (#10266)

* Revert "Revert "remove forced dependency on old bundler (#9395)""

This reverts commit bef984143d.

* plugin management: update internal bundler to 1.17.x APIs

* deps: update dev dependency webmock to version compatible with JRuby 9.2

* spec: update Pack fixture to include manticore version that doesn't conflict

* build: update gradle to version that has Java 11 support

* java11: resolve or suppress deprecation warnings

* Remove superfluous flag opting into ParNew GC implementation

When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.

https://bugs.openjdk.java.net/browse/JDK-8006478
https://openjdk.java.net/jeps/214

* spec: set thread name to example description for easier debugging

* spec: prevent errors in testing specs by checking against skip list before using

* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code

> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)

* qa: by default, run integration against Elastic Stack 6.5.x

To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.

* qa: ignore deprecation warnings when comparing offline pack output

* qa: add Java 9+ support to ChildProcess dev dependency

this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141

* qa: allow connections to localhost in webmock

* bump jrjackson version

* fix filebeat integration tests

* spec: ensure license compliance spec runs first

The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.

Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.

Also updates the rake invocation, although I'm not sure it is used
any more.
2019-02-04 16:36:11 -08:00
Dan Hermann
48ee9987cc
Native support for Java plugins (beta) (#10232) 2019-02-04 11:36:36 -06:00
Colin Surprenant
f08b8c5076
fix agent silent exit upon pipelines reloading (#10346) 2019-01-31 15:48:50 -05:00
João Duarte
25e773556c
remove pipeline output workers setting (#10358) 2019-01-30 10:09:15 +00:00
Dan Hermann
e6d59de458
Update java dependencies (#10340) 2019-01-18 08:16:13 -06:00
Colin Surprenant
528112c67e
correctly handle pipeline actions (#10331) 2019-01-16 14:42:26 -05:00
Colin Surprenant
5591ee7d79 properly set thread name in thread context 2019-01-09 18:12:04 -05:00
Dan Hermann
5701c84b21 Catch only specific exceptions
Fixes #10250
2019-01-04 13:43:37 +00:00
Dan Hermann
9288515b2b Add documentation for java-execution flag and remove any references to beta status
Fixes #10248
2019-01-04 02:48:44 +00:00
Dan Hermann
19554259b8 add option for PQ checkpoint retry
Fixes #10234
2019-01-02 21:15:38 +00:00
Dan Hermann
2a45a8fd9d
Resolve javadoc warnings not seen on 1.8.0_181 (#10244) 2019-01-02 13:46:50 -06:00
Dan Hermann
76a1d0a664
Eliminate compiler and javadoc warnings (#10241) 2019-01-02 07:16:19 -06:00
Dan Hermann
4152f89022 Remove unused code and cleanup javadoc comments
Fixes #10233
2019-01-02 12:40:56 +00:00
Dan Hermann
134818e69e
Remove witness classes (#10240) 2019-01-01 15:03:56 -06:00
Dan Hermann
47e1bd252f Increase default number of threads reported by hot_threads
Fixes #10218
2018-12-18 16:54:12 +00:00
Dan Hermann
69c54bd482 Add list tag for Codec javadoc
Fixes #10219
2018-12-18 14:20:19 +00:00
Dan Hermann
e89501e47e Java plugin API
Fixes #10216
2018-12-18 00:39:59 +00:00
dmitriisilin
d5b5f9a4fc Add Zing JVM to the list of supported garbage collectors.
Fixes #10051
2018-12-10 10:30:43 +00:00
Joao Duarte
e4159d0fb6 setting a list config to empty array should not result in nil
if a plugin author defines a config parameter as:

  config :retryable_codes, :validate => :number, :list => true, :default => [429]

And a user configures the plugin as:

  plugin { retryable_codes => [] }

It's expected that the retryable_codes parameter is an empty array and not a nil object

This PR changes this behaviour to generate an empty array

Fixes #10179
2018-11-27 14:00:26 +00:00
Colin Surprenant
5af825cb62
upgrade rack to version 1.6.11 (#10148) 2018-11-16 13:28:44 -05:00
Dan Hermann
e0a125f3b9 add common base class for generated Dataset classes
Fixes #10136
2018-11-09 19:02:21 +00:00
Dan Hermann
dbf241285e generalize method parameters
Fixes #10133
2018-11-08 22:20:53 +00:00
Dan Hermann
4bdff7a9fc replace for-loop with static method calls
Fixes #10133
2018-11-08 22:20:52 +00:00
Guy Boertje
1f7a369993
Add VERSION_QUALIFIER support for use by release manager (#10117)
* Add VERSION_QUALIFIER support for use by release manager

* Make the gem build processes aware of the version qualifier

* Try debugging xpack ci test failure.

* Try to use the artifacts-api.elastic.co api for ES download.

* It builds/tests locally now.

* add some comments explaining the artifacts-api and the version string

* cahnges requested in review.

Fixes #9956
2018-11-07 22:39:52 +00:00
João Duarte
fe7607abd4
only start monitoring pipeline after valid license (#10106)
Avoid unecessary logging errors and resource usage by only starting
the monitoring pipeline if we can validate the license.
2018-11-07 17:28:48 +00:00
Colin Surprenant
cc2d54bc16
synchronize ruby pipeline initialization to fix concurrency bug (#10113) 2018-11-02 19:19:31 -04:00
Dan Hermann
7c9ce4c285 count unused space in page files towards current PQ size
Fixes #10052
2018-10-30 16:35:38 +00:00
Dan Hermann
0adb99c399 Handle equality checks on list and map types
Fixes #10074
2018-10-25 15:04:54 +00:00
Joao Duarte
935e7cd8ae make java execution beta
Fixes #10063
2018-10-24 09:19:35 +00:00
Guy Boertje
3fdee027d9
Add SC and EB input TLS support for the Logstash ArcSight module (#10056)
* Add SC and EB input TLS support for the Logstash ArcSight module
I added the ssl for the smart connector (tcp) and ssl and sasl for the
event broker.
Needs verification on a current stack.

* So this is the actual extent of changes (I thought it was to simple)
Fixed the docs omission
Fixed the load path issue
Fixed the ERB binding context problem
Added some basic happy path tests

* remove module settings in logstash.yml

* Overwrite my logstash.yml with the content on master

* Add comments to yaml fixtures
2018-10-23 09:12:12 +01:00
Dan Hermann
96ae3fc158 Handle equality comparison in Java execution where one or more fields is null
Fixes #9843.

Fixes #10039
2018-10-04 13:17:07 +00:00
Dan Hermann
6c71b50747 move gem-dependent test to rubyTests target
Fixes #10026
2018-09-27 14:49:57 +00:00
Guy Boertje
0e1b630502 Fix for failing spec in os_spec.rb
Fixes #10022

Fixes #10024
2018-09-26 17:45:22 +00:00
Colin Surprenant
1329e67174
run Agent specs using both memory and persisted queue (#10009)
* useless throws declaration

* run all specs using both memory and PQ
2018-09-25 12:33:51 -04:00
Guy Boertje
cdc1015629
Make cgroups more robust and provide the override similar to ES (#9999)
* Make cgroups more robust and provide the override similar to ES
and Kibana

This should go out as soon as possible and backport to 5.6

* refactor as per comments

Fixes #6797
Fixes #6824 
Fixes #7928
Fixes #8792
Fixes #9907
2018-09-22 12:13:32 +01:00
Mårten Svantesson
1a4bdd604f Fixes MissingConverterException when receiving data with the rabbitmq input plugin (#9984)
* logstash-plugins/logstash-input-rabbitmq#112 Fixes most cases of MissingConverterException when receiving data with the rabbitmq input plugin.

This is adding support for Byte, Short and (most importantly) Date. But to decrease verbosity I use the functionality in fallbackConvert to decode subclasses of the given classes.
See method ValueReader.readFieldValue in java package amqp-client for which types can appear over AMQP (except that LongString is changed to String by March Hare).

* logstash-plugins/logstash-input-rabbitmq#112 Fixes most cases of MissingConverterException when receiving data with the rabbitmq input plugin.

Be verbose instead of using functionality in fallbackConvert for Byte, Short, Integer and Long.
2018-09-19 10:49:02 +01:00
Colin Surprenant
069a42d35e
move queue opening bits from constructor to open_queue method (#9985) 2018-09-12 18:34:54 -04:00
Donal Diamond
234add03dc make worker thread names visbile to OS, fixes #9968
Fixes #9973
2018-09-12 01:02:20 +00:00
Colin Surprenant
5a49102b89
avoid race condition when initializing events and pipelines metrics (#9950) 2018-09-04 13:37:40 -04:00
Ry Biesemeyer
796eb3e4b3 extract kibana and elasticsearch client ssl config
shared helper does not ignore ssl options when user fails to supply
the `${MODULE}.var.${TARGET}.ssl.enabled=true` directive.

Fixes: https://github.com/elastic/logstash/issues/9664

Fixes #9945
2018-09-04 16:50:25 +00:00
Dan Hermann
62273f7f56 increase timeout for long-running PQ tests
Fixes #9926
2018-08-21 14:17:58 +00:00
Joao Duarte
59d7f86416 disable testWithRealSecondJvm
Fixes #9917
2018-08-17 17:02:38 +00:00
Joao Duarte
ea467150ad disable queueStableUnderStress tests
Fixes #9919
2018-08-17 17:02:17 +00:00
Joao Duarte
103677d0ce
disable reachMaxUnread 2018-08-16 16:08:01 +01:00
Joao Duarte
0ebad1932c add missing import
Fixes #9909
2018-08-16 13:11:12 +00:00
Joao Duarte
97515d6ee6 disable frequently failing test concurrentWritesTest
Fixes #9909
2018-08-16 13:11:12 +00:00
Joao Duarte
32027a44a9 disable spec/logstash/agent_spec.rb:409
Fixes #9908
2018-08-16 12:53:17 +00:00