* Starting to audit tests
* Additional field checking in stats
* Add epehemeral id
* More tests
* Test new structure of pipeline report
* Add default_metadata testing
* Add node command tests
* add jvm
* test no mutate
* Add check for graph flag
* Break apart test per review suggestion
* Remove test that doesn't test much
We need a way for a plugin to register simple metadata about external
resources it connects to in order to implement a Monitoring feature in which
an Elasticsearch Output Plugin can store the connected cluster's uuid (#10602)
Here, we add a generic `LogStash::PluginMetadata` along with a registry, and
expose an accessor on `LogStash::Plugin#plugin_metadata` so that instances
can access their own metadata object.
Fixes#10691
This commit fixes a ClassCastException which happens when
a plugin has the `enable_metric` setting set to false - a
NullMetricExt is assumed, but that is only created when
'metric.collect' is set to 'false' in the Logstash configuration,
not when an individual plugin disables its metrics.
Fixes#10538
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
if a plugin author defines a config parameter as:
config :retryable_codes, :validate => :number, :list => true, :default => [429]
And a user configures the plugin as:
plugin { retryable_codes => [] }
It's expected that the retryable_codes parameter is an empty array and not a nil object
This PR changes this behaviour to generate an empty array
Fixes#10179
* Make cgroups more robust and provide the override similar to ES
and Kibana
This should go out as soon as possible and backport to 5.6
* refactor as per comments
Fixes#6797Fixes#6824Fixes#7928Fixes#8792Fixes#9907
* noop: swap expected/actual in tests so failures read correctly
* field_reference: fixes crash and adds many edge-case test cases
Some pathological inputs can cause `FieldReference#parse(CharSequence)` to
throw an `ArrayIndexOutOfBounds` exception, which hard-crashes the entire
Logstash process because this Java-based exception is not caught by the Ruby
plugins.
Instead, proactively throw a new runtime exception indicating that we
encountered illegal syntax, and adjust the ruby event wrapper to re-throw
a ruby RuntimeException that can be handled in the normal way.
Additionally, this commit adds specs for the current behaviour of a wide
variety of illegal-syntax inputs, to ensure we don't accidentally change
the current behaviour.
* field_reference: break legacy tokenizer out, lay groundwork for strict-mode
Extracts the bulk of the current `FieldReference#parse(CharSequence)` method
out into its own `LegacyTokenizer` class verbatim to simplify subsequent
review as we add a strict-mode tokenizer.
* field_reference: add strict-mode and compat-mode
Adds a strict-mode FieldReference tokenizer, which throws an illegal syntax
exception when it encounters illegal syntax.
Adds a compat-mode FieldReference tokenizer, which _warns_ when it encounters
an input that would produce a different output under the strict-mode parser,
but otherwise behaves the same as the legacy-mode parser
Enables users to opt-in to the strict-mode parsing, or to opt-out of the
warnings using a command line flag or a settings-file directive
* field_reference (BREAKING): remove LEGACY and COMPAT for 7.0
This spec had a race, the start_agent invocation this depends on doesn't wait
until the state has converged at least once.
An example failure is here: https://logstash-ci.elastic.co/job/elastic+logstash+6.x+multijob-unix-compatibility/os=debian/121/console
The message was:
```
22:34:08 Failures:
22:34:08
22:34:08 1) LogStash::Agent Agent execute options when `config.reload.automatic` is set to`FALSE` and successfully load the config converge only once
22:34:08 Failure/Error: expect(source_loader.fetch_count).to eq(1)
22:34:08
22:34:08 expected: 1
22:34:08 got: 0
22:34:08
22:34:08 (compared using ==)
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/logstash-core/spec/logstash/agent/converge_spec.rb:120:in `block in (root)'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb:50:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb:43:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/lib/bootstrap/rspec.rb:13:in `<main>'
22:34:08
22:34:08 Finished in 5 minutes 27 seconds (files took 8.75 seconds to load)
22:34:08 2877 examples, 1 failure, 5 pending
```
Fixes#9643
These agent specs occasionally time out. This removes our use of the unreliable Timeout gem
and also lengthens the waiting period before failure
Fixes#9639
I actually am not sure if this matters, but it seems like it won't hurt,
and is a more sane ordering of things. I have a suspicion that in some cases
the agents get stuck and don't shut down because of this ordering, but I can't
prove it
Fixes#9628
the path.plugin setting is a multi valued option, clamp uses
an implicit appender method for adding values to an array.
because we patch Clamp to use LogStash::SETTINGS underneath,
we must tune both reading/writing and appending values.
This last wasn't being customized, so this path redefines the
append_method method to add an element to a setting's value
Fixes#8887
