* noop: swap expected/actual in tests so failures read correctly
* field_reference: fixes crash and adds many edge-case test cases
Some pathological inputs can cause `FieldReference#parse(CharSequence)` to
throw an `ArrayIndexOutOfBounds` exception, which hard-crashes the entire
Logstash process because this Java-based exception is not caught by the Ruby
plugins.
Instead, proactively throw a new runtime exception indicating that we
encountered illegal syntax, and adjust the ruby event wrapper to re-throw
a ruby RuntimeException that can be handled in the normal way.
Additionally, this commit adds specs for the current behaviour of a wide
variety of illegal-syntax inputs, to ensure we don't accidentally change
the current behaviour.
* field_reference: break legacy tokenizer out, lay groundwork for strict-mode
Extracts the bulk of the current `FieldReference#parse(CharSequence)` method
out into its own `LegacyTokenizer` class verbatim to simplify subsequent
review as we add a strict-mode tokenizer.
* field_reference: add strict-mode and compat-mode
Adds a strict-mode FieldReference tokenizer, which throws an illegal syntax
exception when it encounters illegal syntax.
Adds a compat-mode FieldReference tokenizer, which _warns_ when it encounters
an input that would produce a different output under the strict-mode parser,
but otherwise behaves the same as the legacy-mode parser
Enables users to opt-in to the strict-mode parsing, or to opt-out of the
warnings using a command line flag or a settings-file directive
* field_reference (BREAKING): remove LEGACY and COMPAT for 7.0
This makes it consistent with the rest of the class.
I noticed we have some test failures due to @logger not existing. Probably an issue
from when the logger was javafied. This is causing test failures like: https://logstash-ci.elastic.co/job/elastic+logstash+6.x+multijob-unix-compatibility/os=oraclelinux/140/consoleFull
Will open a separate issue about the missing @logger var
```
An exception happened when converging configuration {:exception=>NoMethodError, :message=>"undefined method `error' for nil:NilClass", :backtrace=>["/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/config/source_loader.rb:55:in `block in fetch'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/config/source_loader.rb:54:in `fetch'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/agent.rb:133:in `converge_state_and_update'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/lib/logstash/agent.rb:84:in `execute'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/logstash-core/spec/support/shared_contexts.rb:26:in `block in (root)'", "org/jruby/RubyBasicObject.java:1728:in `instance_exec'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:350:in `run'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:509:in `block in run_owned_hooks_for'", "org/jruby/RubyArray.java:1734:in `each'", "/var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/oraclelinux/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:508:in `run_owned_hooks_for'", "/var/
```
Fixes#9691
It just blocks and doesn't handle the concurrency situation. One can think of the network of connected pipelines as a DAG (We explicitly ask users not to create cycles in our docs). In fact there are two different correct answers to the pipeline shutdown and reload problem.
When reloading pipelines we should assume the user understands whatever changes they're making to the topology. If a downstream pipeline is to be removed, we can assume that's intentional. We don't lose any data from upstream pipelines since those block until that downstream pipeline is restored. To accomplish this none of the `PipelineBus` methods block by default.
When shutting down Logstash we must: 1.) not lose in-flight data, and 2.) not deadlock. The only way to do both is to shutdown the pipelines in order. All state changes happen simultaneously on all piping via multithreading. As a result we don't need to implement a Topological sort or other algorithm to order dependencies, we simply issue a shutdown to all pipelines, and have ones that are dependent block waiting for upstream pipelines.
This patch also correctly handles the case where multiple signals cause pipeline actions to be created simultaneously. If we see two concurrent creates or stops, one of those actions becomes a noop.
Currently the Logstash plugin API has lifecycle methods for starting and stopping, but not reloading. We need to call a different method when a `pipeline` input is stopped due to a pipeline reload vs an agent shutdown. Ideally, we would enrich our plugin API. In the interest of expedience here, however, I added a flag to the `PipelineBus` that changes the shutdown mode of `pipeline` inputs, to be either blocking or non-blocking. It is switched to blocking if a shutdown is triggered.
This also reverts b78d32dede in favor of a better more concurrent approach without mutexes
This is a forward port of https://github.com/elastic/logstash/pull/9650 to master / 6.x
Fixes#9656
Currently, any invocation of clone, including pipeline->pipeline comms and the clone filter
won't actually clone the string data, just the structure. We may be missing other value types too.
This is a start however.
Fixes#9652Fixes#9653
This spec had a race, the start_agent invocation this depends on doesn't wait
until the state has converged at least once.
An example failure is here: https://logstash-ci.elastic.co/job/elastic+logstash+6.x+multijob-unix-compatibility/os=debian/121/console
The message was:
```
22:34:08 Failures:
22:34:08
22:34:08 1) LogStash::Agent Agent execute options when `config.reload.automatic` is set to`FALSE` and successfully load the config converge only once
22:34:08 Failure/Error: expect(source_loader.fetch_count).to eq(1)
22:34:08
22:34:08 expected: 1
22:34:08 got: 0
22:34:08
22:34:08 (compared using ==)
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/logstash-core/spec/logstash/agent/converge_spec.rb:120:in `block in (root)'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb:50:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb:43:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb'
22:34:08 # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/lib/bootstrap/rspec.rb:13:in `<main>'
22:34:08
22:34:08 Finished in 5 minutes 27 seconds (files took 8.75 seconds to load)
22:34:08 2877 examples, 1 failure, 5 pending
```
Fixes#9643
These agent specs occasionally time out. This removes our use of the unreliable Timeout gem
and also lengthens the waiting period before failure
Fixes#9639
I actually am not sure if this matters, but it seems like it won't hurt,
and is a more sane ordering of things. I have a suspicion that in some cases
the agents get stuck and don't shut down because of this ordering, but I can't
prove it
Fixes#9628
