Prior to this a single worker could slurp down multiple shutdown messages, this prevents that from happening by using a flag that can't be overconsumed.
Fixes#9285
Although `LogStash::JavaBasePipeline#plugin_threads_info` attempts to only report on
living threads, it is possible for a thread to die/complete after the check and before
we attempt to extract status in `LogStash::Util#thread_info`; since `Thread#backtrace`
can return `nil` when the thread is no longer living, this causes a crash.
By falling back to an empty array when the backtrace cannot be ascertained from the
given thread, we can emit a thread status even if the thread has already completed.
Fixes#9097
As seen in https://logstash-ci.elastic.co/job/elastic+logstash+pull-request+multijob-ruby-unit-tests/111/consoleFull
sometimes specs fail due to this failing invariant.
The exact error was:
```
[ERROR] 2018-03-14 23:26:12.985 [[main]-pipeline-manager] agent - An exception happened when converging configuration {:exception=>LogStash::InvalidSourceLoaderSettingError, :message=>"Can't find an appropriate config loader with current settings", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/config/source_loader.rb:55:in `fetch'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/opt/logstash/logstash-core/spec/support/shared_contexts.rb:26:in `block in (root)'", "org/jruby/RubyBasicObject.java:1728:in `instance_exec'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:350:in `run'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:509:in `block in run_owned_hooks_for'", "org/jruby/RubyArray.java:1734:in `each'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:508:in `run_owned_hooks_for'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/hooks.rb:460:in `run'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/example_group.rb:537:in `block in run_before_context_hooks'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/memoized_helpers.rb:186:in `block in isolate_for_context_hook'", "org/jruby/RubyBasicObject.java:1728:in `instance_exec'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/memoized_helpers.rb:182:in `isolate_for_context_hook'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/example_group.rb:536:in `run_before_context_hooks'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/example_group.rb:589:in `run'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/runner.rb:118:in `block in run_specs'", "org/jruby/RubyArray.java:2486:in `map'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/runner.rb:118:in `block in run_specs'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/configuration.rb:1926:in `with_suite_hooks'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/runner.rb:113:in `block in run_specs'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/reporter.rb:79:in `report'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/runner.rb:112:in `run_specs'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/runner.rb:87:in `run'", "/tmp/vendor/bundle/jruby/2.3.0/gems/rspec-core-3.7.1/lib/rspec/core/runner.rb:71:in `run'", "/opt/logstash/lib/bootstrap/rspec.rb:13:in `<main>'"]}
```
I think more detail here would be useful, hence the additional log line. Hopefully if/when this happens again we'll have enough to actually debug it.
Fixes#9243
This is in preparation for our opening of x-pack https://www.elastic.co/products/x-pack/open .
X-Pack can be disabled by setting the environment variable OSS=true, or by removing the X-Pack folder.
Fixes#9222
Fixes#8657
Part 1: API and JavaKeyStore implementation (#8657)
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings.
Part 2: Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support (#8659)
* Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support
* Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation.
* Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton.
* Introduce random default password plus obfuscation. Best attempt at security through obscurity.
* Corrections / better support for x-JVM modification.
Part 3: Secret Store: SecretStore, SecretStoreFactory, JavaKeystore - refacactor (#8745)
* Adds more CRUD like operations for SecretStore API
* SecretStoreFactory Mirror API's CRUD operations
* Adds 'exists' to API to allow command line warning 'Overwrite ?'
* Minor readabiliy
Part 4: Integrate secret store with Logstash core (#8905)
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java.
Part 5: Hardening and test fixes (this PR)
Fixes#8935
* Created `org.logstash.Logstash` as entrypoint
* Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this)
* Adjusted `bat` and `sh` entry point wrappers
* Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly)
* Flattened `.jar` path to make it a little less bothersome to build the `-cp` string
* Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself)
* No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle
Fixes#8161
This will allow users to override the pipeline id from the default, "main", to something else while running pipelines via either the -e or -f options.
Fixes#8868
The old order made no sense since Password / SafeURI objects were not wrapped in their to_s suppressing containers.
Only remove the obsolete params after they've been detected
Fixes#8785
Specifically this change explictly checks for the Array type to merge into and pushes into the array instead of relying only on the + concat.
Fixes#8827Fixes#8831
