Fixes#8657
Part 1: API and JavaKeyStore implementation (#8657)
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings.
Part 2: Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support (#8659)
* Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support
* Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation.
* Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton.
* Introduce random default password plus obfuscation. Best attempt at security through obscurity.
* Corrections / better support for x-JVM modification.
Part 3: Secret Store: SecretStore, SecretStoreFactory, JavaKeystore - refacactor (#8745)
* Adds more CRUD like operations for SecretStore API
* SecretStoreFactory Mirror API's CRUD operations
* Adds 'exists' to API to allow command line warning 'Overwrite ?'
* Minor readabiliy
Part 4: Integrate secret store with Logstash core (#8905)
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java.
Part 5: Hardening and test fixes (this PR)
Fixes#8935
* Compile conditional datasets
* Correctly typecast Dataset implementations in generated code using a custom ClassLoader
* Optimize away redundant calls to get the current Ruby thread context
* Make field naming and typing in generated code automatic
* Modularize code a lot more
* Fix and spec for #8757
* Fixed by caching the result of a filter Dataset correctly, it was missing the boolean flag that triggered just returning the internal cache
Fixes#8691
The old order made no sense since Password / SafeURI objects were not wrapped in their to_s suppressing containers.
Only remove the obsolete params after they've been detected
Fixes#8785
Specifically this change explictly checks for the Array type to merge into and pushes into the array instead of relying only on the + concat.
Fixes#8827Fixes#8831
a 50ms delay helps creating more full batches without practical added latency see #8707
a 64mb page helps PQ perfmance related to a large page size see #8702#8707
A regression was introduced in the new LSCL/LIR implementation in 6.0, in
which configurations that had nested hashes produced valid treetop-ASTs that
could not produce a valid LIR; the values in an `LSCL::AST::Hash` were
assumed to implement `ValueExpression`, but `LSCL::AST::Hash#expr` produced
an unwrapped `RubyHash`.
By making `LSCL::AST::Hash#expr` emit a `ValueExpression` _representing_ a
`RubyHash`, and relying on the existing code that deals with `ValueExpression`
appropriately, we can re-enable deeply-nested hashes in configs.
Resolves: elastic/logstash#8701
Resolves: logstash-plugins/logstash-output-http#77
Resolves: sw-jung/logstash-filter-memoize#1
Fixes#8750
This is in support of a future change to also support substitution from the secret store in addition to the environment variables.
Part of #8353Fixes#8699
A [build failure][] of the grok plugin when run through LIR/lscl indicates that
there is an expectation for multiple Attributes of the same name to be merged
together.
This commit ports the failing spec in the plugin to an abstraction that can be
tested within logstash-core, and adds a caveat to the LSCL LIR-builder to
ensure that we merge the hashes in a way that is compatible with legacy
behaviour.
NOTE: when multiple Attributes of the same name are used in a single config,
it's possible to create configurations that circumvent `AST::Hash`'s ability to
report duplicate keys.
[build failure]: 293778268Fixes#8597
If modules configuration has elasticsearch credentials set, but not
kibana, attempt to use elasticsearch credentials for Kibana login.
Fixes#8422Fixes#8482
The custom serializer introduced in #8371 did change the behavior or JSON
serialization when one of the field was a RubyNil values.
The orignal behavior was to serialize RubyNil values into their
Javascript Null. The PR #8371 changed that behavior to instead
serialized an empty string and changed the expected type.
This commit revert the behavior to serialize to Null and also tests
around the events class to prevent that.
Fixes#8386
* Fixes#8344 - Ruby Unit Test can hang with no information as to why
* Fixes#7227 - MetricStore::MetricNotFound
* Fixes#8204 - `config.reload.automatic` is set to`FALSE
I am only certain it fixes#8344 and #7227, since I can not reproduce #8204.
Fixes#8376
Some of the test were still depending on the multiline filter, this
filter is now shipped anymore with Logstash. I've keep the same logic
but created a simple dummy filter.
Fixes#8284
Password settings for Modules supplied via a yml file were being wrapped inside a
LogStash::Utils::Password object twice during startup - once in from_yaml, and once
in validate. Disallow this from happening by skipping the wrap if the password is
already wrapped
This should complete the fix for #8224Fixes#8258
use rm_rf to delete dir and shutdown pipeline after run
avoid the use of rescue nil, bad practice
do not mock close as it prevents closing the file an prevents removing it on Windows
cleanup temporary files and relax file name check for Windows
fix paths handling for Windows
flag the read file string as UTF-8 which is what we expect
fix Windows issues
ignore load_average on windows
windows safe URI
cleanup and fix file handling for windows
wait for pipeline shutdown to complete
revert to original puts
cleanups
use environment for windows platform check
fix hash path
wait for pipeline thread to complete after shutdown
* add newlines to generated json
* Implement cloud.id and cloud.auth settings merge to module settings
* Fixes from review plus convert to using Password for any Module Setting
* Review changes
* update modules.asciidoc to include a section on Cloud
* Capitalize Id
* remove unnecessesary require lines
* Add --setup phase for modules
Without the `--setup` flag, the index template for elasticsearch, and the index-pattern, saved searches, visualizations, and dashboards for Kibana will not be published.
fixes#7959
