Fixes#8657
Part 1: API and JavaKeyStore implementation (#8657)
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings.
Part 2: Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support (#8659)
* Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support
* Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation.
* Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton.
* Introduce random default password plus obfuscation. Best attempt at security through obscurity.
* Corrections / better support for x-JVM modification.
Part 3: Secret Store: SecretStore, SecretStoreFactory, JavaKeystore - refacactor (#8745)
* Adds more CRUD like operations for SecretStore API
* SecretStoreFactory Mirror API's CRUD operations
* Adds 'exists' to API to allow command line warning 'Overwrite ?'
* Minor readabiliy
Part 4: Integrate secret store with Logstash core (#8905)
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java.
Part 5: Hardening and test fixes (this PR)
Fixes#8935
This change keeps the previous behavior but implements it using multiple
pipelines (pipelines.yml) instead of with `path.config`.
* The default pipeline name is still "main"
* Default config path is still /etc/logstash/conf.d/*.conf
Other small changes included:
* Specify fpm workdir. Otherwise rpmbuild is given an empty value for
its temporary directory. It's unclear if this is a bug in fpm or in my
environment, but I like the change anyway to put fpm's build activity
in the build directory.
* DRY's up the common files going in /etc/logstash for both rpm and deb
packages.
For #8893Fixes#8894
* Created `org.logstash.Logstash` as entrypoint
* Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this)
* Adjusted `bat` and `sh` entry point wrappers
* Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly)
* Flattened `.jar` path to make it a little less bothersome to build the `-cp` string
* Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself)
* No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle
Fixes#8161
The stack version constant in index-shared1.asciidoc should not be
updated during a version bump, as this impacts the current version
of the doc on elastic.co. Instead, create a separate rake task
to bump the version.
See #8637 for details of this happening
Fixes#8648
* `clean` actually cleans all dynamically created files, in particular it now properly cleans the generated Gemfile so
that changes to the Gemfile.template reflect in a rerun of `bundler`
* `rubyTests` and `test` are now one-off and will automatically bootstrap JRuby and Gems required by the tests if necessary
* Fixed Readme to document the now much simpler test targets
* All rake tasks remain unchanged and still work exactly as they did before
Fixes#8583
Remove no longer nececessary sync task
Simplify other tasks to reflect this removal
Change update_version_file to keep comments in yaml file after task run
Fixes#8410
final (I hope) fixes for consolidated versioning.
After testing with rake artifact:zip, need to try both files.
Add desc to rake test:install-* tasks, tired of
not seeing them in rake -vT
changes requested via review
Fixes#8373Fixes#8388
Our intent is to ship all commercially supported[1] plugins with
Logstash. By some mistakes, we have omitted a few plugins, and this
commit resolves those omissions.
Fixes#8287Fixes#8318
Instead of using rake to bootstrap we are slowly moving part of our
infra to gradle this is the first step taken, I didn't change the calls,
I just made the `rake bootstrap` call the `gradle bootstrap`
Closes: #8234Fixes#8238
This commit includes:
* A base Dockerfile and script to push to a Docker repo
* A per-build Dockerfile (derived from the base)
* Updates to the test scripts to allow for more parallel builds
* Docker wrappers for the tests scripts
* Update for the integration test readme to manually run the tests
* Clean up the output of the Java tests
* Remove offline tag for tests (no longer needed that we don't use docker dependent services)
This commit does NOT include:
* Changes needed for the CI system to use Docker
Fixes#8223
add option for different jruby runtime version
relax jruby-ssl version to >= 0.9.20
rename jruby-runtime to jruby-runtime-override and add sha1 IGNORE option
use JRuby released version of 9.1.13.0
* add newlines to generated json
* Implement cloud.id and cloud.auth settings merge to module settings
* Fixes from review plus convert to using Password for any Module Setting
* Review changes
* update modules.asciidoc to include a section on Cloud
* Capitalize Id
* remove unnecessesary require lines
When a file is listed as a configuration file the package manager
will ask the user about what to do about local modifications when
the package is upgraded and the file has changed upstream. If not
listed as a configuration file any local modifications will silently
be lost.
Based on google analytics from docs pageviews, we are updating the bundled
plugins to remove plugins which doesn't have to be in the default package.
They can still be installed via bin/logstash-plugin
Using a recursive algorithm was causing issue with SSL connection to
rubygems when we were manually settings variable to NIL.
This commit uses a simple loop and will try to install the plugins in
sequence and it will give a bit more information at the STDOUT level.
Fixes#7562
Make sure when you set a logstash version that we are not wiping
the content of the versions file but instead merge it so we preserve
jruby information.
Fixes#7434
We have multiple places to declare the jruby version, this commit
centralize everything into the `version.yml` file. This is now used by
the ruby part and the gradle build file to correctly fetch the jruby
library.
Fixes#7434
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
This PR is based on the proposal from #6846, it does the following
- Change the format of the plugin help page
- Create an external file for each plugin type that will get included by the plugin
- Add a script to generate the PR on each plugin repo with the static doc
- Add a script `ci/ci_docs.sh` that will be used to generate the list of plugins that can be installed in logstash from all the available plugins from the community organization
This PR is the building block for change in https://github.com/elastic/docs-tools/issues/3Fixes#6941
