* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
```Error: Could not find or load main class Files\Java\jdk1.8.0_172 org.logstash.Logstash -e input```
I had C:\Program Files\Java\jdk1.8.0_172 as part of my classpath, which caused this error. After adding " arround %CLASSPATH% it worked fine for me
Fixes#9898
I think we should not check if a command exists which another command, in here 'which'.
Instead we should use a build-in method, in order that script still works in some environments
that, such as a basic docker image, do not have a "which" command.
Fixes#8880
Fixes#8657
Part 1: API and JavaKeyStore implementation (#8657)
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings.
Part 2: Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support (#8659)
* Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support
* Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation.
* Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton.
* Introduce random default password plus obfuscation. Best attempt at security through obscurity.
* Corrections / better support for x-JVM modification.
Part 3: Secret Store: SecretStore, SecretStoreFactory, JavaKeystore - refacactor (#8745)
* Adds more CRUD like operations for SecretStore API
* SecretStoreFactory Mirror API's CRUD operations
* Adds 'exists' to API to allow command line warning 'Overwrite ?'
* Minor readabiliy
Part 4: Integrate secret store with Logstash core (#8905)
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java.
Part 5: Hardening and test fixes (this PR)
Fixes#8935
* Created `org.logstash.Logstash` as entrypoint
* Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this)
* Adjusted `bat` and `sh` entry point wrappers
* Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly)
* Flattened `.jar` path to make it a little less bothersome to build the `-cp` string
* Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself)
* No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle
Fixes#8161
final (I hope) fixes for consolidated versioning.
After testing with rake artifact:zip, need to try both files.
Add desc to rake test:install-* tasks, tired of
not seeing them in rake -vT
changes requested via review
Fixes#8373
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
This change introduce a test for cross-process/jvm file locking
using the FileLockFactory class. It uses a client program that
will lock a file aftwards the test case proceed.
Fixes#6610Fixes#7117
Some variables that hold path in Linux shell scripts
are not properly quoted, thus undefined behaviors may
show up (e.g. a link path with space).
This change enclose variables with quotes in bin/logstash,
bin/logstash.lib.sh and, bin/system-install scripts.
Fixes#6596Fixes#6877
This will now put ENV variables from the `startup.options` file into `/etc/default/logstash` or `/etc/sysconfig/logstash` (or whatever service name you chose), and use the updated pleaserun to ensure these are honored at start time for whichever init system you use (systemd, upstart, SysV).
fixes#6482Fixes#6660
use new bin/ruby and bundler without :development
refactor to DRY and use expected exception
added original Apache 2.0 license and some cosmetics
exclude bin/lock from packaging
rename variables
Now you can specify 3 arguments, each dependent on the previous:
1. full path to the `startup.options` file
2. the startup variant (sysv, upstart, systemd)
3. the version of the startup variant (usually unnecessary, can safely be left blank)
Present behavior is preserved. If `system-install` is called with no arguments, it autodetects and installs the detected variant.
Fixes#6338
This will work if `--path.settings` is properly used on the command line
Remove unused count variable
It isn't strictly necessary
Add double-quotes around $@
This should protect against strange spaces
fixes#6379Fixes#6382
The original shell script throws a error when calling it with "bin/logstash help"
# ./logstash help
ERROR: Unknown command 'help'
See: 'bin/logstash --help'
Fixes#6303
add queue.max_acked_checkpoint and queue.checkpoint_rate settings
now using checkpoint.max_acks, checkpoint.max_writes and checkpoint.max_interval
rename options
wip rework checkpointing
refactored full acked pages handling on acking and recovery
correclty close queue
proper queue open/recovery
checkpoint dump utility
checkpoint on writes
removed debug code and added missing newline
added better comment on contiguous checkpoints
fix spec for new pipeline setting
Update after-install scripts
Use /usr/lib/logstash instead of /opt/logstash
More revisions.
Using `/usr/share/logstash` instead of `/usr/lib/logstash`
Track these new files
More work in progress.
Proper arg parsing obtained
WIP: Use jvm.options like ES
Remove facter dependency
Due to an update from @jordansissel on pleaserun, facter is no longer necessary
Revert Gemfile to proper version
Latest work in progress.
I added a simple test to logstash.lib.sh to catch empty strings in java options.
Added `startup.options`, and touched up `jvm.options`
Edited the package scripts to make things play nice with new paths.
Tweaked `system-install` to allow users to provide their own `startup.options` file.
Patch up merge failure
Changes to work with the new logstash.yml
Add fpm dependency (build fails without it)
Fix missing gems in Gemfile
Don't know how these got lost
Use /etc/logstash for configuration in packages
Update the instructions/documentation for...
...startup.options
Merging proposed changes
Remove mustache and clamp
They're covered by pleaserun anyway
Prune config dir from /usr/share/logstash...
...but only when using RPM/DEB packages
Fixes#5341
