OpenJDK versions 11.0.15+10, 17.0.3+7 introduced new functionality to allow Java to enable
strict path checking. This included disallowing the use of : in any place other than directly
after the drive letter. Unfortunately, this check had the side effect of breaking compatibility
with special device paths, such as NUL:, which in turn, prevents Logstash from starting.
This feature was gated by the use of the jdk.io.File.enableADS property with a value of true
disabling the check.
This property was introduced with the default value of false, which prevents logstash
from starting in a Windows environment. While the next release is anticipated to set this
value to true, this commit explicitly sets that value to enable Logstash to be able
to start correctly.
Relates: #14066
Backport #13306 to branch 7.x
(cherry picked from commit 7395641a43)
----
This commit applies all the changes needed to run Logstash on JDK 17:
- opens access to module java.base for packages sun.nio.ch and java.io to run the application and to execute the tests
- removes SecurityManager classes used during Logstash startup
- fix exception type catched in JavaKeyStore tampering test
Related to meta issue #13306
- moved parsing of jvm.options file into Java code
- chnaged the parsing code to consider conditional notation to bind the applicability of certain JVM flags to specific JVM versions
- changed the launch scripts (.sh and .bat) to use the options string composition
- binded CMS flags to JVM specifications 8-14
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
Simple test for dead letter queue integration tests:
Attempt to write invalid entries to elastic search, fail and
remove invalid field. Verify that mutated entry exists in es
Not for committing - has different jvm.options to improve stability
to ensure that the tests pass in CI.
Fixes#7882Fixes#8026
This changes the Java-wide from the default of /dev/random. The reason
for this chnage is that /dev/random can block if there is insufficient
entropy available, and this blocking can cause Logstash to stall on
startup for many many minutes.
Fixes#6117.
This scenario (no entropy) seems especially prone to occur on virtual
machines.
Related: https://github.com/jruby/jruby/wiki/Improving-startup-time#ensure-your-system-has-adequate-entropy
On Windows, I was not sure if this would work, but the JDK source tree
says this:
On Windows systems, specifying the URLs "file:/dev/random" or
"file:/dev/urandom" will enable the native Microsoft CryptoAPI seeding
mechanism for SHA1PRNG.
So defaulting to /dev/urandom should still do the right thing on
Windows.
I also checked to see if other unix-likes had /dev/urandom, and
according to Wikipedia[1], Linux, FreeBSD, OpenBSD, OSX, Solaris,
NetBSD, and others all have urandom.
[1] https://en.wikipedia.org/wiki//dev/randomFixes#7469
This fix is for packages only (RPM/DEB)
This puts the requisite changes into /etc/logstash/logstash.yml so that starting logstash from the command-line, e.g. `/usr/share/logstash/bin/logstash --path.settings /etc/logstash` will read the pipeline from `/etc/logstash/conf.d` and log to `/var/log/logstash/logstash.log`.
fixes#5379
Comment out tmpdir in jvm.options
The absence of the tmpdir can cause JRuby to fail
Fixes#5383
Update after-install scripts
Use /usr/lib/logstash instead of /opt/logstash
More revisions.
Using `/usr/share/logstash` instead of `/usr/lib/logstash`
Track these new files
More work in progress.
Proper arg parsing obtained
WIP: Use jvm.options like ES
Remove facter dependency
Due to an update from @jordansissel on pleaserun, facter is no longer necessary
Revert Gemfile to proper version
Latest work in progress.
I added a simple test to logstash.lib.sh to catch empty strings in java options.
Added `startup.options`, and touched up `jvm.options`
Edited the package scripts to make things play nice with new paths.
Tweaked `system-install` to allow users to provide their own `startup.options` file.
Patch up merge failure
Changes to work with the new logstash.yml
Add fpm dependency (build fails without it)
Fix missing gems in Gemfile
Don't know how these got lost
Use /etc/logstash for configuration in packages
Update the instructions/documentation for...
...startup.options
Merging proposed changes
Remove mustache and clamp
They're covered by pleaserun anyway
Prune config dir from /usr/share/logstash...
...but only when using RPM/DEB packages
Fixes#5341
