- moved parsing of jvm.options file into Java code
- chnaged the parsing code to consider conditional notation to bind the applicability of certain JVM flags to specific JVM versions
- changed the launch scripts (.sh and .bat) to use the options string composition
- binded CMS flags to JVM specifications 8-14
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
I think we should not check if a command exists which another command, in here 'which'.
Instead we should use a build-in method, in order that script still works in some environments
that, such as a basic docker image, do not have a "which" command.
Fixes#8880
* Created `org.logstash.Logstash` as entrypoint
* Safely handle `Ruby` runtime (which sadly is still a singleton, moving away from that will require a few iterations on top of this)
* Adjusted `bat` and `sh` entry point wrappers
* Verified manually that performance is unchanged (i.e. all Java opts are still loaded properly)
* Flattened `.jar` path to make it a little less bothersome to build the `-cp` string
* Retained ability to load jars from Ruby via the global `$LS_JARS_LOADED` variable hack, to keep plugin specs that load LS as a `.gem` functional (like e.g. the ITs in LS itself)
* No need for the gem jars magic anymore, the downloading and moving into place of jars is now all handled by Gradle
Fixes#8161
Some variables that hold path in Linux shell scripts
are not properly quoted, thus undefined behaviors may
show up (e.g. a link path with space).
This change enclose variables with quotes in bin/logstash,
bin/logstash.lib.sh and, bin/system-install scripts.
Fixes#6596Fixes#6877
This will work if `--path.settings` is properly used on the command line
Remove unused count variable
It isn't strictly necessary
Add double-quotes around $@
This should protect against strange spaces
fixes#6379Fixes#6382
Update after-install scripts
Use /usr/lib/logstash instead of /opt/logstash
More revisions.
Using `/usr/share/logstash` instead of `/usr/lib/logstash`
Track these new files
More work in progress.
Proper arg parsing obtained
WIP: Use jvm.options like ES
Remove facter dependency
Due to an update from @jordansissel on pleaserun, facter is no longer necessary
Revert Gemfile to proper version
Latest work in progress.
I added a simple test to logstash.lib.sh to catch empty strings in java options.
Added `startup.options`, and touched up `jvm.options`
Edited the package scripts to make things play nice with new paths.
Tweaked `system-install` to allow users to provide their own `startup.options` file.
Patch up merge failure
Changes to work with the new logstash.yml
Add fpm dependency (build fails without it)
Fix missing gems in Gemfile
Don't know how these got lost
Use /etc/logstash for configuration in packages
Update the instructions/documentation for...
...startup.options
Merging proposed changes
Remove mustache and clamp
They're covered by pleaserun anyway
Prune config dir from /usr/share/logstash...
...but only when using RPM/DEB packages
Fixes#5341
These changes allow Logstash to be launched from a symlink _only_ when the `readlink`, or suitable version of `stat` binaries are present. An error message will result if Logstash is launched from a symlink and the path cannot be discovered by one of these two methods. The message will advise the user to use the full path instead of a symlink to launch Logstash.
fixes#4291
because HeapDumpPath is set, if it contains spaces the jruby_opts
function will create two strings appended by -J
even if the list of jruby opts is constructed using replacement
shell script will not escape the path correct to the heap dump path
option needs to be added as late as possible
this is a *nix fix only, windows is not affected
Fixes#4303
This PR change the name of the variable to set to change the maximum
heap size on window to `LS_HEAP_SIZE` this replace the LS_MIN_MEM and
LS_MAX_MEM option.
Also We change the default size of the heap from 500m to 1g
Fixes#4085
remove unused load path
use either local core lib dir or logstash-core gem
include spec in logstash-core
do not include lib, spec and locales dirs in package root
environment bootstrapping in lib/bootstrap.rb
added comments
move pluginmanager out of logstash-core
kafla installation solved by pluginmanager refactor
refactor bootstrap code
refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher
add bin/rspec
cut by half the bin/plugin and bin/rspec startup time
fix drip context
use printf instead of tr
updated Windows bin scripts
missing gemspec in gemspec
use gem instead of gemspec so our plugin manager can correctly install logstash-core
generate packages including locally built logstash-core gem
move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core
path leftover
clean help agent help usage message and remore rspec in available command
comments cosmetics
update Bundler dependency, all recent testing have been made with 1.9.4
updated .lock file with regard to updated Gemfile
cleanup gemspec, Gemfile and regen .lock file
added progress output
avoid dual require on version
closes#3096
updated the packaged to enable JAVA_OPTS to override the defaults
ammended JAVA_OPTS being exported
ammend warning wording
ammend the last commit
fix wording again
Fixes#2942
remove stale code
cleanup the bash conditions for when a variable is defined or not
make sure LS_JAVA_OPTS override JAVA_OPTS when need
ammend unused var
rollback to previous use case of appending vars to JAVA_OPTS in the case of having LS_JAVA_OPTS
Fixes#2942
The new way to build is using 'rake' and not 'make'.
For some some attempt at compatibility, any 'make' invocation will
invoke rake. `make foo` will invoke `rake foo`.
This is for #1640
Major changes:
* rake is used instead of make. This allows us to more easily improve,
debug, and test the build tooling.
* The build process can be bootstrapped from any Ruby that has
Rake available. Bootstrapping rake is automatic will download JRuby and
re-execute itself under JRuby.
* rpm and deb packages no longer list any java as dependencies. This
is the result of user discomfort and discussions which concluded
that there is no good way to "depend" on Java via rpm or deb.
Internal changes:
* Vendoring paths are project-based. Each vendor directory is now
specific as vendor/<project>/... For example, JRuby is dumped into
vendor/jruby/...
Other changes:
* We don't use jruby-complete anymore. We now use bin/jruby as the entry point.
This improves the startup time of Logstash on my laptop by approximately
1.5 seconds (~25% speedup of startup time). Testing `bin/logstash
version` speed; 5 runs each:
* Before patch: 7.27 7.55 7.26 7.42 7.47
* After patch: 5.54 5.97 5.67 5.58 5.54
Smaller changes:
- removed 'env' subcommand
- Remove 'deps' subcommand. Replaced by 'rake vendor:all'
- Add gem_home method for this platform's GEM_HOME
- .travis.yml removed because we haven't used Travis in a long time.
- bin/logstash exits nonzero with a useful message if JRuby can't be found.
Notes:
Invoke bundler via system() call instead of Bundler::CLI.
After some discussion in #bundler on freenode (very helpful!) on how
to do what we need to do with bundler, I was encouraged to not use the
bundler api from ruby, so system() it is.
Patch slightly modified to fit the new model of packaging (ruby instead
of shell invocation), but original author is obazoud in #1008.
Richard Pijnenburg helped with some of the code and was lovely in
helping test it.
Fixes#1640
deprecated USE_JRUBY environment var
added USE_RUBY and USE_DRIP environment vars
centralized gem path handling in LogStash::Environment
use consistent discovery strategy
This fixes the following broken in #1189:
% bin/logstash
=> Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.
This helps ensure we provide only our own gem directory for rubygems.
Helps improve LOGSTASH-1918. Before this patch, my local ruby install
was accidentally providing 'rack' so I never noticed this problem.
Drip allows for faster startup time by pre-starting JVMs and keeping a
pool of them around for use on-demand. More details here:
https://github.com/flatland/drip
Tested with this config:
* 'input { generator { count => 1 } } output { null { } }'
Without drip:
% time bin/logstash -e '...'
13.19s user 2.94s system 170% cpu 9.458 total
With drip (install drip, then set 'JAVACMD=drip' in env):
% time JAVACMD=drip bin/logstash -e '...'
0.03s user 0.46s system 6% cpu 7.008 total
2.5 seconds faster (27% faster)
