When run in debug mode, #invoke was returning an instance of UI::Shell rather
than a string, causing the plugin to crash when `<<` was called on.
This commit ensures that a string is returned regardless of whether debug is set
Fixes: #14131
Fixes an integration test that verifies the capabilities of CLI tool to install a not bundled plugin.
Move away from logstash-input-google_cloud_storage which depends indirectly to OS's package named shared-mime-info, which is not always available.
Cleanly teardown an integration test that made fall other integration tests.
In some cases the CI integration tests fails because the launched Logstash can't find a gem named `mimemagic`. This gem is installed during a CLI plugin test (install of `logstash-input-google_cloud_storage` plugins kicks in that `mimemagic`).
* fix: respect LS_JAVA_OPTS environment even when optionsfile missing
* Fixed integration tests
* Added unit test to cover the fix
* Wipe commented code
* Removed redundant log in a path that could never be reached
* Moved jvm.options checks into only one place
* javaopts: provide injection point for environment string
Co-authored-by: andsel <selva.andre@gmail.com>
After the fix of unlocking ecs_compatibility_support version in plugin update (#13218), `logstash-plugin install` has a problem of installing non default plugin.
This commit removes `Bundler.setup` in install path to avoid Gemfile froze by bundler
Fixed: #13404
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Removes the usage of JAVA_HOME completely which is not anymore used for JDK path resolution.
Updated all the Logstash launching scripts to use only LS_JAVA_HOME as environment variable to
determine the JDK to use to launch Logstash. JAVA_HOME is abandoned for this job.
This commit modifies the launch scripts to take care of the LS_JAVA_HOME giving precedence over the JAVA_HOME, which is still used it the first is not found.
logstash-keystore integration tests spawn a Java process, which by default uses the system JDK generally exposed with JAVA_HOME environment variable. It could be that this JDK is not the one selected with the build system variable BUILD_JAVA_HOME.
This commit uses the JDK defined in BUILD_JAVA_HOME if present.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fix: missing password dependency require
which causes `bin/logstash-keystore` to fail with an error:
```
ERROR: Failed to load settings file from "path.settings". Aborting...
path.setting=/logstash-7.12.0/config, exception=NameError,
message=>uninitialized constant LogStash::Util::Password
```
* Fix: review all LS parts depending on Password
* Test: bin/logstash-keystore create/list
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
The 'prepare_offline_spec.rb' is failing due to a change in the warning message
from JDK11 to JDK14, and JAVA_TOOL_OPTIONS being passed in as an environment
variable by Jenkins, which was not happening before due to the dockerized
environment.
* simplify the plugins-metadata.json file
* sort and update the plugin list in the rakelib/plugins-metadata.json
* remove dependency on twitter input for testing
* sorted Gemfile.template (grouped by group)
* remove default plugins from Gemfile.template
Fixes#10509
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
* update bundler to 1.17.1
This commit required some tweaking of how we setup Bundler
due to changes in reset behaviour, an internal variable name change,
and the Bundler::Settings api changing.
This commit includes:
* A base Dockerfile and script to push to a Docker repo
* A per-build Dockerfile (derived from the base)
* Updates to the test scripts to allow for more parallel builds
* Docker wrappers for the tests scripts
* Update for the integration test readme to manually run the tests
* Clean up the output of the Java tests
* Remove offline tag for tests (no longer needed that we don't use docker dependent services)
This commit does NOT include:
* Changes needed for the CI system to use Docker
Fixes#8223
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
Logstash's plugin manager will now follow proxy configuration from the environment.
If you configure `http_proxy` and `https_proxy`, the manager will now use theses information for all the ruby http
connection and will also pass that information down to maven.
Fixes: #6619, #6528Fixes#6825
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.
Definition:
Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.
PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.
CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.
Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats
How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip
Fixes#6404
A pack in this context is a *bundle* of plugins that can be distributed outside of rubygems; it is similar to what ES and kibana are doing, and
the user interface is modeled after them. See https://www.elastic.co/downloads/x-pack
**Do not mix it with the `bin/logstash-plugin pack/unpack` command.**
- it contains one or more plugins that need to be installed
- it is self-contains with the gems and the needed jars
- it is distributed as a zip file
- the file structure needs to follow some rules.
- As a reserved name name on elastic.co download http server
- `bin/plugin install logstash-mypack` will check on the download server if a pack for the current specific logstash version exist and it will be downloaded, if it doesn't exist we fallback on rubygems.
- The file on the server will follow this convention `logstash-mypack-{LOGSTASH_VERSION}.zip`
- As a fully qualified url
- `bin/plugin install http://test.abc/logstash-mypack.zip`, if it exists it will be downloaded and installed if it does not we raise an error.
- As a local file
- `bin/plugin install file:///tmp/logstash-mypack.zip`, if it exists it will be installed
Fixes#6168
