Revert "work around jruby-5642 during package installation on jdk11 (#10658)"
This reverts commit 033c896330.
skip the bundler-1.16.6 files when unpacking jruby
Fixes#10674
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
* update bundler to 1.17.1
This commit required some tweaking of how we setup Bundler
due to changes in reset behaviour, an internal variable name change,
and the Bundler::Settings api changing.
* List action shows integration and provided plugins
* List action shows integration when searching by inner plugin
* Install action removed plugins provided by integration
* Install action aborts if plugin is provided by an integration
* Remove action aborts if plugin is provided by an integration
Fixes#9802Fixes#9811
As reported in #9132, if one sets `http_proxy=myproxy:8080` the URL
gets parsed to scheme=myproxy and host=nil. This results in a
NullPointerException.
To solve it, we check that the URL is valid as HTTP/HTTPs and raise a
slightly better error message.
Fixes#9132.
Fixes#9230
Fixes#8657
Part 1: API and JavaKeyStore implementation (#8657)
Introduces the API to read/write/delete sensitive data from a secure store and includes a Java KeyStore implementation. Note - this commit does NOT integrate with the Logstash configuration or settings.
Part 2: Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support (#8659)
* Secret Store: SecretStoreFactory, SecureConfig, Obfuscation and X-JVM support
* Introduce a SecretStoreFactory to allow runtime definition of SecretStore implementation.
* Introduce a SecureConfig to allow simple configuration of different SecretStore implementaiton.
* Introduce random default password plus obfuscation. Best attempt at security through obscurity.
* Corrections / better support for x-JVM modification.
Part 3: Secret Store: SecretStore, SecretStoreFactory, JavaKeystore - refacactor (#8745)
* Adds more CRUD like operations for SecretStore API
* SecretStoreFactory Mirror API's CRUD operations
* Adds 'exists' to API to allow command line warning 'Overwrite ?'
* Minor readabiliy
Part 4: Integrate secret store with Logstash core (#8905)
This change introduces the command line tooling and hooks needed to allow Logstash to use the secret store. This change hooks into the same logic that the does the environment variable substitution. The commnad line mirrors the Elasticsearch command line, and is implemented primarily in Java.
Part 5: Hardening and test fixes (this PR)
Fixes#8935
* allow installing a plugin using proxy without having to use the `--no-verify` flag
* allow the use of proxies that have basic authentication
* add tests for proxy handling in plugin manager
Fixes#8435
To understand why we can now remove this gem we have to go back at the
history of LS 1.5 and the choice we made back them. In the begining
plugins depending on external files like the `user-agent` or the `filter-geoip`
were not bundling theses files in the gem.
The `file-dependencies` was providing a new hook when plugins were
installed to trigger the file download.
In the context of making the plugins work in an offline environment we
need to make sure that every plugins external resources are present in
the gem.
We have currently a few plugins that require external files and they all
bundle it. So I think its safe to remove that feature, also the hook was
not triggered correctly. So it was a NOOP.
Fixes#8298
use rm_rf to delete dir and shutdown pipeline after run
avoid the use of rescue nil, bad practice
do not mock close as it prevents closing the file an prevents removing it on Windows
cleanup temporary files and relax file name check for Windows
fix paths handling for Windows
flag the read file string as UTF-8 which is what we expect
fix Windows issues
ignore load_average on windows
windows safe URI
cleanup and fix file handling for windows
wait for pipeline shutdown to complete
revert to original puts
cleanups
use environment for windows platform check
fix hash path
wait for pipeline thread to complete after shutdown
* Update default license
The current value issues a warning when building:
WARNING: license value 'Apache License (2.0)' is invalid. Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
Did you mean 'Apache-2.0'?
* Update license
* Update license
* Update license
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
On some Linux system like debian, the `HTTP_PROXY` are set to an empty
string and was causing the plugin manager to raise an exception. We now
strip the string and check if it is empty before trying to configure the
proxies.
Fixes: #7045Fixes#7065
Logstash's plugin manager will now follow proxy configuration from the environment.
If you configure `http_proxy` and `https_proxy`, the manager will now use theses information for all the ruby http
connection and will also pass that information down to maven.
Fixes: #6619, #6528Fixes#6825
This PR changes the behavior of the command when you were specify a directory
instead of a filename, if the directory exists it would have deleted the
directory and the files in it.
The new flow and validation is now safer, if you specify a directory
Logstash will tell you to specify a complete filename.
If the ZIP extension is missing, the command line will ask you to make
sure you target a right filename.
If the output file already exist it will *not* delete it but instead
will return a warning message, you can force a delete by using the
`--overwrite` option.
Fixes: #6862Fixes#6861
This PR fix an annoyance when running the `bin/logstash-plugin install
--no-verify` without any plugins, the command was making an unnecessary
call to the artifacts web server.
Fixes#6826
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.
Definition:
Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.
PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.
CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.
Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats
How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip
Fixes#6404
The are a difference between the gems installed and the available spec
loaded in memory making bundler remove the gem that we have just
updated.
Fixes: #6339Fixes#6340
**Motivation**: We have decided to rewrite the documentation generator to add more features
to our documentation and to fix some outstanding bugs.
This PR addresses the following problem:
- Hard to add new features to the code base.
- The code was tied with Logstash.
- No easy way to publish the doc for a single plugin
- The plugin author was not able to test their documentation changes
- The reported errors were hard to understand.
- No easy way to automate it.
- the minimal requirement on base classes.
Fixes#5720
A pack in this context is a *bundle* of plugins that can be distributed outside of rubygems; it is similar to what ES and kibana are doing, and
the user interface is modeled after them. See https://www.elastic.co/downloads/x-pack
**Do not mix it with the `bin/logstash-plugin pack/unpack` command.**
- it contains one or more plugins that need to be installed
- it is self-contains with the gems and the needed jars
- it is distributed as a zip file
- the file structure needs to follow some rules.
- As a reserved name name on elastic.co download http server
- `bin/plugin install logstash-mypack` will check on the download server if a pack for the current specific logstash version exist and it will be downloaded, if it doesn't exist we fallback on rubygems.
- The file on the server will follow this convention `logstash-mypack-{LOGSTASH_VERSION}.zip`
- As a fully qualified url
- `bin/plugin install http://test.abc/logstash-mypack.zip`, if it exists it will be downloaded and installed if it does not we raise an error.
- As a local file
- `bin/plugin install file:///tmp/logstash-mypack.zip`, if it exists it will be installed
Fixes#6168
