Reimplements `LogStash::Setting::String` Ruby setting class into the `org.logstash.settings.SettingString` and exposes it through `java_import` as `LogStash::Setting::SettingString`.
Updates the rspec tests in two ways:
- logging mock is now converted to real Log4J appender that spy log line that are later verified
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
* licenses: allow elv2, standard abbreviation for Elastic License version 2
* json-dump: reduce unicode normalization cost
Since the underlying JrJackson now properly (and efficiently) encodes the
UTF-8 transcode of whichever strings it is given, we no longer need to
pre-normalize to UTF-8 in ruby _except_ when the string is flagged as BINARY
because we have alternate behaviour to preserve valid UTF-8 sequences.
By emitting a _copy_ of binary-flagged strings that have been re-flagged as
UTF-8, we allow the downstream (efficient) encoding operation in jrjackson
to produce equivalent behaviour at much lower cost.
* cleanup: remove orphan unicode normalizer
* Ruby code coverage with SimpleCov json formatter
* [CI] Send Java and ruby tests to sonarqube simultaneously
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Added compiled classes to artifacts
* Test change
* Removed test changes
* Returned back ENABLE_SONARQUBE condition
* Removed debug line
* Diable Ruby coverage if ENABLE_SONARQUBE is not true
* Run sonar scan on pull requests and onn push to main
* Run sonar can on release branches
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
Set of changes to make Logstash compatible to JRuby 9.4.
Bundle JRuby 9.4.3.0
- Redefine space token in `LSCL` and `grammar` treetop from `_` which would generated methods in the form `def _0` (deprecated since `2.7`) to `sc`.
- `I18n.t` method doesn't accept hash as second argument
- `URI.encode` has been replaced with same functionality with `URI::Parser.new.escape`
- `YAML.load` needs explicit `fallback: false` to return false when the yaml string is empty (or contains only comments)
- JRuby's `JavaClass` has been removed, now it can use `java.lang.Class` directly
- explicitly require gem `thwait` to satisfy `require "thwait"` (In `Gemfile.template` and `logstash-core/logstash-core.gemspec`)
- fix not args `clone` to be `def clone(*args)`
- fix `Enumeration.each_slice` which from `Ruby 3.1` is [chainable](https://rubyreferences.github.io/rubychanges/3.1.html#enumerableeach_cons-and-each_slice-return-a-receiver) and doesn't return `nil`. JRuby fixed in https://github.com/jruby/jruby/issues/7015
- Expanded `Down.download` arguments map ca16bbed3c302006967413eb9d3862f2da81f7ae
- Avoid to pass `nil` in the list of couples used in `Hash[ <list of couples> ]` which from Ruby `3.0` generates an `ArgumentError`
- Removed space not allowed between method name and parentheses `initialize (` is forbidden. 29b607dcdef98f81a73ad171639fd13aaa65e243
- With [Ruby 2.7 the `Kernel#open`](https://rubyreferences.github.io/rubychanges/2.7.html#network-and-web) doesn't fallback to `URI#open`, fixed test code that used that to verify open port. e5b70de54c5301f51a767da67294092af0cfafdc
- Avoid to drop `rdoc/` folder from vendored JRuby else `bin/logstash -i irb` would crash, commit b71f73e9c6edb81a7b7ae1305047e506f61c6e8c
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Revert "Revert "Unpin bundler to allow 2.4 (#14894)" (#14942)"
This reverts commit 5e3038a3d3.
* prevent bundler 2.4 from blowing up memory during ./gradlew generatePluginsVersion
for some reason, instances of Bundler::Dsl are retained in
org.jruby.MetaClass. Each Bundler::Dsl will retain Bundler::SourceList
which is > 150MB. This commit sets the internal state of each Dsl's
source list so the large objects can be collected.
The proper fix would be to either reuse a single Dsl object or fix the
code upstream to avoid having Dsl objects retained by org.jruby.MetaClass.
* skip input cloudwatch during generatePluginsVersion
the new bundler 2.4's resolution algorithm seems to struggle in certain
situations, like resolving combination of "logstash-input-cloudwatch" and
"logstash-integration-aws".
Until the issue is solved let's skip it.
This reverts commit 6d08a7c1cd.
Bundler 2.4 is accumulating memory during the `generateDocsVersion` task
outside of the known Bundler 2.3 DepProxy class.
Until the memory mitigation patch is adapted to Bundler 2.4 it's best to revert 6d08a7c1cd.
* A structural change for aliased plugin declarations to support doc header replaces.
* YAML file structure made to be more readable, make AliasPlugin class immutable.
* Do not expose alias doc header replaces since we are not using them. Unit tests failure fix.
Fix gem installer tests to enable unpinning the version of bundler
This commit removes changes the gem installer to use real gems, rather than
use `allow_instance_of` during testing, which appears to be problematic with the
latest version of bundler
* Fix bundler handling of 'without'
Prior to this change, the values set in `set_local` are ignored when invoking
bundler via the command line, as is used with `invoke!`. This commit sets those
values in `ENV` variables instead, fixing the functionality to not install
development gems.
* Update bundler spec to check ENV variable
* Added test to ensure kramdown gem not vendored
* Re-add set_local setting to play nice with `expand_logstash_mixin_dependencies`
* logstash service needs to be installed
* gem_vendored? needs to use full path to vendor files
* use `stdout` from `cat` command to generate spec temporary file
* Removed unnecessary support for supplying a block from #gem_vendored?
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit fixes the `logstash-plugin update` command which fail to update plugin
that depends on a new version of logstash-mixin-ecs_compatibility_support.
It resolves logstash-* dependencies and puts them in bundler update command.
Fixed: #13181
Added test to cover the installation of aliased plugins when exists a gem with same name but that's not a Logstash plugin.
In this case the alias is resolved to the original, skipping the gem retrieved from RubyGems.
Remove hard coded alias definitions in favor of yaml descriptor file.
Introduce a single point of aliases definition (logstash-core/src/main/resources/org/logstash/plugins/AliasRegistry.yml), checksum and copy it around to be used by Logstash and by Logstash's plugin management tool.
The descriptor yml file contains a checksum to verify it's not changed accidentally in a deployment of Logstash, if the verification phase fail Logstash avoid to start and plugin management tool avoid to operate.
The signing and copying around is managed by a specific Gradle task invoked during the build.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes#12831
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
* update bundler to 1.17.1
This commit required some tweaking of how we setup Bundler
due to changes in reset behaviour, an internal variable name change,
and the Bundler::Settings api changing.
In an effort to stabilise our specs and reduce the amount of overflow from
each spec to the next, this patch temporarily sets the default queue type
to be the memory queue.
Fixes#9601
As reported in #9132, if one sets `http_proxy=myproxy:8080` the URL
gets parsed to scheme=myproxy and host=nil. This results in a
NullPointerException.
To solve it, we check that the URL is valid as HTTP/HTTPs and raise a
slightly better error message.
Fixes#9132.
Fixes#9230
* allow installing a plugin using proxy without having to use the `--no-verify` flag
* allow the use of proxies that have basic authentication
* add tests for proxy handling in plugin manager
Fixes#8435
use rm_rf to delete dir and shutdown pipeline after run
avoid the use of rescue nil, bad practice
do not mock close as it prevents closing the file an prevents removing it on Windows
cleanup temporary files and relax file name check for Windows
fix paths handling for Windows
flag the read file string as UTF-8 which is what we expect
fix Windows issues
ignore load_average on windows
windows safe URI
cleanup and fix file handling for windows
wait for pipeline shutdown to complete
revert to original puts
cleanups
use environment for windows platform check
fix hash path
wait for pipeline thread to complete after shutdown
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
When building an offline pack we do a real network call to rubygems to
test the structure of the created package, something the network can be
a bit unreliable and SSLError can be raised, when theses errors happen
we will retry them.
Fix: #7251Fixes#7254
Instead of downloading all the filter plugins we reduce the scope to
just download the `logstash-filter-x*` files, this will make the test
faster and reduce the risk of running into IOError.
Ref: #6904Fixes#7211
We have seen unreliable run in the past on travis or jenkins, in that
test we actually test the creation of a physical package we require
internet access to correctly download the gems, this commit make sure we
retries IOErrors.
Fixes: #6904Fixes#7211
This is a big chang, it:
1. Moves API specs out of their special hierarchy
2. Removes the API spec spec_helper
3. Reactivates that stats command spec (that was accidentally not being
run before due to it not having _spec as a suffix
This was required to fix the preceeding commit, where we added a
before(:each) hook to the spec_helper that wasn't being picked up in
some cases due to the existence of two spec helpers and a $LOAD_PATH
that could change.
Fixes#7132
