* Adds tasks to add bundled JDK to tar file used to run integration tests
* Uses `RUNTIME_JAVA_HOME` environment variable to control whether bundled JDK or
alternative is to be used
* Updates logstash service helper to respect value of `RUNTIME_JAVA_HOME`
* Requires updates to jenkins repo to set `RUNTIME_JAVA_HOME` correctly only for
integration tests that expect to use a custom version of Java, such as the JDK
matrix tests.
* Fix version of java used to retrieve logstash version in integration tests
Prior to this commit, the system java would be used to retrieve logstash
version in integration tests, leading to test failures with IT environments
that have java 1.8 as system java
* Actually fix `test_port` this time
Use bash `/dev/tcp` to test ports rather than attempting to use `nc` and
`ruby`
When running certain integration tests, a test against a given port is
performed to ensure that certain dependent services are up. Currently,
these are tests are done either via `nc` or `ruby` if no `nc` is provisioned
on the build nodes. The current `ruby` implenentation attempts to use a system
ruby before using the ruby script shipped with Logstash. This commit removes the
use of the system jruby - certain build boxes are still using java8 as their system
java, which causes builds to fail, as java 11 is expected
Open the ability to use Ruby codec inside Java plugins.
Java plugins need subclasses of Java `co.elastic.logstash.api.Codec` class to properly work. This PR implements an adapter for Ruby codecs to be wrapped into a Java's Codec subclass.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Sets the LS_JAVA_HOME environment variable for the environments used to spawn Logstash process in integration tests.
The JDK matrix testing is based on selecting the desired JDK to run the tests, through the BUILD_JAVA_HOME.
However, when the integration tests spawn a Logstash process this setting was missed.
When the Bash script executes the vendored Ruby it has to use proper `GEM_HOME` to avoid the overwrite that happens inside the logstash.lib.sh
3064f7d0c3/bin/logstash.lib.sh (L161-L165)
Sets `LS_JAVA_HOME` of the spawned logstash to use the same `java.home`
that the test is running under, rather than default to the system JDK, which
would result in the spawned logstash running under a different JDK to that
intended in the test
This commit modifies the launch scripts to take care of the LS_JAVA_HOME giving precedence over the JAVA_HOME, which is still used it the first is not found.
Update releases list, aligning with 7.14.0 and 6.8.18. With ES 7.14.0 and security on by default, this PR alsoswithed off security on ES in integration tests
* fpm to 1.13.0 which allows building packages with java 11 + jruby 9.2
* childprocess to 4.x + remove monkey patches
* clamp to 1.x to unlock fpm 1.13.0
This commit fixes up some IT flakiness which has been presenting mostly
in recent DLQ test failures, it includes the following improvements:
* A recent change to Elasticsearch has required the cluster setting
`action.destructive_requires_name` to be set to `false` to enable the use
of destruction actions with wildcards. This commit sets this before
tests on Elasticsearch and DLQ tests
* Adds some extra safety to the `have_hits` rspec matcher
This commit updates the kafka setup scripts to ensure that the kafka setup is clean between builds, by
setting an explicit zookeeper data directory to be cleaned each time, and correctly overriding `log.dirs`
instead of `log.dir` to ensure that the kafka logs are written and wiped in a consistent place each time,
which helps when using the non-immutable images used in arm64 tests.
This commit clears the `JAVA_HOME` variable when starting Elasticsearch
to force it to use the bundled version of the JDK, rather than the
default `JAVA_HOME` from the machine Logstash integration tests are being
run on, and removes the likelihood of tests failing to run due to `JAVA_HOME`
being set to a non-compliant JDK.
Integration tests may fail during elasticsearch teardown, as currently
the stop_es function sends a `SIGTERM` to Elasticsearch, but does not
wait for the process to exit. That can lead to issues when deleting
data directories from a still running process. This commit adds
wait functionality to `stop_es` to wait for a short period of time,
sending a `SIGKILL` if Elasticsearch does not terminate in time.
Carrying on from the work done in #11958, update the gradle build to download
the same version of Elasticsearch as is specified in the logstash version.yml file.
This commit updates the standard integration tests to use the same version of
Elasticsearch that is already downloaded for x-pack integration tests, and also
fixes integration tests to allow for the different responses around hits generated
by different versions of Elasticsearch.
This commit changes the download to pull the version of beats based on the version pulled from the branch rather than from an environment variable, or 6.5.4.
This commit also moves the download logic of Filebeat fromfilebeat_setup.sh to build.gradle in order to use the artifacts API in the same way as the downloadEs task, and does some refactoring to DRY up the artifact download tasks.
This commit also fixes the beats integration test to replace the use of a removed setting.
This commit also sets retries to 3 for the download tasks, using 'retries' functionality from gradle download task plugin
Try system ruby, then LS_HOME/bin/ruby, then relative path from
script to LS_HOME/bin/ruby. Use LS_RUBY_HOME variable to avoid
testing again on subsequent attempts to wait for port.
When running filebeats integration tests on centos-7, the tests
fail due to permsisions checks on the temporary configuration file
created for the test. This commit sets strict permissions checks
to false in order for the tests to be able to succeed.
A previous commit attempted to fix this issue by adding Shellwords.escape to setup_script and teardown_script locations, but File.exists? returns false when called against a filename escaped by Shellwords.escape. This commit localizes the escaping to where the
file is executed.
This commit also adds Shellwords.escape to teardown script runner and the method used to execute logstash to retrieve version. This is to enable tests to run correctly when Jenkins creates execution environments with folders named with &&, eg centos-7&&immutable
Fixture test scripts use `nc` to wait for the port to determine
whether a test fixture is up and running. This commit adds a fall
back option to use ruby if `nc` is not available - it is not installed
on Jenkins centos worker nodes.
Escape test fixture service scripts to avoid test failures when run in
Jenkins using multiple yaml configuration files, which causes directories
to be constructed like `centos-7&&immutable` which cause issues with
the service runners cutting off directory locations before '&&'
Kafka teardown script can exit with failure, typically when trying to
stop the broker. This commit logs the error code if the scripts fail
rather than crash out causing build failure.
Fixes#11905
* Fail fast when integration test fixtures cannot be setup.
Currently when integration test fixture setup fails, the tests proceed as usual,
resulting in noisy logs where the source of the error is not easy to discover.
This commit will raise an exception and log the exit status if the script fails to
setup or teardown.
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
> KIP-176 removes the `--new-consumer` option for all consumer based tools.
> This option is redundant since the new consumer is automatically used if
> `--bootstrap-server` is defined.
>
> -- [Kafka 2.0.0 Changelog](https://kafka.apache.org/documentation/#upgrade_200_notable)
Fixes#10217
Change the data and log directories to be in /tmp/ls_integrations
(matching the kafka service folder structure), and delete
them in the teardown script.
Fixes#8528
