Carrying on from the work done in #11958, update the gradle build to download
the same version of Elasticsearch as is specified in the logstash version.yml file.
This commit updates the standard integration tests to use the same version of
Elasticsearch that is already downloaded for x-pack integration tests, and also
fixes integration tests to allow for the different responses around hits generated
by different versions of Elasticsearch.
JSON.load allows the creation of complex objects, and should not
be given untrusted input. This commit changes the only three uses
of JSON.load in the codebase, which aren't user facing or present
in bundled product, so not really an attact vector.
Clean backport of #11958
This commit changes the download to pull the version of beats based on the version pulled from the branch rather than from an environment variable, or 6.5.4.
This commit also moves the download logic of Filebeat fromfilebeat_setup.sh to build.gradle in order to use the artifacts API in the same way as the downloadEs task, and does some refactoring to DRY up the artifact download tasks.
This commit also fixes the beats integration test to replace the use of a removed setting.
This commit also sets retries to 3 for the download tasks, using 'retries' functionality from gradle download task plugin
This work breaks out the JVM setting info into a new section, and
expands and updates the content. It adds new subheadings to make
scanning the content easier.
ruby produces a LocalJumpError: unexpected return
error if there's a return in a block so this changes just uses
the value of the last expression as the value of the block
The 'prepare_offline_spec.rb' is failing due to a change in the warning message
from JDK11 to JDK14, and JAVA_TOOL_OPTIONS being passed in as an environment
variable by Jenkins, which was not happening before due to the dockerized
environment.
Fixes#11933
Backport of #11931
Escape test fixture service scripts to avoid test failures when run in
Jenkins using multiple yaml configuration files, which causes directories
to be constructed like `centos-7&&immutable` which cause issues with
the service runners cutting off directory locations before '&&'
This commit deviates from the original commit by not setting @setup_script
and @teardown_script variable with the Shellwords escape, as this was removed
in a subsequent commit (#11944)
Try system ruby, then LS_HOME/bin/ruby, then relative path from
script to LS_HOME/bin/ruby. Use LS_RUBY_HOME variable to avoid
testing again on subsequent attempts to wait for port.
This adds the .ci/matrix-runtime-javas.yml file that defines all
the JDKs logstash could be tested against. This is meant to be
used for the Matrix Combinations Jenkins plugin to be able to
select which JDK to test against dynamically.
Avoid to reassing the subdocument for queue metrics preferring a merge
With PR #10576 the PluginsStats.report(stats) overwrites the subsection related to queue instead of merge with newly created entries.
Fixes#11970
Some QA tests reads the FEATURE_FLAG environment variable, for example to test PQ functionality.
This PR passthrough the environement variable inside the Docker instance.
Fixes#11970
openjdk14 appears to be the only version of java14 installed on jenkins windows
worker nodes, so use this instead of zulu14 and adoptopenjdk14
Backport of #11971
Backport of #11944
A previous commit attempted to fix this issue by adding Shellwords.escape to setup_script and teardown_script locations, but File.exists? returns false when called against a filename escaped by Shellwords.escape. This commit localizes the escaping to where the
file is executed.
This commit also adds Shellwords.escape to teardown script runner and the method used to execute logstash to retrieve version. This is to enable tests to run correctly when Jenkins creates execution environments with folders named with &&, eg centos-7&&immutable
When running filebeats integration tests on centos-7, the tests
fail due to permsisions checks on the temporary configuration file
created for the test. This commit sets strict permissions checks
to false in order for the tests to be able to succeed.
Fixes#11949
Fixture test scripts use `nc` to wait for the port to determine
whether a test fixture is up and running. This commit adds a fall
back option to sleep if `nc` is not available - it is not installed
on Jenkins centos worker nodes.
Fixes#11942
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
Kafka teardown script can exit with failure, typically when trying to
stop the broker. This commit logs the error code if the scripts fail
rather than crash out causing build failure.
Fixes#11905
`RubyString#gsub` requires a (Ruby) frame to be present.
The method attempts to set a backref for the current caller's frame.
When the frame stack is empty there isn't really a place to set $~.
This can happen when a LogStash::Util::Loggable#logger is retrieved,
from the input worker thread while not being nested in any block.
Fixes#11874
