This commit modifies the launch scripts to take care of the LS_JAVA_HOME giving precedence over the JAVA_HOME, which is still used it the first is not found.
Update releases list, aligning with 7.14.0 and 6.8.18. With ES 7.14.0 and security on by default, this PR alsoswithed off security on ES in integration tests
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
## Release notes
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
This PR contains commits attempting to fix the broken acceptance tests:
* Fix the set of test platforms used to run unix acceptance tests
Modernizes the list of OS's used in acceptance tests, to the most modern OS's available at https://app.vagrantup.com/elastic,. This removes the centos-6 platform from the build, which is past end-of-life and fails vagrant bootstrapping, causing the build to fail.
This is more of band-aid than anything - in the longer term, we should remove these vagrant based tests completely, and rely
on the build infrastructure to perform OS-based acceptance tests.
* Fix regexes for plugin list tests. …
Fixes tests to support the plugin alias feature. This introduced a new format for
entries emitted by `bin/logstash-plugin list`:
eg
```
└── logstash-input-elastic_agent (alias)
```
This commit fixes the test to account for this change, and whitespace variances.
* noop: avoid declaring default value in config file
* docs: ecs compatibility from 7.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* ecs: on by default
We know that ECS version 8 will release along-side Logstash 8.0, but its scope
is still coming into focus. In this changeset, we change the default value
of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a
significantly closer approximation to what will eventually ship in Logstash
8.0.0.
* docs: ecs from 8.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Because the "Fatal Error" specs specifically inject fatal errors during
execution, and do so by reacting to a "poison" event, the fatal error prevents
the poison event from being ACK'd in the underlying queue.
By specifying a one-off temporary data directory in these specs and cleaning up
after ourselves, we ensure that a PQ containing un-ACK'd events isn't leaked to
the next spec to run.
* fpm to 1.13.0 which allows building packages with java 11 + jruby 9.2
* childprocess to 4.x + remove monkey patches
* clamp to 1.x to unlock fpm 1.13.0
Avoid the creation of log4j routing appender for log events without the `pipeline.id` fishtag.
In this way no spurious log file named "pipeline_${ctx:pipeline.id}.log" and logs are not duplicated with main Logstash log file.
Remove an useful dynamic creation of appender's log file which leverages the `log.format` property
also when it's explicit by the appender itself.
Log4j configuration leverages the placeholder `${sys:ls.log.format}` to compose the name of the log file.
This generates some not evident conflicts in log4j internals, these conflicts became evident when enabling the `pipeline.separate_logs` feature is enabled and the log4j appender definitions contains both json and plain format.
The problem is that under those circumstances the rollover of the log file doesn't happen.
This commit also add a test against the production log4j configuration, to avoid future regressions.
logstash-keystore integration tests spawn a Java process, which by default uses the system JDK generally exposed with JAVA_HOME environment variable. It could be that this JDK is not the one selected with the build system variable BUILD_JAVA_HOME.
This commit uses the JDK defined in BUILD_JAVA_HOME if present.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fix: missing password dependency require
which causes `bin/logstash-keystore` to fail with an error:
```
ERROR: Failed to load settings file from "path.settings". Aborting...
path.setting=/logstash-7.12.0/config, exception=NameError,
message=>uninitialized constant LogStash::Util::Password
```
* Fix: review all LS parts depending on Password
* Test: bin/logstash-keystore create/list
This commit fixes up some IT flakiness which has been presenting mostly
in recent DLQ test failures, it includes the following improvements:
* A recent change to Elasticsearch has required the cluster setting
`action.destructive_requires_name` to be set to `false` to enable the use
of destruction actions with wildcards. This commit sets this before
tests on Elasticsearch and DLQ tests
* Adds some extra safety to the `have_hits` rspec matcher
Docker container integration tests relating to the java process were
failing due to the introduction of the new JVM option parser. This
commit waits for logstash to start before testing that the logstash
java process is being run as expected
- replaces all scripted filters with custom Java implementation
- implemented routing appender per pipeline in Java
- adapted log4j configuration shipped with Logstash
- exit the Logstash process if it detects an scripted log4j configuration
Currently, LS does not respect fatal errors such as java.lang.OutOfMemoryError and continues executing.
This is dangerous since JVM errors are a legitimate reason to halt the process and not continue processing.
Additionally:
- make sure we log the full stack-trace on fatal errors
- halt the JVM wout executing finalizers/hook (scissors on how ES handles uncaught exceptions)
- also, we should now be aware of a potentially unexpectedly dying thread
these labels are required for redhat openshift certification.
These commit reintroduces the labels for the ubi8 image only, and adds
acceptance tests to ensure these labels are correct and not inherited
Removing the freeform description labels left the container metadata
without a description label. This commit adds a description under the
"org.opencontainers.image.description" label
This commit fixes two issues with the docker metadata:
Removes non-OCI compliant freeform metadata labels
Uses a consistent build date for all the docker images and dockerfiles
Additionally, this commit adds a `build_docker_ubi8` rake task to enable
`ci/docker_acceptance_tests.sh` to run with no options to build all
docker images for the architecture.
to avoid gems being resolved from the usual LS GEM_HOME
this is problematic for gems such as jruby-openssl which are loaded
during boot (by RGs/Bundler) and thus activated in Bundler from a
different GEM_HOME. if such gem is updated it won't end up being
install-ed in the --path location as it's found on the GEM_HOME!
+ Fix: gem conflict 1.3.6 required by core
this is due now isolating GEM_HOME on `bundle install --path`
+ Refactor: we do not need LS_GEM_HOME/PATH
+ avoid pinning jruby-openssl to 0.10.4
resolves GH-12299 (reverting GH-12300)
* Write DLQ entries to temp file first
This commit changes the DLQ writer to write to a temporary file
which will be renamed on "completion", to avoid the possibility
of the DLQ reader reading an incomplete DLQ segment. The temp file
will be renamed and made available, either when the capacity of this
segment is reached, or if a configurable 'flush interval' has elapsed
since the last event reached the dead letter queue.
This commit fixes#8022, #10275, #10967
This commit replaces #11127
This commit adds the ability for the docker build to build artifacts for multiple architectures.
By default, the target architecture is inferred from the architecture of the machine the build is being
run from - running the build from an aarch64 machine will build an aarch64 docker image, while building
from an x86_64 machine will build an x86_64 docker image.
This can be overridden by setting the environment variable DOCKER_ARCHITECTURE to either `x86_64` or
`aarch64`.
This commit also updates the integration tests to test against the architecture from the machine the test
is being run on, and includes the target architecture in the test description.
Changed Linux creation artifacts (tar.gz/deb/rpm) to include the ARM JDK.
Extracted common parts of artifact.rake into functions to be shared between ARM and Intel bundling tasks
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
This commit includes the required changes to pass RedHat docker image certification.
This includes:
Moving license files to /licenses folder
Adding required base labels for name, description, vendor and summary
Relates: https://github.com/elastic/dev/issues/1287
This commit adds the rake docker_ubi8 rake task, and associated
changes to the docker template and makefiles.
This commit also refactors the acceptance tests to extract xpack tests
into a helper class to allow the same tests to be used in both 'full'
and 'ubi8' docker image tests
* Introduce integration tests for docker
This commit adds integration tests for the Logstash docker images. Previous
integration tests were removed in https://github.com/elastic/logstash/pull/10693,
due to the tests being non functional.
The commit adds image and container tests. The image tests check the contents and the
metadata of the image; the container tests check the logstash process, and includes tests
ensuring that logstash runs, and is configurable.
This test also adds a ci script to allow the tests to be run on jenkins, and to split the
running of these tests up based on the image type and includes updates to the rake tasks to
support this.
This commit updates the kafka setup scripts to ensure that the kafka setup is clean between builds, by
setting an explicit zookeeper data directory to be cleaned each time, and correctly overriding `log.dirs`
instead of `log.dir` to ensure that the kafka logs are written and wiped in a consistent place each time,
which helps when using the non-immutable images used in arm64 tests.
This commit clears the `JAVA_HOME` variable when starting Elasticsearch
to force it to use the bundled version of the JDK, rather than the
default `JAVA_HOME` from the machine Logstash integration tests are being
run on, and removes the likelihood of tests failing to run due to `JAVA_HOME`
being set to a non-compliant JDK.
Integration tests may fail during elasticsearch teardown, as currently
the stop_es function sends a `SIGTERM` to Elasticsearch, but does not
wait for the process to exit. That can lead to issues when deleting
data directories from a still running process. This commit adds
wait functionality to `stop_es` to wait for a short period of time,
sending a `SIGKILL` if Elasticsearch does not terminate in time.
Carrying on from the work done in #11958, update the gradle build to download
the same version of Elasticsearch as is specified in the logstash version.yml file.
This commit updates the standard integration tests to use the same version of
Elasticsearch that is already downloaded for x-pack integration tests, and also
fixes integration tests to allow for the different responses around hits generated
by different versions of Elasticsearch.
JSON.load allows the creation of complex objects, and should not
be given untrusted input. This commit changes the only three uses
of JSON.load in the codebase, which aren't user facing or present
in bundled product, so not really an attact vector.
This commit changes the download to pull the version of beats based on the version pulled from the branch rather than from an environment variable, or 6.5.4.
This commit also moves the download logic of Filebeat fromfilebeat_setup.sh to build.gradle in order to use the artifacts API in the same way as the downloadEs task, and does some refactoring to DRY up the artifact download tasks.
This commit also fixes the beats integration test to replace the use of a removed setting.
This commit also sets retries to 3 for the download tasks, using 'retries' functionality from gradle download task plugin
Try system ruby, then LS_HOME/bin/ruby, then relative path from
script to LS_HOME/bin/ruby. Use LS_RUBY_HOME variable to avoid
testing again on subsequent attempts to wait for port.
When running filebeats integration tests on centos-7, the tests
fail due to permsisions checks on the temporary configuration file
created for the test. This commit sets strict permissions checks
to false in order for the tests to be able to succeed.
A previous commit attempted to fix this issue by adding Shellwords.escape to setup_script and teardown_script locations, but File.exists? returns false when called against a filename escaped by Shellwords.escape. This commit localizes the escaping to where the
file is executed.
This commit also adds Shellwords.escape to teardown script runner and the method used to execute logstash to retrieve version. This is to enable tests to run correctly when Jenkins creates execution environments with folders named with &&, eg centos-7&&immutable
Fixture test scripts use `nc` to wait for the port to determine
whether a test fixture is up and running. This commit adds a fall
back option to use ruby if `nc` is not available - it is not installed
on Jenkins centos worker nodes.
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
