The acceptance tests are used to test a variety of scenarios installing, running and updating
logstash on a variety of Linux distributions.
The vagrant distributions we test on are old, and use java8 by default. This commit removes
the java8 tests and changes the tests to only test the jdk included distribution.
This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env.
The substitution happens after syntax parsing and before graph execution.
Fixed: #5115
Fixes an integration test that verifies the capabilities of CLI tool to install a not bundled plugin.
Move away from logstash-input-google_cloud_storage which depends indirectly to OS's package named shared-mime-info, which is not always available.
* Add integration test for Elasticsearch using defaults
Currently, the elasticsearch integration test turns off `data_streams`
and `ecs_compatibility`, hiding the out of the box experience for
ingesting into elasticsearch
* Removed debugging statement
Cleanly teardown an integration test that made fall other integration tests.
In some cases the CI integration tests fails because the launched Logstash can't find a gem named `mimemagic`. This gem is installed during a CLI plugin test (install of `logstash-input-google_cloud_storage` plugins kicks in that `mimemagic`).
When the Bash script executes the vendored Ruby it has to use proper `GEM_HOME` to avoid the overwrite that happens inside the logstash.lib.sh
3064f7d0c3/bin/logstash.lib.sh (L161-L165)
* fix: respect LS_JAVA_OPTS environment even when optionsfile missing
* Fixed integration tests
* Added unit test to cover the fix
* Wipe commented code
* Removed redundant log in a path that could never be reached
* Moved jvm.options checks into only one place
* javaopts: provide injection point for environment string
Co-authored-by: andsel <selva.andre@gmail.com>
Fixes the issue #8752 in event.out counter. When a pipeline contains a drop filter the total out events counter should count only the events that reached the out stage.
This PR changes CompiledExecution.compute() interface to return the number of events that effectively reached the end of the pipeline. This change is used in WorkerLoop to update correctly the event.out metric, instead of relying on the batch's size.
After the fix of unlocking ecs_compatibility_support version in plugin update (#13218), `logstash-plugin install` has a problem of installing non default plugin.
This commit removes `Bundler.setup` in install path to avoid Gemfile froze by bundler
Fixed: #13404
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Update logstash docker to use ubuntu 20.04 base image
* Correctly set locale for ubuntu docker image
* tiny typo fix: ubunto -> ubuntu
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Docker integration tests stability improvements
This commit contains numerous fixes to improve the stability of the docker integration tests
* Patch Excon::UnixSocket
Socket.new running on arm64 on Ubuntu 18.04, causes an immediate SIGSEGV error and crash on
that OS, and, as far as I can tell, only that OS. `TCPSocket.new`,`UDPSocket.new` and
`UNIXSocket.new` do not. This commit patches the UnixSocket of the Excon library to
do the absolute simplest thing possible to avoid this error.
* Ensure that container is deleted even if #kill fails
* Add extra waits to handle the incremental way the payload returned by the monitoring
API increases as logstash starts up and pipelines load.
* Use pyenv to ensure the same version of python is used across different jenkins workers
* Add container logs to help diagnose failed test.
* Update the pipeline definition on multi-pipeline integration test
This was causing a pipeline to halt after startup causing intermittent test failures.
* Remove `;` to ensure failures are propagated appropriately
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fix bundler handling of 'without'
Prior to this change, the values set in `set_local` are ignored when invoking
bundler via the command line, as is used with `invoke!`. This commit sets those
values in `ENV` variables instead, fixing the functionality to not install
development gems.
* Update bundler spec to check ENV variable
* Added test to ensure kramdown gem not vendored
* Re-add set_local setting to play nice with `expand_logstash_mixin_dependencies`
* logstash service needs to be installed
* gem_vendored? needs to use full path to vendor files
* use `stdout` from `cat` command to generate spec temporary file
* Removed unnecessary support for supplying a block from #gem_vendored?
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* settings: add "deprecated alias" support
A deprecated alias provides a path for renaming a setting.
- When a deprecated alias is set on its own, a deprecation notice is emitted
but fetching the canonical setting value will reflect the value set with the
deprecated alias.
- When both the canonical setting (new name) and the deprecated alias (old
name) are specified, it is an error condition.
- When the value of the deprecated alias is queried, a warning is emitted to
the logger and only the value explicitly set to the deprecated alias is
returned.
Additionally, some relevant cleanup is also included:
- Starting Logstash with invalid settings no longer results in the obtuse "An
unexpected error occurred" with backtrace and exception data obscuring the
issue. Instead, a simple message is emitted indicating that the settings are
invalid along with the originating exception's message.
- The various settings implementations share a common logger, instead of each
implementation class providing its own. This is aimed to reduce noise from
the logs and to ensure specs validating logging do not need to tie so
closely to implementation details.
* settings: add password-wrapped setting
* settings: make any setting type capable of being nullable
* settings: add `Settings#names` to power programatic iteration
* cli: route CLI-flag deprecations in to deprecation logger
* settings: group API-related settings under `api.*`
retains deprecated aliases, and is fully backward-compatible.
* webserver: cleanup orphaned attr accessors for never-set ivars
* api: pull settings extraction down from agent
This net-no-change refactor introduces a new method `WebServer#from_settings`
that bridges the gap between Logstash settings and Puma-related options, so
that future additions to the API settings don't add complexity to the Agent.
It also has the benefit of initializing the API Rack App and just ONCE, instead
of once per attempted HTTP port.
* api: add optional TLS/SSL
* docs: reference API security settings
* api: when configured securely, bind to all available interfaces by default
* cleanup: remove unused cert artifacts
* tests: generate fresh webserver certificates
* certs: actually add the binary keystores 🤦
This commit applies all the changes needed to run Logstash on JDK 17:
- opens access to module java.base for packages sun.nio.ch and java.io to run the application and to execute the tests
- removes SecurityManager classes used during Logstash startup
- fix exception type catched in JavaKeyStore tampering test
Related to meta issue #13306
Sets `LS_JAVA_HOME` of the spawned logstash to use the same `java.home`
that the test is running under, rather than default to the system JDK, which
would result in the spawned logstash running under a different JDK to that
intended in the test
Removes the usage of JAVA_HOME completely which is not anymore used for JDK path resolution.
Updated all the Logstash launching scripts to use only LS_JAVA_HOME as environment variable to
determine the JDK to use to launch Logstash. JAVA_HOME is abandoned for this job.
This commit:
- Updates the Gradle wrapper to version 7.2
- Remove the deprecated jcenter and where it was used to retrieve Gradle's plugins it switches to gradlePluginPortal
- Insert an explicit dependency from test task to the log4j.properties manipulation task ("copyProductionLog4jConfiguration") used in integration
This commit modifies the launch scripts to take care of the LS_JAVA_HOME giving precedence over the JAVA_HOME, which is still used it the first is not found.
Update releases list, aligning with 7.14.0 and 6.8.18. With ES 7.14.0 and security on by default, this PR alsoswithed off security on ES in integration tests
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
## Release notes
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
This PR contains commits attempting to fix the broken acceptance tests:
* Fix the set of test platforms used to run unix acceptance tests
Modernizes the list of OS's used in acceptance tests, to the most modern OS's available at https://app.vagrantup.com/elastic,. This removes the centos-6 platform from the build, which is past end-of-life and fails vagrant bootstrapping, causing the build to fail.
This is more of band-aid than anything - in the longer term, we should remove these vagrant based tests completely, and rely
on the build infrastructure to perform OS-based acceptance tests.
* Fix regexes for plugin list tests. …
Fixes tests to support the plugin alias feature. This introduced a new format for
entries emitted by `bin/logstash-plugin list`:
eg
```
└── logstash-input-elastic_agent (alias)
```
This commit fixes the test to account for this change, and whitespace variances.
* noop: avoid declaring default value in config file
* docs: ecs compatibility from 7.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* ecs: on by default
We know that ECS version 8 will release along-side Logstash 8.0, but its scope
is still coming into focus. In this changeset, we change the default value
of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a
significantly closer approximation to what will eventually ship in Logstash
8.0.0.
* docs: ecs from 8.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Because the "Fatal Error" specs specifically inject fatal errors during
execution, and do so by reacting to a "poison" event, the fatal error prevents
the poison event from being ACK'd in the underlying queue.
By specifying a one-off temporary data directory in these specs and cleaning up
after ourselves, we ensure that a PQ containing un-ACK'd events isn't leaked to
the next spec to run.
* fpm to 1.13.0 which allows building packages with java 11 + jruby 9.2
* childprocess to 4.x + remove monkey patches
* clamp to 1.x to unlock fpm 1.13.0
Avoid the creation of log4j routing appender for log events without the `pipeline.id` fishtag.
In this way no spurious log file named "pipeline_${ctx:pipeline.id}.log" and logs are not duplicated with main Logstash log file.
Remove an useful dynamic creation of appender's log file which leverages the `log.format` property
also when it's explicit by the appender itself.
Log4j configuration leverages the placeholder `${sys:ls.log.format}` to compose the name of the log file.
This generates some not evident conflicts in log4j internals, these conflicts became evident when enabling the `pipeline.separate_logs` feature is enabled and the log4j appender definitions contains both json and plain format.
The problem is that under those circumstances the rollover of the log file doesn't happen.
This commit also add a test against the production log4j configuration, to avoid future regressions.
logstash-keystore integration tests spawn a Java process, which by default uses the system JDK generally exposed with JAVA_HOME environment variable. It could be that this JDK is not the one selected with the build system variable BUILD_JAVA_HOME.
This commit uses the JDK defined in BUILD_JAVA_HOME if present.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fix: missing password dependency require
which causes `bin/logstash-keystore` to fail with an error:
```
ERROR: Failed to load settings file from "path.settings". Aborting...
path.setting=/logstash-7.12.0/config, exception=NameError,
message=>uninitialized constant LogStash::Util::Password
```
* Fix: review all LS parts depending on Password
* Test: bin/logstash-keystore create/list
This commit fixes up some IT flakiness which has been presenting mostly
in recent DLQ test failures, it includes the following improvements:
* A recent change to Elasticsearch has required the cluster setting
`action.destructive_requires_name` to be set to `false` to enable the use
of destruction actions with wildcards. This commit sets this before
tests on Elasticsearch and DLQ tests
* Adds some extra safety to the `have_hits` rspec matcher
Docker container integration tests relating to the java process were
failing due to the introduction of the new JVM option parser. This
commit waits for logstash to start before testing that the logstash
java process is being run as expected
- replaces all scripted filters with custom Java implementation
- implemented routing appender per pipeline in Java
- adapted log4j configuration shipped with Logstash
- exit the Logstash process if it detects an scripted log4j configuration
Currently, LS does not respect fatal errors such as java.lang.OutOfMemoryError and continues executing.
This is dangerous since JVM errors are a legitimate reason to halt the process and not continue processing.
Additionally:
- make sure we log the full stack-trace on fatal errors
- halt the JVM wout executing finalizers/hook (scissors on how ES handles uncaught exceptions)
- also, we should now be aware of a potentially unexpectedly dying thread
these labels are required for redhat openshift certification.
These commit reintroduces the labels for the ubi8 image only, and adds
acceptance tests to ensure these labels are correct and not inherited
Removing the freeform description labels left the container metadata
without a description label. This commit adds a description under the
"org.opencontainers.image.description" label
This commit fixes two issues with the docker metadata:
Removes non-OCI compliant freeform metadata labels
Uses a consistent build date for all the docker images and dockerfiles
Additionally, this commit adds a `build_docker_ubi8` rake task to enable
`ci/docker_acceptance_tests.sh` to run with no options to build all
docker images for the architecture.
to avoid gems being resolved from the usual LS GEM_HOME
this is problematic for gems such as jruby-openssl which are loaded
during boot (by RGs/Bundler) and thus activated in Bundler from a
different GEM_HOME. if such gem is updated it won't end up being
install-ed in the --path location as it's found on the GEM_HOME!
+ Fix: gem conflict 1.3.6 required by core
this is due now isolating GEM_HOME on `bundle install --path`
+ Refactor: we do not need LS_GEM_HOME/PATH
+ avoid pinning jruby-openssl to 0.10.4
resolves GH-12299 (reverting GH-12300)
* Write DLQ entries to temp file first
This commit changes the DLQ writer to write to a temporary file
which will be renamed on "completion", to avoid the possibility
of the DLQ reader reading an incomplete DLQ segment. The temp file
will be renamed and made available, either when the capacity of this
segment is reached, or if a configurable 'flush interval' has elapsed
since the last event reached the dead letter queue.
This commit fixes#8022, #10275, #10967
This commit replaces #11127
