PR#15900 missed a few more places where Logstash is installed but
a working minimal pipeline config is added.
This commit fixes that and stabilizes all acceptance tests, thus
minizing the need for time consuming BK retries of corresponding
steps.
Relates #15900
Relates https://github.com/elastic/logstash/issues/15784
(cherry picked from commit 54f73e5d22)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit fixes the startup of the Logstash service during packaging
tests by adding a minimal pipeline config. Without it, the service was
flapping from start to start and vice versa causing test flakiness.
Relates https://github.com/elastic/logstash/issues/15784
(cherry picked from commit b66dc7f460)
The current mechanism of discovering the latest released version per
branch (via ARTIFACTS_API) isn't foolproof near the time of a new
release, as it may be pick a version that hasn't been released
yet. This leads to failures[^1] of the packaging upgrade tests, as we
attempt to download a package file that doesn't exist yet.
This commit switches to an API that that is more up to date regarding
the release version truth.
[^1]: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/125#018d319b-9a33-4306-b7f2-5b41937a8881/1033-1125
(cherry picked from commit 15e19a96c2)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Backport PR #13351 to 7.16 branch. Original message:
* Fix bundler handling of 'without'
Prior to this change, the values set in `set_local` are ignored when invoking
bundler via the command line, as is used with `invoke!`. This commit sets those
values in `ENV` variables instead, fixing the functionality to not install
development gems.
* Update bundler spec to check ENV variable
* Added test to ensure kramdown gem not vendored
* Re-add set_local setting to play nice with `expand_logstash_mixin_dependencies`
* logstash service needs to be installed
* gem_vendored? needs to use full path to vendor files
* use `stdout` from `cat` command to generate spec temporary file
* Removed unnecessary support for supplying a block from #gem_vendored?
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* Backport PR #13015 to 7.x: Bundler: freeze lockfile on run, and "normalize" platform on plugin changes
Backport PR #13015 to 7.x branch. Original Message:
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
(cherry picked from commit 4707cb)
* Remove code pinning bundler to ~> 1.17
Backport PR #13071 to 7.x branch. Original message:
This PR contains commits attempting to fix the broken acceptance tests:
* Fix the set of test platforms used to run unix acceptance tests
Modernizes the list of OS's used in acceptance tests, to the most modern OS's available at https://app.vagrantup.com/elastic,. This removes the centos-6 platform from the build, which is past end-of-life and fails vagrant bootstrapping, causing the build to fail.
This is more of band-aid than anything - in the longer term, we should remove these vagrant based tests completely, and rely
on the build infrastructure to perform OS-based acceptance tests.
* Fix regexes for plugin list tests. …
Fixes tests to support the plugin alias feature. This introduced a new format for
entries emitted by `bin/logstash-plugin list`:
eg
```
└── logstash-input-elastic_agent (alias)
```
This commit fixes the test to account for this change, and whitespace variances.
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
* List action shows integration and provided plugins
* List action shows integration when searching by inner plugin
* Install action removed plugins provided by integration
* Install action aborts if plugin is provided by an integration
* Remove action aborts if plugin is provided by an integration
Fixes#9802Fixes#9811
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
This commit fixes an issue when the debian 8 machine was not correctly bootstrapped and made the suite,
We also disable the test introduced by #6879 they need to be redone so they actually test the content fo the generated gem.
Fixes: #7123Fixes#7187
This change adds an acceptance test that creates a plugin template
for each type with logstash-plugin generate command,
then proceed to check they can be installed.
Fixes#6183.
Fixes#6879
* make sure downloaded logstashes use the new (arch independant files name)
* add specific privileged bootstrap script for debian 8 that is able to install java8 for this distro
* wrap the bootstrap scripts within the platform class, like this logic is in one specific place to select platform specific components
* add platform specific bootstrap file for sles-12 and also scripts to download latest logstash rpm version
* add custom bootstrap script for ubuntu-1404 to be sure it loads all ca certificates into the system
* make the necessary changes to compile with the new package system
* correct logstash path usage
* remove unnecessary dependancy to rake inside the acceptance test framework
* fix script execution under ubuntu environments, making sure it runs on bash
Fixes#5350
Some other minor changes include:
* add back the user level boostrap scripts, useful in that case to pull latest released LS
* cleanup formatting for specs and fixed bootstrap scripts for redhats
* make the latest logstash version available from the current platform config file
* make sure debian based machines use java8
* make sure to use hostname and not just ip:port address when reporting the names
Fixes#5350
Also:
* Refactored the specs organization to make the resoning behind simplier,
introducing the idea of an artifact subject that it wraps the
interactions with the platform.
* Add methods to destroy, bootstrap and halt a list of machines either all of them or the ones listed under a given platform name.
* Introduced more clear namespacing in the rakefile.
* Updated the list of available platforms for acceptance testing, including latest ubuntu, oel, fedora, debian and suse versions
Fixes#5350
This change introduce a new command ci/ci_acceptance which is
responsable of building the packages, bootstraping the acceptance test
environment, launching the VMs and running the tests.
You can use the following command to target specific platform.
```
ci/ci_acceptance.sh all
ci/ci_acceptance.sh centos
ci/ci_acceptance.sh debian
```
This PR also add a new rake task to build all the artifacts in a single
run called rake artifact:all it make sure we only install the plugin
only once and make the build process faster.
Fixes#5350
