This commit allows separate running of Java and Ruby tests on Windows i.e. the same way as we currently do on unix (unit_tests.sh) via a cli argument.
If no argument has been supplied, both tests are run (as it does now).
The wrapper script is also rewritten from old batch style script to Powershell.
This work allows us to split the existing Windows CI job in a subsequent PR to separate steps, as we currently do on Linux.
Relates: https://github.com/elastic/logstash/issues/15566
(cherry picked from commit 8ac55184b8)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the follow up PR after #15466, which migrates away
the remaining aarch64 acceptance test Jenkins jobs to Buildkite.
Relates:
- #15466
- https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit c384190718)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
On ARM architecture UBI8 Docker images aren't created, so avoid to create empty tar.gz files.
(cherry picked from commit 7a39d97055)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [DRA] Force docker save to save directly on file instead of pipe to another command loosing the execution error code
(cherry picked from commit 2e5e49d10d)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Fix the docker image building and upload process:
* Builds ubi8 on x86_64.
* Uploads ironbank and ubi8 context files from x86_64 only.
(cherry picked from commit 2e8bd20cf5)
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
Updates the dra_docker.sh script to upload also docker-build-context.tar.gz files
(cherry picked from commit 6ad5690a8c)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Ensures the DRA build script surfaces a rake error, instead of allowing the build to continue.
This ensures that the build doesn't continue if any of the steps fails.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
(cherry picked from commit 17d0bb5ffb)
* Generalize docker image building
* Rename and add ability to pass the architecture as a parameter
* Handle ARCH env variable
(cherry picked from commit 6ba5cc112f)
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
Version 7.17 doesn't generate Darwin aarch64 artifacts. Don't download these artifacts from the GCS bucket, given that we don't build Darwin for that release.
(cherry picked from commit 9c7b7b7454)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* DRA: Improve shell scripts for debuggability (#14654). The changes remove some code duplication by introducing a common file that can be sourced between all scripts. It also improves debuggability by adding better messages.
* Fix dra_common sourcing (#14657). Fixes the source of dra_common.sh. It will now first check the directory of the file from which this dra_common.sh script is being called. This allows the common script to be sourced regardless of where the sourcing script is being called from.
* Fix sourcing on dra_upload (#14659). Fix sourcing on dra_upload.sh
* DRA: Handle env variables better
* Moved the addition of SNAPSHOT suffix to the version after the VERSION_QUALIFIER
* Fix badly assigned variable, version qualifier has to be appended also to PLAIN_STACK_VERSION and not RELEASE_VER
Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit db6a7bc619)
The version passed to the release-manager doesn't need the SNAPSHOT particle because already handled by the --workflow="snapshot", if inserted make the release manager to search for artifacts named as 8.5.0-SNAPSHOT-SNAPSHOT
(cherry picked from commit b8792107ad)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Do not move out artifacts from the build/ former to ensure the upload doesn't fail.
(cherry picked from commit 363adad3b6)
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
Avoid to leverage on git local commands to guess the local branch, it switches to listing the branches and checking against the the stack version. If doesn't exists it's main
(cherry picked from commit 11ecaaea5a)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Switch branch selector from major.minor to read the current branch name
(cherry picked from commit ff8afb2293)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Handle the WORKFLOW_TYPE enviroment variable used to select the kind of artifacts to generate and consequently adapt the version name.
If the WORKFLOW_TYPE has a value assigned other then empty string it's assumed to be snapshot and so it generates snapshot artifacts else the release ones.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
(cherry picked from commit d8d690079a)
Update DRA scripts to use the version qualifier in stack_version variable for alpha and beta builds
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit 3075029b27)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Extract the branch name passed to release-manager from version and not from git current branch
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
(cherry picked from commit d3b92ec20c)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This is a **not clean backport** of the following PRs to `8.5` branch:
- Introduces a bash script to build all the artifacts and dependencies report. #14522
- Save docker images as tar.gz files move the CSV dependency report in the path that's expected by release-manager #14552
- Split ci scripts into ARM and x86 ones #14567
- Uses the gsutil tool to upload all the generated artifacts into an intermediate collector bucket. #14568
- Collect all artifacts created and upload to GCP with release-manager #14584
* Backport PR #13015 to 7.x: Bundler: freeze lockfile on run, and "normalize" platform on plugin changes
Backport PR #13015 to 7.x branch. Original Message:
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
(cherry picked from commit 4707cb)
* Remove code pinning bundler to ~> 1.17
This commit adds the rake docker_ubi8 rake task, and associated
changes to the docker template and makefiles.
This commit also refactors the acceptance tests to extract xpack tests
into a helper class to allow the same tests to be used in both 'full'
and 'ubi8' docker image tests
Clean backport of #12135
This commit adds integration tests for the Logstash docker images. Previous
integration tests were removed in https://github.com/elastic/logstash/pull/10693,
due to the tests being non functional.
The commit adds image and container tests. The image tests check the contents and the
metadata of the image; the container tests check the logstash process, and includes tests
ensuring that logstash runs, and is configurable.
This test also adds a ci script to allow the tests to be run on jenkins, and to split the
running of these tests up based on the image type and includes updates to the rake tasks to
support this.
Some QA tests reads the FEATURE_FLAG environment variable, for example to test PQ functionality.
This PR passthrough the environement variable inside the Docker instance.
Fixes#11970
* Defined the versions of JDK to use in test build separated by OS (#11768)
* Added JDK 11 and 14 to Unix testing matrix (#11801)
- OpenJDK14 AdoptOpenJDK11 Zulu11
- OpenJDK14 AdoptOpenJDK14 Zulu14
Close#11801
Co-authored-by: Andrea Selva <andrea.selva@elastic.co>
* Backport of #11742. Not a clean backport as 7.x had not previously been upgraded to 5.6.4 as master had been.
* Update gradle version to 6.3
Gradle versions prior to 6.3 cannot run under JDK14.
This commit upgrades the version of Gradle to 6.3, and removes all deprecation warnings that can currently be removed.
Changes include:
* Increase gradle memory to 2g
* Increase gradle memory in the license check job to 2g
* Replace use of `testCompile`
* Replace `runtime` with `runtimeOnly`
* Remove`compile` depedencies from gradle files
* Replace deprecated archive methods
* Fix dependencies report build
* Make jruby dependencies 'api', fix archiveVersion
* Set `duplicatesStrategy` for all tasks of type Copy
* Use `configureEach` for global 'withType' calls
** Use the recommended Tasks API calls
(https://blog.gradle.org/preview-avoiding-task-configuration-time)
* Run `./gradlew wrapper` earlier to improve caching
* Use copy with chown for resources that need to be run during `./gradlew wrapper`
DRYs up cleanup code and prevents error that is raised when we attempt
to `bundle exec rake qa:vm:halt` when the bundle has not yet been installed.
Fixes#9803
This correctly CDs back and forth between dirs and tracks things
with variables to make it easier on the brain than relative paths like '..'
Fixes#9657
This does a few small things:
1. Prefer invoking rake tasks via ./gradlew vs rake. This prevents Java from segfaulting on docker for unknown reasons
2. Remove use of COPYING.csv which has already been removed
Fixes#9583
