This commit adds the packaging tests (that were refactored in #15696
to not rely on Vagrant) in a new "acceptance" group.
Relates: https://github.com/elastic/ingest-dev/issues/1722
(cherry picked from commit 16c4d8827e)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds the Docker acceptance tests in the acceptance phase
of the exhaustive tests pipeline.
- Relates: https://github.com/elastic/ingest-dev/issues/1722
(cherry picked from commit fca1fccb66)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit enables running the exhaustive tests Buildkite pipeline
(i.e. the equivalent to the `main` Jenkins tests) ; the trigger is
code events, i.e. direct pushes, merge commits and creation of new branches.
CI is skipped if changes are only related to files under `docs/`.
(cherry picked from commit c8726b79f3)
This commit adds the compatibility tier for the Exhaustive tests suite.
Specifically, we introduce two new groups (running in parallel) for Linux and Windows compat tests.
Linux picks one OS per family from [^] and likewise Windows one of the three available choices from the same file.
We also support manual override, if user chooses to, by setting `LINUX_OS` or `WINDOWS_OS` as env vars in the Buildkite build prompt (in this case there is no randomization, and only one OS can be defined for Linux and Windows respectively).
For example:
```
LINUX_OS=rhel-9
WINDOWS_OS=windows=216
```
Relates:
- https://github.com/elastic/ingest-dev/issues/1722
[^1]: 4d6bd955e6/.buildkite/scripts/common/vm-images.json
(cherry picked from commit d42b938f81)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
So far we've been using images from the -qa GCP image project throughput
the development of the Logstash Linux JDK matrix pipeline for quicker
iteration.
As we have scheduled weekly builds of those images that promote to
prod[^1] we can now switch to the prod version of the GCP images.
[^1]: https://buildkite.com/elastic/ci-vm-images/builds/2888
Relates https://github.com/elastic/ingest-dev/issues/1725
(cherry picked from commit e259e04e53)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
The last part of the Logstash JDK matrix CI migration from Jenkins to
Buildkite is AmazonLinux 2023.
While we have a working image[^1], this is the only step that requires
a agent that runs on AWS.
This commit refactors the builder to support GCP or AWS agents depending
on the OS.
[^1]: https://github.com/elastic/ci-agent-images/pull/441
(cherry picked from commit 8fa3bd0d7f)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Add missing yaml-language-server definition to Buildkite pipeline files (static and dynamic generated) for consistency and to ease spotting errors with editors.
(cherry picked from commit cd01abb1c7)
Backport of #15590
This commit adds JDK matrix Buildkite pipelines for
Windows 2022, 2019 and 2016.
It also makes the groups easier to read (on both Linux and Windows
pipelines) by removing the os-jdk prefix from the job labels.
`testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty`
fails on Windows Buildkite agents and it's a test issue tracked in
https://github.com/elastic/logstash/issues/15562.
Relates:
- https://github.com/elastic/logstash/pull/15539
- https://github.com/elastic/ingest-dev/issues/1725
(cherry picked from commit 0ede19a0e1)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the first part of the migration of JDK matrix tests
from Jenkins to Buildkite. There will be two separate pipelines, for
Linux and Windows.
Linux is currently limited to Ubuntu 22.04 and 20.04, but
additional operating systems will be added outside of the Logstash
repository seamlessly through additional VM images.
Steps are created dynamically and the underlying script is meant to be
common for Linux and Windows. Windows is currently a stub and
will be added in a follow up PR.
Relates:
- https://github.com/elastic/ingest-dev/issues/1725
- https://github.com/elastic/ci-agent-images/pull/424
(cherry picked from commit 956bf483f2)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the follow up PR after #15466, which migrates away
the remaining aarch64 acceptance test Jenkins jobs to Buildkite.
Relates:
- #15466
- https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit c384190718)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the first part of migrating away the aarch64 Jenkins
jobs to Buildkite. It adds a group of exhaustive test steps in the
aarch64 pipeline.
The java unit tests are temporarily disabled as they run SonarQube
scans which need to be associated with pull requests.
Relates:
https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit 36656de4f0)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This is a backport of the initial Pull Request pipeline for Buildkite.
While currently we haven't migrated all PR jobs from Jenkins, this is needed so PRs against non `main` branches don't fail this check (also giving us the possibility to test functionality against non `main` branches).
Relates:
- #15402
- #15413
- #15415
- #15421
- https://github.com/elastic/ingest-dev/issues/1721
## Release notes
[rn:skip]
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
(cherry picked from commit 94dfd4773b)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
We seem to be hitting a weird issue with the psych gem when running in a container:
```
> Task :bootstrap
2023-09-27T15:13:23.844Z [Execution worker Thread 5] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
Error Errno::ENOENT, retrying 1/10
No such file or directory - /buildkite/builds/bk-agent-prod-k8s-1695827479442731913/elastic/logstash-dra-pipeline-ci/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/deps.lst
```
This commit works around this problem by running inside a VM. The underlying root cause (and that gradle should be erroring in this case) will be investigated separately.
We've spotted network flakiness downloading artifacts with gradle (connection resets) when using GCP/t2a on us-central1.
This commit switches to AWS Graviton instance types for building the aarch64 artifacts.
This commit adds support for building + publishing DRA (-SNAPSHOT for now) artifacts for Logstash. It builds on top of #15312 and therefore only targets the `main` branch and is intended to be run manually during a trial period before we retire the corresponding Jenkins job.
The structure is similar to Jenkins:
1. Three steps runs in parallel to build packages, x86_64 docker and aarch64 docker artifacts.
2. Once 1. is successfully done, use release manager to publish the artifacts.
We generate the pipeline steps for 1. and 2. dynamically (with a simple Python script) to avoid repetition for future PRs: we will add a new pipeline in a follow up PR for -STAGING.
The actual shell scripts are simplified copies from the existing `dra*` scripts under https://github.com/elastic/logstash/tree/main/ci; the simplification comes from native support for copying artifacts between steps in Buildkite and
not having to use an intermediate bucket.
Relates: https://github.com/elastic/ingest-dev/issues/1720
Blocked by: https://github.com/elastic/ci/pull/2312/files
* Snyk scans Logstash container vulnerabilities.
* Exclude integ test and tools when Snyk scanning.
* Remote repo url fix for main branch.
* Update .buildkite/scripts/snyk/report.sh
Simplify the logic to retrieve the version from `versions.yml`
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Add backstage definition for Snyk Report pipeline.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* This commit includes number of changes:
- adds initial pipeline steps, each step proceeds one branch report job
- a script to resolve current and previous release latest branches from the release URL. Noting that it considers the branches from snapshots where upcoming releases will be our interest.
- pseudo logic to download logstash, build and report to Snyk. In order to script fully work we need a logstash machine to access Buildkite vault, which I am working.
* Source optimization and use Buildkite Vault to access to Snyk token.
* Add git branch and head to Snyk project tag.
* Dynamically resolve latest branches instead of manually defining.
* Update .buildkite/scripts/snyk/report.sh
Improve comment readability.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
