This commit uses the new --qualifier parameter in the release manager
for publishing dra artifacts. Additionally, simplifies the expected
variables to rely on a simple `VERSION_QUALIFIER`.
Snapshot builds are skipped when VERSION_QUALIFIER is set.
Finally, for helping to test DRA PRs, we also allow passing the `DRA_BRANCH` option/env var
to override BUILDKITE_BRANCH.
Closes https://github.com/elastic/ingest-dev/issues/4856
Backported from #16907 cherry picked from 9385cfac5a
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
This commit adds support for building + publishing DRA (-SNAPSHOT for now) artifacts for Logstash. It builds on top of #15312 and therefore only targets the `main` branch and is intended to be run manually during a trial period before we retire the corresponding Jenkins job.
The structure is similar to Jenkins:
1. Three steps runs in parallel to build packages, x86_64 docker and aarch64 docker artifacts.
2. Once 1. is successfully done, use release manager to publish the artifacts.
We generate the pipeline steps for 1. and 2. dynamically (with a simple Python script) to avoid repetition for future PRs: we will add a new pipeline in a follow up PR for -STAGING.
The actual shell scripts are simplified copies from the existing `dra*` scripts under https://github.com/elastic/logstash/tree/main/ci; the simplification comes from native support for copying artifacts between steps in Buildkite and
not having to use an intermediate bucket.
Relates: https://github.com/elastic/ingest-dev/issues/1720
Blocked by: https://github.com/elastic/ci/pull/2312/files
Add initial buildkite pipeline for DRA artifacts.
This is a "hello-world" pipeline skeleton required to iterate on migrating the actual DRA jobs.
This commit also updates the backstage component for logstash to become a top level entity with the
necessary annotations and linked dependencies to pipelines.
Relates https://github.com/elastic/ingest-dev/issues/1720
