Defines a Buildkite pipeline to run specs tests on tier1 and tier2 plugins, using container images. The tasks are divided by plugin type for each tier, to run them in parallel. Once all tier1 testing is completed then starts with tier2.
This commit adds a Buildkite resource for pull requests and
the two simple jobs, license checking and ruby unit tests that are
already part of the Jenkins PR multi job setup.
As this is WiP, slack notifications aren't enabled.
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
We seem to be hitting a weird issue with the psych gem when running in a container:
```
> Task :bootstrap
2023-09-27T15:13:23.844Z [Execution worker Thread 5] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
Error Errno::ENOENT, retrying 1/10
No such file or directory - /buildkite/builds/bk-agent-prod-k8s-1695827479442731913/elastic/logstash-dra-pipeline-ci/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/deps.lst
```
This commit works around this problem by running inside a VM. The underlying root cause (and that gradle should be erroring in this case) will be investigated separately.
We've spotted network flakiness downloading artifacts with gradle (connection resets) when using GCP/t2a on us-central1.
This commit switches to AWS Graviton instance types for building the aarch64 artifacts.
This commit adds support for building + publishing DRA (-SNAPSHOT for now) artifacts for Logstash. It builds on top of #15312 and therefore only targets the `main` branch and is intended to be run manually during a trial period before we retire the corresponding Jenkins job.
The structure is similar to Jenkins:
1. Three steps runs in parallel to build packages, x86_64 docker and aarch64 docker artifacts.
2. Once 1. is successfully done, use release manager to publish the artifacts.
We generate the pipeline steps for 1. and 2. dynamically (with a simple Python script) to avoid repetition for future PRs: we will add a new pipeline in a follow up PR for -STAGING.
The actual shell scripts are simplified copies from the existing `dra*` scripts under https://github.com/elastic/logstash/tree/main/ci; the simplification comes from native support for copying artifacts between steps in Buildkite and
not having to use an intermediate bucket.
Relates: https://github.com/elastic/ingest-dev/issues/1720
Blocked by: https://github.com/elastic/ci/pull/2312/files
Add initial buildkite pipeline for DRA artifacts.
This is a "hello-world" pipeline skeleton required to iterate on migrating the actual DRA jobs.
This commit also updates the backstage component for logstash to become a top level entity with the
necessary annotations and linked dependencies to pipelines.
Relates https://github.com/elastic/ingest-dev/issues/1720
This commit changes the serverless tests to use a restricted API key with necessary permissions and disables Logstash legacy monitoring test.
Fixes: #15234
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Snyk scans Logstash container vulnerabilities.
* Exclude integ test and tools when Snyk scanning.
* Remote repo url fix for main branch.
* Update .buildkite/scripts/snyk/report.sh
Simplify the logic to retrieve the version from `versions.yml`
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Add backstage definition for Snyk Report pipeline.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
This will eventually replace the pipeline definition in the catalog repo, using the local file instead.
---------
Co-authored-by: elastic-backstage-prod[bot] <123036547+elastic-backstage-prod[bot]@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* This commit includes number of changes:
- adds initial pipeline steps, each step proceeds one branch report job
- a script to resolve current and previous release latest branches from the release URL. Noting that it considers the branches from snapshots where upcoming releases will be our interest.
- pseudo logic to download logstash, build and report to Snyk. In order to script fully work we need a logstash machine to access Buildkite vault, which I am working.
* Source optimization and use Buildkite Vault to access to Snyk token.
* Add git branch and head to Snyk project tag.
* Dynamically resolve latest branches instead of manually defining.
* Update .buildkite/scripts/snyk/report.sh
Improve comment readability.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
