This commit adds a skeleton Buildkite pipeline for the Exhaustive tests
suite.
(cherry picked from commit db50983ab5)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This removes the dependency on jackson's dataformat-yaml. Since there's only a single place where this library is used in core: to load the plugin alias definition, the code can be replaced by the underlying snakeyaml.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
So far we've been using images from the -qa GCP image project throughput
the development of the Logstash Linux JDK matrix pipeline for quicker
iteration.
As we have scheduled weekly builds of those images that promote to
prod[^1] we can now switch to the prod version of the GCP images.
[^1]: https://buildkite.com/elastic/ci-vm-images/builds/2888
Relates https://github.com/elastic/ingest-dev/issues/1725
(cherry picked from commit e259e04e53)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
The last part of the Logstash JDK matrix CI migration from Jenkins to
Buildkite is AmazonLinux 2023.
While we have a working image[^1], this is the only step that requires
a agent that runs on AWS.
This commit refactors the builder to support GCP or AWS agents depending
on the OS.
[^1]: https://github.com/elastic/ci-agent-images/pull/441
(cherry picked from commit 8fa3bd0d7f)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Add missing yaml-language-server definition to Buildkite pipeline files (static and dynamic generated) for consistency and to ease spotting errors with editors.
(cherry picked from commit cd01abb1c7)
Backport of #15590
Fix typo for image name of Rocky Linux 8 for JDK matrix jobs.
(cherry picked from commit ce63ea4a51)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds JDK matrix Buildkite pipelines for
Windows 2022, 2019 and 2016.
It also makes the groups easier to read (on both Linux and Windows
pipelines) by removing the os-jdk prefix from the job labels.
`testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty`
fails on Windows Buildkite agents and it's a test issue tracked in
https://github.com/elastic/logstash/issues/15562.
Relates:
- https://github.com/elastic/logstash/pull/15539
- https://github.com/elastic/ingest-dev/issues/1725
(cherry picked from commit 0ede19a0e1)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the first part of the migration of JDK matrix tests
from Jenkins to Buildkite. There will be two separate pipelines, for
Linux and Windows.
Linux is currently limited to Ubuntu 22.04 and 20.04, but
additional operating systems will be added outside of the Logstash
repository seamlessly through additional VM images.
Steps are created dynamically and the underlying script is meant to be
common for Linux and Windows. Windows is currently a stub and
will be added in a follow up PR.
Relates:
- https://github.com/elastic/ingest-dev/issues/1725
- https://github.com/elastic/ci-agent-images/pull/424
(cherry picked from commit 956bf483f2)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Adds filtering on Logstash output message in an integration tests when setting LS_JAVA_HOME environment variable.
(cherry picked from commit 5af14f4e1c)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit splits the generic Buildkite pipelines introduced
in #15520 for JDK tests to separate pipelines for Linux and Windows.
(cherry picked from commit 07147b3e40)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* Adapted the JDK's download URL creation to intereact with Elastic catalog to get metadata, and return the catalog download link instead of directly pointing to Adoptium API
* Silenced the Download task of JDK to print the full url
(cherry picked from commit 73daec05ed)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit is the follow up PR after #15466, which migrates away
the remaining aarch64 acceptance test Jenkins jobs to Buildkite.
Relates:
- #15466
- https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit c384190718)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861
Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message.
(cherry picked from commit 90964fb559)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
PR#15466 skipped the Java unit tests as on the `main` and `8.11`
branches they attempted to run sonar scans (which are only meant to
run for PRs).
This commit re-enables the Java unit tests, taking advantage of #15486,
disabling the sonar scan part of the test suite.
(cherry picked from commit 16da966290)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit introduces a way to optionally disable SonarQube scanning
and coverage reports when running Java unit tests. The integration
was introduced in #15279, however, there are cases (e.g. running Java
unit tests outside of PRs) where we don't want this integration.
Disabling can be achieved by setting the env var `ENABLE_SONARQUBE`
to `false`.
(cherry picked from commit b1ab2fad3e)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the first part of migrating away the aarch64 Jenkins
jobs to Buildkite. It adds a group of exhaustive test steps in the
aarch64 pipeline.
The java unit tests are temporarily disabled as they run SonarQube
scans which need to be associated with pull requests.
Relates:
https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit 36656de4f0)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This is a backport of the initial Pull Request pipeline for Buildkite.
While currently we haven't migrated all PR jobs from Jenkins, this is needed so PRs against non `main` branches don't fail this check (also giving us the possibility to test functionality against non `main` branches).
Relates:
- #15402
- #15413
- #15415
- #15421
- https://github.com/elastic/ingest-dev/issues/1721
## Release notes
[rn:skip]
The new psych 5.1.1 gem seems to not work when installed in JRuby 9.4 (used in main).
This change pins the version back to 5.1.0 until ruby/psych#655 is sorted.
(cherry picked from commit 401d166a89)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Add known imap and email plugin issues section to Logstash 8.10+ versions.
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit a88f82e77f)
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
(cherry picked from commit 94dfd4773b)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit 70081bbcac)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
