Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 43614ede50)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
As a follow up to #15861 this commit splits the current unit tests step
for the Windows JDK matrix pipeline to two that run
Java and Ruby unit tests separately.
Closes https://github.com/elastic/logstash/issues/15566
(cherry picked from commit c0c213d17e)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
PR#15900 missed a few more places where Logstash is installed but
a working minimal pipeline config is added.
This commit fixes that and stabilizes all acceptance tests, thus
minizing the need for time consuming BK retries of corresponding
steps.
Relates #15900
Relates https://github.com/elastic/logstash/issues/15784
(cherry picked from commit 54f73e5d22)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit tightens the checks for the status
output of the Logstash OS service to specifically
scan for `org.logstash.Logstash` rather than
only the jdk path.
The reason is that the startup script first runs
an options parser, and then the logstash process
itself, both referencing the JDK path.
Closes https://github.com/elastic/ingest-dev/issues/2950
(cherry picked from commit eedccea33f)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Logstash on ECK requires openssl command to build TLS keystore.
This commit adds `microdnf install -y openssl` to ensure the command exists in ubi image.
Updates Netty's configuration of maxOrder to a previously proven value, if not already customised by the user.
Adds a step to the JvmOption parsing tool, which is used to compose the JVM options string to pass down to Logstash at startup.
The added step rework the parsed options to set the allocator max order -Dio.netty.allocator.maxOrder=11 so that the maximum pooled buffer is up to 16MB and not 4MB.
This option is added iff it's not yet specified by the user
(cherry picked from commit 52ce3ff8f6)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Update the env2yaml to have a go.mod instead of relying on disabling go modules, otherwise building with golang 1.22 will fail in the future.
This change also directly uses the golang image to build the binary removing the need for an intermediate image.
(cherry picked from commit 5c3e64d591)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes the startup of the Logstash service during packaging
tests by adding a minimal pipeline config. Without it, the service was
flapping from start to start and vice versa causing test flakiness.
Relates https://github.com/elastic/logstash/issues/15784
(cherry picked from commit b66dc7f460)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Similarly to #15874, this commit adds retries
to another group, the acceptance/docker to reduce
build noise from transient issues.
(cherry picked from commit 2fc3f4c21f)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds Debian 12 (Bookworm) to the
Linux JDK matrix pipeline and Compat Phase of the
exhaustive pipeline respectively.
Relates https://github.com/elastic/ingest-dev/issues/2871
(cherry picked from commit fedcf58c48)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit allows separate running of Java and Ruby tests on Windows i.e. the same way as we currently do on unix (unit_tests.sh) via a cli argument.
If no argument has been supplied, both tests are run (as it does now).
The wrapper script is also rewritten from old batch style script to Powershell.
This work allows us to split the existing Windows CI job in a subsequent PR to separate steps, as we currently do on Linux.
Relates: https://github.com/elastic/logstash/issues/15566
(cherry picked from commit 8ac55184b8)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds retries to the steps of the Linux + Windows JDK matrix
pipeline steps to avoid notification noise due to transient network
errors.
(cherry picked from commit 3b747d86b8)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
As a follow up to #15787 we also add Buildkite retries for the
exhaustive pipeline / compatibility group steps to prevent
failures due to flakiness.
(cherry picked from commit 88a32cca81)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds (up to 3) retries for all steps of the `serverless-integration-testing`
pipeline as a stop-gap measure to prevent network related transient failures.
(cherry picked from commit 0d808ed708)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit mutes the DLQ test:
`testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty`
when running on Windows.
Closes https://github.com/elastic/logstash/issues/15768
(cherry picked from commit c33afd4cd0)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit fixes IT failures that frequently occur after
version bumps due to missing unified release snapshot builds for
the new version.
This commit uses project specific DRA snapshot URLs for ES and Filebeat
in all cases apart from release builds.
Closes#2825
(cherry picked from commit d74fea4b55)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
The current mechanism of discovering the latest released version per
branch (via ARTIFACTS_API) isn't foolproof near the time of a new
release, as it may be pick a version that hasn't been released
yet. This leads to failures[^1] of the packaging upgrade tests, as we
attempt to download a package file that doesn't exist yet.
This commit switches to an API that that is more up to date regarding
the release version truth.
[^1]: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/125#018d319b-9a33-4306-b7f2-5b41937a8881/1033-1125
(cherry picked from commit 15e19a96c2)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit makes the generated DRA URL easily accessible via
a Buildkite annotation.
Closes https://github.com/elastic/ingest-dev/issues/2608
(cherry picked from commit c5cb1fe2ed)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit fixes the flaky IT test:
`install non bundle plugin successfully installs the plugin with debug enabled`
by being a bit more lenient with the output which can get garbled by Bundler.
Closes#15801
(cherry picked from commit fc09ad4112)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds the packaging tests (that were refactored in #15696
to not rely on Vagrant) in a new "acceptance" group.
Relates: https://github.com/elastic/ingest-dev/issues/1722
(cherry picked from commit 16c4d8827e)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit modernizes the qa/acceptance (packaging) test framework by
moving away from Vagrant and having the tests operate locally.
As we are migrating to Buildkite, the expectation is that those tests
will run on dedicated vms thus removing the necessity of vagrant.
Relates: https://github.com/elastic/ingest-dev/issues/1722
(cherry picked from commit cebe4a7537)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds the Docker acceptance tests in the acceptance phase
of the exhaustive tests pipeline.
- Relates: https://github.com/elastic/ingest-dev/issues/1722
(cherry picked from commit fca1fccb66)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
There is occasional flakiness mainly with IT tests requiring us to
manually retry such failures when we raise PR (or the first
group of the exhaustive suite, which runs the same steps).
This commit adds up to 3 retries for all the steps of the PR
pipeline.
(cherry picked from commit 739e8a3ef0)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit enables running the exhaustive tests Buildkite pipeline
(i.e. the equivalent to the `main` Jenkins tests) ; the trigger is
code events, i.e. direct pushes, merge commits and creation of new branches.
CI is skipped if changes are only related to files under `docs/`.
(cherry picked from commit c8726b79f3)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Introduces a new interface named SchedulerService to abstract from the ScheduledExecutorService to execute the DLQ flushes of segments. Abstracting from time provides a benefit in testing, where the test doesn't have to wait for things to happen, but those things could happen synchronously.
(cherry picked from commit 6b22cc8dd1)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit added a few jvm.options properties to configure the Jackson read constraints defaults (Maximum Number value length, Maximum String value length, and Maximum Nesting depth).
(cherry picked from commit a21ced0946)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
This commit pins the `childprocess` gem to version `4` since version `5.0.0` of
https://github.com/enkessler/childprocess/pull/175 seems to have broken JRuby support for spawning.
Closes https://github.com/elastic/logstash/issues/15757
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit 9f1d55c6a2)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit adds annotations for Java unit tests (in the pull request pipeline) helping
identify failing unit tests quickly.
(cherry picked from commit 286088915f)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit fixes how the keystore tool handle the command's options, including validation for unknown options, and adding the --stdin flag to the add command.
(cherry picked from commit 41ec183f09)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
PR#15729 missed the input step. As a result when the job is triggered
the steps are executed, but the pause icon still shows in the job
requiring manual unblock[^1]
This commit also skips the input step when the job is triggered from
the scheduler pipeline.
[^1] https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/86
(cherry picked from commit a8b64a32e9)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
The recent PRs #15668 and #15705 refactored jobs with a custom schedule
to leverage a centralized trigger pipeline.
An unexpected sideffect of this is that the conditional for the wait
step doesn't work anymore.
This commit skips the wait step when the JDK matrix pipelines get triggered
from another pipeline.
(cherry picked from commit 82ac474b13)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is a pre-requisite for adding unit + IT tests in a
dedicated phase of the Exhaustive tests pipeline.
It refactors the tests currently used by PR jobs, so that they become
reusable.
(cherry picked from commit 03d7b59f2a)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
