PR#15900 missed a few more places where Logstash is installed but
a working minimal pipeline config is added.
This commit fixes that and stabilizes all acceptance tests, thus
minizing the need for time consuming BK retries of corresponding
steps.
Relates #15900
Relates https://github.com/elastic/logstash/issues/15784
This commit tightens the checks for the status
output of the Logstash OS service to specifically
scan for `org.logstash.Logstash` rather than
only the jdk path.
The reason is that the startup script first runs
an options parser, and then the logstash process
itself, both referencing the JDK path.
Closes https://github.com/elastic/ingest-dev/issues/2950
* Exclude plugins feature in OSS distributions.
* Set elastic_integration plugin default.
* Remove non-OSS plugins after installing default plugins.
* Testing local can't find gem bundler (= 2.3.26) issue.
* Include extract non-OSS plugins logic indocker build operations.
* Only default plugins can be excluded from OSS distros.
* Simplification: instead conditional check, use intersection to make OSS exlucluded plugin list.
* Gem and specification files still stay after removing the plugin. This change removes the stayed files.
* Rename oss-exclude to skip-oss to align namings with other params.
* Make intersection method simpler.
* [Test] Temporary excluding elastic integration plugin from default plugin list.
* Sets elastic_integration plugin default back. When removing locally installed gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle.
* Test the build order, remove plugin from cache logic seems invalid since we don't pack the cache.
Our shared CI infrastructure relies on this file being here with a specific
format, expanding a given `ELASTIC_STACK_VERSION` environment variable and
an optional `SNAPSHOT` environment variable into the information necessary
to target either the most recent published release or a snapshot representing
a candidate for the _next_ possible release in the series.
When we cut a new minor branch from `main` (at feature freeze), the tip of
that branch contains unreleased code that is in-flight for the next release.
We need to have a way to reference artifacts from this in-flight release,
instead of skipping over them while a minor release is in flight.
This change affects the _semantics_ of what is tested during a minor stack
release's feature-freeze period, ensuring that automated tests throughout
the Logstash ecosystem continue to run against the pending release with NO
changes required in those plugins.
It also introduces a new `snapshots.main` entry, which refers to snapshot
artifacts built from the un-feature-frozen `main` branch, so that plugins
can _optionally_ test against the _next_ minor in advance of its feature
freeze, as they currently do unintentionally in builds that target
`ELASTIC_STACK_VERSION=8.x SNAPSHOT=true`.
Logstash on ECK requires openssl command to build TLS keystore.
This commit adds `microdnf install -y openssl` to ensure the command exists in ubi image.
Updates Netty's configuration of maxOrder to a previously proven value, if not already customised by the user.
Adds a step to the JvmOption parsing tool, which is used to compose the JVM options string to pass down to Logstash at startup.
The added step rework the parsed options to set the allocator max order -Dio.netty.allocator.maxOrder=11 so that the maximum pooled buffer is up to 16MB and not 4MB.
This option is added iff it's not yet specified by the user
Update the env2yaml to have a go.mod instead of relying on disabling go modules, otherwise building with golang 1.22 will fail in the future.
This change also directly uses the golang image to build the binary removing the need for an intermediate image.
* Fix the latest 8.12 version, to the actually latest released
* Update version fo 7.17
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit fixes the startup of the Logstash service during packaging
tests by adding a minimal pipeline config. Without it, the service was
flapping from start to start and vice versa causing test flakiness.
Relates https://github.com/elastic/logstash/issues/15784
This commit allows separate running of Java and Ruby tests on Windows i.e. the same way as we currently do on unix (unit_tests.sh) via a cli argument.
If no argument has been supplied, both tests are run (as it does now).
The wrapper script is also rewritten from old batch style script to Powershell.
This work allows us to split the existing Windows CI job in a subsequent PR to separate steps, as we currently do on Linux.
Relates: https://github.com/elastic/logstash/issues/15566
Currently for pipelines that have steps with auto retries, if there
is a failure of a step, even if the subsequent retry succeeds, we
received Slack alerts.
This commits uses new functionality available in the Buildkite PR Bot
to avoid Slack notifications in those cases and changes the settings
for all pipelines where we have, or expect to have, retries.
This commit adds a schedule to run the exhaustive pipeline
(biweekly, every other Wednesday @2AM UTC) and the aarch64
(weekly, every Monday@2AM UTC).
Closes https://github.com/elastic/ingest-dev/issues/2852
This commit adds (up to 3) retries for all steps of the `serverless-integration-testing`
pipeline as a stop-gap measure to prevent network related transient failures.
* Describe how to troubleshoot connectivity problems at HTTP level with Azure Blob Storage
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This commit fixes IT failures that frequently occur after
version bumps due to missing unified release snapshot builds for
the new version.
This commit uses project specific DRA snapshot URLs for ES and Filebeat
in all cases apart from release builds.
Closes#2825
The current mechanism of discovering the latest released version per
branch (via ARTIFACTS_API) isn't foolproof near the time of a new
release, as it may be pick a version that hasn't been released
yet. This leads to failures[^1] of the packaging upgrade tests, as we
attempt to download a package file that doesn't exist yet.
This commit switches to an API that that is more up to date regarding
the release version truth.
[^1]: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/125#018d319b-9a33-4306-b7f2-5b41937a8881/1033-1125
This commit fixes the flaky IT test:
`install non bundle plugin successfully installs the plugin with debug enabled`
by being a bit more lenient with the output which can get garbled by Bundler.
Closes#15801
* Ruby code coverage with SimpleCov json formatter
* [CI] Send Java and ruby tests to sonarqube simultaneously
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Added compiled classes to artifacts
* Test change
* Removed test changes
* Returned back ENABLE_SONARQUBE condition
* Removed debug line
* Diable Ruby coverage if ENABLE_SONARQUBE is not true
* Run sonar scan on pull requests and onn push to main
* Run sonar can on release branches
