This commit adds JDK matrix Buildkite pipelines for
Windows 2022, 2019 and 2016.
It also makes the groups easier to read (on both Linux and Windows
pipelines) by removing the os-jdk prefix from the job labels.
`testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty`
fails on Windows Buildkite agents and it's a test issue tracked in
https://github.com/elastic/logstash/issues/15562.
Relates:
- https://github.com/elastic/logstash/pull/15539
- https://github.com/elastic/ingest-dev/issues/1725
(cherry picked from commit 0ede19a0e1)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the first part of the migration of JDK matrix tests
from Jenkins to Buildkite. There will be two separate pipelines, for
Linux and Windows.
Linux is currently limited to Ubuntu 22.04 and 20.04, but
additional operating systems will be added outside of the Logstash
repository seamlessly through additional VM images.
Steps are created dynamically and the underlying script is meant to be
common for Linux and Windows. Windows is currently a stub and
will be added in a follow up PR.
Relates:
- https://github.com/elastic/ingest-dev/issues/1725
- https://github.com/elastic/ci-agent-images/pull/424
(cherry picked from commit 956bf483f2)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit splits the generic Buildkite pipelines introduced
in #15520 for JDK tests to separate pipelines for Linux and Windows.
(cherry picked from commit 07147b3e40)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the follow up PR after #15466, which migrates away
the remaining aarch64 acceptance test Jenkins jobs to Buildkite.
Relates:
- #15466
- https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit c384190718)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
PR#15466 skipped the Java unit tests as on the `main` and `8.11`
branches they attempted to run sonar scans (which are only meant to
run for PRs).
This commit re-enables the Java unit tests, taking advantage of #15486,
disabling the sonar scan part of the test suite.
(cherry picked from commit 16da966290)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This commit is the first part of migrating away the aarch64 Jenkins
jobs to Buildkite. It adds a group of exhaustive test steps in the
aarch64 pipeline.
The java unit tests are temporarily disabled as they run SonarQube
scans which need to be associated with pull requests.
Relates:
https://github.com/elastic/ingest-dev/issues/1724
(cherry picked from commit 36656de4f0)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
This is a backport of the initial Pull Request pipeline for Buildkite.
While currently we haven't migrated all PR jobs from Jenkins, this is needed so PRs against non `main` branches don't fail this step (also giving us the possibility to test functionality against non `main` branches).
Relates:
- #15402
- #15413
- #15415
- #15421
- https://github.com/elastic/ingest-dev/issues/1721
## Release notes
[rn:skip]
PR#15392 (which was a backport to `8.10` branch for the Buildkite DRA
scripts) had a bug.
This commit moves the dra scripts under the right directory as used
in the other branches.
* Add known imap and email plugin issues section to Logstash 8.10+ versions.
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This is a backport of the DRA pipeline and related scripts from:
- #15366
- #15365
- #15356
- #15352
- #15344
- #15343
- #15337
- #15312
Note that it's a manual backport because some PRs (e.g. #15312) contain files (`catalog-info.yaml`) that should only
live on the `main` branch.
Expands the description of memory used by Logstash, dividing the heap and non-heap; describing in details which parts composes the non-heap, how to size it and list the JVM settings that can be used to properly define this space.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit b31c429d19)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit adds null guard to get the native thread when constructing pipeline report
Fix: #15300
(cherry picked from commit cd78558121)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
This commit adds a call to identify serverless cluster before fetching `/_xpack` to verify the license and supported features.
When it is serverless, the license checker uses hardcoded xpack info instead of calling xpack API. The internal pipeline of monitoring is disabled and logs error when configured to use legacy monitoring.
Fixes:
ingest-dev#2303
ingest-dev#2284
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
(cherry picked from commit 6b2fa20622)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
Revert PR #15267 which updated Puma in Gemfile.template and correctly place it in logstash-core.gemspec, beeing a dependency of Logstash core.
(cherry picked from commit 58e23fbdc3)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
The DeadLetterQueueWriter#updateOldestSegmentReference was sorting the existing segments by the lexicographic file names. For example, if there are a few segments on the DQL path with index 8, 9 and 10, it was choosing the 10 as the oldest segment instead of 8.
This commit changed it to comparing the segment ID as numbers, so the file deletion will happen following the proper order.
(cherry picked from commit 499091824b)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Pipeline to pipeline communication acked queue improvements.
* Handle InterruptedException exception in input back, warn message improvement when in-flight events are partially sent and other minor such as descriptive logs, etc improvements.
* Apply suggestions from code review
Check if queue is open after thread acquires the lock.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Apply suggestions from code review
Unite test case improvement: use `assertThrows` when validating the exception.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Pulling off of introducing wrap with operations.
* Introduce functional interface to broadly use to catch the exception types.
* Addressing comments: do not retry sending inflight events on case. We still throw if we get error when opening queue.
* will not be reached input retry logic, removing.
* Move queue close check after thread acquiring a lock. Make read next page interface private since it is an internal use purpose.
* Apply suggestions from code review
Leave a comment for the write lock and remove unnecessary warning with `ensure_delivery=>false`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Remove unused method, check if current thread acquired lock when accessing next page.
* pq: getting possibly-shared access to next read page is illegal
The private `Queue#nextReadPage()` method requires that the caller has
exclusive ownership of the lock, and failing to have the lock is an
illegal state that cannot be recoverd from; it would leak the
effectively-private `Page` to a caller that cannot reliably use it
without corrupting other callers.
Both callers of this private method already call it with exclusive
access, so this safeguard is merely to prevent future development from
breaking the expectation unknowingly.
As such, we throw an `IllegalStateException`.
* pq: use shared queue-closed check for block and non-block reads
By moving the closed-check from the blocking `Queue#readBatch` to the
shared private `Queue#nextReadPage`, we ensure that both blocking reads
by `Queue#readBatch` and non-blocking reads by `Queue#nonBlockReadBatch`
behave the same when the queue has been closed.
* pq: make exception message constants descriptive
* p2p: clarify comment about cumulating retry behaviour
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit moves the Files.size(...) call into the try catch block, that way, when the oldest segment is deleted by the DeadLetterQueueReader, no NoSuchFileException will be thrown up, and the writer will gracefully update the oldest segment on the next updateOldestSegmentReference invocation (scheduled flush, entry write, delete expired, etc).
It also adds the volatile keyword to the shared mutable variables, making sure that all the changes will be instantly visible among all the running threads (scheduler & writer).
This commit changes the serverless tests to use a restricted API key with necessary permissions and disables Logstash legacy monitoring test.
Fixes: #15234
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
