* Add known imap and email plugin issues section to Logstash 8.10+ versions.
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit a88f82e77f)
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
(cherry picked from commit 94dfd4773b)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit 70081bbcac)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
We seem to be hitting a weird issue with the psych gem when running in a container:
```
> Task :bootstrap
2023-09-27T15:13:23.844Z [Execution worker Thread 5] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
Error Errno::ENOENT, retrying 1/10
No such file or directory - /buildkite/builds/bk-agent-prod-k8s-1695827479442731913/elastic/logstash-dra-pipeline-ci/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/deps.lst
```
This commit works around this problem by running inside a VM. The underlying root cause (and that gradle should be erroring in this case) will be investigated separately.
We've spotted network flakiness downloading artifacts with gradle (connection resets) when using GCP/t2a on us-central1.
This commit switches to AWS Graviton instance types for building the aarch64 artifacts.
This commit adds support for building + publishing DRA (-SNAPSHOT for now) artifacts for Logstash. It builds on top of #15312 and therefore only targets the `main` branch and is intended to be run manually during a trial period before we retire the corresponding Jenkins job.
The structure is similar to Jenkins:
1. Three steps runs in parallel to build packages, x86_64 docker and aarch64 docker artifacts.
2. Once 1. is successfully done, use release manager to publish the artifacts.
We generate the pipeline steps for 1. and 2. dynamically (with a simple Python script) to avoid repetition for future PRs: we will add a new pipeline in a follow up PR for -STAGING.
The actual shell scripts are simplified copies from the existing `dra*` scripts under https://github.com/elastic/logstash/tree/main/ci; the simplification comes from native support for copying artifacts between steps in Buildkite and
not having to use an intermediate bucket.
Relates: https://github.com/elastic/ingest-dev/issues/1720
Blocked by: https://github.com/elastic/ci/pull/2312/files
Expands the description of memory used by Logstash, dividing the heap and non-heap; describing in details which parts composes the non-heap, how to size it and list the JVM settings that can be used to properly define this space.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Since the DRA build for 8.10.1 was made with 82daae80bb , this fix didn't get in.
(cherry picked from commit aa9265665e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Add initial buildkite pipeline for DRA artifacts.
This is a "hello-world" pipeline skeleton required to iterate on migrating the actual DRA jobs.
This commit also updates the backstage component for logstash to become a top level entity with the
necessary annotations and linked dependencies to pipelines.
Relates https://github.com/elastic/ingest-dev/issues/1720
This commit adds a call to identify serverless cluster before fetching `/_xpack` to verify the license and supported features.
When it is serverless, the license checker uses hardcoded xpack info instead of calling xpack API. The internal pipeline of monitoring is disabled and logs error when configured to use legacy monitoring.
Fixes:
ingest-dev#2303
ingest-dev#2284
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
The DeadLetterQueueWriter#updateOldestSegmentReference was sorting the existing segments by the lexicographic file names. For example, if there are a few segments on the DQL path with index 8, 9 and 10, it was choosing the 10 as the oldest segment instead of 8.
This commit changed it to comparing the segment ID as numbers, so the file deletion will happen following the proper order.
* Pipeline to pipeline communication acked queue improvements.
* Handle InterruptedException exception in input back, warn message improvement when in-flight events are partially sent and other minor such as descriptive logs, etc improvements.
* Apply suggestions from code review
Check if queue is open after thread acquires the lock.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Apply suggestions from code review
Unite test case improvement: use `assertThrows` when validating the exception.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Pulling off of introducing wrap with operations.
* Introduce functional interface to broadly use to catch the exception types.
* Addressing comments: do not retry sending inflight events on case. We still throw if we get error when opening queue.
* will not be reached input retry logic, removing.
* Move queue close check after thread acquiring a lock. Make read next page interface private since it is an internal use purpose.
* Apply suggestions from code review
Leave a comment for the write lock and remove unnecessary warning with `ensure_delivery=>false`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Remove unused method, check if current thread acquired lock when accessing next page.
* pq: getting possibly-shared access to next read page is illegal
The private `Queue#nextReadPage()` method requires that the caller has
exclusive ownership of the lock, and failing to have the lock is an
illegal state that cannot be recoverd from; it would leak the
effectively-private `Page` to a caller that cannot reliably use it
without corrupting other callers.
Both callers of this private method already call it with exclusive
access, so this safeguard is merely to prevent future development from
breaking the expectation unknowingly.
As such, we throw an `IllegalStateException`.
* pq: use shared queue-closed check for block and non-block reads
By moving the closed-check from the blocking `Queue#readBatch` to the
shared private `Queue#nextReadPage`, we ensure that both blocking reads
by `Queue#readBatch` and non-blocking reads by `Queue#nonBlockReadBatch`
behave the same when the queue has been closed.
* pq: make exception message constants descriptive
* p2p: clarify comment about cumulating retry behaviour
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit moves the Files.size(...) call into the try catch block, that way, when the oldest segment is deleted by the DeadLetterQueueReader, no NoSuchFileException will be thrown up, and the writer will gracefully update the oldest segment on the next updateOldestSegmentReference invocation (scheduled flush, entry write, delete expired, etc).
It also adds the volatile keyword to the shared mutable variables, making sure that all the changes will be instantly visible among all the running threads (scheduler & writer).
This commit changes the serverless tests to use a restricted API key with necessary permissions and disables Logstash legacy monitoring test.
Fixes: #15234
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
