* upgrades logstash-mixin-aws to 5.1.0
* removes unused aws-sdk-v1 dependency.
* upgrades json version to 2.6.3
* upgrades fpm to 1.14.1 where 13.x versions.
This commit adds the missing method `worker_threads_draining?` to ruby pipeline which is added in #13934 to java pipeline for log msg improvement
Fixed: #15010
On ARM architecture UBI8 Docker images aren't created, so avoid to create empty tar.gz files.
(cherry picked from commit 7a39d97055)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [DRA] Force docker save to save directly on file instead of pipe to another command loosing the execution error code
(cherry picked from commit 2e5e49d10d)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
During stalled shutdowns while waiting for in-flight batches to complete,
our shutdown watcher emits helpful information about what work is in flight,
including the actual threads and plugins that are still executing.
Since ~6.3.0, the `inflight_count` metric in this log message has always
been `0`, in part because of two somewhat-overlapping bugs:
- elastic/logstash#8987 and elastic/logstash#9056 (7.0, 6.3) changed
the `inflight_batches` map provided by the queue read clients to index
batches by native thread id, but pipeline reporter continued to
attempt to extract by ruby thread object. Because it does not find
the thread in the "batch map", it reports zero.
- elastic/logstash#9111 (7.0, 6.3) changed the _value_ stored in
the `inflight_batches` map provided by a new common queue read client
from an object responding to `#size` to a java `QueueBatch` which
does not respond to `size`. If our pipeline reporter had been able to
look up the queue batch, it would have failed with a `NoMethodError`.
We resolve the issue by (1) extracting the batch from our "batch map" using
the native thread id and (2) safely extracting the value from a `QueueBatch`
before falling through to `Object#size` or 0.
(cherry picked from commit 4941c25f32)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Note this is a manual cherry-pick backport, as it did not backport cleanly. This backport includes some changed/additional code to
the original PR:
* Added additional null check in seekToNextEvent that was previously not present in 7.17, but is required for this PR
* Filter is added, but surrounding code is slightly different, but the intent is the same.
**Backport PR #14605 to 8.5 branch, original message:**
---
<!-- Type of change
Please label this PR with the release version and one of the following labels, depending on the scope of your change:
- bug
- enhancement
- breaking change
- doc
-->
<!-- Add content to appear in [Release Notes](https://www.elastic.co/guide/en/logstash/current/releasenotes.html), or add [rn:skip] to leave this PR out of release notes -->
Fix DLQ fails to start due to read 1 byte file
<!-- Mandatory
Explain here the changes you made on the PR. Please explain the WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
Example:
Expose 'xpack.monitoring.elasticsearch.proxy' in the docker environment variables and update logstash.yml to surface this config option.
This commit exposes the 'xpack.monitoring.elasticsearch.proxy' variable in the docker by adding it in env2yaml.go, which translates from
being an environment variable to a proper yaml config.
Additionally, this PR exposes this setting for both xpack monitoring & management to the logstash.yml file.
-->
This commit ignores DLQ files that contain only the version number. These files have no content and should be skipped.
Mapping 1 byte DLQ files to buffer causes java.lang.IllegalArgumentException: newPosition < 0: (-1 < 0)
User is unable to start the pipeline using dead_letter_queue input
<!-- Mandatory
Explain here the WHY or the IMPACT to the user, or the rationale/motivation for the changes.
Example:
This PR fixes an issue that was preventing the docker image from using the proxy setting when sending xpack monitoring information.
and/or
This PR now allows the user to define the xpack monitoring proxy setting in the docker container.
-->
<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to have the PR approved
List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->
- [ ] My code follows the style guidelines of this project
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
- [ ] I have added tests that prove my fix is effective or that my feature works
<!-- Recommended
Add a checklist of things that are required to be reviewed in order to have the PR approved
-->
- [x] manually test the 1 byte file and can start Logstash
<!-- Recommended
Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
-->
follow the reproducer of #14599
<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.
- Closes#123
- Relates #123
- Requires #123
- Superseeds #123
-->
- Fixed: #14599
<!-- Recommended
Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.
If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
-->
<!-- Optional
Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
-->
<!-- Recommended
Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
-->
* [Doc] Document the usage of LS_JAVA_OPTS environment variable
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
(cherry picked from commit 9242105c3c)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Starting with Log4j2 2.6 if a subclass of MessageFactory associated with an Logger instance
is not subclass of MessageFactory2, then it's wrapped with MessageFactory2Adapter.
This trigger a log4j warn log that, when a class subclasses LogStash::Plugin for example, is noisy and report about
a Logger is not associated with the default MessagedFactory (LogstashMessageFactory) every time a subclass of Plugins is instantiated.
This commit adapt LogstashMessageFactory to implement the MessagedFactory2 instead of the older MessageFactory to avoid the wrapping with the adapter class.
(cherry picked from commit 05bfaff799)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Fix the docker image building and upload process:
* Builds ubi8 on x86_64.
* Uploads ironbank and ubi8 context files from x86_64 only.
(cherry picked from commit 2e8bd20cf5)
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
Updates the dra_docker.sh script to upload also docker-build-context.tar.gz files
(cherry picked from commit 6ad5690a8c)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Ensures the DRA build script surfaces a rake error, instead of allowing the build to continue.
This ensures that the build doesn't continue if any of the steps fails.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
(cherry picked from commit 17d0bb5ffb)
* Generalize docker image building
* Rename and add ability to pass the architecture as a parameter
* Handle ARCH env variable
(cherry picked from commit 6ba5cc112f)
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
Version 7.17 doesn't generate Darwin aarch64 artifacts. Don't download these artifacts from the GCS bucket, given that we don't build Darwin for that release.
(cherry picked from commit 9c7b7b7454)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* DRA: Improve shell scripts for debuggability (#14654). The changes remove some code duplication by introducing a common file that can be sourced between all scripts. It also improves debuggability by adding better messages.
* Fix dra_common sourcing (#14657). Fixes the source of dra_common.sh. It will now first check the directory of the file from which this dra_common.sh script is being called. This allows the common script to be sourced regardless of where the sourcing script is being called from.
* Fix sourcing on dra_upload (#14659). Fix sourcing on dra_upload.sh
* DRA: Handle env variables better
* Moved the addition of SNAPSHOT suffix to the version after the VERSION_QUALIFIER
* Fix badly assigned variable, version qualifier has to be appended also to PLAIN_STACK_VERSION and not RELEASE_VER
Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit db6a7bc619)
The version passed to the release-manager doesn't need the SNAPSHOT particle because already handled by the --workflow="snapshot", if inserted make the release manager to search for artifacts named as 8.5.0-SNAPSHOT-SNAPSHOT
(cherry picked from commit b8792107ad)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Do not move out artifacts from the build/ former to ensure the upload doesn't fail.
(cherry picked from commit 363adad3b6)
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
