* qa: don't bypass plugin manger tests on linux
* add gradle task to build gem fixtures for integration tests
(cherry picked from commit 0f81816311)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit adds Windows 2025 to the Windows JDK matrix and exhaustive tests pipelines.
(cherry picked from commit 4d52b7258d)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
In https://github.com/elastic/logstash/pull/17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
(cherry picked from commit 227c0d8150)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.
(cherry picked from commit 91258c3f98)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit updates the acceptance tests to expect messages in the updated
format for removing plugins. See https://github.com/elastic/logstash/pull/17030
for change.
(cherry picked from commit e8e24a0397)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
This commit updates the acceptance tests to expect messages in the updated
format for removing plugins. See https://github.com/elastic/logstash/pull/17030
for change.
(cherry picked from commit e094054c0e)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Removal works in a single pass by finding plugins that would have unmet
dependencies if all of the specified plugins were to be removed, and
proceeding with the removal only if no conflicts were created.
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-remove-multiple ✘) }
> ╰─● bin/logstash-plugin remove logstash-input-syslog logstash-filter-grok
> Using system java: /Users/rye/.jenv/shims/java
> Resolving dependencies......
> Successfully removed logstash-input-syslog
> Successfully removed logstash-filter-grok
> [success (00:00:05)]
~~~
(cherry picked from commit 089558801e)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* spec: improve ls2ls spec
- fixes upstream/downstream convention
- upstream: the sending logstash (has an LS output)
- downstream: the receiving logstash (has an LS input)
- helper `run_logstash_instance` yields the `LogstashService` instance
and handles the teardown.
- pass the pipeline id and node name to the LS instances via command line
flags to make logging easier to differentiate
- use the generator input's sequence id to ensure that the _actual_ events
generated are received by the downstream pipeline
* start with port-offset 100
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit 9abad6609c)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* qa: use clean expansion of LS tarball per fixture instance
Because QA tests can _modify_ the Logstash installation (e.g. those that
invoke the plugin manager), it is important that the service wrapper
begins with a clean expansion of the logstash tarball.
* qa: enable safe reuse of ls_home in ls_to_ls tests
(cherry picked from commit d20eb4dbcb)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Currently the deserialization is behind the readBatch's lock, so any large batch will take time deserializing, causing any other Queue writer (e.g. netty executor threads) and any other Queue reader (pipeline worker) to block.
This commit moves the deserialization out of the lock, allowing multiple pipeline workers to deserialize batches concurrently.
- add intermediate batch-holder from `Queue` methods
- make the intermediate batch-holder a private inner class of `Queue` with a descriptive name `SerializedBatchHolder`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit 637f447b88)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Starting from es-output 12.0.2, a 404 response is treated as an error. Previously, central pipeline management considered 404 as an empty pipeline, not an error.
This commit restores the expected behavior by handling 404 gracefully and logs a user-friendly message.
It also removes the redundant cache of pipeline in CPM
Fixes: #17035
(cherry picked from commit e896cd727d)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Update release notes for 9.0.0
* First clean up: non end user related changes.
* Start cleanup of generated content
* Humanize github commit notes.
* Remove same items shipped in 8.x, simplify SSL removed settings.
* Categorize breaking change section and standardize the notes.
* Move base image change to breaking change section, plugin releases included in 8.x removed.
* Remove Elastic_enterprise_search plugin change.
* Make a single simple line if posssible.
* Update docs/static/releasenotes.asciidoc
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Move avoiding JSON log collision to breaking change section.
* Apply suggestions from code review
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* github-action: Add AsciiDoc freeze warning (#16969)
* Add 9.0 branch to the CI branches definition (#16997)
* Core version bump to 9.1.0 (#16991)
* Use centralized source of truth for active branches (#17063)
This commit simplifies the DRA process in Logstash by removing the need to maintain a separate file for the active branches, and instead rely on a centrally maintained file containing source of truth.
While at it, we refactor/simplify the creation of an array with the versions in `.buildkite/scripts/snyk/resolve_stack_version.sh`.
* Fix conflicts
---------
Co-authored-by: Jan Calanog <nejcalanog@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit preserves spaces in values, ensuring that multi-word strings are stored as intended.
Prior to this change, `logstash-keystore` incorrectly handled values containing spaces,
causing only the first word to be stored.
(cherry picked from commit 5573b5ad77)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
PR #17006 revealed that the `VERSION_QUALIFIER` env var gets honored in
various scripts when present but empty.
This shouldn't be the case as the DRA process is designed to gracefully
ignore empty values for this variable.
This commit changes various ruby scripts to not treat "" as truthy.
Bash scripts (used by CI etc.) are already ok with this as part of
refactorings done in #16907.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit c7204fd7d6)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Backport PR #16968 to 9.0 branch, original message:
----
Permit to use effectively the tokenizer also in context where a line is bigger than a limit.
Fixes an issues related to token size limit error, when the offending token was bigger than the input fragment in happened that the tokenzer wasn't unable to recover the token stream from the first delimiter after the offending token but messed things, loosing part of tokens.
## How solve the problem
This is a second take to fix the processing of tokens from the tokenizer after a buffer full error. The first try #16482 was rollbacked to the encoding error #16694.
The first try failed on returning the tokens in the same encoding of the input.
This PR does a couple of things:
- accumulates the tokens, so that after a full condition can resume with the next tokens after the offending one.
- respect the encoding of the input string. Use `concat` method instead of `addAll`, which avoid to convert RubyString to String and back to RubyString. When return the head `StringBuilder` it enforce the encoding with the input charset.
(cherry picked from commit 1c8cf546c2)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This reverts commit ac56d99553.
As mentioned in https://github.com/elastic/logstash/pull/16995#issuecomment-2626811054 testing revealed
that even an empty VERSION_QUALIFIER env var currently gets honored for snapshot builds (resulting in an unnecessary empty "-" suffix)
so there's a little more prerequisite work needed.
To avoid manual invocations specifying the version qualifier for
prereleases in this commit we leverage a centralized version of truth
for the version qualifier.
We also honor the `DRA_BRANCH` BK var for the generated step names
(the functionality was already there, it's just a cosmetic improvement).
Log input has been deprecated in filebeat 9.0.0 and throws an error if it's present in the configuration.
This commit switches the configuration to the "filestream" input.
This commit removed irrelevant warning for internal pipeline, such as monitoring pipeline.
Monitoring pipeline is expected to be one worker. The warning is not useful
Fixes: #13298
Reimplements `LogStash::Setting::String` Ruby setting class into the `org.logstash.settings.SettingString` and exposes it through `java_import` as `LogStash::Setting::SettingString`.
Updates the rspec tests in two ways:
- logging mock is now converted to real Log4J appender that spy log line that are later verified
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2
Related to: https://github.com/jruby/jruby/issues/8579
This commit uses the new --qualifier parameter in the release manager
for publishing dra artifacts. Additionally, simplifies the expected
variables to rely on a simple `VERSION_QUALIFIER`.
Snapshot builds are skipped when VERSION_QUALIFIER is set.
Finally, for helping to test DRA PRs, we also allow passing the `DRA_BRANCH` option/env var
to override BUILDKITE_BRANCH.
Closes https://github.com/elastic/ingest-dev/issues/4856
* Add a new pattern %{{TIME_NOW}} to `event.sprintf` to generate a fresh timestamp.
The timestamp is represented as a string in the default ISO 8601 format
For example,
```
input {
heartbeat {
add_field => { "heartbeat_time" => "%{{TIME_NOW}}" }
}
}
```
VERSION_QUALIFIER was already observed in rake artifacts task but only to influence the name of the artifact.
This commit ensure that the qualifier is also displayed in the cli and in the http api.
