* Upgrade elasticsearch-ruby client. (#17161)
* Fix Faraday removed basic auth option and apply the ES client module name change.
(cherry picked from commit e748488e4a)
* Update elasticsearch-ruby client in gemfile lock.
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
The https://github.com/elastic/logstash/pull/17310 PR changed the rake task for
artifact creation to use shellwords from standard library. The acceptance tests
need to explitily load that library. This commit updates the rake file to handle
loading the required code.
(cherry picked from commit 964468f922)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Previously when rake would shell out the output would be lost. This
made debugging CI logs difficult. This commit updates the stack with
improved message surfacing on error.
(cherry picked from commit 0d931a502a)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.
This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue
Fixes: #17209
(cherry picked from commit ba5f21576c)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
migrate benchmark to the new artifacts-api
(cherry picked from commit d61a83abbe)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Additional cleanify changes to ls2ls integ tests: replace heartbeat-input with reload option, set queue drain to get consistent result.
(cherry picked from commit 1e06eea86e)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* tests: ls2ls delay checking until events have been processed
* Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec.
* Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable.
---------
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit 73ffa243bf)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit refactors the `PersistedQueueConfigValidator` class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
https://github.com/elastic/logstash/issues/14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.
(cherry picked from commit 062154494a)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
This commit migrates gradle task to the new artifacts-api
- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
with priority given in that order.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 0a745686f6)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Fixed an issue where the `/_node/stats` API displayed empty pipeline metrics
when X-Pack monitoring was enabled
(cherry picked from commit 86785815bd)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* build full docker image from dockerfiles during docker acceptance tests
(cherry picked from commit fecfc7c602)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* qa: don't bypass plugin manger tests on linux
* add gradle task to build gem fixtures for integration tests
(cherry picked from commit 0f81816311)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit adds Windows 2025 to the Windows JDK matrix and exhaustive tests pipelines.
(cherry picked from commit 4d52b7258d)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
In https://github.com/elastic/logstash/pull/17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
(cherry picked from commit 227c0d8150)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.
(cherry picked from commit 91258c3f98)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit updates the acceptance tests to expect messages in the updated
format for removing plugins. See https://github.com/elastic/logstash/pull/17030
for change.
(cherry picked from commit e8e24a0397)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
This commit updates the acceptance tests to expect messages in the updated
format for removing plugins. See https://github.com/elastic/logstash/pull/17030
for change.
(cherry picked from commit e094054c0e)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Removal works in a single pass by finding plugins that would have unmet
dependencies if all of the specified plugins were to be removed, and
proceeding with the removal only if no conflicts were created.
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-remove-multiple ✘) }
> ╰─● bin/logstash-plugin remove logstash-input-syslog logstash-filter-grok
> Using system java: /Users/rye/.jenv/shims/java
> Resolving dependencies......
> Successfully removed logstash-input-syslog
> Successfully removed logstash-filter-grok
> [success (00:00:05)]
~~~
(cherry picked from commit 089558801e)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* spec: improve ls2ls spec
- fixes upstream/downstream convention
- upstream: the sending logstash (has an LS output)
- downstream: the receiving logstash (has an LS input)
- helper `run_logstash_instance` yields the `LogstashService` instance
and handles the teardown.
- pass the pipeline id and node name to the LS instances via command line
flags to make logging easier to differentiate
- use the generator input's sequence id to ensure that the _actual_ events
generated are received by the downstream pipeline
* start with port-offset 100
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit 9abad6609c)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* qa: use clean expansion of LS tarball per fixture instance
Because QA tests can _modify_ the Logstash installation (e.g. those that
invoke the plugin manager), it is important that the service wrapper
begins with a clean expansion of the logstash tarball.
* qa: enable safe reuse of ls_home in ls_to_ls tests
(cherry picked from commit d20eb4dbcb)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Currently the deserialization is behind the readBatch's lock, so any large batch will take time deserializing, causing any other Queue writer (e.g. netty executor threads) and any other Queue reader (pipeline worker) to block.
This commit moves the deserialization out of the lock, allowing multiple pipeline workers to deserialize batches concurrently.
- add intermediate batch-holder from `Queue` methods
- make the intermediate batch-holder a private inner class of `Queue` with a descriptive name `SerializedBatchHolder`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit 637f447b88)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Starting from es-output 12.0.2, a 404 response is treated as an error. Previously, central pipeline management considered 404 as an empty pipeline, not an error.
This commit restores the expected behavior by handling 404 gracefully and logs a user-friendly message.
It also removes the redundant cache of pipeline in CPM
Fixes: #17035
(cherry picked from commit e896cd727d)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Update release notes for 9.0.0
* First clean up: non end user related changes.
* Start cleanup of generated content
* Humanize github commit notes.
* Remove same items shipped in 8.x, simplify SSL removed settings.
* Categorize breaking change section and standardize the notes.
* Move base image change to breaking change section, plugin releases included in 8.x removed.
* Remove Elastic_enterprise_search plugin change.
* Make a single simple line if posssible.
* Update docs/static/releasenotes.asciidoc
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Move avoiding JSON log collision to breaking change section.
* Apply suggestions from code review
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* github-action: Add AsciiDoc freeze warning (#16969)
* Add 9.0 branch to the CI branches definition (#16997)
* Core version bump to 9.1.0 (#16991)
* Use centralized source of truth for active branches (#17063)
This commit simplifies the DRA process in Logstash by removing the need to maintain a separate file for the active branches, and instead rely on a centrally maintained file containing source of truth.
While at it, we refactor/simplify the creation of an array with the versions in `.buildkite/scripts/snyk/resolve_stack_version.sh`.
* Fix conflicts
---------
Co-authored-by: Jan Calanog <nejcalanog@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit preserves spaces in values, ensuring that multi-word strings are stored as intended.
Prior to this change, `logstash-keystore` incorrectly handled values containing spaces,
causing only the first word to be stored.
(cherry picked from commit 5573b5ad77)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
PR #17006 revealed that the `VERSION_QUALIFIER` env var gets honored in
various scripts when present but empty.
This shouldn't be the case as the DRA process is designed to gracefully
ignore empty values for this variable.
This commit changes various ruby scripts to not treat "" as truthy.
Bash scripts (used by CI etc.) are already ok with this as part of
refactorings done in #16907.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit c7204fd7d6)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Backport PR #16968 to 9.0 branch, original message:
----
Permit to use effectively the tokenizer also in context where a line is bigger than a limit.
Fixes an issues related to token size limit error, when the offending token was bigger than the input fragment in happened that the tokenzer wasn't unable to recover the token stream from the first delimiter after the offending token but messed things, loosing part of tokens.
## How solve the problem
This is a second take to fix the processing of tokens from the tokenizer after a buffer full error. The first try #16482 was rollbacked to the encoding error #16694.
The first try failed on returning the tokens in the same encoding of the input.
This PR does a couple of things:
- accumulates the tokens, so that after a full condition can resume with the next tokens after the offending one.
- respect the encoding of the input string. Use `concat` method instead of `addAll`, which avoid to convert RubyString to String and back to RubyString. When return the head `StringBuilder` it enforce the encoding with the input charset.
(cherry picked from commit 1c8cf546c2)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This reverts commit ac56d99553.
As mentioned in https://github.com/elastic/logstash/pull/16995#issuecomment-2626811054 testing revealed
that even an empty VERSION_QUALIFIER env var currently gets honored for snapshot builds (resulting in an unnecessary empty "-" suffix)
so there's a little more prerequisite work needed.
To avoid manual invocations specifying the version qualifier for
prereleases in this commit we leverage a centralized version of truth
for the version qualifier.
We also honor the `DRA_BRANCH` BK var for the generated step names
(the functionality was already there, it's just a cosmetic improvement).
