Use long instead of int type to keep the length of the first token.
The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test 9c0e50faac/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java (L123).
To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.
(cherry picked from commit afde43f918)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.
This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue
Fixes: #17209
(cherry picked from commit ba5f21576c)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
This commit migrates gradle task to the new artifacts-api
- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
with priority given in that order.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 0a745686f6)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Update patch plugin versions in gemfile lock
* Remove universal-java-17 to remain consistent with previous version
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
This commit adds Windows 2025 to the Windows JDK matrix and exhaustive tests pipelines.
(cherry picked from commit 4d52b7258d)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Currently the deserialization is behind the readBatch's lock, so any large batch will take time deserializing, causing any other Queue writer (e.g. netty executor threads) and any other Queue reader (pipeline worker) to block.
This commit moves the deserialization out of the lock, allowing multiple pipeline workers to deserialize batches concurrently.
- add intermediate batch-holder from `Queue` methods
- make the intermediate batch-holder a private inner class of `Queue` with a descriptive name `SerializedBatchHolder`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit 637f447b88)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Starting from es-output 12.0.2, a 404 response is treated as an error. Previously, central pipeline management considered 404 as an empty pipeline, not an error.
This commit restores the expected behavior by handling 404 gracefully and logs a user-friendly message.
It also removes the redundant cache of pipeline in CPM
Fixes: #17035
(cherry picked from commit e896cd727d)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
This commit allows Buildkite to capture any heap dumps produced
during DRA builds.
(cherry picked from commit 78c34465dc)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
PR #17006 revealed that the `VERSION_QUALIFIER` env var gets honored in
various scripts when present but empty.
This shouldn't be the case as the DRA process is designed to gracefully
ignore empty values for this variable.
This commit changes various ruby scripts to not treat "" as truthy.
Bash scripts (used by CI etc.) are already ok with this as part of
refactorings done in #16907.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit c7204fd7d6)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
VERSION_QUALIFIER was already observed in rake artifacts task but only to influence the name of the artifact.
This commit ensure that the qualifier is also displayed in the cli and in the http api.
(cherry picked from commit 00f8b91c35)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Update patch plugin versions in gemfile lock
* pull in minors from ES input and filter
* remove addition of java 17 platform
already covered by `java` platform
* pull in minor from http input to get fixed netty thread names
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Backport PR #16968 to 8.16 branch, original message:
----
Permit to use effectively the tokenizer also in context where a line is bigger than a limit.
Fixes an issues related to token size limit error, when the offending token was bigger than the input fragment in happened that the tokenzer wasn't unable to recover the token stream from the first delimiter after the offending token but messed things, loosing part of tokens.
## How solve the problem
This is a second take to fix the processing of tokens from the tokenizer after a buffer full error. The first try #16482 was rollbacked to the encoding error #16694.
The first try failed on returning the tokens in the same encoding of the input.
This PR does a couple of things:
- accumulates the tokens, so that after a full condition can resume with the next tokens after the offending one.
- respect the encoding of the input string. Use `concat` method instead of `addAll`, which avoid to convert RubyString to String and back to RubyString. When return the head `StringBuilder` it enforce the encoding with the input charset.
(cherry picked from commit 1c8cf546c2)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit removed irrelevant warning for internal pipeline, such as monitoring pipeline.
Monitoring pipeline is expected to be one worker. The warning is not useful
Fixes: #13298
(cherry picked from commit 3f41828ebb)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Non clean backport of #16959 from 8.x to 8.16 (which is a non clean backport of #16576 to 8.x)
Changes respect to 8.x:
- in 8.x the `with_deprecated_alias` receives also a parameter specifying the version where the setting is removed. In this case we removed the extra parameter to the method a0378c05cb/logstash-core/lib/logstash/environment.rb (L79)
- fixed the test removing the check for the remove version a0378c05cb/logstash-core/spec/logstash/settings/setting_with_deprecated_alias_spec.rb (L157)
----
Reimplements `LogStash::Setting::String` Ruby setting class into the `org.logstash.settings.SettingString` and exposes it through `java_import` as `LogStash::Setting::SettingString`.
Updates the rspec tests in two ways:
- logging mock is now converted to real Log4J appender that spy log line that are later verified
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
* Fixed the rename of NullableString to SettingNullableString
* Fixed runner test to use real spy logger from Java Settings instead of mock test double
Backport of #16907 cherry picked from 9385cfa
This commit uses the new --qualifier parameter in the release manager
for publishing dra artifacts. Additionally, simplifies the expected
variables to rely on a simple `VERSION_QUALIFIER`.
Snapshot builds are skipped when VERSION_QUALIFIER is set.
Finally, for helping to test DRA PRs, we also allow passing the `DRA_BRANCH` option/env var
to override BUILDKITE_BRANCH.
Closes https://github.com/elastic/ingest-dev/issues/4856
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2
Related to: https://github.com/jruby/jruby/issues/8579
(cherry picked from commit 52b7fb0ae6)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
When Logstash 8.12.0 added increased Jackson stream read constraints to
jvm.options, assumptions about the existence of that file's contents
were invalidated. This led to issues like #16683.
This change ensures Logstash applies defaults from config at runtime:
- MAX_STRING_LENGTH: 200_000_000
- MAX_NUMBER_LENGTH: 10_000
- MAX_NESTING_DEPTH: 1_000
These match the jvm.options defaults and are applied even when config
is missing. Config values still override these defaults when present.
(cherry picked from commit cc608eb88b)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Because a `break` escapes a `begin`...`end` block, we must not use a `break` in order to ensure that the explicitly set value gets memoized to avoid lock contention.
> ~~~ ruby
> def fake_sync(&block)
> puts "FAKE_SYNC:enter"
> val = yield
> puts "FAKE_SYNC:return(#{val})"
> return val
> ensure
> puts "FAKE_SYNC:ensure"
> end
>
> fake_sync do
> @ivar = begin
> puts("BE:begin")
> break :break
>
> val = :ret
> puts("BE:return(#{val})")
> val
> ensure
> puts("BE:ensure")
> end
> end
> ~~~
Note: no `FAKE_SYNC:return`:
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash (main ✔) }
> ╰─● ruby break-esc.rb
> FAKE_SYNC:enter
> BE:begin
> BE:ensure
> FAKE_SYNC:ensure
> [success]
> ~~~
(cherry picked from commit 01c8e8bb55)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
