The logstash-input-elastic_serverless_forwarder plugin has been promoted from
tech preview to GA. This commit updates the 8.x stream of logstash to use the
new 1.x series of the plugin.
This commit updates the curl scripts that interact with Kibana's
`api/logstash/pipeline/*` endpoin. Additionally this adds the header to any curl
that interacts with elasticsearch API as well.
(cherry picked from commit 188d9e7ed8)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
This commit updates the Kibana client to add a `x-elastic-product-origin` header
with the value `logstash` for any requests originating from logstash to indicate
which internal product is using the API.
* Major-pin default plugins
pins all default plugins _optimistically_ to the major.minor in the head of 8.17's `Gemfile.*.lock`, in order to avoid accidentally shipping _any_ plugin's breaking changes in the 8.x series.
* remove duplication of es output
* dont require plugins in Gemfile
* give more memory to tests. 2gb instead of 512mb
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: João Duarte <jsvduarte@gmail.com>
Moves the application of jackson defaults overrides into pure java, and
applies them statically _before_ the `org.logstash.ObjectMappers` has a chance
to start initializing object mappers that rely on the defaults.
We replace the runner's invocation (which was too late to be fully applied) with
a _verification_ that the configured defaults have been applied.
(cherry picked from commit 202d07cbbf)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
A number of plugins include breaking changes in 9.x, which, without
restriction, will be included in development `8.x` branches that do
not include a lockfile.
This commit adds these to the Gemfile.template to avoid being introduced
in 8.x development branches
A new transative dependency on the `logger` gem has been added through sinatra 4.1.0. Update the
license checker to ensure this is accounted for.
(cherry picked from commit e0ed994ab1)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* pipeline bus: add deadlock test for unlisten/unregisterSender
* pipeline bus: eliminate deadlock
Moves the sync-to-notify out of the `AddressStateMapping#mutate`'s effective
synchronous block to eliminate a race condition where unlistening to an address
and unregistering a sender could deadlock.
It is safe to notify an AddressState's attached input without exclusive access
to the AddressState, because notifying an input that has since been disconnected
is net zero harm.
Updates release notes for `8.14.x` to call for an update to a subsequent minor for fix performance regression in JSON decoding.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit 52836f8caf)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This fixes the offline install problem of the logstash-input-java_filter_example off-line install.
(cherry picked from commit 113585d4a5)
Co-authored-by: Nicole Albee <2642763+a03nikki@users.noreply.github.com>
The DRA build failed because the required jars were missing, as they had been removed during the Docker build process.
(cherry picked from commit 00da72378b)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
there's no need to build jdk-less and windows tarballs for docker images
so this change simplifies the build process.
It should reduce the time spent needed to build docker images.
(cherry picked from commit 9eced9a106)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Switch execution agent of JDK availability check pipeline from vm-agent to container-agent.
Moves the schedule definition from the `Logstash Pipeline Scheduler` pipeline into the pipeline definition, adding a schedule also for `8.x` branch.
(cherry picked from commit c602b851bf)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Updates the existing `createElasticCatalogDownloadUrl` method to use the precise version retrieved `versions.yml` to download the JDK instead of using the latest of major version. This makes the build reproducible again.
Defines a new Gradle `checkNewJdkVersion` task to check if there is a new JDK version available from JVM catalog matching the same major of the current branch.
Creates a new Buildkite pipeline to execute a `bash` script to run the Gradle task; plus it also update the `catalog-info.yaml` with the new pipeline and a trigger to execute every week.
(cherry picked from commit ed5874bc27)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
The current max inflight error message focuses on a single pipeline and on a maximum amount of 10k events regardless of the heap size.
The new warning will take into account all loaded pipelines and also consider the heap size, giving a warning if the total number of events consumes 10% or more of the total heap.
For the purpose of the warning events are assumed to be 2KB as it a normal size for a small log entry.
(cherry picked from commit ca19f0029e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
- refactor deprecated alias to support obsoleted version
- add deprecation log for http.* config
(cherry picked from commit 3f0ad12d06)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Fixes the behaviour of the tokenizer to be able to work properly when buffer full conditions are met.
Updates BufferedTokenizerExt so that can accumulate token fragments coming from different data segments. When a "buffer full" condition is matched, it record this state in a local field so that on next data segment it can consume all the token fragments till the next token delimiter.
Updated the accumulation variable from RubyArray containing strings to a StringBuilder which contains the head token, plus the remaining token fragments are stored in the input array.
Furthermore it translates the `buftok_spec` tests into JUnit tests.
(cherry picked from commit 85493ce864)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Adds a JMH benchmark to measure the peformances of BufferedTokenizerExt.
Update also Gradle build script to remove CMS GC flags and fix deprecations for Gradle 9.0.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit b6f16c8b81)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
On 8.x series log a deprecation log if the user didn't explicitly specify a selection for pipeline.buffer.type. Before this change the default was silently set to direct, after this change if not explicitly defined, the default is still direct but log a deprecation log.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
- add `obsoleted_version` and remove `deprecated_msg` from `deprecated_option` for consistent warning message
(cherry picked from commit 8cd0fa8767)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* [test] Fix xpack test to check for http_address stats only if the webserver is enabled (#16525)
Set the 'api.enabled' setting to reflect the flag webserver_enabled and consequently test for http_address presence in settings iff the web server is enabled.
(cherry picked from commit 648472106f)
* Update also the global LogStash::SETTINGS's 'api.enabled' setting value becuase used in the constructor of StatsEventFactory and needs to be in synch with the settings provided to the Agent constructor
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Update Settings to_hash method to also skip Java DeprecatedAlias and not just the Ruby one.
With PR #15679 was introduced org.logstash.settings.DeprecatedAlias which mirrors the behaviour of Ruby class Setting::DeprecatedAlias. The equality check at Logstash::Settings, as descibed in #16505 (comment), is implemented comparing the maps.
The conversion of Settings to the corresponding maps filtered out the Ruby implementation of DeprecatedAlias but not the Java one.
This PR adds also the Java one to the list of filter.
(cherry picked from commit 5d4825f000)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [Spacetime] Reimplement config Setting classe in java (#15679)
Reimplement the root Ruby Setting class in Java and use it from the Ruby one moving the original Ruby class to a shell wrapping the Java instance.
In particular create a new symmetric hierarchy (at the time just for `Setting`, `Coercible` and `Boolean` classes) to the Ruby one, moving also the feature for setting deprecation. In this way the new `org.logstash.settings.Boolean` is syntactically and semantically equivalent to the old Ruby Boolean class, which replaces.
(cherry picked from commit 61de60fe26)
* Adds supress warnings related to this-escape for Java Settings classes
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
