This commit adds licences for bouncycastle jars that are added for the
observability SRE container artifact. It re-enables the previously disabled
license check and adds a new one running in fips mode.
* Fix empty node stats pipelines (#17185) (#17197)
Fixed an issue where the `/_node/stats` API displayed empty pipeline metrics
when X-Pack monitoring was enabled
(cherry picked from commit 86785815bd)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Update z_rubycheck.rake to no longer inject Xmx1g (#17211)
This allows the environment variable JRUBY_OPTS to be used for setting properties like Xmx
original pr: #16420
(cherry picked from commit f562f37df2)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Improve warning for insufficient file resources for PQ max_bytes (#16656) (#17222)
This commit refactors the `PersistedQueueConfigValidator` class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
https://github.com/elastic/logstash/issues/14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.
(cherry picked from commit 062154494a)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* gradle task migrate to the new artifacts-api (#17232) (#17236)
This commit migrates gradle task to the new artifacts-api
- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
with priority given in that order.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 0a745686f6)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* tests: ls2ls delay checking until events have been processed (#17167) (#17252)
* tests: ls2ls delay checking until events have been processed
* Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec.
* Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable.
---------
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit 73ffa243bf)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Additional cleanify changes to ls2ls integ tests (#17246) (#17255)
* Additional cleanify changes to ls2ls integ tests: replace heartbeat-input with reload option, set queue drain to get consistent result.
(cherry picked from commit 1e06eea86e)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* [8.x] Reimplement LogStash::Numeric setting in Java (backport #17127) (#17273)
This is an automatic backport of pull request #17127 done by [Mergify](https://mergify.com).
----
* Reimplement LogStash::Numeric setting in Java (#17127)
Reimplements `LogStash::Setting::Numeric` Ruby setting class into the `org.logstash.settings.NumericSetting` and exposes it through `java_import` as `LogStash::Setting::NumericSetting`.
Updates the rspec tests:
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
(cherry picked from commit 07a3c8e73b)
* Fixed reference of SettingNumeric class (on main modules were removed)
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [CI] Health report integration tests use the new artifacts-api (#17274) (#17277)
migrate to the new artifacts-api
(cherry picked from commit feb2b92ba2)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Backport 17203 and 17267 8.x (#17270)
* Pluginmanager clean after mutate (#17203)
* pluginmanager: always clean after mutate
* pluginmanager: don't skip updating plugins installed with --version
* pr feedback
(cherry picked from commit 8c96913807)
* Pluginmanager install preserve (#17267)
* tests: integration tests for pluginmanager install --preserve
* fix regression where pluginmanager's install --preserve flag didn't
* [Backport 8.x] benchmark script (#17283)
This commit cherry-picked the missing becnhmark script PRs
The deprecated artifacts-api is removed
[CI] benchmark uses the new artifacts-api (#17224)
[CI] benchmark readme (#16783)
Introduce a new flag to explicitly permit legacy monitoring (#16586) (Only take the benchmark script)
[ci] fix wrong queue type in benchmark marathon (#16465)
[CI] fix benchmark marathon (#16447)
[CI] benchmark dashboard and pipeline for testing against multiple versions (#16421)
* Fix pqcheck and pqrepair on Windows (#17210) (#17259)
A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.
This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue
Fixes: #17209
(cherry picked from commit ba5f21576c)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Shareable function for partitioning integration tests (#17223) (#17303)
For the fedramp high work https://github.com/elastic/logstash/pull/17038/files a
use case for multiple scripts consuming the partitioning functionality emerged.
As we look to more advanced partitioning we want to ensure that the
functionality will be consumable from multiple scripts.
See https://github.com/elastic/logstash/pull/17219#issuecomment-2698650296
(cherry picked from commit d916972877)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* [8.x] Surface failures from nested rake/shell tasks (backport #17310) (#17317)
* Surface failures from nested rake/shell tasks (#17310)
Previously when rake would shell out the output would be lost. This
made debugging CI logs difficult. This commit updates the stack with
improved message surfacing on error.
(cherry picked from commit 0d931a502a)
# Conflicts:
# rubyUtils.gradle
* Extend ruby linting tasks to handle file inputs (#16660)
This commit extends the gradle and rake tasks to pass through a list of files
for rubocop to lint. This allows more specificity and fine grained control for
linting when the consumer of the tasks only wishes to lint a select few files.
* Ensure shellwords library is loaded
Without this depending on task load order `Shellwords` may not be available.
---------
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* Forward Port of Release notes for `8.16.5` and `8.17.3` (#17187), (#17188) (#17266) (#17321)
* Forward Port of Release notes for 8.17.3 (#17187)
* Update release notes for 8.17.3
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Forward Port of Release notes for 8.16.5 (#17188)
* Update release notes for 8.16.5
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
(cherry picked from commit 63e8fd1d21)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Add Deprecation tag to arcsight module (#17331)
* [8.x] Upgrade elasticsearch-ruby client. (backport #17161) (#17306)
* Upgrade elasticsearch-ruby client. (#17161)
* Fix Faraday removed basic auth option and apply the ES client module name change.
(cherry picked from commit e748488e4a)
* Apply the required changes in elasticsearch_client.rb after upgrading the elasticsearch-ruby client to 8.x
* Swallow the exception and make non-connectable client when ES client raises connection refuses exception.
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
* Removed unused configHash computation that can be replaced by PipelineConfig.configHash() (#17336) (#17345)
Removed unused configHash computation happening in AbstractPipeline and used only in tests replaced by PipelineConfig.configHash() invocation
(cherry picked from commit 787fd2c62f)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Use org.logstash.common.Util to hashing by default to SHA256 (#17346) (#17352)
Removes the usage fo Apache Commons Codec MessgeDigest to use internal Util class with embodies hashing methods.
(cherry picked from commit 9c0e50faac)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Added test to verify the int overflow happen (#17353) (#17354)
Use long instead of int type to keep the length of the first token.
The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test 9c0e50faac/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java (L123).
To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.
(cherry picked from commit afde43f918)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [8.x] add ci shared qualified-version script (backport #17311) (#17348)
* add ci shared qualified-version script (#17311)
* ci: add shareable script for generating qualified version
* ci: use shared script to generate qualified version
(cherry picked from commit 10b5a84f84)
# Conflicts:
# .buildkite/scripts/dra/build_docker.sh
* resolve merge conflict
---------
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* tests: make integration split quantity configurable (#17219) (#17367)
* tests: make integration split quantity configurable
Refactors shared splitter bash function to take a list of files on stdin
and split into a configurable number of partitions, emitting only those from
the currently-selected partition to stdout.
Also refactors the only caller in the integration_tests launcher script to
accept an optional partition_count parameter (defaulting to `2` for backward-
compatibility), to provide the list of specs to the function's stdin, and to
output relevant information about the quantity of partition splits and which
was selected.
* ci: run integration tests in 3 parts
(cherry picked from commit 3e0f488df2)
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* Update buildkite with new patterns from 8.x
This commit updates the buildkite definitions to be compatible with the
upstream 8.x branch. Specificially:
- Split integration tests for fips into 3 runners.
- Use the new shared bash helper for computing QUALIFIED_VERSION
It also continues standardization of using a "fedrampHighMode" for indicating
the tests should be running in the context of our custom image for the SRE team.
* Bug fix: Actually use shared integration_tests.sh file
After refactoring to use the same script, I forgot to actually use it
in the buildkite definition...
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
JvmOptionsParser adds support for ${VAR:default} syntax when parsing jvm.options
- allow dynamic resolution of environment variables in the jvm.options file
- enables fallback to default value when the environment variable is not set
(cherry picked from commit ef36df6b81)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
A new transative dependency on the `logger` gem has been added through sinatra 4.1.0. Update the
license checker to ensure this is accounted for.
(cherry picked from commit e0ed994ab1)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
Bugfix to parse correctly Java options when the environment variable LS_JAVA_OPTS contains multiple definitions separated by space character.
Adapt the parsing of `LS_JAVA_OPTS` environment variable to split by space various definitions it can contains.
Given that JRuby comes with ruby-maven-libs 3.3.9 this commit upgrades the gem to 3.8.9 and ensures files from 3.3.9 are not included in the distribution.
Adds a global new task named javaTests which groups both :logstash-core:javaTests and :jvm-options-parser:test to include the JvmOptionParser unit test in javaTests phase.
Updates Netty's configuration of maxOrder to a previously proven value, if not already customised by the user.
Adds a step to the JvmOption parsing tool, which is used to compose the JVM options string to pass down to Logstash at startup.
The added step rework the parsed options to set the allocator max order -Dio.netty.allocator.maxOrder=11 so that the maximum pooled buffer is up to 16MB and not 4MB.
This option is added iff it's not yet specified by the user
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
Set of changes to make Logstash compatible to JRuby 9.4.
Bundle JRuby 9.4.3.0
- Redefine space token in `LSCL` and `grammar` treetop from `_` which would generated methods in the form `def _0` (deprecated since `2.7`) to `sc`.
- `I18n.t` method doesn't accept hash as second argument
- `URI.encode` has been replaced with same functionality with `URI::Parser.new.escape`
- `YAML.load` needs explicit `fallback: false` to return false when the yaml string is empty (or contains only comments)
- JRuby's `JavaClass` has been removed, now it can use `java.lang.Class` directly
- explicitly require gem `thwait` to satisfy `require "thwait"` (In `Gemfile.template` and `logstash-core/logstash-core.gemspec`)
- fix not args `clone` to be `def clone(*args)`
- fix `Enumeration.each_slice` which from `Ruby 3.1` is [chainable](https://rubyreferences.github.io/rubychanges/3.1.html#enumerableeach_cons-and-each_slice-return-a-receiver) and doesn't return `nil`. JRuby fixed in https://github.com/jruby/jruby/issues/7015
- Expanded `Down.download` arguments map ca16bbed3c302006967413eb9d3862f2da81f7ae
- Avoid to pass `nil` in the list of couples used in `Hash[ <list of couples> ]` which from Ruby `3.0` generates an `ArgumentError`
- Removed space not allowed between method name and parentheses `initialize (` is forbidden. 29b607dcdef98f81a73ad171639fd13aaa65e243
- With [Ruby 2.7 the `Kernel#open`](https://rubyreferences.github.io/rubychanges/2.7.html#network-and-web) doesn't fallback to `URI#open`, fixed test code that used that to verify open port. e5b70de54c5301f51a767da67294092af0cfafdc
- Avoid to drop `rdoc/` folder from vendored JRuby else `bin/logstash -i irb` would crash, commit b71f73e9c6edb81a7b7ae1305047e506f61c6e8c
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Flow metrics: initial implementation (#14509)
* metrics: eliminate race condition when registering metrics
Ensure our fast-lookup and store tables cannot diverge in a race condition
by wrapping mutation of both in a single mutex and appropriately handle
another thread winning the race to the lock by using the value that it
persisted instead of writing our own.
* metrics: guard against intermediate namespace conflicts
- ensures our safeguard that prevents using an existing metric as a namespace
is applied to _intermediate_ nodes, not just the tail-node, eliminating a
potential crash when sending `fetch_or_store` to a metric object that is not
expected to respond to `fetch_or_store`.
- uses the atomic `Concurrent::Map#compute_if_absent` instead of the
non-atomic `Concurrent::Map#fetch_or_store`, which is prone to
last-write-wins during contention (as-written, this method is only
executed under lock and not subject to contention)
- uses `Enumerable#reduce` to eliminate the need for recursion
* flow: introduce auto-advancing UptimeMetric
* flow: introduce FlowMetric with minimal current/lifetime rates
* flow: initialize pipeline metrics at pipeline start
* Controller and service layer implementation for flow metrics. (#14514)
* Controller and service layer implementation for flow metrics.
* Add flow metrics to unit test and benchmark cli definitions.
* flow: fix tests for metric types to accomodate new one
* Renaming concurrency and backpressure metrics.
Rename `concurrency` to `worker_concurrency ` and `backpressure` to `queue_backpressure` to provide proper scope naming.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* metric: register flow metrics only when we have a collector (#14529)
the collector is absent when the pipeline is run in test with a
NullMetricExt, or when the pipeline is explicitly configured to
not collect metrics using `metric.collect: false`.
* Unit tests and integration tests added for flow metrics. (#14527)
* Unit tests and integration tests added for flow metrics.
* Node stat spec and pipeline spec metric updates.
* Metric keys statically imported, implicit error expectation added in metric spec.
* Fix node status API spec after renaming flow metrics.
* Removing flow metric from PipelinesInfo DS (used in peridoci metric snapshot), integration QA updates.
* metric: register flow metrics only when we have a collector (#14529)
the collector is absent when the pipeline is run in test with a
NullMetricExt, or when the pipeline is explicitly configured to
not collect metrics using `metric.collect: false`.
* Unit tests and integration tests added for flow metrics.
* Node stat spec and pipeline spec metric updates.
* Metric keys statically imported, implicit error expectation added in metric spec.
* Fix node status API spec after renaming flow metrics.
* Removing flow metric from PipelinesInfo DS (used in peridoci metric snapshot), integration QA updates.
* Rebasing with feature branch.
* metric: register flow metrics only when we have a collector
the collector is absent when the pipeline is run in test with a
NullMetricExt, or when the pipeline is explicitly configured to
not collect metrics using `metric.collect: false`.
* Apply suggestions from code review
Integration tests updated to test capturing the flow metrics.
* Flow metrics expectation updated in tegration tests.
* flow: refine integration expectations for reloads/monitoring
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Co-authored-by: Mashhur <mashhur.sattorov@gmail.com>
* metric: add ScaledView with sub-unit precision to UptimeMetric (#14525)
* metric: add ScaledView with sub-unit precision to UptimeMetric
By presenting a _view_ of our metric that maintains sub-unit precision,
we prevent jitter that can be caused by our periodic poller not running at
exactly our configured cadence.
This is especially important as the UptimeMetric is used as the _denominator_ of
several flow metrics, and a capture at 4.999s that truncates to 4s, causes the
rate to be over-reported by ~25%.
The `UptimeMetric.ScaledView` implements `Metric<Number>`, so its full
lossless `BigDecimal` value is accessible to our `FlowMetric` at query time.
* metrics: reduce window for too-frequent-captures bug and document it
* fixup: provide mocked clock to flow metric
* Flow metrics cleanup (#14535)
* flow metrics: code-style and readability pass
* remove unused imports
* cleanup: simplify usage of internal helpers
* flow: migrate internals to use OptionalDouble
* Flow metrics global (#14539)
* flow: add global top-level flows
* docs: add flow metrics
* Top level flow metrics unit tests added. (#14540)
* Top level flow metrics unit tests added.
* Add unit tests when config reloads, make sure top-level flow metrics didn't get reset.
* Apply suggestions from code review
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Validating against Hash test cases updated.
* For the safety check against exact type in unit tests.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* docs: section links and clarity in node stats API flow metrics
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@gmail.com>
* Manually import CSV gem to work around thread leak
The version of CSV included in jruby distributions >= `9.3.0.0` and
< `9.3.8.0` include a bug that triggers a thread leak when calling
`parse_line` in CSV files.
This commit updates the version of the CSV gem to the latest version,
which includes fixes, and remains compatible with ruby 2.6
Relates: https://github.com/jruby/jruby/issues/7346
* Add CSV license info
The logstash-integration-aws plugin replaces the underlying individual plugins related to AWS.
This commit also updates the necessary files around license checking.
With JDK 18, the Javac lint checking was expanded to raise an error on serializable subclasses that contains instance fields which are not serializable themselves.
This could be solved suppressing the warning or marking the field as transient. The majority of classes with this problem inherit transitively from Serializable but are not intended to be serialized (from Java's serialization mechanism) because has the serialVersionUID = 1L which is just a fix to make linter happy, but are not effectively used in a serialization context.
This commit also removes a finalize method, that could be safely be removed.
In jackson-databind 2.10, enabling Default Typing requires having a type validator, and while there's an "allow all" validator called LaissezFaireSubTypeValidator, this commit also tightens the validation a bit by narrowing down the allowed classes.
The default typing validator is only applied to the ObjectMapper for CBOR, which is used in the DLQ, leaving the one for JSON as-is.
Other changes:
* make ingest-converter use versions.yml for jackson-databind
* update jrjackson
* Add support for ca_trusted_fingerprint in Apache HTTP and Manticore
Adds a module `LogStash::Plugins::CATrustedFingerprintSupport`, which can be
included in a plugin class to add a `ca_trusted_fingerprint` option to create
an Apache SSL TrustStrategy that can be used to bypass the TrustManager when
a matching certificate is found on the chain.
This commit updates the version of jruby used in Logstash to `9.3.4.0`.
* Updates the references of `jruby` from `9.2.20.1` to `9.3.4.0`
* Updates references/locations of ruby from `2.5.0` to `2.6.0`
* Updates java imports including `org.logstash.util` to be quoted
* Without quoting the name of the import, the following error is observed in tests:
* `java.lang.NoClassDefFoundError: org/logstash/Util (wrong name: org/logstash/util)`
* Maybe an instance of https://github.com/jruby/jruby/issues/4861
* Adds a monkey patch to `require` to resolve compatibility issue between latest `jruby` and `polyglot` gem
* The addition of https://github.com/jruby/jruby/pull/7145 to disallow circular
causes, will throw when `polyglot` is thrown into the mix, and stop logstash from
starting and building - any gems that use an exception to determine whether or not
to load the native gem, will trigger the code added in that commit.
* This commit adds a monkey patch of `require` to rollback the circular cause exception
back to the original cause.
* Removes the use of the deprecated `JavaClass`
* Adds additional `require time` in `generate_build_metadata`
* Rewrites a test helper to avoid potentially calling `~>` on `FalseClass`
Co-authored-by: Joao Duarte <jsvduarte@gmail.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit moves the JvmOptionParser into its own gradle project.
This enables the JvmOptionParser to remain compatible with Java 1.8 to present a helpful error message to a user attempting to start Logstash using older versions of Java, while allowing the main Logstash code base to freely use idiomatic Java 11 features.
