Updates only test code to be able to run a test that consumes big memory if:
- the physical memory is bigger than the requested Java heap
- JDK version is greater than or equal to 21.
The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error.
(cherry picked from commit 075fdb4152)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Use long instead of int type to keep the length of the first token.
The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test 9c0e50faac/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java (L123).
To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.
(cherry picked from commit afde43f918)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Removes the usage fo Apache Commons Codec MessgeDigest to use internal Util class with embodies hashing methods.
(cherry picked from commit 9c0e50faac)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [health] bootstrap HealthObserver from agent to API (#16141)
* [health] bootstrap HealthObserver from agent to API
* specs: mocked agent needs health observer
* add license headers
* Merge `main` into `feature/health-report-api` (#16397)
* Add GH vault plugin bot to allowed list (#16301)
* regenerate webserver test certificates (#16331)
* correctly handle stack overflow errors during pipeline compilation (#16323)
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* Doc: Reposition worker-utilization in doc (#16335)
* settings: add support for observing settings after post-process hooks (#16339)
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
* fix line used to determine ES is up (#16349)
* add retries to snyk buildkite job (#16343)
* Fix 8.13.1 release notes (#16363)
make a note of the fix that went to 8.13.1: #16026
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16347)
* [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Doc: Add SNMP integration to breaking changes (#16374)
* deprecate java less-than 17 (#16370)
* Exclude substitution refinement on pipelines.yml (#16375)
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Doc: Forwardport 8.15.0 release notes to main (#16388)
* Removing 8.14 from ci/branches.json as we have 8.15. (#16390)
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Squashed merge from 8.x
* Failure injector plugin implementation. (#16466)
* Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes.
* Fix the degrate method name typo.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Add explanation to the config params and rebuild plugin gem.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Health report integration tests bootstrapper and initial tests implementation (#16467)
* Health Report integration tests bootstrapper and initial slow start scenario implementation.
* Apply suggestions from code review
Renaming expectation check method name.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Changed to branch concept, YAML structure simplified as changed to Dict.
* Apply suggestions from code review
Reflect `help_url` to the integration test.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* health api: expose `GET /_health_report` with pipelines/*/status probe (#16398)
Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the
resulting report status into the other API responses, replacing their hard-coded `green`
with a meaningful status indication.
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* docs: health report API, and diagnosis links (feature-targeted) (#16518)
* docs: health report API, and diagnosis links
* Remove plus-for-passthrough markers
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* merge 8.x into feature branch... (#16519)
* Add GH vault plugin bot to allowed list (#16301)
* regenerate webserver test certificates (#16331)
* correctly handle stack overflow errors during pipeline compilation (#16323)
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* Doc: Reposition worker-utilization in doc (#16335)
* settings: add support for observing settings after post-process hooks (#16339)
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
* fix line used to determine ES is up (#16349)
* add retries to snyk buildkite job (#16343)
* Fix 8.13.1 release notes (#16363)
make a note of the fix that went to 8.13.1: #16026
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16347)
* [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Doc: Add SNMP integration to breaking changes (#16374)
* deprecate java less-than 17 (#16370)
* Exclude substitution refinement on pipelines.yml (#16375)
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Doc: Forwardport 8.15.0 release notes to main (#16388)
* Removing 8.14 from ci/branches.json as we have 8.15. (#16390)
* Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408)
Fix: #16406
* Generate Dataset code with meaningful fields names (#16386)
This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields.
Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`.
Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`.
Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields.
* Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322)
This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it.
This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch.
Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node.
Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16426)
* Release notes for 8.15.1 (#16405) (#16427)
* Update release notes for 8.15.1
* update release note
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co>
(cherry picked from commit 2fca7e39e8)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430)
Make inner field of ConditionalEvaluationError transient to be avoided during serialization.
(cherry picked from commit bb7ecc203f)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* use gnu tar compatible minitar to generate tar artifact (#16432) (#16434)
Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters.
For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem.
(cherry picked from commit 69f0fa54ca)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* account for the 8.x in DRA publishing task (#16436) (#16440)
the current DRA publishing task computes the branch from the version
contained in the version.yml
This is done by taking the major.minor and confirming that a branch
exists with that name.
However this pattern won't be applicable for 8.x, as that branch
currently points to 8.16.0 and there is no 8.16 branch.
This commit falls back to reading the buildkite injected
BUILDKITE_BRANCH variable.
(cherry picked from commit 17dba9f829)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459)
* Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.
* Add a docker acceptance integration test.
(cherry picked from commit 7c64c7394b)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Known issue for 8.15.1 related to env vars references (#16455) (#16469)
(cherry picked from commit b54caf3fd8)
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
* bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480)
(cherry picked from commit 51cca7320e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Release notes for 8.15.2 (#16471) (#16478)
Co-authored-by: andsel <selva.andre@gmail.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit 01dc76f3b5)
* Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488)
(cherry picked from commit 8368c00367)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491)
(cherry picked from commit fd1de39005)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Don't use an older JRuby with oraclelinux-7 (#16499) (#16501)
A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images
and removed JRuby 9.4.5.0 and some older versions.
This ended up breaking exhaustive test on Oracle Linux 7 that hard coded
JRuby 9.4.5.0.
PR https://github.com/elastic/logstash/pull/16489 worked around the
problem by pinning to the new JRuby, but actually we don't
need the conditional anymore since the original issue
https://github.com/jruby/jruby/issues/7579#issuecomment-1425885324 has
been resolved and none of our releasable branches (apart from 7.17 which
uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`.
Therefore, this commit removes conditional setting of JRuby for
OracleLinux 7 agents in exhaustive tests (and relies on whatever
`/.ruby-version` defines).
(cherry picked from commit 07c01f8231)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* Improve pipeline bootstrap error logs (#16495) (#16504)
This PR adds the cause errors details on the pipeline converge state error logs
(cherry picked from commit e84fb458ce)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511)
(cherry picked from commit 5195332bc6)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Make health report test runner script executable. (#16446) (#16512)
(cherry picked from commit 2ebf2658ff)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493)
* DLQ-ing events that trigger an conditional evaluation error. (#16423)
When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline.
This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it.
(cherry picked from commit b69d993d71)
* Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* add deprecation log for `--event_api.tags.illegal` (#16507) (#16515)
- move `--event_api.tags.illegal` from option to deprecated_option
- add deprecation log when the flag is explicitly used
relates: #16356
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit a4eddb8a2a)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* fix: restore support for unicode pipeline- and plugin-id's
JRuby's `Ruby#newSymbol(String)` throws an exception when provided a `String`
that contains characters outside of lower-ASCII because JRuby internals expect
"the incoming String to be one of our mangled ISO-8859-1 strings" as noted in
a comment on jruby/jruby#6217.
Instead, we use `Ruby#newString(String)` to create a new `RubyString` (which
works properly), and then rely on `RubyString#intern` to get our `RubySymbol`.
This fixes a regression introduced in the 8.7 series in which pipeline id's
are consistently represented as ruby symbols in the metrics store, and ensures
similar issue does not exist when specifying a plugin id that contains
characters above the lower-ASCII plane.
* fix: use properly-encoded RubySymbol in PipelineConfig
We cannot rely on `RubySymbol#toString` to produce a properly-encoded `String`
whe the string contains characters above the lower-ASCII plane because the
result is effectively a binary ruby-internal marshal of the bytes that only
holds when the symbol contains lower-ASCII.
Instead, we can use the internally-memoizing `RubySymbol#name` to get a
properly-encoded `RubyString`, and `RubyString#asJavaString()` to get a
properly-encoded java-`String`.
* fix: properly serialize unicode pipeline names in API output
Jackson's JSON serializer leaks the JRuby-internal byte structure of Symbols,
which only aligns with the byte-structure of the symbol's actual string when
that string is wholly-comprised of lower-ASCII characters.
By pre-converting Symbols to Strings, we ensure that the result is readable
and useful.
* spec: bypass monitoring specs for unicode pipeline ids when PQ enabled
* p2p: extract interface from v1 pipeline bus
* p2p: extract pipeline push to abstract
* p2p: add opt-in unblocked "v2" implementation
Adds a v2 implementation that does not synchronize on the sender so that
multiple workers can send events through a common `pipeline` output instance
simultaneously.
In this implementation, an `AddressStateMapping` provides synchronized
mutation and cleanup of the underlying `AddressState`, and allows only
queryable mutable views (`AddressState.ReadOnly`) to escape encapsulation.
The implementation also holds indentity-keyed mapping from `PipelineOutput`s
to the set of `AddressState.ReadOnly`s it is regested as a sender for so
that they can be quickly resolved at runtime.
* p2p: more tests for pipeline restart behaviour
* p2p: make v2 pipeline bus the default
This removes the dependency on jackson's dataformat-yaml. Since there's only a single place where this library is used in core: to load the plugin alias definition, the code can be replaced by the underlying snakeyaml.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 93d8a9da32)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Modifies the logic used by the scheduled task flusher so that execute age policy also in case the current (head) segments is not stale (haven't received any write, and the segment is empty).
This means that generally used finalize segment logic is applied plus a reinforcement step to grant the age policy is respected.
However this PR:
- introduced new debug log lines, improving the description of the context when a segment is finalized (because the DLQ is closing or because the segment file has reached its maximum size or because the flush interval expiration). This is done with the introduction of `SealReason` enumeration.
- introduces `Awaitility` test dependency to improve the testing of asychronous conditions.
* live timers: introduce API boundary
Introduces an API boundary for timers as a first-class metric, as described
in elastic/logstash#14675, and migrates all known internal timers to use the
new API boundary for tracked execution.
Please refer to the specification for details on motivations.
This commit is net zero change to behaviour, and introduces a single new
undocumented setting `metric.timers` to `logstash.yml`, which presently only
takes its default value `delayed` to indicate that delayed committing of
execution time is acceptable.
It implements the new `TimerMetric` API in a way that is also net-zero-change.
Tracked executions are still performed by marking a start time, performing
the tracked execution, and incrementing an underlying long-type counter with
the number of elapsed milliseconds _after_ execution has completed. This means
that long-running execution is still missing from the metric until it has
completed.
The new Timer API is available to both the Ruby- and the Java-based plugin APIs
* timer metrics: sub-package and add baseline tests
* WIP: move execution metric ownership out of queue
* noop: remove useless abstract method
Our `AbstractMetric` implements `Metric` and does not need to declare
an abstract override of `Metric#getType`. Doing so prevents interfaces
from providing a default override for all implementers.
* timer metric tests: extract util, refactor for reuse
* timers: accumulate milli-excess-nanos
* live timers: single-checkpoint implementation
* timer metric: use explicit type parameters to make intent clear
* remove unused imports
* use safe int conversion
* test fixup: use given name for tested metric
* test helper: TimerMetricFactory prefers nanotime supplier
* timers: flesh out test coverage, incl live-timers
* test: move validation of queue-read metrics to ObservedExecution
* flow: support non-moving denominator (±infinity)
* metrics: add metric config pass-through to env2yaml
Update the following java dependencies:
* org.reflections:reflections
* commons-codec:commons-codec
* com.google.guava:guava
* com.google.googlejavaformat:google-java-format
* org.javassist:javassist
The goal of these updates is to not fall behind and avoid surprises when an upgrade is necessary due to a security issue.
* Add support for ca_trusted_fingerprint in Apache HTTP and Manticore
Adds a module `LogStash::Plugins::CATrustedFingerprintSupport`, which can be
included in a plugin class to add a `ca_trusted_fingerprint` option to create
an Apache SSL TrustStrategy that can be used to bypass the TrustManager when
a matching certificate is found on the chain.
This commit moves the JvmOptionParser into its own gradle project.
This enables the JvmOptionParser to remain compatible with Java 1.8 to present a helpful error message to a user attempting to start Logstash using older versions of Java, while allowing the main Logstash code base to freely use idiomatic Java 11 features.
* Make AliasRegistry a singleton
The current implementation of AliasRegistry will create an instance of the Alias Registry for each
pipeline, which appears to potentially result in situations, such as in #13996, where multiple pipelines
are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with
the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when
reading the yaml file as the JarFile has been closed after the initial thread completed accessing.
This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry.
Relates: #13996, #13666
* Update reflections library to 0.9.12 to avoid multi threading bug
Earlier versions of the reflections library used in the plugin registry would
use caches on JarUrlConnection, which when closed would also close the jar file
for other resources using it, such as the AliasRegistry. This, combined with the
fact that the AliasRegistry could be created simultaneously by many threads/pipelines
could cause issues during AliasRegistry creation leading to failure to create a
pipeline.
* Avoid use of URLConnection caching when getting alias yaml resource
* Use idiomatic ruby when accessing Java getInstance method
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* add nanoseconds support
Migrates internals of `org.logstash.Timestamp` from legacy `org.joda.time.*`
which is limited to millisecond-precision to modern `java.time.Instant`,
allowing us to retain nanosecond granularity of `@timestamp` values.
Timestamps that are generated by Logstash (such as when creating an event that
does _not_ have a `@timestamp` field) will be generated at the highest precision
available to the JVM and/or platform (in many cases, this is microseconds).
Timestamps that are _parsed_ from user input will capture the entire provided
precision, up to and including nanosecond granularity.
Throughout the flow in the pipeline, including serialization to PQ, DLQ, and
JSON, will retain all available precision.
BREAKING: This produces an effectively-breaking change to the serialization
format of both the persistent queue (PQ) and dead-letter queue (DLQ),
as the serialized format this changeset contains a higher granularity
of timestamp than previous releases of Logstash were capable of
parsing without error.
As such, it _MUST NOT_ be back-ported to the 7.x series.
Remove hard coded alias definitions in favor of yaml descriptor file.
Introduce a single point of aliases definition (logstash-core/src/main/resources/org/logstash/plugins/AliasRegistry.yml), checksum and copy it around to be used by Logstash and by Logstash's plugin management tool.
The descriptor yml file contains a checksum to verify it's not changed accidentally in a deployment of Logstash, if the verification phase fail Logstash avoid to start and plugin management tool avoid to operate.
The signing and copying around is managed by a specific Gradle task invoked during the build.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes#12831
Having the jar around would allow us to fine tune logging for libraries
such as manticore's http-client (4.5) using LS's `log4j2.properties`
e.g.
```
logger.apache_http_headers.name = org.apache.http.headers
logger.apache_http_headers.level = DEBUG
```
... to log http headers for each request
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
* Update gradle version to 6.3
Gradle versions prior to 6.3 cannot run under JDK14.
This commit upgrades the version of Gradle to 6.3, and removes all deprecation warnings that can currently be removed.
Changes include:
* Increase gradle memory to 2g
* Increase gradle memory in the license check job to 2g
* Replace use of `testCompile`
* Replace `runtime` with `runtimeOnly`
* Remove`compile` depedencies from gradle files
* Replace deprecated archive methods
* Fix dependencies report build
* Make jruby dependencies 'api', fix archiveVersion
* Set `duplicatesStrategy` for all tasks of type Copy
* Use `configureEach` for global 'withType' calls
** Use the recommended Tasks API calls
(https://blog.gradle.org/preview-avoiding-task-configuration-time)
* Run `./gradlew wrapper` earlier to improve caching
* Use copy with chown for resources that need to be run during `./gradlew wrapper`
