Adds log.format.json.fix_duplicate_message_fields feature flag to rename the clashing fields when json logging format (log.format) is selected.
In case two message fields clashes on structured log message, then the second is renamed attaching _1 suffix to the field name.
By default the feature is disabled and requires user to explicitly enable the behaviour.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Wipe out comment lines if config comment contains.
* Remove substitution var process when loading the YAML, instead align on the generic approach which LSCL happens during the pipeline compile.
* Update logstash-core/src/main/java/org/logstash/config/ir/PipelineConfig.java
Put the logging config back as it is being used with composed configs.
Introduce a new setting named `pipeline.buffer.type` which could be valued direct or heap to enable the allocation on Java heap.
The processing of the setting is done in `LogStash::Runner#execute` and sets the Java properties considered by Netty to disable the direct allocation: `io.netty.noPreferDirect`.
However, if that system property is already configured explicitly by the user (because set in `jvm.options`or `LS_JAVA_OPTS`) the setting doesn't take place and warning log is reported, respecting the user's will.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Given that JRuby comes with ruby-maven-libs 3.3.9 this commit upgrades the gem to 3.8.9 and ensures files from 3.3.9 are not included in the distribution.
Remove unused method createFlushScheduler and add some debug logs to the finalizeSegment method to better follow what happens during its execution in case of problems.
This PR adds a shutdown method to the SchedulerService class used to handle actions to be executed on a certain cadence. In particular is used to execute scheduled finalization of DLQ head segment.
Updates the close method of the DLQ writer to invoke this additional shutdown on the service instance.
Adds a burning of time condition to avoid a collision of time which, under certain circumstances, would fail the test.
The sealing of a segment happens if the segment is considered as stale, which requires 2 conditions:
- the segment must have received a write.
- the time of the last write must exceed the flush interval.
In this failing test, the flush interval is set to ZERO because of the synchronicity of the test, to avoid time dependency. However, with coarse grain timer resolution, could happen that the last write coincide with the time of the stale check, so fail the seal condition.
* Ruby code coverage with SimpleCov json formatter
* [CI] Send Java and ruby tests to sonarqube simultaneously
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Added compiled classes to artifacts
* Test change
* Removed test changes
* Returned back ENABLE_SONARQUBE condition
* Removed debug line
* Diable Ruby coverage if ENABLE_SONARQUBE is not true
* Run sonar scan on pull requests and onn push to main
* Run sonar can on release branches
This commit added a few jvm.options properties to configure the Jackson read constraints defaults (Maximum Number value length, Maximum String value length, and Maximum Nesting depth).
This commit fixes how the keystore tool handle the command's options, including validation for unknown options, and adding the --stdin flag to the add command.
Introduces a new interface named SchedulerService to abstract from the ScheduledExecutorService to execute the DLQ flushes of segments. Abstracting from time provides a benefit in testing, where the test doesn't have to wait for things to happen, but those things could happen synchronously.
In DLQ unit testing sometime the DLQ writer is started explicitly without starting the segments flushers. In such cases the test 's logs contains exceptions which could lead to think that the test fails silently.
Avoid to invoke scheduledFlusher's shutdown when it's not started (such behaviour is present only in tests).
This commit added support to add and remove multiple keystore keys in a single operation. It also fixed the empty value validation for editing existing key values and added ASCII validation for values.
This removes the dependency on jackson's dataformat-yaml. Since there's only a single place where this library is used in core: to load the plugin alias definition, the code can be replaced by the underlying snakeyaml.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 93d8a9da32)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes the issue with Jackson > 2.15 and `log.format=json`: "Can not write a field name, expecting a value", by adding a default serializers for JRuby's objects.
* geoip: extract database manager to stand-alone feature
Introduces an Elastic-licensed GeoipDatabaseManagement tool that can be used
by ANY plugin running on Elastic-licensed Logstash to retrieve a subscription
to a GeoIP database that ensures EULA-compliance and frequent updates, and
migrates the previous Elastic-licensed code-in-Logstash-core extension to
the Geoip Filter to use this new tool, requiring ZERO changes to in-the-wild
versions of the plugin.
The implementation of the new tool follows the previous implementation as
closely as possible, but presents a new interface that ensures that a
consumer can ATOMICALLY subscribe to a database path without risk that the
subscriber will receive an update or expiry before it is finished applying
the initial value:
~~~ ruby
geoip_manager = LogStash::GeoipDatabaseManagement::Manager.instance
subscription = geoip_manager.subscribe('City')
subscription.observe(construct: ->(initial_dbinfo){ },
on_update: ->(updated_dbinfo){ },
on_expire: ->( _ ){ })
subscription.release!
~~~
* docs: link in geoip database manager docs
* docs: reorganize pending 'geoip database management' feature
* docs: link to geoip pages from feature index
* geoip: add SubscriptionObserver "interface"
simplifies using Subscription#observe from Java
* geoip: fixup SubscriptionObserver after rename
* geoip: quacking like a SubscriptionObserver is enough
* geoip: simplify constants of legacy geoip filter extension
* geoip: bump logging level to debug for non-actionable log
* geoip: refine log message to omit non-actionable info
* re-enable invokedynamic (was disabled to avoid upstream bug)
* geoip: resolve testing fall-out from filter extension's "private" constants removal
* geoip: consistently use `DataPath#resolve` internally, too
Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861
Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message.
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
The DeadLetterQueueWriter#updateOldestSegmentReference was sorting the existing segments by the lexicographic file names. For example, if there are a few segments on the DQL path with index 8, 9 and 10, it was choosing the 10 as the oldest segment instead of 8.
This commit changed it to comparing the segment ID as numbers, so the file deletion will happen following the proper order.
* Pipeline to pipeline communication acked queue improvements.
* Handle InterruptedException exception in input back, warn message improvement when in-flight events are partially sent and other minor such as descriptive logs, etc improvements.
* Apply suggestions from code review
Check if queue is open after thread acquires the lock.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Apply suggestions from code review
Unite test case improvement: use `assertThrows` when validating the exception.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Pulling off of introducing wrap with operations.
* Introduce functional interface to broadly use to catch the exception types.
* Addressing comments: do not retry sending inflight events on case. We still throw if we get error when opening queue.
* will not be reached input retry logic, removing.
* Move queue close check after thread acquiring a lock. Make read next page interface private since it is an internal use purpose.
* Apply suggestions from code review
Leave a comment for the write lock and remove unnecessary warning with `ensure_delivery=>false`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Remove unused method, check if current thread acquired lock when accessing next page.
* pq: getting possibly-shared access to next read page is illegal
The private `Queue#nextReadPage()` method requires that the caller has
exclusive ownership of the lock, and failing to have the lock is an
illegal state that cannot be recoverd from; it would leak the
effectively-private `Page` to a caller that cannot reliably use it
without corrupting other callers.
Both callers of this private method already call it with exclusive
access, so this safeguard is merely to prevent future development from
breaking the expectation unknowingly.
As such, we throw an `IllegalStateException`.
* pq: use shared queue-closed check for block and non-block reads
By moving the closed-check from the blocking `Queue#readBatch` to the
shared private `Queue#nextReadPage`, we ensure that both blocking reads
by `Queue#readBatch` and non-blocking reads by `Queue#nonBlockReadBatch`
behave the same when the queue has been closed.
* pq: make exception message constants descriptive
* p2p: clarify comment about cumulating retry behaviour
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit moves the Files.size(...) call into the try catch block, that way, when the oldest segment is deleted by the DeadLetterQueueReader, no NoSuchFileException will be thrown up, and the writer will gracefully update the oldest segment on the next updateOldestSegmentReference invocation (scheduled flush, entry write, delete expired, etc).
It also adds the volatile keyword to the shared mutable variables, making sure that all the changes will be instantly visible among all the running threads (scheduler & writer).
This commit updates the puma gem from version 5 to the latest version 6.3.
A few breaking changes were introduced in Puma 6.0.0, which required some refactoring on the Logstash side, especially to adapt it to the "Extracted LogWriter from Events #2798" changes.
Before this PR, all the logs generated by Puma were using the debug level, even the ones that were actually errors and needed attention/action from the users. This commit also changes the log level as following:
error(...): changed from debug to error
unknown_error(..): changed from debug to error
Set of changes to make Logstash compatible to JRuby 9.4.
Bundle JRuby 9.4.3.0
- Redefine space token in `LSCL` and `grammar` treetop from `_` which would generated methods in the form `def _0` (deprecated since `2.7`) to `sc`.
- `I18n.t` method doesn't accept hash as second argument
- `URI.encode` has been replaced with same functionality with `URI::Parser.new.escape`
- `YAML.load` needs explicit `fallback: false` to return false when the yaml string is empty (or contains only comments)
- JRuby's `JavaClass` has been removed, now it can use `java.lang.Class` directly
- explicitly require gem `thwait` to satisfy `require "thwait"` (In `Gemfile.template` and `logstash-core/logstash-core.gemspec`)
- fix not args `clone` to be `def clone(*args)`
- fix `Enumeration.each_slice` which from `Ruby 3.1` is [chainable](https://rubyreferences.github.io/rubychanges/3.1.html#enumerableeach_cons-and-each_slice-return-a-receiver) and doesn't return `nil`. JRuby fixed in https://github.com/jruby/jruby/issues/7015
- Expanded `Down.download` arguments map ca16bbed3c302006967413eb9d3862f2da81f7ae
- Avoid to pass `nil` in the list of couples used in `Hash[ <list of couples> ]` which from Ruby `3.0` generates an `ArgumentError`
- Removed space not allowed between method name and parentheses `initialize (` is forbidden. 29b607dcdef98f81a73ad171639fd13aaa65e243
- With [Ruby 2.7 the `Kernel#open`](https://rubyreferences.github.io/rubychanges/2.7.html#network-and-web) doesn't fallback to `URI#open`, fixed test code that used that to verify open port. e5b70de54c5301f51a767da67294092af0cfafdc
- Avoid to drop `rdoc/` folder from vendored JRuby else `bin/logstash -i irb` would crash, commit b71f73e9c6edb81a7b7ae1305047e506f61c6e8c
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit changed the java_pipeline.rb to include the pipeline/main thread on the stalling threads info list, that way, Logstash can provide users with more helpful information when the stalling thread is the pipeline/main one.
This commit adds missing Elasticsearch SSL settings and replaces deprecated options being used on `xpack.monitoring.*` and `xpack.management.*` settings:
Changes:
- Updated deprecated monitoring and management Elasticsearch's SSL settings so no warnings are logged.
- Added monitoring settings support for file-based certificates and for the cipher suites: `xpack.monitoring.elasticsearch.ssl.certificate`, `xpack.monitoring.elasticsearch.ssl.key`, and `xpack.monitoring.elasticsearch.ssl.cipher_suites`.
- Added management settings support for file-based certificates and for the cipher suites: `xpack.management.elasticsearch.ssl.certificate`, `xpack.management.elasticsearch.ssl.key`, and `xpack.management.elasticsearch.ssl.cipher_suites`.
