- moved parsing of jvm.options file into Java code
- chnaged the parsing code to consider conditional notation to bind the applicability of certain JVM flags to specific JVM versions
- changed the launch scripts (.sh and .bat) to use the options string composition
- binded CMS flags to JVM specifications 8-14
- replaces all scripted filters with custom Java implementation
- implemented routing appender per pipeline in Java
- adapted log4j configuration shipped with Logstash
- print a warn message if it detects an scripted log4j configuration and continue the execution (#12591)
Currently, LS does not respect fatal errors such as java.lang.OutOfMemoryError and continues executing.
This is dangerous since JVM errors are a legitimate reason to halt the process and not continue processing.
Additionally:
- make sure we log the full stack-trace on fatal errors
- halt the JVM wout executing finalizers/hook (scissors on how ES handles uncaught exceptions)
- also, we should now be aware of a potentially unexpectedly dying thread
Back-port of #12470
When the PQ creates a new page and allocates a memory-mapped buffer, the
underlying file is zero'd out to full page capacity and the version byte is
written to the buffer.
If Logstash crashes or is shut down before any elements have been pushed into
the queue page, we have no guarantees that the version marker has been
persisted to the storage device. A subsequent attempt to load an all-zeros
queue page will result in an obscure error message and failure to load:
~~~
AbstractPipelineExt - Logstash failed to create queue.
org.logstash.ackedqueue.io.MmapPageIOV2$PageIOInvalidVersionException: Expected page version=2 but found version=0
~~~
By sending `MappedByteBuffer#force()` immediately after the version has been
added to the buffer, we can shrink the window in which a crash can leave the
queue on disk in a corrupt state.
This commit updates the license information for the license dependency report.
Specifically, this adds a notice for racc, a different version of which is now
pulled in by nokogiri from the version included with jruby.
* plugin: adds `:validate => :field_reference`
Provide plugins a way of validating that an input is a literal field-reference.
This is useful for input plugins that implement a `target` or other
non-interpolated directive, and allows these plugins to reject invalid
configuration before start-up instead of at run-time.
Plugins should not use this named validator directly, as doing so would cause
validation to fail with "Unknown validator" when the plugin is run on older
releases of Logstash. Instead, plugins should use the `validator_support`
adapter mixin that provides back-ports when necessary.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Clean backport of #12498
these labels are required for redhat openshift certification.
These commit reintroduces the labels for the ubi8 image only, and adds
acceptance tests to ensure these labels are correct and not inherited
The ModulesSettingArray is responsible to obfuscate password in arrays of settings.
The test are still in Ruby to proove the interoperability with Ruby code that used the previous version.
Added method to mimic .first and .last methods of Ruby Array
(cherry picked from commit fa3891953d)
There is two Password classes that almost does the same thing. One in Ruby (LogStash::Util::Password) and one in Java (co.elastic.logstash.api.Password).
This commit drop the the Ruby implementation to import the Java version in the LogStash::Util so that existing Ruby code haven't to be changed, works as it is.
(cherry picked from commit ca81a8f4a3)
Clean backport of #12447, #12452
This commit fixes two issues with the docker metadata:
Removes non-OCI compliant freeform metadata labels
Uses a consistent build date for all the docker images and dockerfiles
Additionally, this commit adds a `build_docker_ubi8` rake task to enable
`ci/docker_acceptance_tests.sh` to run with no options to build all
docker images for the architecture.
Removing the freeform description labels left the container metadata
without a description label. This commit adds a description under the
"org.opencontainers.image.description" label
Clean backport of #12426
Where available, this commit adds information from getSourceWithMetadata to the
error message of UnexpectedTypeException, dropping down to `toString`
if not, giving more context to find where the issue is caused in the configuration.
Clean backport of #12394
This commit adds context to the pipeline to pipeline input and output
plugins by adding a string containing the `address` field to the input
plugin, and an array containing the `send_to` field to the output plugin.
This helps gain a picture of how pipeline to pipeline enabled configurations
are communicating with each other, without having to refer back to the pipeline
definition
add wildcard support in xpack pipeline id
do the pattern matching with glob
add warning msg to wildcard with legacy api
check invalid pipeline in bootstrap
test cases for invalid checking
Fixed: #10558
Restructures troubleshooting docs in preparation for expanding content
Adds info for plugin tracing to help users track down plugins that might be causing problems
Co-authored-by: João Duarte jsvd@users.noreply.github.com
Backports: #12270Fixes: #12228
Moves Cloud info to Configuration section to make it more obvious and easier to find
Expands content for using cloud id and cloud auth outside of modules
Moves module-specific info into modules section
Backports #11884 to 7.x
Clean backport of #12346
This commit adds an extra optional column 'sourceURL' to the license report. This
column contains a pointer to the source code, which is optional for most dependencies,
but a requirement for some, such as the Red Hat Universal Base Image.
This commit also populates the 'copyright' field, which previously was an used
column in the CSV definition
Relates #12297
to avoid gems being resolved from the usual LS GEM_HOME
this is problematic for gems such as jruby-openssl which are loaded
during boot (by RGs/Bundler) and thus activated in Bundler from a
different GEM_HOME. if such gem is updated it won't end up being
install-ed in the --path location as it's found on the GEM_HOME!
+ Fix: gem conflict 1.3.6 required by core
this is due now isolating GEM_HOME on `bundle install --path`
+ Refactor: we do not need LS_GEM_HOME/PATH
+ avoid pinning jruby-openssl to 0.10.4
resolves GH-12299 (reverting GH-12301)
Clean backport of #12335
When deleting temporary files created by the DLQ writer to store data before moving to their
final location, Windows may leave these files in a "delete pending" state, where the files
are somewhat in a state of limbo, where they result of `Files.exist(filename)` is `false`,
but the result of `filename.toFile().exists()` is true. When files are in this state, a new
file with the same name cannot be created, which causes the DLQ test used to ensure that
closing and reopening the DLQ (in such events as a pipeline restart) to fail.
This commit moves the temporary file to an alternative location before deletion, ensuring that
the "pending delete" status does not interrupt with the DLQ startup
This addresses an incomplete fix in #12019 starting in 7.8.1 where upon catching a worker exception (to avoid crashing the whole logstash per #12306) the input plugin(s) are not terminated prior to closing the pipeline leading to the input plugin(s) continuing execution and failing with IllegalStateException & Tried to write to a closed queue since closing the pipeline also correctly closes the queue.
Clean backport of #12304
This commit changes the DLQ writer to write to a temporary file
which will be renamed on "completion", to avoid the possibility
of the DLQ reader reading an incomplete DLQ segment. The temp file
will be renamed and made available, either when the capacity of this
segment is reached, or if a configurable 'flush interval' has elapsed
since the last event reached the dead letter queue.
This commit fixes#8022, #10275, #10967
This commit replaces #11127
Clean backport of #12314
The `shouldRunAfter` specified in the main script body was causing the runIntegrationTests
task to be evaluated even when it should not have been, causing unnecessary failures
when artifacts required only for integration tests are unavailable.
This can be removed, because the `shouldRunAfter` relationship for the `runIntegrationTests`
task is already defined in the task body.
* replace direct hidden indices access with system indices api
* fulfill backward compatibility
* fix log msg, rename class, simplify response handling
* modularise fetcher
