* This PR allows the agent to stop pipeline by pipeline_id instead of fetching the full set of pipelines from elasticsearch and compute the pipeline actions internally
Fixed: #12560
* Fix sub second `config.reload.interval`
Due to a change in #11803, using `to_seconds` to normalize values of `config.reload.interval`
would resolve to a value of 0 causing issues in tests where short reload intervals were desired.
This commit uses the `to_nanos` method to preserve the previous functionality.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
- replaces all scripted filters with custom Java implementation
- implemented routing appender per pipeline in Java
- adapted log4j configuration shipped with Logstash
- exit the Logstash process if it detects an scripted log4j configuration
* plugin: adds `:validate => :field_reference`
Provide plugins a way of validating that an input is a literal field-reference.
This is useful for input plugins that implement a `target` or other
non-interpolated directive, and allows these plugins to reject invalid
configuration before start-up instead of at run-time.
Plugins should not use this named validator directly, as doing so would cause
validation to fail with "Unknown validator" when the plugin is run on older
releases of Logstash. Instead, plugins should use the `validator_support`
adapter mixin that provides back-ports when necessary.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
The ModulesSettingArray is responsible to obfuscate password in arrays of settings.
The test are still in Ruby to proove the interoperability with Ruby code that used the previous version.
Added method to mimic .first and .last methods of Ruby Array
There is two Password classes that almost does the same thing. One in Ruby (LogStash::Util::Password) and one in Java (co.elastic.logstash.api.Password).
This commit drop the the Ruby implementation to import the Java version in the LogStash::Util so that existing Ruby code haven't to be changed, works as it is.
This commit adds context to the pipeline to pipeline input and output
plugins by adding a string containing the `address` field to the input
plugin, and an array containing the `send_to` field to the output plugin.
This helps gain a picture of how pipeline to pipeline enabled configurations
are communicating with each other, without having to refer back to the pipeline
definition
* Add optional sourceURL to license report CSV
This commit adds an extra optional column 'sourceURL' to the license report. This
column contains a pointer to the source code, which is optional for most dependencies,
but a requirement for some, such as the Red Hat Universal Base Image.
This commit also populates the 'copyright' field, which previously was an used
column in the CSV definition
Relates #12297
This addresses an incomplete fix in #12019 starting in 7.8.1 where upon catching a worker exception (to avoid crashing the whole logstash per #12306) the input plugin(s) are not terminated prior to closing the pipeline leading to the input plugin(s) continuing execution and failing with IllegalStateException & Tried to write to a closed queue since closing the pipeline also correctly closes the queue.
* Write DLQ entries to temp file first
This commit changes the DLQ writer to write to a temporary file
which will be renamed on "completion", to avoid the possibility
of the DLQ reader reading an incomplete DLQ segment. The temp file
will be renamed and made available, either when the capacity of this
segment is reached, or if a configurable 'flush interval' has elapsed
since the last event reached the dead letter queue.
This commit fixes#8022, #10275, #10967
This commit replaces #11127
Implements a plugin `ecs_compatibility` option, whose default value is powered
by the pipeline-level setting `pipeline.ecs_compatibility`, in line with the
proposal in elastic/logstash#11623:
In order to increase the confidence a user has when upgrading Logstash, this
implementation uses the deprecation logger to warn when `ecs_compatibility` is
used without an explicit directive.
For now, as we continue to add ECS Compatibility Modes, an opting into a
specific ECS Compatibility mode at a pipeline level is considered a BETA
feature. All plugins using the [ECS Compatibility Support][] adapter will
use the setting correctly, but pipelines configured in this way do not
guarantee consistent behaviour across minor versions of Logstash or the
plugins it bundles (e.g., upgraded plugins that have newly-implemented an ECS
Compatibility mode will use the pipeline-level setting as a default, causing
them to potentially behave differently after the upgrade).
This change-set also includes a significant amount of work within the
`PluginFactory`, which allows us to ensure that pipeline-level settings are
available to a Logstash plugin _before_ its `initialize` is executed,
including the maintaining of context for codecs that are routinely cloned.
* JEE: instantiate codecs only once
* PluginFactory: use passed FilterDelegator class
* PluginFactory: require engine name in init
* NOOP: remove useless secondary plugin factory interface
* PluginFactory: simplify, compute java args only when necessary
* PluginFactory: accept explicit id when vertex unavailable
* PluginFactory: make source optional, args required
* PluginFactory: threadsafe refactor of id duplicate tracking
* PluginFactory: make id extraction/geration more abstract/understandable
* PluginFactory: extract or generate ID when source not available
* PluginFactory: inject ExecutionContext before initializing plugins
* Codec: propagate execution_context and metric to clones
* Plugin: intercept string-specified codecs and propagate execution_context
* Plugin: implement `ecs_compatibility` for all plugins
* Plugin: deprecate use of `Config::Mixin::DSL::validate_value(String, :codec)`
A pipeline in the process of being created was not marked as such in the pipeline registry resulting in a situation where a slow to initialize pipeline could be recreated on state convergence resulting in a PQ LockException because that pipeline was already existing and held the PQ lock. Replace native Java concurency with Ruby Mutex for simpler and straighforward implementation.
The worker threads were not correctly monitored for a worker loop exception resulting in a complete logstash crash upon any exception even when multiple pipelines are running. Now only the failed pipeline is terminated. If pipeline reloading is enabled, it is possible to edit the config and have that failed pipeline reloaded.
The creation of a Ruby thread from Java seems to be a trigger
for jruby/jruby#6207.
Pipeline#shutdown now blocks on the ShutdownWatcher#start, which will wait for
pipeline.finished_execution? to be true.
This removes the need for the pattern:
`pipeline.shutdown { block } && pipeline.thread.join`
And can be replaced with just `pipeline.shutdown`
To avoid having `shutdown` blocked waiting for ready? when pipeline crashes too quickly,
this method returns immediately if finished_execution? is true.
Most uses of pipeline#run have also been replaced by pipeline#start
since the latter will block until the pipeline is ready, again avoiding
the pattern:
`pipeline.run && sleep 0.1 until pipeline.ready?`
Pipeline tests have been changed according to these two changes.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* plugin config: support space-deliminated URIs on list-type params
Since whitespace is illegal in URIs, we can safely use it as a delimiter when
validating `list`-type `URI` params, enabling the expansion of an arbitrary
list of URIs from a single Environment- or Keystore-variable.
Resolves: https://github.com/elastic/logstash/issues/8157
Resolves: https://github.com/elastic/logstash/issues/6366
* Doc: Create section for cross-plugin functionality and add space delimiters
Co-authored-by: Karen Metts <karen.metts@elastic.co>
ruby produces a LocalJumpError: unexpected return
error if there's a return in a block so this changes just uses
the value of the last expression as the value of the block
Avoid to reassing the subdocument for queue metrics preferring a merge
With PR #10576 the PluginsStats.report(stats) overwrites the subsection related to queue instead of merge with newly created entries.
Often when browsing logstash logs for debugging purposes we miss the
information about the Java version and platform being used.
Printing the global RUBY_PLATFORM gives us all of this information
plus the JRuby version as well.
Fixes#11852
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
In some workflows such as simple file manipulation, starting a webserver is
unnecessary overhead, and we should be able to avoid it.
Here we introduce a new parameter `http.enabled`, which defaults to `true` to
maintain the existing functionality.
Resolves: elastic/logstash#9408
Closes: elastic/logstash#11525
Co-authored-by: Benoit Dupont <benoit.dupont@gmail.com>
Fixes#11533
We have "required" units for a variety of `TimeValue` settings when they are
provided as a `String`, but unquoted values in YAML have been passed through as
Integers, where we long assumed nanosecond units. This frequently leads to
surprise (e.g., when `config.reload.interval` is set to `60`, we consume 100%
of CPU in a tight loop trying to reload and re-parse the configs every 60
nanoseconds).
By making the setting retain the TimeValue object for the entirety of its
lifecycle, we can issue a deprecation notice the first time an Integer value is
encountered. As a secondary benefit, our usage of the setting value in code
becomes more clear since we are empowered to ask `TimeValue` for a numeric
value in a specific scale.
Fixes#11803
The separator vertices are an implementation detail of the serialized
output of the LIR, and are not meaningful to the pipeline viewer.
This commit removes the separator vertices, and reworks the edges to
account for this.
Fixes#11779
Loading a Java Keystore can take anywhere from ~0.3s to upwards of 3s, so the
pattern of loading one per variable we need to replace adds a significant
amount of overhead on pipelines that use these variables, whether or not they
are provided by the keystore.
By providing a private, constant, lazy singleton, we ensure that we don't
incur the cost of repeatedly building the keystore.
Fixes#10794
cleanup RubyArray "rawtypes"
remove all LinkedHashSet from batch and queue classes
avoid processing empty batches in Java worker loop
cleanup AckedReadBatch and MemoryReadBatch
refactor Ruby worker loop similar to Java Execution to not use batch merge
remove QueueBatch merge and replace LinkedHashSet with ArrayList
