As we close in on the availability of 8.0.0 alphas, we are reassessing which
breaking changes are _necessary_, and which are merely _desired_. And while
we would love to be in a world where ECS was on by default, and have put
substantial effort into designing an upgrade path that would be as simple as
possible, we have determined that the time may not be right to change the
default value out of under our users.
This change restores the default value for `pipeline.ecs_compatibility` to
`disabled`, ensuring pipelines will continue running in Logstash 8 as they have
in Logstash 7 without modification. We will still encourage our users to be
explicit about which behaviour they desire, and will revisit making ECS on by
default at a later date.
* noop: avoid declaring default value in config file
* docs: ecs compatibility from 7.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* ecs: on by default
We know that ECS version 8 will release along-side Logstash 8.0, but its scope
is still coming into focus. In this changeset, we change the default value
of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a
significantly closer approximation to what will eventually ship in Logstash
8.0.0.
* docs: ecs from 8.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Introcuce the concept of alias for a plugin.
Creates an AliasRegistry to map plugin aliases to original plugins.
If a real plugin with same name of the an alias is present in the system, then the real plugin take precedence during the
instantiation of the pipeline.
Simplified the error handling in class lookup
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This change fix the behavior of considering as "running" also pipelines that are still in "loading", both "loading" and "running" is considered as "not terminated".
Fixed a flakyness in tests due to different ways to looks at the same thing: pipeline status.
The pipeline status is determined by both `pipeline.running?` and by `agent.pipelines_running`.
The first checks for an atomic boolean in pipeline object, the second check for the status in PipelineRegistry.
Fixes#12190
* This PR allows the agent to stop pipeline by pipeline_id instead of fetching the full set of pipelines from elasticsearch and compute the pipeline actions internally
Fixed: #12560
* Fix sub second `config.reload.interval`
Due to a change in #11803, using `to_seconds` to normalize values of `config.reload.interval`
would resolve to a value of 0 causing issues in tests where short reload intervals were desired.
This commit uses the `to_nanos` method to preserve the previous functionality.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* plugin: adds `:validate => :field_reference`
Provide plugins a way of validating that an input is a literal field-reference.
This is useful for input plugins that implement a `target` or other
non-interpolated directive, and allows these plugins to reject invalid
configuration before start-up instead of at run-time.
Plugins should not use this named validator directly, as doing so would cause
validation to fail with "Unknown validator" when the plugin is run on older
releases of Logstash. Instead, plugins should use the `validator_support`
adapter mixin that provides back-ports when necessary.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
The ModulesSettingArray is responsible to obfuscate password in arrays of settings.
The test are still in Ruby to proove the interoperability with Ruby code that used the previous version.
Added method to mimic .first and .last methods of Ruby Array
This commit adds context to the pipeline to pipeline input and output
plugins by adding a string containing the `address` field to the input
plugin, and an array containing the `send_to` field to the output plugin.
This helps gain a picture of how pipeline to pipeline enabled configurations
are communicating with each other, without having to refer back to the pipeline
definition
This addresses an incomplete fix in #12019 starting in 7.8.1 where upon catching a worker exception (to avoid crashing the whole logstash per #12306) the input plugin(s) are not terminated prior to closing the pipeline leading to the input plugin(s) continuing execution and failing with IllegalStateException & Tried to write to a closed queue since closing the pipeline also correctly closes the queue.
Implements a plugin `ecs_compatibility` option, whose default value is powered
by the pipeline-level setting `pipeline.ecs_compatibility`, in line with the
proposal in elastic/logstash#11623:
In order to increase the confidence a user has when upgrading Logstash, this
implementation uses the deprecation logger to warn when `ecs_compatibility` is
used without an explicit directive.
For now, as we continue to add ECS Compatibility Modes, an opting into a
specific ECS Compatibility mode at a pipeline level is considered a BETA
feature. All plugins using the [ECS Compatibility Support][] adapter will
use the setting correctly, but pipelines configured in this way do not
guarantee consistent behaviour across minor versions of Logstash or the
plugins it bundles (e.g., upgraded plugins that have newly-implemented an ECS
Compatibility mode will use the pipeline-level setting as a default, causing
them to potentially behave differently after the upgrade).
This change-set also includes a significant amount of work within the
`PluginFactory`, which allows us to ensure that pipeline-level settings are
available to a Logstash plugin _before_ its `initialize` is executed,
including the maintaining of context for codecs that are routinely cloned.
* JEE: instantiate codecs only once
* PluginFactory: use passed FilterDelegator class
* PluginFactory: require engine name in init
* NOOP: remove useless secondary plugin factory interface
* PluginFactory: simplify, compute java args only when necessary
* PluginFactory: accept explicit id when vertex unavailable
* PluginFactory: make source optional, args required
* PluginFactory: threadsafe refactor of id duplicate tracking
* PluginFactory: make id extraction/geration more abstract/understandable
* PluginFactory: extract or generate ID when source not available
* PluginFactory: inject ExecutionContext before initializing plugins
* Codec: propagate execution_context and metric to clones
* Plugin: intercept string-specified codecs and propagate execution_context
* Plugin: implement `ecs_compatibility` for all plugins
* Plugin: deprecate use of `Config::Mixin::DSL::validate_value(String, :codec)`
A pipeline in the process of being created was not marked as such in the pipeline registry resulting in a situation where a slow to initialize pipeline could be recreated on state convergence resulting in a PQ LockException because that pipeline was already existing and held the PQ lock. Replace native Java concurency with Ruby Mutex for simpler and straighforward implementation.
The worker threads were not correctly monitored for a worker loop exception resulting in a complete logstash crash upon any exception even when multiple pipelines are running. Now only the failed pipeline is terminated. If pipeline reloading is enabled, it is possible to edit the config and have that failed pipeline reloaded.
Many DLQ and PQ tests were run with default settings for their size,
or otherwise set to values such as 1GB.
This meant that the container/machine the test ran on needed a lot of disk space
(i.e. 1GB for the test + 1GB for OS and Logstash + some more free space).
This commit drops the disk space requirement overall by a factor of 10 (e.g. 1GB to 100MB)
this test only called "pipeline.start" and expected an output to receive
the "do_close" method call. Therefore it relies on a race condition that
the pipeline shuts down quickly enough, which could fail.
This change ensures the pipeline fully terminated before making the
assertion
This commit also removes an unused let from the time when logstash
output plugins had workers.
This test relies on the generator input sending 10 events and observing
the output. Calling shutdown immediately after start can cause the
plugin to abort earily.
This change waits for the execution to finish before performing the
test assertions.
The creation of a Ruby thread from Java seems to be a trigger
for jruby/jruby#6207.
Pipeline#shutdown now blocks on the ShutdownWatcher#start, which will wait for
pipeline.finished_execution? to be true.
This removes the need for the pattern:
`pipeline.shutdown { block } && pipeline.thread.join`
And can be replaced with just `pipeline.shutdown`
To avoid having `shutdown` blocked waiting for ready? when pipeline crashes too quickly,
this method returns immediately if finished_execution? is true.
Most uses of pipeline#run have also been replaced by pipeline#start
since the latter will block until the pipeline is ready, again avoiding
the pattern:
`pipeline.run && sleep 0.1 until pipeline.ready?`
Pipeline tests have been changed according to these two changes.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* plugin config: support space-deliminated URIs on list-type params
Since whitespace is illegal in URIs, we can safely use it as a delimiter when
validating `list`-type `URI` params, enabling the expansion of an arbitrary
list of URIs from a single Environment- or Keystore-variable.
Resolves: https://github.com/elastic/logstash/issues/8157
Resolves: https://github.com/elastic/logstash/issues/6366
* Doc: Create section for cross-plugin functionality and add space delimiters
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
We have "required" units for a variety of `TimeValue` settings when they are
provided as a `String`, but unquoted values in YAML have been passed through as
Integers, where we long assumed nanosecond units. This frequently leads to
surprise (e.g., when `config.reload.interval` is set to `60`, we consume 100%
of CPU in a tight loop trying to reload and re-parse the configs every 60
nanoseconds).
By making the setting retain the TimeValue object for the entirety of its
lifecycle, we can issue a deprecation notice the first time an Integer value is
encountered. As a secondary benefit, our usage of the setting value in code
becomes more clear since we are empowered to ask `TimeValue` for a numeric
value in a specific scale.
Fixes#11803
Loading a Java Keystore can take anywhere from ~0.3s to upwards of 3s, so the
pattern of loading one per variable we need to replace adds a significant
amount of overhead on pipelines that use these variables, whether or not they
are provided by the keystore.
By providing a private, constant, lazy singleton, we ensure that we don't
incur the cost of repeatedly building the keystore.
Fixes#10794
cleanup RubyArray "rawtypes"
remove all LinkedHashSet from batch and queue classes
avoid processing empty batches in Java worker loop
cleanup AckedReadBatch and MemoryReadBatch
refactor Ruby worker loop similar to Java Execution to not use batch merge
remove QueueBatch merge and replace LinkedHashSet with ArrayList
reuse rubyArray for single element batches
rename preserveBatchOrder to preserveEventOrder
allow boolean and string values for the pipeline.ordered setting, reorg validation
update docs
yml typo
Update docs/static/running-logstash-command-line.asciidoc
Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>
Update docs/static/running-logstash-command-line.asciidoc
Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>
java execution specs and spec support
docs corrections per review
typo
close not shutdown
Ruby pipeline spec
