* Fix keystore thread safety
This commit is intended to fix thread safety issues with the JavaKeystore implementation of the secret store.
From reading the code, it appears that thread safety for the keystore was intended to be provided by
a ReentrantReadWriteLock, a read lock for accessing secrets from the keystore, and a write lock for updating
secrets in the keystore.
In practice, this was insufficient, the act of accessing a secret from the keystore involved the mutation of
a shared keyStore object - the keyStore is `load`ed every time a secret is retrieved from the store.
Previous to https://github.com/elastic/logstash/pull/10794, this did not matter, each pipeline held its
own instance of the secret store, effectively meaning that only a single thread would ever access a key
store at any one time. This PR moved to using a shared keystore instance for substitution variables,
exposing the lack of thread safety in the JavaKeystore class.
This commit is intended to be the simplest change to fix the underlying issue, and does not address whether
we *need* to reload the secrets every time they are read.
Relates #12229
when running a pipeline with ordered execution, flushes on the pipeline
were no longer being called when compute is called with an empty batch, causing
issues with the aggregate filter, for example, not being able to push events on
timeout.
Our internal representation of the composite config file needs only to inject
newline delimiters if they are missing, and to avoid doing so if they are
present. This allows `PipelineConfig#sourceReferences()`, used to map back from
the composite line/column to source file/column, to correctly track an offset
using the source fragments `SourceWithMetadata#getLinesCount()`.
Fixes: #12155
A pipeline in the process of being created was not marked as such in the pipeline registry resulting in a situation where a slow to initialize pipeline could be recreated on state convergence resulting in a PQ LockException because that pipeline was already existing and held the PQ lock. Replace native Java concurency with Ruby Mutex for simpler and straighforward implementation.
The worker threads were not correctly monitored for a worker loop exception resulting in a complete logstash crash upon any exception even when multiple pipelines are running. Now only the failed pipeline is terminated. If pipeline reloading is enabled, it is possible to edit the config and have that failed pipeline reloaded.
Many DLQ and PQ tests were run with default settings for their size,
or otherwise set to values such as 1GB.
This meant that the container/machine the test ran on needed a lot of disk space
(i.e. 1GB for the test + 1GB for OS and Logstash + some more free space).
This commit drops the disk space requirement overall by a factor of 10 (e.g. 1GB to 100MB)
this test only called "pipeline.start" and expected an output to receive
the "do_close" method call. Therefore it relies on a race condition that
the pipeline shuts down quickly enough, which could fail.
This change ensures the pipeline fully terminated before making the
assertion
This commit also removes an unused let from the time when logstash
output plugins had workers.
This test relies on the generator input sending 10 events and observing
the output. Calling shutdown immediately after start can cause the
plugin to abort earily.
This change waits for the execution to finish before performing the
test assertions.
The creation of a Ruby thread from Java seems to be a trigger
for jruby/jruby#6207.
Pipeline#shutdown now blocks on the ShutdownWatcher#start, which will wait for
pipeline.finished_execution? to be true.
This removes the need for the pattern:
`pipeline.shutdown { block } && pipeline.thread.join`
And can be replaced with just `pipeline.shutdown`
To avoid having `shutdown` blocked waiting for ready? when pipeline crashes too quickly,
this method returns immediately if finished_execution? is true.
Most uses of pipeline#run have also been replaced by pipeline#start
since the latter will block until the pipeline is ready, again avoiding
the pattern:
`pipeline.run && sleep 0.1 until pipeline.ready?`
Pipeline tests have been changed according to these two changes.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
The reduction from 500 to 100 is based on observations where 06d7f01fd
reduced the number of generated classes by about an order of magnitude
especially on very large pipelines (e.g. from ~600 to ~30).
Generated implementations of `Dataset` often have fields referencing
other specifically-generated `Dataset`, despite only using public methods
(`compute` and `clear`) defined on the `Dataset` interface.
By allowing the code generation to reference the interface instead of
the specific implementation, we eliminate the need for the compiler to chain
parent class loaders indefinitely, thereby eliminating the need for a global
mutual exclusion when compiling.
This change moves the locking semantics from the compiler to the non-evicting
cache itself, relying on tried-and-true `ConcurrentHashMap#computeIfAbsent`
to minimize synchronization and cache-priming stampedes.
This also vastly reduces the scope of `Dataset` implementations that we need
to generate, because datasets will no longer need to reference the specific
implementation details of all "downstream" datasets and will therefore be more
likely to match an implementation that has already been compiled and cached.
* plugin config: support space-deliminated URIs on list-type params
Since whitespace is illegal in URIs, we can safely use it as a delimiter when
validating `list`-type `URI` params, enabling the expansion of an arbitrary
list of URIs from a single Environment- or Keystore-variable.
Resolves: https://github.com/elastic/logstash/issues/8157
Resolves: https://github.com/elastic/logstash/issues/6366
* Doc: Create section for cross-plugin functionality and add space delimiters
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Changed ComputeStepSyntaxElement to generate Java code to retrieve the plugin's id by a method instead of hardcoding the value in the generated code.
This permit to share more compiled classes, that differs only by plugin.id and speed up the pipeline compilation.
The change has been secured by future regression with unit test that track pipeline compilations times.
Co-authored-by: Andrea Selva <andsel@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes: #12031
Having the jar around would allow us to fine tune logging for libraries
such as manticore's http-client (4.5) using LS's `log4j2.properties`
e.g.
```
logger.apache_http_headers.name = org.apache.http.headers
logger.apache_http_headers.level = DEBUG
```
... to log http headers for each request
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
ruby produces a LocalJumpError: unexpected return
error if there's a return in a block so this changes just uses
the value of the last expression as the value of the block
Avoid to reassing the subdocument for queue metrics preferring a merge
With PR #10576 the PluginsStats.report(stats) overwrites the subsection related to queue instead of merge with newly created entries.
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
`RubyString#gsub` requires a (Ruby) frame to be present.
The method attempts to set a backref for the current caller's frame.
When the frame stack is empty there isn't really a place to set $~.
This can happen when a LogStash::Util::Loggable#logger is retrieved,
from the input worker thread while not being nested in any block.
Fixes#11874
for Strings with copy-on-write semantics when deep cloning.
motivated by "big" events reaching plugins such as split,
which might produce several new events out of a single one.
Fixes#11794
Often when browsing logstash logs for debugging purposes we miss the
information about the Java version and platform being used.
Printing the global RUBY_PLATFORM gives us all of this information
plus the JRuby version as well.
Fixes#11852
In time comparison of LocalDateTime the isBefore is strict, so in case two instants has the same millisecond, it fails in test (happens in Windows tests)
Close: 11862
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
* Avoid using deprecated `getSystemCpuLoad` method with JDK14
JDK14 prefers the use of `getCpuLoad` over `getSystemCpuLoad`. This commit
reworks the call to use reflection to use the appropriate method call
depending on the version of the JDK being used.
* Added warning message if OperatingSystemMXBean not available
* Also corrected spacing after code review comments.
In some workflows such as simple file manipulation, starting a webserver is
unnecessary overhead, and we should be able to avoid it.
Here we introduce a new parameter `http.enabled`, which defaults to `true` to
maintain the existing functionality.
Resolves: elastic/logstash#9408
Closes: elastic/logstash#11525
Co-authored-by: Benoit Dupont <benoit.dupont@gmail.com>
Fixes#11533
We have "required" units for a variety of `TimeValue` settings when they are
provided as a `String`, but unquoted values in YAML have been passed through as
Integers, where we long assumed nanosecond units. This frequently leads to
surprise (e.g., when `config.reload.interval` is set to `60`, we consume 100%
of CPU in a tight loop trying to reload and re-parse the configs every 60
nanoseconds).
By making the setting retain the TimeValue object for the entirety of its
lifecycle, we can issue a deprecation notice the first time an Integer value is
encountered. As a secondary benefit, our usage of the setting value in code
becomes more clear since we are empowered to ask `TimeValue` for a numeric
value in a specific scale.
Fixes#11803
