If #stop is called on Puma before #run is finished - for example, when an
incorrect configuration is specified, Puma can be left with threads hanging
around.
This is due to a check/notify pipe used to signal state changes not being created
until halfway through the #run method, enabling a window where #stop can be called
before the pipe has been created, leading to the run thread not exiting. Prior to
jruby-9.3.x, this would tend to be benign and Logstash would exit normally. However,
jruby-9.3.x introduced a tear down as part of the shutdown mechanism, which will join
active threads on shutdown. This puma issue would cause an indefinite hang on exit
when this condition is triggered.
Updates the `internalReceive` method implementation in Pipeline Input to catch exception error and return the position where the stream was interrupted. Modify the EventBus's batch forwarding logic to handle errors from Pipeline Input and apply retry logic only from last error position in the batch of events.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Prior to this commit, attempting to use a custom java plugin installed from rubygems would
fail as the `files` field used to find jar files in the gem is not populated for such gems
This commit uses the `full_require_path` to find the jar file for a native jar plugin,
which should work regardless of where the gem was installed from, locally or via rubygems
Additionally, this commit will only attempt to locate the jar file inside the gem when it
is required (only when the experimental separate plugin class loader is used)
This commit builds on the work done in #13888
Relates: #13888, #11305
Co-authored-by: twosom <two_somang@icloud.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Exposes dead_letter_queue.storage_policy configuration setting to explicitly enable the drop_older behavior in DLQs.
Moving from a drop_newer to a drop_older behavior has impact both on the writer side and to the reader side.
The implementation leverage the fact that a complete DLQ segment can be removed to free up space; on the writer side when the dead_letter_queue.max_bytes limit is reached it has to remove old segments.
On the reader side, the consuming has to be adapted to don't expect a continuous flow of segments, it could face an hole due to removal of tail segments.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Prior to the change, pipeline `stop` and `delete` happen in two converge cycles, which
has a gap letting the stopped pipeline compare with the same pipeline definition
in central pipeline management, hence Logstash see the stopped pipeline as graceful finish
and not to delete in registry
This commit creates StopAndDelete action to delete running pipeline in one converge cycle
Fixed: #14017
* Make AliasRegistry a singleton
The current implementation of AliasRegistry will create an instance of the Alias Registry for each
pipeline, which appears to potentially result in situations, such as in #13996, where multiple pipelines
are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with
the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when
reading the yaml file as the JarFile has been closed after the initial thread completed accessing.
This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry.
Relates: #13996, #13666
* Update reflections library to 0.9.12 to avoid multi threading bug
Earlier versions of the reflections library used in the plugin registry would
use caches on JarUrlConnection, which when closed would also close the jar file
for other resources using it, such as the AliasRegistry. This, combined with the
fact that the AliasRegistry could be created simultaneously by many threads/pipelines
could cause issues during AliasRegistry creation leading to failure to create a
pipeline.
* Avoid use of URLConnection caching when getting alias yaml resource
* Use idiomatic ruby when accessing Java getInstance method
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
As pointed out in this post merge review comment, there is a window
where we could miss a pipeline transitioning from 'loading' to 'running'
in the original fix, as separate calls are made to the pipeline registry.
This commit fixes that by making a single call to the pipeline registry which
allows for also returning pipelines in the `loading` state.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
When logstash is run without automatic reloading, it is still possible to reload configurations
by using 'SIGHUP'. This functionality was broken in #12444, which split non-terminated pipelines
into "loading" and "running" states. The call `no_pipelines?` in agent#execute would no longer
find pipelines in a "loading" state, causing the loop to exit, and logstash to shutdown. This
commit tests for pipelines in a "loading" state to restore functionality
Certain errors during pipeline converge will emit an IllegalStateException - for example attempting
to reference an environment variable that does not exist. Attempting to add a java exception to the
converge result would result in an uncaught exception in a pipeline thread leading to an unclean
shutdown.
This had the effect of,prior to this commit, Logstash behaviour varying depending on the class of
error that caused a pipeline not to start - an invalid pipeline configuration would still enable
logstash to start other pipelines in a multiple pipeline configuration, or automatic reloading
enabling changes to the configuration to allow the pipeline to start(in multi- and single pipeline
configurations).
However, a missing environment variable would cause Logstash to crash hard no matter what.
This was discovered when updating to jruby-9.3.3.0, when new clean up code joining existing
threads to perform a graceful shutdown was stuck indefinitely, due to the webserver shutdown
code not being called, due to the unclean shutdown.
Fix the class reference `Instrument::MetricStore::MetricNotFound` to be `LogStash::Instrument::MetricStore::MetricNotFound` and do not raise a constant not found reference when the point "/_node/stats" is queried and no pipelines are defined.
This commit guards against reading a value from a stat that may not be
populated before an attempt is made to retrieve its value, potentially causing
a `undefined method `[]' for nil:NilClass`.
This commit changes `queue.checkpoint.retry` to `true` by default allowing retry of checkpoint write failure.
Add exponential backoff retry to checkpoint write to mitigate AccessDeniedExcpetion in Windows.
Fixed: #12345
When a pipeline isn't fully initialized, we run the risk of attempting to
format pipeline info that isn't yet fully-shaped. By using safe-fallback
methods like `Hash#dig` and conditional-chaining, we can avoid the spurious
`NoMethodError` caused by sending `[]` to nil.
* api: avoid 5xx when stats/events not yet populated
* Add catching of metrics exception also for retrieve of :queue metrics. No pipelines means no queues created
* Added empty result when no pipelines is present and query for node info API
Co-authored-by: andsel <selva.andre@gmail.com>
* Refactor: logstash boot to know what to do
* Refactor: reduce LOAD_PATH - load jars directly
* Refactor: unused requires in runner
* Refactor: boot - require libs when to be used
* logging: move init into environment's settings post-processor
Ensures that the non-runner command line utilities like `bin/logstash-keystore`
correctly initialize the logger as-configured.
* fixup: ensure we get ruby stdlib URI & File
Sometime the deep_replace could be invoked by plugins, using the LogStash::Config::Mixin#validate.
This method receives a Ruby hash which could contains Java ArrayList instead of Ruby Array.
The iteration method `each_index` is not available for ArrayList, so resort to some form of "plain old way".
The reason why an ArrayList is recognized as a Ruby Array is due to the override classes, like RubyJavaIntegration.JavaCollectionOverride that monkey patches Ruby Array, so that a Java Collection could be seen as a RubyArray but it doesn't implement all the abstractions, like `each_index`.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* ecs: report pipeline's ECS-compatibility with INFO at startup
Because the pipeline-level setting `pipeline.ecs_compatibility` affects the
default behaviour of nearly every plugin in the pipeline, an INFO-level log
message will provide useful hints, especially to our users who upgrade to
Logstash 8 without first reading the breaking changes docs.
For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages:
> ~~~
> [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
> [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
> ~~~
* ecs: make v8 the default for 8.0
* ecs: `pipeline.ecs_compatibility` defaults to `v8`
Related: elastic/logstash#11623
* doc: temporarily remove deep link from breaking changes doc to fix build
* Forwardport #13358 to master: Add warnings for JAVA_HOME/older versions of Java
Forwardport PR #13358 to master branch. Original message:
* Add deprecation warnings for JAVA_HOME/older versions of Java
Logstash 8.0 will remove support for java versions before java 11, this commit
adds entries to the deprecation log warning against this.
Also adds use of `JAVA_HOME` to the deprecation log.
* Change deprecation log entries to warnings to indicate imminent issue
* settings: add "deprecated alias" support
A deprecated alias provides a path for renaming a setting.
- When a deprecated alias is set on its own, a deprecation notice is emitted
but fetching the canonical setting value will reflect the value set with the
deprecated alias.
- When both the canonical setting (new name) and the deprecated alias (old
name) are specified, it is an error condition.
- When the value of the deprecated alias is queried, a warning is emitted to
the logger and only the value explicitly set to the deprecated alias is
returned.
Additionally, some relevant cleanup is also included:
- Starting Logstash with invalid settings no longer results in the obtuse "An
unexpected error occurred" with backtrace and exception data obscuring the
issue. Instead, a simple message is emitted indicating that the settings are
invalid along with the originating exception's message.
- The various settings implementations share a common logger, instead of each
implementation class providing its own. This is aimed to reduce noise from
the logs and to ensure specs validating logging do not need to tie so
closely to implementation details.
* settings: add password-wrapped setting
* settings: make any setting type capable of being nullable
* settings: add `Settings#names` to power programatic iteration
* cli: route CLI-flag deprecations in to deprecation logger
* settings: group API-related settings under `api.*`
retains deprecated aliases, and is fully backward-compatible.
* webserver: cleanup orphaned attr accessors for never-set ivars
* api: pull settings extraction down from agent
This net-no-change refactor introduces a new method `WebServer#from_settings`
that bridges the gap between Logstash settings and Puma-related options, so
that future additions to the API settings don't add complexity to the Agent.
It also has the benefit of initializing the API Rack App and just ONCE, instead
of once per attempted HTTP port.
* api: add optional TLS/SSL
* docs: reference API security settings
* api: when configured securely, bind to all available interfaces by default
* cleanup: remove unused cert artifacts
* tests: generate fresh webserver certificates
* certs: actually add the binary keystores 🤦
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
## Release notes
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
As we close in on the availability of 8.0.0 alphas, we are reassessing which
breaking changes are _necessary_, and which are merely _desired_. And while
we would love to be in a world where ECS was on by default, and have put
substantial effort into designing an upgrade path that would be as simple as
possible, we have determined that the time may not be right to change the
default value out of under our users.
This change restores the default value for `pipeline.ecs_compatibility` to
`disabled`, ensuring pipelines will continue running in Logstash 8 as they have
in Logstash 7 without modification. We will still encourage our users to be
explicit about which behaviour they desire, and will revisit making ECS on by
default at a later date.
* noop: avoid declaring default value in config file
* docs: ecs compatibility from 7.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* ecs: on by default
We know that ECS version 8 will release along-side Logstash 8.0, but its scope
is still coming into focus. In this changeset, we change the default value
of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a
significantly closer approximation to what will eventually ship in Logstash
8.0.0.
* docs: ecs from 8.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* spec: noop refactor of xpack central management
* spec: validate central management settings loading
* central management: allow pipeline.ordered and pipeline.ecs_compatibility settings
Introcuce the concept of alias for a plugin.
Creates an AliasRegistry to map plugin aliases to original plugins.
If a real plugin with same name of the an alias is present in the system, then the real plugin take precedence during the
instantiation of the pipeline.
Simplified the error handling in class lookup
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* Fix: missing password dependency require
which causes `bin/logstash-keystore` to fail with an error:
```
ERROR: Failed to load settings file from "path.settings". Aborting...
path.setting=/logstash-7.12.0/config, exception=NameError,
message=>uninitialized constant LogStash::Util::Password
```
* Fix: review all LS parts depending on Password
* Test: bin/logstash-keystore create/list
This commit avoid an error in gathering monitoring information when webserver is disabled or is not yet started;
which could happen with slow loading pipelines or no pipelines defined from the central management UI.
This change fix the behavior of considering as "running" also pipelines that are still in "loading", both "loading" and "running" is considered as "not terminated".
Fixed a flakyness in tests due to different ways to looks at the same thing: pipeline status.
The pipeline status is determined by both `pipeline.running?` and by `agent.pipelines_running`.
The first checks for an atomic boolean in pipeline object, the second check for the status in PipelineRegistry.
Fixes#12190
