This commit guards against reading a value from a stat that may not be
populated before an attempt is made to retrieve its value, potentially causing
a `undefined method `[]' for nil:NilClass`.
* [Java17] Add `--add-export` settings to restore JDK17 compatibility
After #13700 updated google-java-format dependency, it is now required to add a number
of `--add-export` flags in order to run on JDK17. This commit adds these flags to the
jvm options for a running logstash, and to the tests running on gradle to enable tests
to still work
* Move mandatory JVM options out of `jvm.options` and into JvmOptionsParser
Certain values for the JVM are mandatory for Logstash to function correctly. Rather than
leave them in config/jvm.options where they can be updated, and can cause upgrade issues
for users when we add new mandatory options, move them into code where they cannot be
changed
This commit changes `queue.checkpoint.retry` to `true` by default allowing retry of checkpoint write failure.
Add exponential backoff retry to checkpoint write to mitigate AccessDeniedExcpetion in Windows.
Fixed: #12345
When a pipeline isn't fully initialized, we run the risk of attempting to
format pipeline info that isn't yet fully-shaped. By using safe-fallback
methods like `Hash#dig` and conditional-chaining, we can avoid the spurious
`NoMethodError` caused by sending `[]` to nil.
* api: avoid 5xx when stats/events not yet populated
* Add catching of metrics exception also for retrieve of :queue metrics. No pipelines means no queues created
* Added empty result when no pipelines is present and query for node info API
Co-authored-by: andsel <selva.andre@gmail.com>
A number of plugins reach into Logstash's i18n translations to "helpfully"
communicate certain configuration errors, but rely on translations that were
moved in 00a99c19e5 from logstash.agent to
logstash.runner. Since then, it is possible to hit obtuse error messages about
failing to load a translation instead of the intended helpful message:
~~~
translation missing: en.logstash.agent.configuration.invalid_plugin_register
~~~
By moving the `logstash.agent` definition to _after_ the `logstash.runner`
definition, we can use YAML tooling to name the `logstash.runner.configuration`
node and then merge its contents into `logstash.agent.configuration`. This
effectively allows us to keep a single definition of those translations while
making them available at both addresses.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Refactor: logstash boot to know what to do
* Refactor: reduce LOAD_PATH - load jars directly
* Refactor: unused requires in runner
* Refactor: boot - require libs when to be used
This commit deleted the corrupted zero byte PQ file and recreated the head checkpoint file to get rid of page file size is too small exception
Fixed: #10855
The acceptance tests are used to test a variety of scenarios installing, running and updating
logstash on a variety of Linux distributions.
The vagrant distributions we test on are old, and use java8 by default. This commit removes
the java8 tests and changes the tests to only test the jdk included distribution.
Use same technique for Unix system to extract the CPU percentage and publish to Logstash's metrics collector.
It doesn't retrieve the full set of metric of a Unix system, but the ones that are available from internal JDK class com.sun.management.OperatingSystemMXBean
This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env.
The substitution happens after syntax parsing and before graph execution.
Fixed: #5115
* logging: move init into environment's settings post-processor
Ensures that the non-runner command line utilities like `bin/logstash-keystore`
correctly initialize the logger as-configured.
* fixup: ensure we get ruby stdlib URI & File
* Remove separate JavaVersionChecker executable
The JavaVersionChecker is unnecessary as we already run the JvmOptionsParser before logstash,
providing a convenient location to test the Jvm version before running logstash
* Remove JavaVersionChecker class
* Check exit code from java options parser
* field-reference: cap RUBY_CACHE to 10k entries
Reduces the scope of a memory leak that can be caused by using UUIDs or other
high-cardinality field names by preventing the ruby string _keys_ from being
held by the cache indefinitely.
Note: this may not solve the problem entirely, but certainly limits its impact.
Because ConvertedMap requires individual field names to be interned into
the global String intern pool, their eligibility for GC is JVM-specific
and high-cardinality field names should still be avoided.
* noop: field-reference test refactor to consolodate reflection
* Replace generated cleaner code with concrete implementation
No need to to perform codegen to create JDK specific compatible classes to
call the ByteBuffer cleaner, as we are dropping support for Java 8.
* Remove dead/redundant java 8 code from LogstashJavaCompat
Removes setupByteBuffer, specific test for Java9 and above and updates memory monitor
unit test to only test for values relevant to Java>1.8
* Renamed version utility class to reflect use after removal of methods
Also fixed some dead imports
* fix: respect LS_JAVA_OPTS environment even when optionsfile missing
* Fixed integration tests
* Added unit test to cover the fix
* Wipe commented code
* Removed redundant log in a path that could never be reached
* Moved jvm.options checks into only one place
* javaopts: provide injection point for environment string
Co-authored-by: andsel <selva.andre@gmail.com>
we've seen in the wild a situation where a keystore had a single invalid secret that couldn't be removed since that action did a retrieve before purging. The retrieve would fail causing the purge to never happen.
This commit skips the retrieval but checks if the secret exists to preserve the scenario of reporting an error when removing a secret that doesn't exist.
This commit fixes 2 tests
- Set queue.drain to true in pipeline pq test
- Under certain conditions the pipeline_pq_file_spec test would fail as the pipeline would exit once the generator had generated all of its events, but before the events were processed, leading to the test hanging. This commit adds `queue.drain:true` to the settings to ensure that all of the events are processed before the pipeline is shut down
- Increase the flush delay in dead letter quest testFlushAfterDelay test
- Under certain conditions, the flush delay of 1 second was insufficient, and invalidated a pre-condition assertion that no events had been flushed before the expiry of that delay.
Fixes the issue #8752 in event.out counter. When a pipeline contains a drop filter the total out events counter should count only the events that reached the out stage.
This PR changes CompiledExecution.compute() interface to return the number of events that effectively reached the end of the pipeline. This change is used in WorkerLoop to update correctly the event.out metric, instead of relying on the batch's size.
Sometime the deep_replace could be invoked by plugins, using the LogStash::Config::Mixin#validate.
This method receives a Ruby hash which could contains Java ArrayList instead of Ruby Array.
The iteration method `each_index` is not available for ArrayList, so resort to some form of "plain old way".
The reason why an ArrayList is recognized as a Ruby Array is due to the override classes, like RubyJavaIntegration.JavaCollectionOverride that monkey patches Ruby Array, so that a Java Collection could be seen as a RubyArray but it doesn't implement all the abstractions, like `each_index`.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Provides a "friendly" error message when running Logstash with an incompatible
version of Java
Readded from #13356, incorporated with use of Comparable from #13358
* ecs: report pipeline's ECS-compatibility with INFO at startup
Because the pipeline-level setting `pipeline.ecs_compatibility` affects the
default behaviour of nearly every plugin in the pipeline, an INFO-level log
message will provide useful hints, especially to our users who upgrade to
Logstash 8 without first reading the breaking changes docs.
For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages:
> ~~~
> [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
> [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
> ~~~
* ecs: make v8 the default for 8.0
* ecs: `pipeline.ecs_compatibility` defaults to `v8`
Related: elastic/logstash#11623
* doc: temporarily remove deep link from breaking changes doc to fix build
* Forwardport #13358 to master: Add warnings for JAVA_HOME/older versions of Java
Forwardport PR #13358 to master branch. Original message:
* Add deprecation warnings for JAVA_HOME/older versions of Java
Logstash 8.0 will remove support for java versions before java 11, this commit
adds entries to the deprecation log warning against this.
Also adds use of `JAVA_HOME` to the deprecation log.
* Change deprecation log entries to warnings to indicate imminent issue
* Add JavaVersionChecker to check Java version for compatibility before running
Provides a "friendly" error message when running Logstash with an incompatible
version of Java
* Add version check to Windows
* Improvements
Improve readability of `JavaVersion`
Fix logstash bash script to exit on JavaVersionChecker error
* settings: add "deprecated alias" support
A deprecated alias provides a path for renaming a setting.
- When a deprecated alias is set on its own, a deprecation notice is emitted
but fetching the canonical setting value will reflect the value set with the
deprecated alias.
- When both the canonical setting (new name) and the deprecated alias (old
name) are specified, it is an error condition.
- When the value of the deprecated alias is queried, a warning is emitted to
the logger and only the value explicitly set to the deprecated alias is
returned.
Additionally, some relevant cleanup is also included:
- Starting Logstash with invalid settings no longer results in the obtuse "An
unexpected error occurred" with backtrace and exception data obscuring the
issue. Instead, a simple message is emitted indicating that the settings are
invalid along with the originating exception's message.
- The various settings implementations share a common logger, instead of each
implementation class providing its own. This is aimed to reduce noise from
the logs and to ensure specs validating logging do not need to tie so
closely to implementation details.
* settings: add password-wrapped setting
* settings: make any setting type capable of being nullable
* settings: add `Settings#names` to power programatic iteration
* cli: route CLI-flag deprecations in to deprecation logger
* settings: group API-related settings under `api.*`
retains deprecated aliases, and is fully backward-compatible.
* webserver: cleanup orphaned attr accessors for never-set ivars
* api: pull settings extraction down from agent
This net-no-change refactor introduces a new method `WebServer#from_settings`
that bridges the gap between Logstash settings and Puma-related options, so
that future additions to the API settings don't add complexity to the Agent.
It also has the benefit of initializing the API Rack App and just ONCE, instead
of once per attempted HTTP port.
* api: add optional TLS/SSL
* docs: reference API security settings
* api: when configured securely, bind to all available interfaces by default
* cleanup: remove unused cert artifacts
* tests: generate fresh webserver certificates
* certs: actually add the binary keystores 🤦
This commit applies all the changes needed to run Logstash on JDK 17:
- opens access to module java.base for packages sun.nio.ch and java.io to run the application and to execute the tests
- removes SecurityManager classes used during Logstash startup
- fix exception type catched in JavaKeyStore tampering test
Related to meta issue #13306
