This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env.
The substitution happens after syntax parsing and before graph execution.
Fixed: #5115
* logging: move init into environment's settings post-processor
Ensures that the non-runner command line utilities like `bin/logstash-keystore`
correctly initialize the logger as-configured.
* fixup: ensure we get ruby stdlib URI & File
* Remove separate JavaVersionChecker executable
The JavaVersionChecker is unnecessary as we already run the JvmOptionsParser before logstash,
providing a convenient location to test the Jvm version before running logstash
* Remove JavaVersionChecker class
* Check exit code from java options parser
* field-reference: cap RUBY_CACHE to 10k entries
Reduces the scope of a memory leak that can be caused by using UUIDs or other
high-cardinality field names by preventing the ruby string _keys_ from being
held by the cache indefinitely.
Note: this may not solve the problem entirely, but certainly limits its impact.
Because ConvertedMap requires individual field names to be interned into
the global String intern pool, their eligibility for GC is JVM-specific
and high-cardinality field names should still be avoided.
* noop: field-reference test refactor to consolodate reflection
* Replace generated cleaner code with concrete implementation
No need to to perform codegen to create JDK specific compatible classes to
call the ByteBuffer cleaner, as we are dropping support for Java 8.
* Remove dead/redundant java 8 code from LogstashJavaCompat
Removes setupByteBuffer, specific test for Java9 and above and updates memory monitor
unit test to only test for values relevant to Java>1.8
* Renamed version utility class to reflect use after removal of methods
Also fixed some dead imports
* fix: respect LS_JAVA_OPTS environment even when optionsfile missing
* Fixed integration tests
* Added unit test to cover the fix
* Wipe commented code
* Removed redundant log in a path that could never be reached
* Moved jvm.options checks into only one place
* javaopts: provide injection point for environment string
Co-authored-by: andsel <selva.andre@gmail.com>
we've seen in the wild a situation where a keystore had a single invalid secret that couldn't be removed since that action did a retrieve before purging. The retrieve would fail causing the purge to never happen.
This commit skips the retrieval but checks if the secret exists to preserve the scenario of reporting an error when removing a secret that doesn't exist.
This commit fixes 2 tests
- Set queue.drain to true in pipeline pq test
- Under certain conditions the pipeline_pq_file_spec test would fail as the pipeline would exit once the generator had generated all of its events, but before the events were processed, leading to the test hanging. This commit adds `queue.drain:true` to the settings to ensure that all of the events are processed before the pipeline is shut down
- Increase the flush delay in dead letter quest testFlushAfterDelay test
- Under certain conditions, the flush delay of 1 second was insufficient, and invalidated a pre-condition assertion that no events had been flushed before the expiry of that delay.
Fixes the issue #8752 in event.out counter. When a pipeline contains a drop filter the total out events counter should count only the events that reached the out stage.
This PR changes CompiledExecution.compute() interface to return the number of events that effectively reached the end of the pipeline. This change is used in WorkerLoop to update correctly the event.out metric, instead of relying on the batch's size.
Sometime the deep_replace could be invoked by plugins, using the LogStash::Config::Mixin#validate.
This method receives a Ruby hash which could contains Java ArrayList instead of Ruby Array.
The iteration method `each_index` is not available for ArrayList, so resort to some form of "plain old way".
The reason why an ArrayList is recognized as a Ruby Array is due to the override classes, like RubyJavaIntegration.JavaCollectionOverride that monkey patches Ruby Array, so that a Java Collection could be seen as a RubyArray but it doesn't implement all the abstractions, like `each_index`.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Provides a "friendly" error message when running Logstash with an incompatible
version of Java
Readded from #13356, incorporated with use of Comparable from #13358
* ecs: report pipeline's ECS-compatibility with INFO at startup
Because the pipeline-level setting `pipeline.ecs_compatibility` affects the
default behaviour of nearly every plugin in the pipeline, an INFO-level log
message will provide useful hints, especially to our users who upgrade to
Logstash 8 without first reading the breaking changes docs.
For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages:
> ~~~
> [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
> [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
> ~~~
* ecs: make v8 the default for 8.0
* ecs: `pipeline.ecs_compatibility` defaults to `v8`
Related: elastic/logstash#11623
* doc: temporarily remove deep link from breaking changes doc to fix build
* Forwardport #13358 to master: Add warnings for JAVA_HOME/older versions of Java
Forwardport PR #13358 to master branch. Original message:
* Add deprecation warnings for JAVA_HOME/older versions of Java
Logstash 8.0 will remove support for java versions before java 11, this commit
adds entries to the deprecation log warning against this.
Also adds use of `JAVA_HOME` to the deprecation log.
* Change deprecation log entries to warnings to indicate imminent issue
* Add JavaVersionChecker to check Java version for compatibility before running
Provides a "friendly" error message when running Logstash with an incompatible
version of Java
* Add version check to Windows
* Improvements
Improve readability of `JavaVersion`
Fix logstash bash script to exit on JavaVersionChecker error
* settings: add "deprecated alias" support
A deprecated alias provides a path for renaming a setting.
- When a deprecated alias is set on its own, a deprecation notice is emitted
but fetching the canonical setting value will reflect the value set with the
deprecated alias.
- When both the canonical setting (new name) and the deprecated alias (old
name) are specified, it is an error condition.
- When the value of the deprecated alias is queried, a warning is emitted to
the logger and only the value explicitly set to the deprecated alias is
returned.
Additionally, some relevant cleanup is also included:
- Starting Logstash with invalid settings no longer results in the obtuse "An
unexpected error occurred" with backtrace and exception data obscuring the
issue. Instead, a simple message is emitted indicating that the settings are
invalid along with the originating exception's message.
- The various settings implementations share a common logger, instead of each
implementation class providing its own. This is aimed to reduce noise from
the logs and to ensure specs validating logging do not need to tie so
closely to implementation details.
* settings: add password-wrapped setting
* settings: make any setting type capable of being nullable
* settings: add `Settings#names` to power programatic iteration
* cli: route CLI-flag deprecations in to deprecation logger
* settings: group API-related settings under `api.*`
retains deprecated aliases, and is fully backward-compatible.
* webserver: cleanup orphaned attr accessors for never-set ivars
* api: pull settings extraction down from agent
This net-no-change refactor introduces a new method `WebServer#from_settings`
that bridges the gap between Logstash settings and Puma-related options, so
that future additions to the API settings don't add complexity to the Agent.
It also has the benefit of initializing the API Rack App and just ONCE, instead
of once per attempted HTTP port.
* api: add optional TLS/SSL
* docs: reference API security settings
* api: when configured securely, bind to all available interfaces by default
* cleanup: remove unused cert artifacts
* tests: generate fresh webserver certificates
* certs: actually add the binary keystores 🤦
This commit applies all the changes needed to run Logstash on JDK 17:
- opens access to module java.base for packages sun.nio.ch and java.io to run the application and to execute the tests
- removes SecurityManager classes used during Logstash startup
- fix exception type catched in JavaKeyStore tampering test
Related to meta issue #13306
* add nanoseconds support
Migrates internals of `org.logstash.Timestamp` from legacy `org.joda.time.*`
which is limited to millisecond-precision to modern `java.time.Instant`,
allowing us to retain nanosecond granularity of `@timestamp` values.
Timestamps that are generated by Logstash (such as when creating an event that
does _not_ have a `@timestamp` field) will be generated at the highest precision
available to the JVM and/or platform (in many cases, this is microseconds).
Timestamps that are _parsed_ from user input will capture the entire provided
precision, up to and including nanosecond granularity.
Throughout the flow in the pipeline, including serialization to PQ, DLQ, and
JSON, will retain all available precision.
BREAKING: This produces an effectively-breaking change to the serialization
format of both the persistent queue (PQ) and dead-letter queue (DLQ),
as the serialized format this changeset contains a higher granularity
of timestamp than previous releases of Logstash were capable of
parsing without error.
As such, it _MUST NOT_ be back-ported to the 7.x series.
This commit:
- Updates the Gradle wrapper to version 7.2
- Remove the deprecated jcenter and where it was used to retrieve Gradle's plugins it switches to gradlePluginPortal
- Insert an explicit dependency from test task to the log4j.properties manipulation task ("copyProductionLog4jConfiguration") used in integration
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
## Release notes
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
As we close in on the availability of 8.0.0 alphas, we are reassessing which
breaking changes are _necessary_, and which are merely _desired_. And while
we would love to be in a world where ECS was on by default, and have put
substantial effort into designing an upgrade path that would be as simple as
possible, we have determined that the time may not be right to change the
default value out of under our users.
This change restores the default value for `pipeline.ecs_compatibility` to
`disabled`, ensuring pipelines will continue running in Logstash 8 as they have
in Logstash 7 without modification. We will still encourage our users to be
explicit about which behaviour they desire, and will revisit making ECS on by
default at a later date.
* noop: avoid declaring default value in config file
* docs: ecs compatibility from 7.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* ecs: on by default
We know that ECS version 8 will release along-side Logstash 8.0, but its scope
is still coming into focus. In this changeset, we change the default value
of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a
significantly closer approximation to what will eventually ship in Logstash
8.0.0.
* docs: ecs from 8.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* fpm to 1.13.0 which allows building packages with java 11 + jruby 9.2
* childprocess to 4.x + remove monkey patches
* clamp to 1.x to unlock fpm 1.13.0
Loads the production plugin_aliases.yml definition file and check that every alias has
a properly published gem on RubyGems.
Adds clean up of plugin_aliases.yml files
Fixed task dependency for copyPluginAlias
* spec: noop refactor of xpack central management
* spec: validate central management settings loading
* central management: allow pipeline.ordered and pipeline.ecs_compatibility settings
