* Update logstash_releases.json after 8.17.5
* Update 9.next post 9.0.0 release
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* change base images to ubi9-minimal
* do all env2yaml related copying in 1 COPY
* use -trimpath in go build
* move other COPY to end of dockerfile
* don't run package manager upgrade
* FROM and AS with same case
* ENV x=y instead of ENV x y
* remove indirect config folder
* Disabling some of metricsets may cause dashboards partially display some metrics. This change recommends collecting metricsets to fully visualize metrics on dashboards.
* Paraphrasing sentences and grammar correction.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Update docs/reference/serverless-monitoring-with-elastic-agent.md
* Make recommendation as a tip.
---------
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444
reporting the following error
```
File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
def integration_tests(self, part: int, parts: int) -> JobRetValues:
^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)
This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Fix standalone agent access for agent-driven monitoring
Change incorrect dedicated instructions, and base them on those from running Elastic Agent
in standalone mode.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
The rubocop-ast gem just introduced a new dependency on prism.
- https://rubygems.org/gems/rubocop-ast/versions/1.43.0
In our install default gem rake task we are seeing issues trying to build native
extensions. I see that in upstream jruby they are seeing a similar problem (at
least it is the same failure mode https://github.com/jruby/jruby/pull/8415
This commit pins rubocop-ast to 1.42.0 which is the last version that did not
have an explicit prism dependency.
Updates only test code to be able to run a test that consumes big memory if:
- the physical memory is bigger than the requested Java heap
- JDK version is greater than or equal to 21.
The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error.
* tests: make integration split quantity configurable
Refactors shared splitter bash function to take a list of files on stdin
and split into a configurable number of partitions, emitting only those from
the currently-selected partition to stdout.
Also refactors the only caller in the integration_tests launcher script to
accept an optional partition_count parameter (defaulting to `2` for backward-
compatibility), to provide the list of specs to the function's stdin, and to
output relevant information about the quantity of partition splits and which
was selected.
* ci: run integration tests in 3 parts
Use long instead of int type to keep the length of the first token.
The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test 9c0e50faac/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java (L123).
To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.
This commit migrated API doc to OpenAPI spec
- expand Node Info `/_node/<types>`
- expand Node Stats
- add health report
- break down the share part to components
- add examples
- update authentication
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
The https://github.com/elastic/logstash/pull/17310 PR changed the rake task for
artifact creation to use shellwords from standard library. The acceptance tests
need to explitily load that library. This commit updates the rake file to handle
loading the required code.
Previously when rake would shell out the output would be lost. This
made debugging CI logs difficult. This commit updates the stack with
improved message surfacing on error.
Reimplements `LogStash::Setting::Numeric` Ruby setting class into the `org.logstash.settings.NumericSetting` and exposes it through `java_import` as `LogStash::Setting::NumericSetting`.
Updates the rspec tests:
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.
This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue
Fixes: #17209
* tests: ls2ls delay checking until events have been processed
* Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec.
* Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable.
---------
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit migrates gradle task to the new artifacts-api
- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
with priority given in that order.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit refactors the `PersistedQueueConfigValidator` class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
https://github.com/elastic/logstash/issues/14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.
* plugin manager: add --no-expand flag for list command
Allows us to avoid expanding aliases and integration plugins
* spec: escape expected output in regexp
In https://github.com/elastic/logstash/pull/17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
Removal works in a single pass by finding plugins that would have unmet
dependencies if all of the specified plugins were to be removed, and
proceeding with the removal only if no conflicts were created.
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-remove-multiple ✘) }
> ╰─● bin/logstash-plugin remove logstash-input-syslog logstash-filter-grok
> Using system java: /Users/rye/.jenv/shims/java
> Resolving dependencies......
> Successfully removed logstash-input-syslog
> Successfully removed logstash-filter-grok
> [success (00:00:05)]
~~~
* spec: improve ls2ls spec
- fixes upstream/downstream convention
- upstream: the sending logstash (has an LS output)
- downstream: the receiving logstash (has an LS input)
- helper `run_logstash_instance` yields the `LogstashService` instance
and handles the teardown.
- pass the pipeline id and node name to the LS instances via command line
flags to make logging easier to differentiate
- use the generator input's sequence id to ensure that the _actual_ events
generated are received by the downstream pipeline
* start with port-offset 100
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Currently the deserialization is behind the readBatch's lock, so any large batch will take time deserializing, causing any other Queue writer (e.g. netty executor threads) and any other Queue reader (pipeline worker) to block.
This commit moves the deserialization out of the lock, allowing multiple pipeline workers to deserialize batches concurrently.
- add intermediate batch-holder from `Queue` methods
- make the intermediate batch-holder a private inner class of `Queue` with a descriptive name `SerializedBatchHolder`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Starting from es-output 12.0.2, a 404 response is treated as an error. Previously, central pipeline management considered 404 as an empty pipeline, not an error.
This commit restores the expected behavior by handling 404 gracefully and logs a user-friendly message.
It also removes the redundant cache of pipeline in CPM
Fixes: #17035
* qa: use clean expansion of LS tarball per fixture instance
Because QA tests can _modify_ the Logstash installation (e.g. those that
invoke the plugin manager), it is important that the service wrapper
begins with a clean expansion of the logstash tarball.
* qa: enable safe reuse of ls_home in ls_to_ls tests
This commit simplifies the DRA process in Logstash by removing the need to maintain a separate file for the active branches, and instead rely on a centrally maintained file containing source of truth.
While at it, we refactor/simplify the creation of an array with the versions in `.buildkite/scripts/snyk/resolve_stack_version.sh`.
This commit preserves spaces in values, ensuring that multi-word strings are stored as intended.
Prior to this change, `logstash-keystore` incorrectly handled values containing spaces,
causing only the first word to be stored.
PR #17006 revealed that the `VERSION_QUALIFIER` env var gets honored in
various scripts when present but empty.
This shouldn't be the case as the DRA process is designed to gracefully
ignore empty values for this variable.
This commit changes various ruby scripts to not treat "" as truthy.
Bash scripts (used by CI etc.) are already ok with this as part of
refactorings done in #16907.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Permit to use effectively the tokenizer also in context where a line is bigger than a limit.
Fixes an issues related to token size limit error, when the offending token was bigger than the input fragment in happened that the tokenzer wasn't unable to recover the token stream from the first delimiter after the offending token but messed things, loosing part of tokens.
## How solve the problem
This is a second take to fix the processing of tokens from the tokenizer after a buffer full error. The first try #16482 was rollbacked to the encoding error #16694.
The first try failed on returning the tokens in the same encoding of the input.
This PR does a couple of things:
- accumulates the tokens, so that after a full condition can resume with the next tokens after the offending one.
- respect the encoding of the input string. Use `concat` method instead of `addAll`, which avoid to convert RubyString to String and back to RubyString. When return the head `StringBuilder` it enforce the encoding with the input charset.
Log input has been deprecated in filebeat 9.0.0 and throws an error if it's present in the configuration.
This commit switches the configuration to the "filestream" input.
This commit removed irrelevant warning for internal pipeline, such as monitoring pipeline.
Monitoring pipeline is expected to be one worker. The warning is not useful
Fixes: #13298
Reimplements `LogStash::Setting::String` Ruby setting class into the `org.logstash.settings.SettingString` and exposes it through `java_import` as `LogStash::Setting::SettingString`.
Updates the rspec tests in two ways:
- logging mock is now converted to real Log4J appender that spy log line that are later verified
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2
Related to: https://github.com/jruby/jruby/issues/8579
This commit uses the new --qualifier parameter in the release manager
for publishing dra artifacts. Additionally, simplifies the expected
variables to rely on a simple `VERSION_QUALIFIER`.
Snapshot builds are skipped when VERSION_QUALIFIER is set.
Finally, for helping to test DRA PRs, we also allow passing the `DRA_BRANCH` option/env var
to override BUILDKITE_BRANCH.
Closes https://github.com/elastic/ingest-dev/issues/4856
* Add a new pattern %{{TIME_NOW}} to `event.sprintf` to generate a fresh timestamp.
The timestamp is represented as a string in the default ISO 8601 format
For example,
```
input {
heartbeat {
add_field => { "heartbeat_time" => "%{{TIME_NOW}}" }
}
}
```
VERSION_QUALIFIER was already observed in rake artifacts task but only to influence the name of the artifact.
This commit ensure that the qualifier is also displayed in the cli and in the http api.
The preferred channel for communication about LS is the elastic discussion
forum, this commit updates the source code and readme files to reflect that.
* Doc: Environment variables cannot be in config.string comments due to substitution logic.
* Apply suggestions from code review
Suggestion to revise the statements.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Make sentences present tense.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Revert "Apply Jackson stream read constraints defaults at runtime (#16832)"
This reverts commit cc608eb88b.
* jackson stream read constraints: code-based defaults
refactors stream read constraints to couple default values with their
associated overrides, which allows us to have more descriptive logging
that includes provenance of the value that has been applied.
Previously when the `:deprecated` modifier was used in the plugin config DSL a
log message was sent at `:warn` level to the main logger. This commit updates
that message to be routed *only* to the deprecation logger.
JvmOptionsParser adds support for ${VAR:default} syntax when parsing jvm.options
- allow dynamic resolution of environment variables in the jvm.options file
- enables fallback to default value when the environment variable is not set
When Logstash 8.12.0 added increased Jackson stream read constraints to
jvm.options, assumptions about the existence of that file's contents
were invalidated. This led to issues like #16683.
This change ensures Logstash applies defaults from config at runtime:
- MAX_STRING_LENGTH: 200_000_000
- MAX_NUMBER_LENGTH: 10_000
- MAX_NESTING_DEPTH: 1_000
These match the jvm.options defaults and are applied even when config
is missing. Config values still override these defaults when present.
Because a `break` escapes a `begin`...`end` block, we must not use a `break` in order to ensure that the explicitly set value gets memoized to avoid lock contention.
> ~~~ ruby
> def fake_sync(&block)
> puts "FAKE_SYNC:enter"
> val = yield
> puts "FAKE_SYNC:return(#{val})"
> return val
> ensure
> puts "FAKE_SYNC:ensure"
> end
>
> fake_sync do
> @ivar = begin
> puts("BE:begin")
> break :break
>
> val = :ret
> puts("BE:return(#{val})")
> val
> ensure
> puts("BE:ensure")
> end
> end
> ~~~
Note: no `FAKE_SYNC:return`:
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash (main ✔) }
> ╰─● ruby break-esc.rb
> FAKE_SYNC:enter
> BE:begin
> BE:ensure
> FAKE_SYNC:ensure
> [success]
> ~~~
This commit updates the curl scripts that interact with Kibana's
`api/logstash/pipeline/*` endpoin. Additionally this adds the header to any curl
that interacts with elasticsearch API as well.
Moves the application of jackson defaults overrides into pure java, and
applies them statically _before_ the `org.logstash.ObjectMappers` has a chance
to start initializing object mappers that rely on the defaults.
We replace the runner's invocation (which was too late to be fully applied) with
a _verification_ that the configured defaults have been applied.
* pipeline bus: add deadlock test for unlisten/unregisterSender
* pipeline bus: eliminate deadlock
Moves the sync-to-notify out of the `AddressStateMapping#mutate`'s effective
synchronous block to eliminate a race condition where unlistening to an address
and unregistering a sender could deadlock.
It is safe to notify an AddressState's attached input without exclusive access
to the AddressState, because notifying an input that has since been disconnected
is net zero harm.
(cherry picked from commit 8af6343a26)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Pip install is unnecessary and sometimes hangs with prompt dialog. This commit aligns on agent python version which has a default pip installed.
* Update .buildkite/scripts/health-report-tests/main.sh
Improve readability.
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
---------
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
In https://github.com/elastic/logstash/pull/16586 the module include was
removed. This causes failures in the script when the module is referenced. This
commit re-enables the include for the yaml module.
Introduce a new flag setting `xpack.monitoring.allow_legacy_collection` which eventually enable the legacy monitoring collector.
Update the method to test if monitoring is enabled so that consider also `xpack.monitoring.allow_legacy_collection` to determine if `monitoring.*` settings are valid or not.
By default it's false, the user has to intentionally enable it to continue to use the legacy monitoring settings.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Switch the default value of `pipeline.buffer.type` to use the heap memory instead of direct one.
Change the default value of the setting `pipeline.buffer.type` from direct to heap and update consequently the documentation.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
See naming rational in https://github.com/logstash-plugins/.ci/pull/63#issue-2597373955
After 8.16 is GA, the "releases" entry should become:
```json
"releases": {
"7.current": "7.17.24",
"8.current": "8.16.0",
"8.previous": "8.15.3"
},
```
For snapshots we'll also test against "main", "8.next", and "8.future". The labels are:
- `main`: main branch
- `8.future`: the future 8.x release, i.e. current version of the 8.x branch
- `8.next`: the short lived period between a minor's FF - when the new branch is cut from 8.x - and GA
- `8.current`: the most recent 8.x release
- `8.previous`: the previous, but still supported, 8.x release
This commit extends the gradle and rake tasks to pass through a list of files
for rubocop to lint. This allows more specificity and fine grained control for
linting when the consumer of the tasks only wishes to lint a select few files.
Logstash Docker images, full and OSS, now use UBI image as its base, replacing the previous Ubuntu base.
- change the base image of `full` and `oss` to ubi
- Set locale to C.UTF-8
- remove ubi flavour
- use go image to build env2yaml
- remove redundant and refactor steps
- add support to build image in mac aarch64
- allow customizing ELASTIC_VERSION and LOCAL_ARTIFACTS for test purpose
there's no need to build jdk-less and windows tarballs for docker images
so this change simplifies the build process.
It should reduce the time spent needed to build docker images.
Switch execution agent of JDK availability check pipeline from vm-agent to container-agent.
Moves the schedule definition from the `Logstash Pipeline Scheduler` pipeline into the pipeline definition, adding a schedule also for `8.x` branch.
Updates the existing `createElasticCatalogDownloadUrl` method to use the precise version retrieved `versions.yml` to download the JDK instead of using the latest of major version. This makes the build reproducible again.
Defines a new Gradle `checkNewJdkVersion` task to check if there is a new JDK version available from JVM catalog matching the same major of the current branch.
Creates a new Buildkite pipeline to execute a `bash` script to run the Gradle task; plus it also update the `catalog-info.yaml` with the new pipeline and a trigger to execute every week.
The current max inflight error message focuses on a single pipeline and on a maximum amount of 10k events regardless of the heap size.
The new warning will take into account all loaded pipelines and also consider the heap size, giving a warning if the total number of events consumes 10% or more of the total heap.
For the purpose of the warning events are assumed to be 2KB as it a normal size for a small log entry.
Adds a JMH benchmark to measure the peformances of BufferedTokenizerExt.
Update also Gradle build script to remove CMS GC flags and fix deprecations for Gradle 9.0.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes the behaviour of the tokenizer to be able to work properly when buffer full conditions are met.
Updates BufferedTokenizerExt so that can accumulate token fragments coming from different data segments. When a "buffer full" condition is matched, it record this state in a local field so that on next data segment it can consume all the token fragments till the next token delimiter.
Updated the accumulation variable from RubyArray containing strings to a StringBuilder which contains the head token, plus the remaining token fragments are stored in the input array.
Furthermore it translates the `buftok_spec` tests into JUnit tests.
* Health API: Add 1min 5min backpressure cases and improve Logstash temination logic.
* Apply suggestions from code review
Uncomment accidentally commented sources.
* Update .buildkite/scripts/health-report-tests/tests/slow-start.yaml
No need to wait for LS startup when using slow start scenario.
* Apply suggestions from code review
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Standardize YAML structure and rename wait time to wait_seconds
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Fixes a potential flaky test, due to shared (LogStash:SETTINGS) fixture across the test base.
Forward port the commit 609155a61b used to fix the non clean backport PR #16531 of #16525 to 8.x.
LogStash:SETTINGS is used in the constructor of LogStash::Inputs::Metrics::StatsEventFactory to query the value of api.enabled. This PR keeps updated the value for the setting provided to the Agent constructor and to the StatsEventFactory.
* Utilize default agent for Health API CI. Call python scripts from directly CI step.
* Change BK agent to support both Java and python. Install pip manually and send env vars to subprocess.
* [health] bootstrap HealthObserver from agent to API (#16141)
* [health] bootstrap HealthObserver from agent to API
* specs: mocked agent needs health observer
* add license headers
* Merge `main` into `feature/health-report-api` (#16397)
* Add GH vault plugin bot to allowed list (#16301)
* regenerate webserver test certificates (#16331)
* correctly handle stack overflow errors during pipeline compilation (#16323)
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* Doc: Reposition worker-utilization in doc (#16335)
* settings: add support for observing settings after post-process hooks (#16339)
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
* fix line used to determine ES is up (#16349)
* add retries to snyk buildkite job (#16343)
* Fix 8.13.1 release notes (#16363)
make a note of the fix that went to 8.13.1: #16026
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16347)
* [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Doc: Add SNMP integration to breaking changes (#16374)
* deprecate java less-than 17 (#16370)
* Exclude substitution refinement on pipelines.yml (#16375)
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Doc: Forwardport 8.15.0 release notes to main (#16388)
* Removing 8.14 from ci/branches.json as we have 8.15. (#16390)
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Squashed merge from 8.x
* Failure injector plugin implementation. (#16466)
* Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes.
* Fix the degrate method name typo.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Add explanation to the config params and rebuild plugin gem.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Health report integration tests bootstrapper and initial tests implementation (#16467)
* Health Report integration tests bootstrapper and initial slow start scenario implementation.
* Apply suggestions from code review
Renaming expectation check method name.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Changed to branch concept, YAML structure simplified as changed to Dict.
* Apply suggestions from code review
Reflect `help_url` to the integration test.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* health api: expose `GET /_health_report` with pipelines/*/status probe (#16398)
Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the
resulting report status into the other API responses, replacing their hard-coded `green`
with a meaningful status indication.
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* docs: health report API, and diagnosis links (feature-targeted) (#16518)
* docs: health report API, and diagnosis links
* Remove plus-for-passthrough markers
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* merge 8.x into feature branch... (#16519)
* Add GH vault plugin bot to allowed list (#16301)
* regenerate webserver test certificates (#16331)
* correctly handle stack overflow errors during pipeline compilation (#16323)
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* Doc: Reposition worker-utilization in doc (#16335)
* settings: add support for observing settings after post-process hooks (#16339)
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
* fix line used to determine ES is up (#16349)
* add retries to snyk buildkite job (#16343)
* Fix 8.13.1 release notes (#16363)
make a note of the fix that went to 8.13.1: #16026
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16347)
* [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Doc: Add SNMP integration to breaking changes (#16374)
* deprecate java less-than 17 (#16370)
* Exclude substitution refinement on pipelines.yml (#16375)
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Doc: Forwardport 8.15.0 release notes to main (#16388)
* Removing 8.14 from ci/branches.json as we have 8.15. (#16390)
* Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408)
Fix: #16406
* Generate Dataset code with meaningful fields names (#16386)
This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields.
Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`.
Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`.
Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields.
* Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322)
This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it.
This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch.
Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node.
Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16426)
* Release notes for 8.15.1 (#16405) (#16427)
* Update release notes for 8.15.1
* update release note
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co>
(cherry picked from commit 2fca7e39e8)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430)
Make inner field of ConditionalEvaluationError transient to be avoided during serialization.
(cherry picked from commit bb7ecc203f)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* use gnu tar compatible minitar to generate tar artifact (#16432) (#16434)
Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters.
For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem.
(cherry picked from commit 69f0fa54ca)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* account for the 8.x in DRA publishing task (#16436) (#16440)
the current DRA publishing task computes the branch from the version
contained in the version.yml
This is done by taking the major.minor and confirming that a branch
exists with that name.
However this pattern won't be applicable for 8.x, as that branch
currently points to 8.16.0 and there is no 8.16 branch.
This commit falls back to reading the buildkite injected
BUILDKITE_BRANCH variable.
(cherry picked from commit 17dba9f829)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459)
* Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.
* Add a docker acceptance integration test.
(cherry picked from commit 7c64c7394b)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Known issue for 8.15.1 related to env vars references (#16455) (#16469)
(cherry picked from commit b54caf3fd8)
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
* bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480)
(cherry picked from commit 51cca7320e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Release notes for 8.15.2 (#16471) (#16478)
Co-authored-by: andsel <selva.andre@gmail.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit 01dc76f3b5)
* Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488)
(cherry picked from commit 8368c00367)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491)
(cherry picked from commit fd1de39005)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Don't use an older JRuby with oraclelinux-7 (#16499) (#16501)
A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images
and removed JRuby 9.4.5.0 and some older versions.
This ended up breaking exhaustive test on Oracle Linux 7 that hard coded
JRuby 9.4.5.0.
PR https://github.com/elastic/logstash/pull/16489 worked around the
problem by pinning to the new JRuby, but actually we don't
need the conditional anymore since the original issue
https://github.com/jruby/jruby/issues/7579#issuecomment-1425885324 has
been resolved and none of our releasable branches (apart from 7.17 which
uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`.
Therefore, this commit removes conditional setting of JRuby for
OracleLinux 7 agents in exhaustive tests (and relies on whatever
`/.ruby-version` defines).
(cherry picked from commit 07c01f8231)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* Improve pipeline bootstrap error logs (#16495) (#16504)
This PR adds the cause errors details on the pipeline converge state error logs
(cherry picked from commit e84fb458ce)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511)
(cherry picked from commit 5195332bc6)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Make health report test runner script executable. (#16446) (#16512)
(cherry picked from commit 2ebf2658ff)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493)
* DLQ-ing events that trigger an conditional evaluation error. (#16423)
When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline.
This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it.
(cherry picked from commit b69d993d71)
* Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* add deprecation log for `--event_api.tags.illegal` (#16507) (#16515)
- move `--event_api.tags.illegal` from option to deprecated_option
- add deprecation log when the flag is explicitly used
relates: #16356
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit a4eddb8a2a)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
(cherry picked from commit 7eb5185b4e)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Set the 'api.enabled' setting to reflect the flag webserver_enabled and consequently test for http_address presence in settings iff the web server is enabled.
* Update rspec dependency of the QA package.
* Update qa/Gemfile
Align on rspec 3.13.x
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Fix the QA test failure caused after reflecting Health Report status to the Node stats.
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit 1e5105fcd8)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
- move `--event_api.tags.illegal` from option to deprecated_option
- add deprecation log when the flag is explicitly used
relates: #16356
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Update Settings to_hash method to also skip Java DeprecatedAlias and not just the Ruby one.
With PR #15679 was introduced org.logstash.settings.DeprecatedAlias which mirrors the behaviour of Ruby class Setting::DeprecatedAlias. The equality check at Logstash::Settings, as descibed in #16505 (comment), is implemented comparing the maps.
The conversion of Settings to the corresponding maps filtered out the Ruby implementation of DeprecatedAlias but not the Java one.
This PR adds also the Java one to the list of filter.
A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images
and removed JRuby 9.4.5.0 and some older versions.
This ended up breaking exhaustive test on Oracle Linux 7 that hard coded
JRuby 9.4.5.0.
PR https://github.com/elastic/logstash/pull/16489 worked around the
problem by pinning to the new JRuby, but actually we don't
need the conditional anymore since the original issue
https://github.com/jruby/jruby/issues/7579#issuecomment-1425885324 has
been resolved and none of our releasable branches (apart from 7.17 which
uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`.
Therefore, this commit removes conditional setting of JRuby for
OracleLinux 7 agents in exhaustive tests (and relies on whatever
`/.ruby-version` defines).
Suppress some warnings compared with JDK 21
- this-escape uses this before it is completely initialised.
- avoid a non serialisable DeadLetterQueueWriter field from serialisable instance.
When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline.
This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it.
Reimplement the root Ruby Setting class in Java and use it from the Ruby one moving the original Ruby class to a shell wrapping the Java instance.
In particular create a new symmetric hierarchy (at the time just for `Setting`, `Coercible` and `Boolean` classes) to the Ruby one, moving also the feature for setting deprecation. In this way the new `org.logstash.settings.Boolean` is syntactically and semantically equivalent to the old Ruby Boolean class, which replaces.
- split main.sh to core.sh and main.sh
- rename all.sh to marathon.sh
- fix vault expiry issue for long running task
- fix unable to call main function
- update save object runtime field `release` to return true when `version` contains "SNAPSHOT"
* Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.
* Add a docker acceptance integration test.
If a user runs Logstash with a hosted JDK and not the one bundled with Logstash distribution, like setting a specific LS_JAVA_HOME, which is minor than JDK 17 then Logstash refuses to start. Has to provide at least a JDK 17 or unset the LS_JAVA_HOME and let Logstash uses the bundled JDK.
Updates the jvm.options and JvmOptionsParser to remove support for JDK 11. If the options parser identifies that the running JVM is less than 17, it refuses to start.
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
- add becnhmark dashboard and related save objects
- add one buildkite pipeline to test against multiple versions
- remove null field in json
- add `FLOG_FILE_CNT`, `VAULT_PATH`, `TAGS`
the current DRA publishing task computes the branch from the version
contained in the version.yml
This is done by taking the major.minor and confirming that a branch
exists with that name.
However this pattern won't be applicable for 8.x, as that branch
currently points to 8.16.0 and there is no 8.16 branch.
This commit falls back to reading the buildkite injected
BUILDKITE_BRANCH variable.
Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters.
For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem.
This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it.
This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch.
Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node.
Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields.
Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`.
Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`.
Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields.
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* licenses: allow elv2, standard abbreviation for Elastic License version 2
* json-dump: reduce unicode normalization cost
Since the underlying JrJackson now properly (and efficiently) encodes the
UTF-8 transcode of whichever strings it is given, we no longer need to
pre-normalize to UTF-8 in ruby _except_ when the string is flagged as BINARY
because we have alternate behaviour to preserve valid UTF-8 sequences.
By emitting a _copy_ of binary-flagged strings that have been re-flagged as
UTF-8, we allow the downstream (efficient) encoding operation in jrjackson
to produce equivalent behaviour at much lower cost.
* cleanup: remove orphan unicode normalizer
- fixes the CI benchmark script to always runs against the latest snapshot version
- uses `/v1/versions/$VERSION/builds/latest` to get the latest build id
Fixes: #16307
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit fixed the configuration reload process to clean up the pipeline's metric store, so it does not retain references to failed pipelines components.
* Add RubyEvent#dup support and unit test case to keep Json#dump(Event) safe.
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
---------
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* fix: restore support for unicode pipeline- and plugin-id's
JRuby's `Ruby#newSymbol(String)` throws an exception when provided a `String`
that contains characters outside of lower-ASCII because JRuby internals expect
"the incoming String to be one of our mangled ISO-8859-1 strings" as noted in
a comment on jruby/jruby#6217.
Instead, we use `Ruby#newString(String)` to create a new `RubyString` (which
works properly), and then rely on `RubyString#intern` to get our `RubySymbol`.
This fixes a regression introduced in the 8.7 series in which pipeline id's
are consistently represented as ruby symbols in the metrics store, and ensures
similar issue does not exist when specifying a plugin id that contains
characters above the lower-ASCII plane.
* fix: use properly-encoded RubySymbol in PipelineConfig
We cannot rely on `RubySymbol#toString` to produce a properly-encoded `String`
whe the string contains characters above the lower-ASCII plane because the
result is effectively a binary ruby-internal marshal of the bytes that only
holds when the symbol contains lower-ASCII.
Instead, we can use the internally-memoizing `RubySymbol#name` to get a
properly-encoded `RubyString`, and `RubyString#asJavaString()` to get a
properly-encoded java-`String`.
* fix: properly serialize unicode pipeline names in API output
Jackson's JSON serializer leaks the JRuby-internal byte structure of Symbols,
which only aligns with the byte-structure of the symbol's actual string when
that string is wholly-comprised of lower-ASCII characters.
By pre-converting Symbols to Strings, we ensure that the result is readable
and useful.
* spec: bypass monitoring specs for unicode pipeline ids when PQ enabled
Add a buildkite pipeline to do benchmark.
The script does benchmark by running Filebeats (docker) -> Logstash (docker) -> ES Cloud.
Logstash metrics and benchmark results are sent to the same ES Cloud.
- Secrets store in vault `secret/ci/elastic-logstash/benchmark`
- Use flog (docker) to generate ~2GB logs
- Pull the snapshot docker image of the main branch every day
- Logstash runs two pipelines, main and node_stats
- The main pipeline handles beats ingestion, sending data to the data stream `logs-generic-default`
- It runs for all combinations. (pq + mq) x worker x batch size
- Each test runs for ~7 minutes
- The node_stats pipeline retrieves /_node/stats API every 30s and sends it to the data stream `metrics-nodestats-logstash`
- The script sends a summary of EPS and resource usage to index `benchmark_summary`
The buildkite pipeline accepts ENV variables to customize the test
| Variable Name | Default Value | Comment |
|-----------------|---------------------|----------------------------------------------------|
| FB_VERSION | 8.13.4 | docker tag |
| LS_VERSION | | docker tag |
| LS_JAVA_OPTS | -Xmx2g | by default, Xmx is set to half of memory |
| MULTIPLIERS | 2,4,6 | determine the number of workers (cpu * multiplier) |
| BATCH_SIZES | 125,1000 | |
| CPU | 4 | number of cpu for Logstash container |
| MEM | 4 | number of GB for Logstash container |
| QTYPE | memory | queue type to test -- persisted; memory; all |
| FB_CNT | 4 | number of filebeats to use in benchmark |
To check the result
- `vault read secret/ci/elastic-logstash/benchmark` to get the host and credentials
- `curl -u "$ES_USER:$ES_PW" "$ES_HOST/benchmark_summary/_search"`
Fixes: https://github.com/elastic/ingest-dev/issues/3377
* Rework the logic to delete DLQ eldest segments to be more resilient on file not found errors and avoid to log warn messages that there isn't any action the user can do to solve.
* Fixed test case, when path point to a file that doesn't exist, rely always on path name comparator. Reworked the code to simplify, not needing anymore the tri-state variable
This a refactoring of test fixture.
Avoid mocking the value returned in global SETTINGS constant. Use instead the local setting map instance used in subject creation.
* Add wolfi as an option to the build process
* Add docker acceptance tests for the wolfi image
* Change how tests are done on the java process, due to "ps -C" not being available on wolfi
replaces and closes https://github.com/elastic/logstash/pull/16116
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
* p2p: extract interface from v1 pipeline bus
* p2p: extract pipeline push to abstract
* p2p: add opt-in unblocked "v2" implementation
Adds a v2 implementation that does not synchronize on the sender so that
multiple workers can send events through a common `pipeline` output instance
simultaneously.
In this implementation, an `AddressStateMapping` provides synchronized
mutation and cleanup of the underlying `AddressState`, and allows only
queryable mutable views (`AddressState.ReadOnly`) to escape encapsulation.
The implementation also holds indentity-keyed mapping from `PipelineOutput`s
to the set of `AddressState.ReadOnly`s it is regested as a sender for so
that they can be quickly resolved at runtime.
* p2p: more tests for pipeline restart behaviour
* p2p: make v2 pipeline bus the default
Updates the DLQ reader to create a notification file (`.deleted_segment`) which signal when a segment is deleted in consequence of `clean_consumed` set. Updates the DLQ writer to have a filesystem watch so that can receive the reader's signal and update the exposed metric, loading the size by listing FS segments occupation.
The PR was created to skip resolving environment variable references in comments present in the “config.string” pipelines defined in the pipelines.yml file.
However it introduced a bug that no longer resolves env var references in values of settings like pipeline.batch.size or queue.max_bytes.
For now we’ll revert this PR and create a fix that handles both problems.
Updates the plain, json and pipeline appenders in default config/log4j2.properties to define a delete rule executed during the rollover strategy, which deletes compressed log archives older than 7 days.
Updates the documentation that describe the logging configuration to explain how the rollover file works, how to configure the strategy, in particular how to update to setup space limitation condition on the rollover.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* pq: avoid blocking writer when queue is precisely full
A PQ is considered full (and therefore needs to block before releasing the
writer) when its persisted size on disk _exceeds_ its `queue.max_bytes`
capacity.
This removes an edge-case preemptive block when the persisted size after
writing an event _meets_ its `queue.max_bytes` precisely AND its current
head page has insufficient room to also accept a hypothetical future event.
Fixes: elastic/logstash#16172
* docs: PQ `queue.max_bytes` cannot be less than `queue.page_capacity`
* A logic to handle non-unicode payload in Logstash.
* Well tested and code organized version of the logic.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Upgrade jrjackson to 0.4.20
* Code review: simplify the logic with a standard String#encode interface with replace option.
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Adds section to describe the intended usage of and impact on memory sizing.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
In the grammar definitions for hashes, `whitespace` was replaced with `cs` to allow either whitespace _or_ comments.
Additionally, the grammar definition for comments was previously required to end with a newline, now it can end with a newline _or_ EOF, using the "not anything" treetop rule `!.`.
Co-authored-by: Jonas Lundholm Bertelsen <jonas.lundholm.bertelsen@beumer.com>
Adds log.format.json.fix_duplicate_message_fields feature flag to rename the clashing fields when json logging format (log.format) is selected.
In case two message fields clashes on structured log message, then the second is renamed attaching _1 suffix to the field name.
By default the feature is disabled and requires user to explicitly enable the behaviour.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Bugfix to parse correctly Java options when the environment variable LS_JAVA_OPTS contains multiple definitions separated by space character.
Adapt the parsing of `LS_JAVA_OPTS` environment variable to split by space various definitions it can contains.
* Wipe out comment lines if config comment contains.
* Remove substitution var process when loading the YAML, instead align on the generic approach which LSCL happens during the pipeline compile.
* Update logstash-core/src/main/java/org/logstash/config/ir/PipelineConfig.java
Put the logging config back as it is being used with composed configs.
Introduce a new setting named `pipeline.buffer.type` which could be valued direct or heap to enable the allocation on Java heap.
The processing of the setting is done in `LogStash::Runner#execute` and sets the Java properties considered by Netty to disable the direct allocation: `io.netty.noPreferDirect`.
However, if that system property is already configured explicitly by the user (because set in `jvm.options`or `LS_JAVA_OPTS`) the setting doesn't take place and warning log is reported, respecting the user's will.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Replace 'git show-ref' with 'git rev-parse' to fix the issue where show-ref is not working as expected.
* Use git checkout instead 'git rev-parse'.
* Apply prune dependencies recommended for big projects (like we have multi gradle projects) by Snyk.
* Apply prune repeated dependency option directly to snyk monitor.
* Avoid the exit, continue scanning to the end.
* Remove the debugging.
This commit adds logic to copy the appropriate env2yaml file to the Docker image
* Clean up env2yaml folder
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Change trigger from pull_request_target to pull_request, as the former uses the base branch instead of the PR source code.
This allows simplification of the checkout action (also took the opportunity to bump from v2 to v4).
* Save name came through ENV vars to let Logstash decide using either keystore or ENV value.
* Apply suggestions from code review to simplify array declaration.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Given that JRuby comes with ruby-maven-libs 3.3.9 this commit upgrades the gem to 3.8.9 and ensures files from 3.3.9 are not included in the distribution.
As a follow up to #15861 this commit splits the current unit tests step
for the Windows JDK matrix pipeline to two that run
Java and Ruby unit tests separately.
Closes https://github.com/elastic/logstash/issues/15566
* Add arm64 support for env2yaml
This commit builds env2yaml in arm64 and amd64 flavors, and uses
$TARGETARCH in the Dockerfile to ensure that the correct version is used
when building for alternative architectures
Fixes: #15913
* Add env2yaml executables to build context
* Split `COPY_FILES` for readability
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Remove unused method createFlushScheduler and add some debug logs to the finalizeSegment method to better follow what happens during its execution in case of problems.
This PR adds a shutdown method to the SchedulerService class used to handle actions to be executed on a certain cadence. In particular is used to execute scheduled finalization of DLQ head segment.
Updates the close method of the DLQ writer to invoke this additional shutdown on the service instance.
Adds a burning of time condition to avoid a collision of time which, under certain circumstances, would fail the test.
The sealing of a segment happens if the segment is considered as stale, which requires 2 conditions:
- the segment must have received a write.
- the time of the last write must exceed the flush interval.
In this failing test, the flush interval is set to ZERO because of the synchronicity of the test, to avoid time dependency. However, with coarse grain timer resolution, could happen that the last write coincide with the time of the stale check, so fail the seal condition.
Adds a global new task named javaTests which groups both :logstash-core:javaTests and :jvm-options-parser:test to include the JvmOptionParser unit test in javaTests phase.
PR#15900 missed a few more places where Logstash is installed but
a working minimal pipeline config is added.
This commit fixes that and stabilizes all acceptance tests, thus
minizing the need for time consuming BK retries of corresponding
steps.
Relates #15900
Relates https://github.com/elastic/logstash/issues/15784
This commit tightens the checks for the status
output of the Logstash OS service to specifically
scan for `org.logstash.Logstash` rather than
only the jdk path.
The reason is that the startup script first runs
an options parser, and then the logstash process
itself, both referencing the JDK path.
Closes https://github.com/elastic/ingest-dev/issues/2950
* Exclude plugins feature in OSS distributions.
* Set elastic_integration plugin default.
* Remove non-OSS plugins after installing default plugins.
* Testing local can't find gem bundler (= 2.3.26) issue.
* Include extract non-OSS plugins logic indocker build operations.
* Only default plugins can be excluded from OSS distros.
* Simplification: instead conditional check, use intersection to make OSS exlucluded plugin list.
* Gem and specification files still stay after removing the plugin. This change removes the stayed files.
* Rename oss-exclude to skip-oss to align namings with other params.
* Make intersection method simpler.
* [Test] Temporary excluding elastic integration plugin from default plugin list.
* Sets elastic_integration plugin default back. When removing locally installed gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle.
* Test the build order, remove plugin from cache logic seems invalid since we don't pack the cache.
Our shared CI infrastructure relies on this file being here with a specific
format, expanding a given `ELASTIC_STACK_VERSION` environment variable and
an optional `SNAPSHOT` environment variable into the information necessary
to target either the most recent published release or a snapshot representing
a candidate for the _next_ possible release in the series.
When we cut a new minor branch from `main` (at feature freeze), the tip of
that branch contains unreleased code that is in-flight for the next release.
We need to have a way to reference artifacts from this in-flight release,
instead of skipping over them while a minor release is in flight.
This change affects the _semantics_ of what is tested during a minor stack
release's feature-freeze period, ensuring that automated tests throughout
the Logstash ecosystem continue to run against the pending release with NO
changes required in those plugins.
It also introduces a new `snapshots.main` entry, which refers to snapshot
artifacts built from the un-feature-frozen `main` branch, so that plugins
can _optionally_ test against the _next_ minor in advance of its feature
freeze, as they currently do unintentionally in builds that target
`ELASTIC_STACK_VERSION=8.x SNAPSHOT=true`.
Logstash on ECK requires openssl command to build TLS keystore.
This commit adds `microdnf install -y openssl` to ensure the command exists in ubi image.
Updates Netty's configuration of maxOrder to a previously proven value, if not already customised by the user.
Adds a step to the JvmOption parsing tool, which is used to compose the JVM options string to pass down to Logstash at startup.
The added step rework the parsed options to set the allocator max order -Dio.netty.allocator.maxOrder=11 so that the maximum pooled buffer is up to 16MB and not 4MB.
This option is added iff it's not yet specified by the user
Update the env2yaml to have a go.mod instead of relying on disabling go modules, otherwise building with golang 1.22 will fail in the future.
This change also directly uses the golang image to build the binary removing the need for an intermediate image.
* Fix the latest 8.12 version, to the actually latest released
* Update version fo 7.17
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit fixes the startup of the Logstash service during packaging
tests by adding a minimal pipeline config. Without it, the service was
flapping from start to start and vice versa causing test flakiness.
Relates https://github.com/elastic/logstash/issues/15784
This commit allows separate running of Java and Ruby tests on Windows i.e. the same way as we currently do on unix (unit_tests.sh) via a cli argument.
If no argument has been supplied, both tests are run (as it does now).
The wrapper script is also rewritten from old batch style script to Powershell.
This work allows us to split the existing Windows CI job in a subsequent PR to separate steps, as we currently do on Linux.
Relates: https://github.com/elastic/logstash/issues/15566
Currently for pipelines that have steps with auto retries, if there
is a failure of a step, even if the subsequent retry succeeds, we
received Slack alerts.
This commits uses new functionality available in the Buildkite PR Bot
to avoid Slack notifications in those cases and changes the settings
for all pipelines where we have, or expect to have, retries.
This commit adds a schedule to run the exhaustive pipeline
(biweekly, every other Wednesday @2AM UTC) and the aarch64
(weekly, every Monday@2AM UTC).
Closes https://github.com/elastic/ingest-dev/issues/2852
This commit adds (up to 3) retries for all steps of the `serverless-integration-testing`
pipeline as a stop-gap measure to prevent network related transient failures.
* Describe how to troubleshoot connectivity problems at HTTP level with Azure Blob Storage
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This commit fixes IT failures that frequently occur after
version bumps due to missing unified release snapshot builds for
the new version.
This commit uses project specific DRA snapshot URLs for ES and Filebeat
in all cases apart from release builds.
Closes#2825
The current mechanism of discovering the latest released version per
branch (via ARTIFACTS_API) isn't foolproof near the time of a new
release, as it may be pick a version that hasn't been released
yet. This leads to failures[^1] of the packaging upgrade tests, as we
attempt to download a package file that doesn't exist yet.
This commit switches to an API that that is more up to date regarding
the release version truth.
[^1]: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/125#018d319b-9a33-4306-b7f2-5b41937a8881/1033-1125
This commit fixes the flaky IT test:
`install non bundle plugin successfully installs the plugin with debug enabled`
by being a bit more lenient with the output which can get garbled by Bundler.
Closes#15801
* Ruby code coverage with SimpleCov json formatter
* [CI] Send Java and ruby tests to sonarqube simultaneously
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Added compiled classes to artifacts
* Test change
* Removed test changes
* Returned back ENABLE_SONARQUBE condition
* Removed debug line
* Diable Ruby coverage if ENABLE_SONARQUBE is not true
* Run sonar scan on pull requests and onn push to main
* Run sonar can on release branches
Having enabled branch pushes as triggers for the exhaustive test
pipeline, we have an unwanted sideeffect, that when using
PR automation (@logstashmachine) to create backports, the temporary
branches also triggered exhaustive tests.
This commit skips triggering this pipeline when upstream branches
starting with `backport` get pushed.
There is occasional flakiness mainly with IT tests requiring us to
manually retry such failures when we raise PR (or the first
group of the exhaustive suite, which runs the same steps).
This commit adds up to 3 retries for all the steps of the PR
pipeline.
This commit enables running the exhaustive tests Buildkite pipeline
(i.e. the equivalent to the `main` Jenkins tests) ; the trigger is
code events, i.e. direct pushes, merge commits and creation of new branches.
CI is skipped if changes are only related to files under `docs/`.
This commit added a few jvm.options properties to configure the Jackson read constraints defaults (Maximum Number value length, Maximum String value length, and Maximum Nesting depth).
This commit modernizes the qa/acceptance (packaging) test framework by
moving away from Vagrant and having the tests operate locally.
As we are migrating to Buildkite, the expectation is that those tests
will run on dedicated vms thus removing the necessity of vagrant.
Relates: https://github.com/elastic/ingest-dev/issues/1722
Add the `logstash-filter-elasticsearch_integration` plugin to the list of test_plugins pipeline.
In doing so, reworked the definition of plugins, by not just specifying a plugin name but also adding support leve, type and organization. This necessary for plugins which repository is not part of `logstash-plugins` Github organization but at the same time permit an easier management of the different axes that partitions the plugin landscape.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes how the keystore tool handle the command's options, including validation for unknown options, and adding the --stdin flag to the add command.
- Re-enabled logstash-filter-json and logstash-codec-netflow plugins into the test_plugins BK pipeline.
- Moved codecs tier2 to be executed in separata VM.
- Commented plugin logstash-codec-collectd till is fixed
PR#15729 missed the input step. As a result when the job is triggered
the steps are executed, but the pause icon still shows in the job
requiring manual unblock[^1]
This commit also skips the input step when the job is triggered from
the scheduler pipeline.
[^1] https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/86
The recent PRs #15668 and #15705 refactored jobs with a custom schedule
to leverage a centralized trigger pipeline.
An unexpected sideffect of this is that the conditional for the wait
step doesn't work anymore.
This commit skips the wait step when the JDK matrix pipelines get triggered
from another pipeline.
As a follow up to #15705, this commit shushes Slack alerts for the
common Buildkite scheduling pipeline. This is because the pipeline
scheduler just triggers other pipelines which have their own dedicated
alerts setup, therefore we want to avoid duplicate alerts when
there is a failure with one of the triggered pipelines.
This commit is a pre-requisite for adding unit + IT tests in a
dedicated phase of the Exhaustive tests pipeline.
It refactors the tests currently used by PR jobs, so that they become
reusable.
As an improvement from #15668 / #15700, rather than having one
dedicated side-car scheduling job per pipeline, we move to a single
scheduling job. Various pipelines that need triggering with different
schedules now live under each schedule in the new pipeline.
This reduces the amount of jobs we have to maintain in yaml.
This commit enhances the functionality introduced in #15668 and #15700
by allowing a single Buildkite scheduling job to trigger several
pipelines, in addition to multiple branches which it already does.
We rename the env var PIPELINE_TO_TRIGGER to PIPELINES_TO_TRIGGER
which now supports comma separate values.
This enhancement can be useful for pipelines like JDK matrix which
have variants (Linux and Windows) that we want to trigger with a single
scheduling job, thus reducing unnecessary entries in catalog-info.
This commit fixes a few bugs introduced in #15668 related to paths for
the calling script. We also stop limiting the execution only from the
main branch (to facilitate e.g. tests from PRs) and, finally, remove
the async clause, which is not needed, since by default BK steps are
run in parallel.
This commit is the first making use of #15627 to remove hard coded
branches for the DRA Snapshot build schedule.
With this pattern, we will only need to keep `ci/branches.json` up to date,
as versions evolve, and not need to update/maintain hard coded branches
in `catalog-info.yaml` anymore.
Once this is verified working, we'll add a corresponding schedule
pipeline (in `catalog.info`) for the JDK matrix job.
Relates: https://github.com/elastic/ingest-dev/issues/2664
Now that CI VM images are pre-provisioned with various flavors of
Java 21, we add the option for the corresponding CI job.
Adoptium 17 remains the default pre-selected option.
Relates https://github.com/elastic/ci-agent-images/pull/463
Introduces a new interface named SchedulerService to abstract from the ScheduledExecutorService to execute the DLQ flushes of segments. Abstracting from time provides a benefit in testing, where the test doesn't have to wait for things to happen, but those things could happen synchronously.
Update the test_plugins pipeline script to execute only the unit tests.
Use the vendored JRuby in every Ruby related duty, such as running `bundler` and `gem`.
Temporarily comments plugins that has needs to be fixed and already fails on their Travis CI.
Executes the testing of input tier1 plugins in VM instead of Buildkite agent.
In DLQ unit testing sometime the DLQ writer is started explicitly without starting the segments flushers. In such cases the test 's logs contains exceptions which could lead to think that the test fails silently.
Avoid to invoke scheduledFlusher's shutdown when it's not started (such behaviour is present only in tests).
This commit adds the compatibility tier for the Exhaustive tests suite.
Specifically, we introduce two new groups (running in parallel) for Linux and Windows compat tests.
Linux picks one OS per family from [^] and likewise Windows one of the three available choices from the same file.
We also support manual override, if user chooses to, by setting `LINUX_OS` or `WINDOWS_OS` as env vars in the Buildkite build prompt (in this case there is no randomization, and only one OS can be defined for Linux and Windows respectively).
For example:
```
LINUX_OS=rhel-9
WINDOWS_OS=windows=216
```
Relates:
- https://github.com/elastic/ingest-dev/issues/1722
[^1]: 4d6bd955e6/.buildkite/scripts/common/vm-images.json
This commit added support to add and remove multiple keystore keys in a single operation. It also fixed the empty value validation for editing existing key values and added ASCII validation for values.
The last remaining Jenkins job prior to BK migration is for
exhaustive tests. The compatibility phase seems to be failing
since 57dc14c92
with Java 17.0.2
This commit switches from OpenJDK 17 (whose last release was 17.0.2)
to AdoptiumJDK 17 which actively receives updates and is bundled
in the custom images used by Jenkins.
* Use Java installed BK agent and remove unnecessary git clone operation since repo is already cloned.
* Switch back to normal VM since Logstash BK agent doesn't support docker operations.
This commit adds a maximum default (global) timeout for every pipeline
definition (now that it's possible to define this programmatically in
an RRE).
The default values have been chosen arbitrarily based on intuition
about how much (in the worst case) we should wait for each job to
run until we consider them stuck/failed.
While at it, we update the yaml schema for RREs to point to the
latest commit (rather than a pinned commit that doesn't reflect the
latest changes, e.g. `maximum_default_timeout`).
Relates #15380
This removes the dependency on jackson's dataformat-yaml. Since there's only a single place where this library is used in core: to load the plugin alias definition, the code can be replaced by the underlying snakeyaml.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 93d8a9da32)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes the issue with Jackson > 2.15 and `log.format=json`: "Can not write a field name, expecting a value", by adding a default serializers for JRuby's objects.
This commit enables scheduled runs of the JDK matrix pipelines for
both Windows and Linux, once a week, Tuesdays at 1am UTC, using the
pipeline defaults for OS and JDK.
The last part of the Logstash JDK matrix CI migration from Jenkins to
Buildkite is AmazonLinux 2023.
While we have a working image[^1], this is the only step that requires
a agent that runs on AWS.
This commit refactors the builder to support GCP or AWS agents depending
on the OS.
[^1]: https://github.com/elastic/ci-agent-images/pull/441
Add missing yaml-language-server definition to Buildkite pipeline files
(static and dynamic generated) for consistency and to ease spotting
errors with editors.
This commit adds a global max timeout of 90min for the supported plugins
Buildkite pipeline. This prevents hanging builds (for 24hrs, which is
the default).
Relates: #15380
Build's maximum_timeout_in_minutes and default_timeout_in_minutes are available now through the catalog-info.yaml file.
As this change was made manually before we implemented this in RRE/Terrazzo, they got reverted to the default value (0); thus, I am raising this PR to get it as it was specified before the upgrade (120 mins.)
This commit is the first part of the migration of JDK matrix tests
from Jenkins to Buildkite. There will be two separate pipelines, for
Linux and Windows.
Linux is currently limited to Ubuntu 22.04 and 20.04, but
additional operating systems will be added outside of the Logstash
repository seamlessly through additional VM images.
Steps are created dynamically and the underlying script is meant to be
common for Linux and Windows. Windows is currently a stub and
will be added in a follow up PR.
Relates:
- https://github.com/elastic/ingest-dev/issues/1725
- https://github.com/elastic/ci-agent-images/pull/424
* Update the Logstash to Logstash Native doc to reflect the multiple hosts usage.
* Logstash to Logstash comm page, adding LS-to-LS native HA support.
* Apply suggestions from code review
Refining the context.
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
---------
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
* geoip: extract database manager to stand-alone feature
Introduces an Elastic-licensed GeoipDatabaseManagement tool that can be used
by ANY plugin running on Elastic-licensed Logstash to retrieve a subscription
to a GeoIP database that ensures EULA-compliance and frequent updates, and
migrates the previous Elastic-licensed code-in-Logstash-core extension to
the Geoip Filter to use this new tool, requiring ZERO changes to in-the-wild
versions of the plugin.
The implementation of the new tool follows the previous implementation as
closely as possible, but presents a new interface that ensures that a
consumer can ATOMICALLY subscribe to a database path without risk that the
subscriber will receive an update or expiry before it is finished applying
the initial value:
~~~ ruby
geoip_manager = LogStash::GeoipDatabaseManagement::Manager.instance
subscription = geoip_manager.subscribe('City')
subscription.observe(construct: ->(initial_dbinfo){ },
on_update: ->(updated_dbinfo){ },
on_expire: ->( _ ){ })
subscription.release!
~~~
* docs: link in geoip database manager docs
* docs: reorganize pending 'geoip database management' feature
* docs: link to geoip pages from feature index
* geoip: add SubscriptionObserver "interface"
simplifies using Subscription#observe from Java
* geoip: fixup SubscriptionObserver after rename
* geoip: quacking like a SubscriptionObserver is enough
* geoip: simplify constants of legacy geoip filter extension
* geoip: bump logging level to debug for non-actionable log
* geoip: refine log message to omit non-actionable info
* re-enable invokedynamic (was disabled to avoid upstream bug)
* geoip: resolve testing fall-out from filter extension's "private" constants removal
* geoip: consistently use `DataPath#resolve` internally, too
* Adapted the JDK's download URL creation to intereact with Elastic catalog to get metadata, and return the catalog download link instead of directly pointing to Adoptium API
* Silenced the Download task of JDK to print the full url
Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861
Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message.
PR#15466 skipped the Java unit tests as on the `main` and `8.11`
branches they attempted to run sonar scans (which are only meant to
run for PRs).
This commit re-enables the Java unit tests, taking advantage of #15486,
disabling the sonar scan part of the test suite.
This commit introduces a way to optionally disable SonarQube scanning
and coverage reports when running Java unit tests. The integration
was introduced in #15279, however, there are cases (e.g. running Java
unit tests outside of PRs) where we don't want this integration.
Disabling can be achieved by setting the env var `ENABLE_SONARQUBE`
to `false`.
This commit is the first part of migrating away the aarch64 Jenkins
jobs to Buildkite. It adds a group of exhaustive test steps in the
aarch64 pipeline.
The java unit tests are temporarily disabled as they run SonarQube
scans which need to be associated with pull requests.
Relates:
https://github.com/elastic/ingest-dev/issues/1724
This commit enables automatically updated comments from Buildkite for
PRs. It posts comments to PRs whenever a Buildkite build finishes or
encounters its first failing step and keeps the comment updated.
This commit enables the snapshot job schedule for Buildkite jobs.
They are set to run after Jenkins (scheduled @midnight) so that any
remote/unknown chance of conflict with the release manager is limited.
While at it, we also enable slack notifications for failures to the
same channels as Jenkins.
The new psych 5.1.1 gem seems to not work when installed in JRuby 9.4 (used in main).
This change pins the version back to 5.1.0 until ruby/psych#655 is sorted.
Defines a Buildkite pipeline to run specs tests on tier1 and tier2 plugins, using container images. The tasks are divided by plugin type for each tier, to run them in parallel. Once all tier1 testing is completed then starts with tier2.
This commit adds a Buildkite resource for pull requests and
the two simple jobs, license checking and ruby unit tests that are
already part of the Jenkins PR multi job setup.
As this is WiP, slack notifications aren't enabled.
* Add known imap and email plugin issues section to Logstash 8.10+ versions.
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit a88f82e77f)
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
We seem to be hitting a weird issue with the psych gem when running in a container:
```
> Task :bootstrap
2023-09-27T15:13:23.844Z [Execution worker Thread 5] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
Error Errno::ENOENT, retrying 1/10
No such file or directory - /buildkite/builds/bk-agent-prod-k8s-1695827479442731913/elastic/logstash-dra-pipeline-ci/vendor/bundle/jruby/3.1.0/gems/psych-5.1.0-java/deps.lst
```
This commit works around this problem by running inside a VM. The underlying root cause (and that gradle should be erroring in this case) will be investigated separately.
We've spotted network flakiness downloading artifacts with gradle (connection resets) when using GCP/t2a on us-central1.
This commit switches to AWS Graviton instance types for building the aarch64 artifacts.
This commit adds support for building + publishing DRA (-SNAPSHOT for now) artifacts for Logstash. It builds on top of #15312 and therefore only targets the `main` branch and is intended to be run manually during a trial period before we retire the corresponding Jenkins job.
The structure is similar to Jenkins:
1. Three steps runs in parallel to build packages, x86_64 docker and aarch64 docker artifacts.
2. Once 1. is successfully done, use release manager to publish the artifacts.
We generate the pipeline steps for 1. and 2. dynamically (with a simple Python script) to avoid repetition for future PRs: we will add a new pipeline in a follow up PR for -STAGING.
The actual shell scripts are simplified copies from the existing `dra*` scripts under https://github.com/elastic/logstash/tree/main/ci; the simplification comes from native support for copying artifacts between steps in Buildkite and
not having to use an intermediate bucket.
Relates: https://github.com/elastic/ingest-dev/issues/1720
Blocked by: https://github.com/elastic/ci/pull/2312/files
Expands the description of memory used by Logstash, dividing the heap and non-heap; describing in details which parts composes the non-heap, how to size it and list the JVM settings that can be used to properly define this space.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Since the DRA build for 8.10.1 was made with 82daae80bb , this fix didn't get in.
(cherry picked from commit aa9265665e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Add initial buildkite pipeline for DRA artifacts.
This is a "hello-world" pipeline skeleton required to iterate on migrating the actual DRA jobs.
This commit also updates the backstage component for logstash to become a top level entity with the
necessary annotations and linked dependencies to pipelines.
Relates https://github.com/elastic/ingest-dev/issues/1720
This commit adds a call to identify serverless cluster before fetching `/_xpack` to verify the license and supported features.
When it is serverless, the license checker uses hardcoded xpack info instead of calling xpack API. The internal pipeline of monitoring is disabled and logs error when configured to use legacy monitoring.
Fixes:
ingest-dev#2303
ingest-dev#2284
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
The DeadLetterQueueWriter#updateOldestSegmentReference was sorting the existing segments by the lexicographic file names. For example, if there are a few segments on the DQL path with index 8, 9 and 10, it was choosing the 10 as the oldest segment instead of 8.
This commit changed it to comparing the segment ID as numbers, so the file deletion will happen following the proper order.
* Pipeline to pipeline communication acked queue improvements.
* Handle InterruptedException exception in input back, warn message improvement when in-flight events are partially sent and other minor such as descriptive logs, etc improvements.
* Apply suggestions from code review
Check if queue is open after thread acquires the lock.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Apply suggestions from code review
Unite test case improvement: use `assertThrows` when validating the exception.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Pulling off of introducing wrap with operations.
* Introduce functional interface to broadly use to catch the exception types.
* Addressing comments: do not retry sending inflight events on case. We still throw if we get error when opening queue.
* will not be reached input retry logic, removing.
* Move queue close check after thread acquiring a lock. Make read next page interface private since it is an internal use purpose.
* Apply suggestions from code review
Leave a comment for the write lock and remove unnecessary warning with `ensure_delivery=>false`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Remove unused method, check if current thread acquired lock when accessing next page.
* pq: getting possibly-shared access to next read page is illegal
The private `Queue#nextReadPage()` method requires that the caller has
exclusive ownership of the lock, and failing to have the lock is an
illegal state that cannot be recoverd from; it would leak the
effectively-private `Page` to a caller that cannot reliably use it
without corrupting other callers.
Both callers of this private method already call it with exclusive
access, so this safeguard is merely to prevent future development from
breaking the expectation unknowingly.
As such, we throw an `IllegalStateException`.
* pq: use shared queue-closed check for block and non-block reads
By moving the closed-check from the blocking `Queue#readBatch` to the
shared private `Queue#nextReadPage`, we ensure that both blocking reads
by `Queue#readBatch` and non-blocking reads by `Queue#nonBlockReadBatch`
behave the same when the queue has been closed.
* pq: make exception message constants descriptive
* p2p: clarify comment about cumulating retry behaviour
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit moves the Files.size(...) call into the try catch block, that way, when the oldest segment is deleted by the DeadLetterQueueReader, no NoSuchFileException will be thrown up, and the writer will gracefully update the oldest segment on the next updateOldestSegmentReference invocation (scheduled flush, entry write, delete expired, etc).
It also adds the volatile keyword to the shared mutable variables, making sure that all the changes will be instantly visible among all the running threads (scheduler & writer).
This commit changes the serverless tests to use a restricted API key with necessary permissions and disables Logstash legacy monitoring test.
Fixes: #15234
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Implements a script to test all supported plugins against Logstash/JRuby, in particular it uses the JRuby(and JDK) bundled with Logstash to execute the unit tests, create the gem and install the gem on local Logstash.
This doesn't start a real pipeline run into the local Logstash, but should be enough to reveal macroscopic Ruby syntax compatibility errors.
Refer to PR https://github.com/elastic/logstash/pull/15018 for usage examples.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Adds rubocop as a linter for ruby.:
- Adds .rubocop.yml which defines specific config on top of default
- Adds Gradle task 'lint' task
- Adds 'lint:report' rake task to report Lint Cops.
- Adds 'lint:format' rake task to automatically format the code.
* Snyk scans Logstash container vulnerabilities.
* Exclude integ test and tools when Snyk scanning.
* Remote repo url fix for main branch.
* Update .buildkite/scripts/snyk/report.sh
Simplify the logic to retrieve the version from `versions.yml`
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Add backstage definition for Snyk Report pipeline.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
This commit updates the puma gem from version 5 to the latest version 6.3.
A few breaking changes were introduced in Puma 6.0.0, which required some refactoring on the Logstash side, especially to adapt it to the "Extracted LogWriter from Events #2798" changes.
Before this PR, all the logs generated by Puma were using the debug level, even the ones that were actually errors and needed attention/action from the users. This commit also changes the log level as following:
error(...): changed from debug to error
unknown_error(..): changed from debug to error
When using a proxy with the plugin manager, this template is written to `~/.m2/settings.xml`. The
license header is also copied, which generates invalid XML which maven cannot parse.
Set of changes to make Logstash compatible to JRuby 9.4.
Bundle JRuby 9.4.3.0
- Redefine space token in `LSCL` and `grammar` treetop from `_` which would generated methods in the form `def _0` (deprecated since `2.7`) to `sc`.
- `I18n.t` method doesn't accept hash as second argument
- `URI.encode` has been replaced with same functionality with `URI::Parser.new.escape`
- `YAML.load` needs explicit `fallback: false` to return false when the yaml string is empty (or contains only comments)
- JRuby's `JavaClass` has been removed, now it can use `java.lang.Class` directly
- explicitly require gem `thwait` to satisfy `require "thwait"` (In `Gemfile.template` and `logstash-core/logstash-core.gemspec`)
- fix not args `clone` to be `def clone(*args)`
- fix `Enumeration.each_slice` which from `Ruby 3.1` is [chainable](https://rubyreferences.github.io/rubychanges/3.1.html#enumerableeach_cons-and-each_slice-return-a-receiver) and doesn't return `nil`. JRuby fixed in https://github.com/jruby/jruby/issues/7015
- Expanded `Down.download` arguments map ca16bbed3c302006967413eb9d3862f2da81f7ae
- Avoid to pass `nil` in the list of couples used in `Hash[ <list of couples> ]` which from Ruby `3.0` generates an `ArgumentError`
- Removed space not allowed between method name and parentheses `initialize (` is forbidden. 29b607dcdef98f81a73ad171639fd13aaa65e243
- With [Ruby 2.7 the `Kernel#open`](https://rubyreferences.github.io/rubychanges/2.7.html#network-and-web) doesn't fallback to `URI#open`, fixed test code that used that to verify open port. e5b70de54c5301f51a767da67294092af0cfafdc
- Avoid to drop `rdoc/` folder from vendored JRuby else `bin/logstash -i irb` would crash, commit b71f73e9c6edb81a7b7ae1305047e506f61c6e8c
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This will eventually replace the pipeline definition in the catalog repo, using the local file instead.
---------
Co-authored-by: elastic-backstage-prod[bot] <123036547+elastic-backstage-prod[bot]@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* This commit includes number of changes:
- adds initial pipeline steps, each step proceeds one branch report job
- a script to resolve current and previous release latest branches from the release URL. Noting that it considers the branches from snapshots where upcoming releases will be our interest.
- pseudo logic to download logstash, build and report to Snyk. In order to script fully work we need a logstash machine to access Buildkite vault, which I am working.
* Source optimization and use Buildkite Vault to access to Snyk token.
* Add git branch and head to Snyk project tag.
* Dynamically resolve latest branches instead of manually defining.
* Update .buildkite/scripts/snyk/report.sh
Improve comment readability.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit changed the java_pipeline.rb to include the pipeline/main thread on the stalling threads info list, that way, Logstash can provide users with more helpful information when the stalling thread is the pipeline/main one.
This commit adds missing Elasticsearch SSL settings and replaces deprecated options being used on `xpack.monitoring.*` and `xpack.management.*` settings:
Changes:
- Updated deprecated monitoring and management Elasticsearch's SSL settings so no warnings are logged.
- Added monitoring settings support for file-based certificates and for the cipher suites: `xpack.monitoring.elasticsearch.ssl.certificate`, `xpack.monitoring.elasticsearch.ssl.key`, and `xpack.monitoring.elasticsearch.ssl.cipher_suites`.
- Added management settings support for file-based certificates and for the cipher suites: `xpack.management.elasticsearch.ssl.certificate`, `xpack.management.elasticsearch.ssl.key`, and `xpack.management.elasticsearch.ssl.cipher_suites`.
* docs: fix example block syntax types and truncations
* docs: provide wrapping hints to flow metric tables
* docs: refresh node stats api response examples
include only `current` and `lifetime` metrics that are GA, and not
technology preview metrics.
* docs: use "m(onospace)" modifier for metric name columns
* docs: swap literal column to first
relies on `#guide table td:first-child .literal` having `white-space: nowrap`
With PR #13923 was described how the DLQ storage_policy behaves. With PR #14261 was introduced a duplication of the description in logstash.yml.
This commit resolves the duplication and keeps the latest description that seems more direct expressive.
Modifies the logic used by the scheduled task flusher so that execute age policy also in case the current (head) segments is not stale (haven't received any write, and the segment is empty).
This means that generally used finalize segment logic is applied plus a reinforcement step to grant the age policy is respected.
However this PR:
- introduced new debug log lines, improving the description of the context when a segment is finalized (because the DLQ is closing or because the segment file has reached its maximum size or because the flush interval expiration). This is done with the introduction of `SealReason` enumeration.
- introduces `Awaitility` test dependency to improve the testing of asychronous conditions.
* Revert "Revert "Unpin bundler to allow 2.4 (#14894)" (#14942)"
This reverts commit 5e3038a3d3.
* prevent bundler 2.4 from blowing up memory during ./gradlew generatePluginsVersion
for some reason, instances of Bundler::Dsl are retained in
org.jruby.MetaClass. Each Bundler::Dsl will retain Bundler::SourceList
which is > 150MB. This commit sets the internal state of each Dsl's
source list so the large objects can be collected.
The proper fix would be to either reuse a single Dsl object or fix the
code upstream to avoid having Dsl objects retained by org.jruby.MetaClass.
* skip input cloudwatch during generatePluginsVersion
the new bundler 2.4's resolution algorithm seems to struggle in certain
situations, like resolving combination of "logstash-input-cloudwatch" and
"logstash-integration-aws".
Until the issue is solved let's skip it.
Modify the WorkerLoop to catch the newly introduced exception org.logstash.execution.AbortedBatchException so that an inflight batch could be negatively ACK-ed. This feature is used in combination with PQs to let exit plugins without completing the processing. Any filter and output already executed for the batch will be executed again next time the batch is picked by the persistent queue.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes the system unit TimeoutStopSec option to set 0 on older version that do not support 'infinity'.
Also handles the upgrade case where the OS may not automatically upgrade the systemd file.
In commit, DLQ writer skips searching segments with 1 byte during the search of the oldest segment timestamp in the initialization
Co-authored-by: Andrea Selva <andrea.selva@elastic.co>
This reverts commit 6d08a7c1cd.
Bundler 2.4 is accumulating memory during the `generateDocsVersion` task
outside of the known Bundler 2.3 DepProxy class.
Until the memory mitigation patch is adapted to Bundler 2.4 it's best to revert 6d08a7c1cd.
This commit adds environment variable LOG_STYLE to control log behavior of docker container
`console` - this is the default to output to standout
`file` - log to disk.
Fixed: #14941
Pass the argument in correct order when calls DeadLetterQueueWriter.writeEntry from PluginDeadLetterQueueWriterExt.
Adds a unit test to verify the DLQ writer-read interaction.
* Skip dependency Gemfile.lock during the artifact creation. This will help vulnerability scanners not to confuse while scanning.
* Apply suggestions from code review
Add to exclude `Gemfile`, apply pattern to include shared gems as well.
This PR fixes a bug on DLQ age policy not executed if the current head segment haven't receives any write.
The change update the `flushCheck` method, executed both on DLQ writer close and also by the scheduled flusher, so that the `executeAgeRetentionPolicy` is invoked also when the current writer hasn't received any writes.
Adds some test, and to separate the testing of the close from the scheduled flush a new constructor's parameter is added, and consequently updated builder utility.
import platform-specific ffi binaries from JRuby 9.3.9.0 to overcome jruby/jruby#7579
Since JRuby ignores the files in "lib/jni" and uses the ones inside "lib/jruby.jar" instead, we have to inject the older versions into the new JRuby's "jruby.jar"
fixes#14873
Reject illegal value assigning to `tags` field. Top-level `tags` should only accept string of array of string.
When `tags` got illegal value on event creation, LogStash::Event will rename the field to `_tags` and add a tag `_tagsparsefailure` to `tags`.
When `tags` got illegal value on `set` operation, LogStash::Event will throw exception.
Add a flag `--event_api.tags.illegal` to allow fallback to old logic. There are two options.
`warn` - the old flow that allows illegal value assignment to tags field.
`rename` - the new flow. This is the default value in 8.7
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
The delayed implementation `AfterCompletionTimerMetric` of the `TimerMetric`
interface, introduced along-side that interface's introduction to replicate
the previous (undesired) behaviour, is superceded by an already-merged live-
tracking implementation that is effectively as performant when not under
concurrent contention and still reasonably performant when a single timer is
contended across multiple threads.
The `metric.timers` setting removed here has not been a part of any Logstash
release and can safely be removed without going through the normal deprecation
path; from the user's perspective this removal combined with the previously-
merged work is simply an improvement to the accuracy of the existing timer
metrics exposed via our API.
* Forward port of 8.6 and 8.5.3 release notes to main
* Better phrasing for pipeline level metrics feature
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit adds a new logstash.yml setting "xpack.geoip.downloader.enabled" to disable the GeoIP databases auto-update feature. When disabled, Logstash will fall back to the CC database license indefinitely and delete any previously downloaded EULA databases.
Closes#14724
* Initial effort to initialize plugin flow metrics. Followings are addressed:
- Namespace store is shaped with RubySymbol key but filter and output codecs were using string key. This commit intends to standardize the namespace key with RubySymbol for filter & output codecs.
- Initializes throughput flow metrics for the input plugins.
- Initializes the worker cost per event and worker utilization for the filter and output plugins with only uptime metrics but it should combine with worker count, will be implemented in next commits.
- Fetching codec ID generated in ruby scope is possible but problematic to in Java scope. We will skip codec flow metrics since they are rarely produce the hard times.
* Worker utilization metrics implementation.
- Worker count will be provided as a fraction to the flow metrics. At the time when we fetch the metric value, fraction is applied.
* Unit tests added for fractured extended & simple metrics.
* Code review change requests applied.
- To simplify the scale (or fraction) at metric get value time, we can introduce the wrapper (`UpScaleMetric`) that applies the scale at metric value fetch time.
- Unit test added for `UpScaleMetric`
- We don't touch the codec namespace shape for now since we skipped codec metrics.
- Unused sources removed.
* Worker utilization and worker cost per event explanation added in the documentation.
* Integration test added for plugin-level flow metrics.
* Apply suggestions from code review
- Integration test failure fix: input plugin ID is not always in context config.
- Suggestions to simplify integration test source and rollback to intentional namings.
- Metrics explanation improvement in the doc.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* plugin flow: fix units; pass UptimeMetric and scale when needed
Aligns the units of the newly-introduced plugin metrics with the specification,
and passes our `UptimeMetric` through to the individual helper methods so that
they can scale appropriately for their context and our type-checker can ensure
we don't receive an incorrectly-scaled `Metric<Long>`.
Input `throughput`
------------------
all throughput metrics should be expressed in events-per-second; this
per-plugin scoped view of the pipeline's `input_throughput` flow should be
expressed in the same units.
Filters, Outputs `worker_utilization`
-------------------------------------
> a worker_utilization (duration / (uptime * worker count)) shows what percent
> of available resources an individual plugin instance is taking and can help
> identify where the blocker is.
To achieve this, we need to divide millis used by _millis_ available.
Filters, Outputs `worker_cost_per_event`
----------------------------------------
> we also provide a (to be named) cost-per-event metric (duration / event) to
> surface issues with a plugin that operates on a very small subset of events
> (via conditionals) but contributes disproportionately to the cost of getting
> its events through.
We start with a baseline of seconds-per-event, and acknowledge that this may
need to be scaled to a more understandable number before merging.
* plugin flow: express cost per event in millis per event
The "worker cost per event" metric when expressed as an inverse per-worker
throughput in seconds-per-event produces a range of values that are not
particularly easy to compare at-a-glance, with "nearly free" operations
being expressed in negative-exponent scientific notation and extremely
expensive operations being expressed with single-digits.
By scaling this metric up by a factor of 1000 to "millis per event" or its
eqivalent "seconds per thousand events", the resulting numbers in practice
are easier to make sense of:
+------------------------+--------------+---------------+------------+
| EXAMPLE / SCALE | s/event | ms/event | µs/event |
+------------------------+--------------+---------------+------------+
| no-op mutate @ 12k eps | 8.33e-05 | 0.0833 | 83.3 |
| stdout w/ dots codec | 0.000831 | 0.831 | 831 |
| ES out 1s RTT/125 | 0.008 | 8 | 8000 |
| ES out 30s retries/125 | 0.24 | 240 | 240000 |
| ES filter 1s/event | 1 | 1000 | 1000000 |
| grok 30s timeout | 30 | 30000 | 30000000 |
+------------------------+--------------+---------------+------------+
* plugin flow: reshape docs
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Logstah currently does not support multiple top-level `codec` per plugin. This commit fixes the pipeline compilation ensuring that erroneously configured plugins fail to compile and result in a configuration error.
* live timers: introduce API boundary
Introduces an API boundary for timers as a first-class metric, as described
in elastic/logstash#14675, and migrates all known internal timers to use the
new API boundary for tracked execution.
Please refer to the specification for details on motivations.
This commit is net zero change to behaviour, and introduces a single new
undocumented setting `metric.timers` to `logstash.yml`, which presently only
takes its default value `delayed` to indicate that delayed committing of
execution time is acceptable.
It implements the new `TimerMetric` API in a way that is also net-zero-change.
Tracked executions are still performed by marking a start time, performing
the tracked execution, and incrementing an underlying long-type counter with
the number of elapsed milliseconds _after_ execution has completed. This means
that long-running execution is still missing from the metric until it has
completed.
The new Timer API is available to both the Ruby- and the Java-based plugin APIs
* timer metrics: sub-package and add baseline tests
* WIP: move execution metric ownership out of queue
* noop: remove useless abstract method
Our `AbstractMetric` implements `Metric` and does not need to declare
an abstract override of `Metric#getType`. Doing so prevents interfaces
from providing a default override for all implementers.
* timer metric tests: extract util, refactor for reuse
* timers: accumulate milli-excess-nanos
* live timers: single-checkpoint implementation
* timer metric: use explicit type parameters to make intent clear
* remove unused imports
* use safe int conversion
* test fixup: use given name for tested metric
* test helper: TimerMetricFactory prefers nanotime supplier
* timers: flesh out test coverage, incl live-timers
* test: move validation of queue-read metrics to ObservedExecution
* flow: support non-moving denominator (±infinity)
* metrics: add metric config pass-through to env2yaml
* Remove unnecessary IR graph loggings.
* Add the guides for protecting sensitive info leaking in the logs.
`CONTRIBUTING.md` now contains the guide to prevent sensitive info leaking in the debug logs.
Fixes: #14778
Pull-request: #14779
* Add sample source codes to the `CONTRIBUTING.md`.
* Apply suggestions from code review
Contribution guide for using sensitive data in Logstash improved.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
During stalled shutdowns while waiting for in-flight batches to complete,
our shutdown watcher emits helpful information about what work is in flight,
including the actual threads and plugins that are still executing.
Since ~6.3.0, the `inflight_count` metric in this log message has always
been `0`, in part because of two somewhat-overlapping bugs:
- elastic/logstash#8987 and elastic/logstash#9056 (7.0, 6.3) changed
the `inflight_batches` map provided by the queue read clients to index
batches by native thread id, but pipeline reporter continued to
attempt to extract by ruby thread object. Because it does not find
the thread in the "batch map", it reports zero.
- elastic/logstash#9111 (7.0, 6.3) changed the _value_ stored in
the `inflight_batches` map provided by a new common queue read client
from an object responding to `#size` to a java `QueueBatch` which
does not respond to `size`. If our pipeline reporter had been able to
look up the queue batch, it would have failed with a `NoMethodError`.
We resolve the issue by (1) extracting the batch from our "batch map" using
the native thread id and (2) safely extracting the value from a `QueueBatch`
before falling through to `Object#size` or 0.
This PR presents a getting started guide to working with Logstash in Kubernetes, and comes in three parts:
- A walkthrough of a sample scenario, setting up a Kubernetes cluster with
- Elasticsearch
- Kibana
- Filebeat to read Kubernetes API server logs and sent to Elasticsearch via Logstash
- Logstash
- Metricbeat to monitor Logstash
and walks through the setup of these files
- An annotated guide to these files
- Explanations of how Logstash configuration and settings files map to the Kubernetes environment
Relates #14576Closes#14576
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Starting with Log4j2 2.6 if a subclass of MessageFactory associated with an Logger instance
is not subclass of MessageFactory2, then it's wrapped with MessageFactory2Adapter.
This trigger a log4j warn log that, when a class subclasses LogStash::Plugin for example, is noisy and report about
a Logger is not associated with the default MessagedFactory (LogstashMessageFactory) every time a subclass of Plugins is instantiated.
This commit adapt LogstashMessageFactory to implement the MessagedFactory2 instead of the older MessageFactory to avoid the wrapping with the adapter class.
* source/multilocal: fix detection of empty pipelines.yml
Fixes a regression introduced in elastic/logstash#13883 in which the presence
of an empty `pipelines.yml` file produces an error message indicating that
the file cannot be read.
When either `YAML::load` or `YAML::safe_load` encounter an effectively-empty
payload (such as one that is entirely comments), they use a `fallback` param
to determine what value to emit, with the former emitting `false` and the
latter emitting `nil`.
This is problematic because a _separate_ blind-`rescue nil` causes `nil` to
be bound to the MultiLocal's `@detected_marker`, and we assume that a `nil`
value in the marker means that there was an exception reading the file (such
as a permissions issue or parse failure).
By providing a `fallback: false` directive when parsing the contents, we
ensure that an empty file is reported as such.
* source/multilocal: avoid `rescue nil` that loses helpful context
When the pipelines yaml cannot be read, or can be read but fails to parse,
the MultiLocal#read_pipelines_from_yaml emits a helpful exception including
specifics about why it failed to load or parse, but a blind `rescue nil`
here causes that helpful information to be lost.
When pipeline detection is exceptional, hold onto the helpful exception
so that it can be reported along with the config conflicts.
* source/multilocal: differentiate between reading and parsing failure
* source/multilocal: use translations for conflict messages
* source/multilocal: specs for error conditions
doc for k8s troubleshooting and common issues
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Ensures the DRA build script surfaces a rake error, instead of allowing the build to continue.
This ensures that the build doesn't continue if any of the steps fails.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Version 7.17 doesn't generate Darwin aarch64 artifacts. Don't download these artifacts from the GCS bucket, given that we don't build Darwin for that release.
Exclude Jruby's bundler and rake from the built artifacts. The artifacts don't need to ship with such dependencies. Also, Logstash will bundle its own bundler for plugin management but it is not the one shipped with jruby.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes the source of dra_common.sh. It will now first check the directory of the file from which this dra_common.sh script is being called. This allows the common script to be sourced regardless of where the sourcing script is being called from.
The changes remove some code duplication by introducing a common file that can be sourced between all scripts. It also improves debuggability by adding better messages.
* Collect growth events and bytes metrics if PQ is enabled: Java changes.
* Move queue flow under queue namespace.
* Pipeline level PQ flow metrics: add unit & integration tests.
* Include queue info in node stats sample.
* Apply suggestions from code review
Change uptime precision for PQ growth metrics to uptime seconds since PQ events are based on seconds.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Add safeguard when using lazy delegating gauge type.
* flow metrics: simplify generics of lazy implementation
Enables interface `FlowMetrics::create` to take suppliers that _implement_
a `Metric<? extends Number>` instead of requiring them to be pre-cast, and
avoid unnecessary exposure of the metrics value-type into our lazy init.
* flow metrics: use lazy init for PQ gauge-based metrics
* noop: use enum equality
Avoids routing two enum values through `MetricType#toString()`
and `String#equals()` when they can be compared directly.
* Apply suggestions from code review
Optional.ofNullable used for safe return. Doc includes real tested expected metric values.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* flow metrics: make lazy-init wraper inherit from AbstractMetric
this allows the Jackson serialization annotations to work
* flow metrics: move pipeline queue-based flows into pipeline flow namespace
* Follow up for moving PQ growth metrics under pipeline.*.flow.
- Unit and integration tests are added or fixed.
- Documentation added along with sample response data
* flow: pipeline pq flow rates docs
* Do not expect flow in the queue section of API. Metrics moved to flow section.
Update logstash-core/spec/logstash/api/commands/stats_spec.rb
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Integration test failure fix.
Mistake: `flow_status` should be `pipeline_flow_stats`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Integration test failures fix.
Number should be Numeric in the ruby specs.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Make CI happy.
* api specs: use PQ only where needed
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* DRA: Handle env variables better
* Moved the addition of SNAPSHOT suffix to the version after the VERSION_QUALIFIER
* Fix badly assigned variable, version qualifier has to be appended also to PLAIN_STACK_VERSION and not RELEASE_VER
Co-authored-by: andsel <selva.andre@gmail.com>
The version passed to the release-manager doesn't need the SNAPSHOT particle because already handled by the --workflow="snapshot", if inserted make the release manager to search for artifacts named as 8.5.0-SNAPSHOT-SNAPSHOT
During graph composition vertices may be copied. This caused
getPostQueue to malfunction as the QueueVertex object stored in the
PipelineIR isn't the one present in the graph once it's fully generated.
This object mismatch caused Graph.getSortedVerticesBetween to not find
the QueueVertex since it takes Objects instead of ids.
This commit waits for the graph to be built and then retrieves the
QueueVertex from the graph and sets it in PipelineIR.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Avoid to leverage on git local commands to guess the local branch, it switches to listing the branches and checking against the the stack version. If doesn't exists it's main
Uses the locale-defined decimal style first.
When encountering a failure and the locale-defined decimal style is NOT
the "standard" decimal style, retry the parse operation with the "standard"
decimal style.
* specs: detangle out-of-band pipeline initialization
Our API tests were initializing their pipelines-to-test in an out-of-band
manner that prevented the agent from having complete knowledge of the
pipelines that were running. By providing a ConfigSource to our Agent's
SourceLoader, we can rely on the normal pipeline reload behaviour to ensure
that the agent fully-manages the pipelines in question.
* api: do not emit pipeline that is not fully-initialized
Handle the WORKFLOW_TYPE enviroment variable used to select the kind of artifacts to generate and consequently adapt the version name.
If the WORKFLOW_TYPE has a value assigned other then empty string it's assumed to be snapshot and so it generates snapshot artifacts else the release ones.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Update DRA scripts to use the version qualifier in stack_version variable for alpha and beta builds
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
- Adds a new method to the public API interface
- Pass the call through the JavaFilterDelegatorExt
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Extract the branch name passed to release-manager from version and not from git current branch
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* flow metrics: extract to interface, sharable-comon base, and implementation
In preparation of landing an additional implementation of FlowMetric, we
shuffle the current parts net-unchanged to provide interfaces for `FlowMetric`
and `FlowCapture`, along with a sharable-common `BaseFlowMetric`, and move
our initial implementation to a new `SimpleFlowMetric`, accessible only
through a static factory method on our new `FlowMetric` interface.
* flow-rates: refactor LIFETIME up to sharable base
* util: add SetOnceReference
* flow metrics: tolerate unavailable captures
While the metrics we capture from in the initial release of FlowMetrics
are all backed by `Metric<T extends Number>` whose values are non-null,
we will need to capture from nullable `Gauge<Number>` in order to
support persistent queue size and capacity metrics. This refactor uses
the newly-introduced `SetOnceReference` to defer our baseline lifetime
capture until one is available, and ensures `BaseFlowMetric#doCapture`
creates a capture if-and-only-if non-null values are available from
the provided metrics.
* flow rates: limit precision for readability
* flow metrics: introduce policy-driven extended windows implementation
The new ExtendedFlowMetric is an alternate implementation of the FlowMetric
introduced in Logstash 8.5.0 that is capable of producing windoes for a set of
policies, which dictate the desired retention for the rate along with a
desired resolution.
- `current`: 10s retention, 1s resolution [*]
- `last_1_minute`: one minute retention, at 3s resolution [*]
- `last_5_minutes`: five minutes retention, at 15s resolution
- `last_15_minutes`: fifteen minutes retention, at 30s resolution
- `last_1_hour`: one hour retention, at 60s resolution
- `last_24_hours`: one day retention at 15 minute resolution
A given series may report a range for slightly longer than its configured
retention period, up to the either the series' configured resolution or
our capture rate (currently ~5s), whichever is greater. This approach
allows us to retain sufficient data-points to present meaningful rolling
averages while ensuring that our memory footprint is bounded.
When recording these captures, we first stage the newest capture, and then
promote the previously-staged caputure to the tail of a linked list IFF
the gap between our new capture and the newest promoted capture is larger
than our desired resolution.
When _reading_ these rates, we compact the head of that linked list forward
in time as far as possible without crossing the desired retention barrier,
at which point the head points to the youngest record that is old enough
to satisfy the period for the series.
We also occesionally compact the head during writes, but only if the head
is significantly out-of-date relative to the allowed retention.
As implemented here, this extended flow rates are on by default, but can be
disabled by setting the JVM system property `-Dlogstash.flowMetric=simple`
* flow metrics: provide lazy-initiazed implementation
* flow metrics: append lifetime baseline if available during init
* flow metric tests: continuously monitor combined capture count
* collection of unrelated minor code-review fixes
* collection of even more unrelated minor code-review fixes
Downloads all artifacts generated for ARM and x86 by ci/dra_x86_64.sh ci/dra_aarch64.sh, position in locations expected by the release-manager and invokes it to upload to the global bucket.
Removing the -Djruby.jit.threshold=0 flag, which seems to have been introduced due benchmarking.
Removing the force of AOT means a noticeably faster startup (we do not need to 'compile' every method we bump into). The JIT threshold default is 50 in JRuby 9.2/9.3, there might be other heuristics in the future to better determine hot methods.
* refactor: pull members up from JavaBasePipelineExt to AbstractPipelineExt
* refactor: make `LogStash::JavaPipeline` inherit directly from `AbstractPipeline`
* Flow metrics: initial implementation (#14509)
* metrics: eliminate race condition when registering metrics
Ensure our fast-lookup and store tables cannot diverge in a race condition
by wrapping mutation of both in a single mutex and appropriately handle
another thread winning the race to the lock by using the value that it
persisted instead of writing our own.
* metrics: guard against intermediate namespace conflicts
- ensures our safeguard that prevents using an existing metric as a namespace
is applied to _intermediate_ nodes, not just the tail-node, eliminating a
potential crash when sending `fetch_or_store` to a metric object that is not
expected to respond to `fetch_or_store`.
- uses the atomic `Concurrent::Map#compute_if_absent` instead of the
non-atomic `Concurrent::Map#fetch_or_store`, which is prone to
last-write-wins during contention (as-written, this method is only
executed under lock and not subject to contention)
- uses `Enumerable#reduce` to eliminate the need for recursion
* flow: introduce auto-advancing UptimeMetric
* flow: introduce FlowMetric with minimal current/lifetime rates
* flow: initialize pipeline metrics at pipeline start
* Controller and service layer implementation for flow metrics. (#14514)
* Controller and service layer implementation for flow metrics.
* Add flow metrics to unit test and benchmark cli definitions.
* flow: fix tests for metric types to accomodate new one
* Renaming concurrency and backpressure metrics.
Rename `concurrency` to `worker_concurrency ` and `backpressure` to `queue_backpressure` to provide proper scope naming.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* metric: register flow metrics only when we have a collector (#14529)
the collector is absent when the pipeline is run in test with a
NullMetricExt, or when the pipeline is explicitly configured to
not collect metrics using `metric.collect: false`.
* Unit tests and integration tests added for flow metrics. (#14527)
* Unit tests and integration tests added for flow metrics.
* Node stat spec and pipeline spec metric updates.
* Metric keys statically imported, implicit error expectation added in metric spec.
* Fix node status API spec after renaming flow metrics.
* Removing flow metric from PipelinesInfo DS (used in peridoci metric snapshot), integration QA updates.
* metric: register flow metrics only when we have a collector (#14529)
the collector is absent when the pipeline is run in test with a
NullMetricExt, or when the pipeline is explicitly configured to
not collect metrics using `metric.collect: false`.
* Unit tests and integration tests added for flow metrics.
* Node stat spec and pipeline spec metric updates.
* Metric keys statically imported, implicit error expectation added in metric spec.
* Fix node status API spec after renaming flow metrics.
* Removing flow metric from PipelinesInfo DS (used in peridoci metric snapshot), integration QA updates.
* Rebasing with feature branch.
* metric: register flow metrics only when we have a collector
the collector is absent when the pipeline is run in test with a
NullMetricExt, or when the pipeline is explicitly configured to
not collect metrics using `metric.collect: false`.
* Apply suggestions from code review
Integration tests updated to test capturing the flow metrics.
* Flow metrics expectation updated in tegration tests.
* flow: refine integration expectations for reloads/monitoring
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Co-authored-by: Mashhur <mashhur.sattorov@gmail.com>
* metric: add ScaledView with sub-unit precision to UptimeMetric (#14525)
* metric: add ScaledView with sub-unit precision to UptimeMetric
By presenting a _view_ of our metric that maintains sub-unit precision,
we prevent jitter that can be caused by our periodic poller not running at
exactly our configured cadence.
This is especially important as the UptimeMetric is used as the _denominator_ of
several flow metrics, and a capture at 4.999s that truncates to 4s, causes the
rate to be over-reported by ~25%.
The `UptimeMetric.ScaledView` implements `Metric<Number>`, so its full
lossless `BigDecimal` value is accessible to our `FlowMetric` at query time.
* metrics: reduce window for too-frequent-captures bug and document it
* fixup: provide mocked clock to flow metric
* Flow metrics cleanup (#14535)
* flow metrics: code-style and readability pass
* remove unused imports
* cleanup: simplify usage of internal helpers
* flow: migrate internals to use OptionalDouble
* Flow metrics global (#14539)
* flow: add global top-level flows
* docs: add flow metrics
* Top level flow metrics unit tests added. (#14540)
* Top level flow metrics unit tests added.
* Add unit tests when config reloads, make sure top-level flow metrics didn't get reset.
* Apply suggestions from code review
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Validating against Hash test cases updated.
* For the safety check against exact type in unit tests.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* docs: section links and clarity in node stats API flow metrics
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@gmail.com>
The work done in #14355 to fix the keystore and plugin scripts erroneously included
%JAVA_OPTS% in the command to invoke `jruby`, rather than as an environment variable, which
prevented these scripts from operating correctly, due to `jruby.exe` not handling Java options,
causing the scripts to crash on invocation.
This commit removes the %JAVA_OPTS% - this is already set as an environment variable, and allows the
scripts to operate correctly
These are taken from the filebeat-modules doc with minor changes to
localise to winlogbeat and to remove the fully worked example.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Manually import CSV gem to work around thread leak
The version of CSV included in jruby distributions >= `9.3.0.0` and
< `9.3.8.0` include a bug that triggers a thread leak when calling
`parse_line` in CSV files.
This commit updates the version of the CSV gem to the latest version,
which includes fixes, and remains compatible with ruby 2.6
Relates: https://github.com/jruby/jruby/issues/7346
* Add CSV license info
Remove support for DES-CBC3-SHA.
Also removes unnecessary exclusions for EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC3-SHA and KRB5-DES-CBC3-SHA since there's already a "!DES" rule.
SettingsImpl.checkpointRetry is hardcoded to false in builder. Prior to this change, users are unable to set queue.checkpoint.retry to true to enable Windows retry on PQ AccessDeniedException in checkpoint.
Fixed: #14486
* Adds tasks to add bundled JDK to tar file used to run integration tests
* Uses `RUNTIME_JAVA_HOME` environment variable to control whether bundled JDK or
alternative is to be used
* Updates logstash service helper to respect value of `RUNTIME_JAVA_HOME`
* Requires updates to jenkins repo to set `RUNTIME_JAVA_HOME` correctly only for
integration tests that expect to use a custom version of Java, such as the JDK
matrix tests.
Update feature-request.md to provide a nicer description.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: hendry-lim <48344515+hendry-lim@users.noreply.github.com>
* pin jar-dependencies to 0.4.1
CI red with `Gem::LoadError: You have already activated jar-dependencies 0.4.1, but your Gemfile requires jar-dependencies 0.4.2`
* add comment
Open the access to java.lang and java.util packages in java.base module.
With Gradle 7.5 these packages aren't anymore implicitly opened in test tasks. With this commit, those packages are opened for all test tasks, also on the buildSrc.
Release branches are expected to have a lock file whose name is `Gemfile.jruby-X.Y.lock.release`, where `X.Y` is the Ruby language version.
This Ruby language version differs from JRuby version: for example, JRuby 9.2 is compatible with Ruby 2.5, while JRuby 9.3 is compatible with Ruby 2.6.
This commit makes the version bump agnostic to the version. Since there will be a lock file already present when the action is run, we only need to replace it, so there's no need to know the exact file name.
When using Bundler 2.3.19, doing a "bin/logstash-plugin uninstall <plugin>"
will crash, failing to find gems in the :build group.
Until we know more about why, pin bundler to 2.3.19
Documents the clean_consumed setting, which allows the user to remove processed/consumed DLQ events .
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Update the following java dependencies:
* org.reflections:reflections
* commons-codec:commons-codec
* com.google.guava:guava
* com.google.googlejavaformat:google-java-format
* org.javassist:javassist
The goal of these updates is to not fall behind and avoid surprises when an upgrade is necessary due to a security issue.
The logstash-integration-aws plugin replaces the underlying individual plugins related to AWS.
This commit also updates the necessary files around license checking.
* Fix version of java used to retrieve logstash version in integration tests
Prior to this commit, the system java would be used to retrieve logstash
version in integration tests, leading to test failures with IT environments
that have java 1.8 as system java
* Actually fix `test_port` this time
Use bash `/dev/tcp` to test ports rather than attempting to use `nc` and
`ruby`
* Apply suggestions from code review
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
Prior to this commit, the java filter delegator would return "java" to the
`#threadsafe?` call when determining whether a filter is considered thread safe.
This is correct for the _outputs_, where this prefix is used to construct the
appropriate delegator object, but not for filters, where a simple true/false is
required. This commit replaces the `getCurrency` call with a `isThreadsafe` call
to make it clearer that a boolean call is required.
This test has been broken on Windows since the jruby 9.3 update, and is
similar to an issue we saw when referencing other classes in the
org.logstash.util namespace
A new class of build node has recently been added which appears to have fewer
memory resources than previous nodes, causing integration test failures. The
gradle task for integration tests doesn't appear to require 4gb, so this commit
reduces the -Xmx setting to 4gb
When running certain integration tests, a test against a given port is
performed to ensure that certain dependent services are up. Currently,
these are tests are done either via `nc` or `ruby` if no `nc` is provisioned
on the build nodes. The current `ruby` implenentation attempts to use a system
ruby before using the ruby script shipped with Logstash. This commit removes the
use of the system jruby - certain build boxes are still using java8 as their system
java, which causes builds to fail, as java 11 is expected
The Module is broken with the current version. The Type needs to be changed from syslog to _doc to fix the issue.
* remove dangling setting and add arcsight index suffixes
* add tests for new suffix in arcsight module
Co-authored-by: Tobias Schröer <tobias@schroeer.ch>
* Fix windows scripts to also pull settings from JvmOptionsParser
Prior to this commit, the windows version of the keystore and plugin scripts
diverged from the bash version of these scripts, as they did not pick up Jvm Options
from the JvmOptionsParser, leading to certain mandatory settings not being picked up,
breaking compatibility with Windows on certain versions of the JDK. This PR ensures
that these scripts also use the JvmOptionsParser.
This does not resolve the issue described in #14176, which will be looked at in a future
PR
Relates: #14354
* Add goto :eof as per code review suggestion
Open the ability to use Ruby codec inside Java plugins.
Java plugins need subclasses of Java `co.elastic.logstash.api.Codec` class to properly work. This PR implements an adapter for Ruby codecs to be wrapped into a Java's Codec subclass.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Increments the counters of consumed events and consumed events every time a segment is completely read from the reader side.
Publish the collected data to the input plugin through the newly added method `segmentsDeleted(numberOfSegments, numberOfEvents` to `SegmentListener`.
Bugfix for DLQ when writer and reader with clean_consumed feature are working on same queue.
The writer side keeps the size of DLQ in memory variable and when exceed it applies the storage policy, so essentially drop older or never events.
If on the opposite side of the DLQ there is a read, with clean_consumed feature enabled, that delete the segments it processes, then the writer side could have a not consistent information about the DLQ size.
This commit adds a filesystem check when it recognizes that "DLQ full" condition is reached, to avoid consider false positive as valid data.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Adds the metric measure `expired_events` to node stats graph to track the number of events removed by age retention pruning policy.
In the _node/stats API, under the path `pipelines.<pipeline upstream>.dead_letter_queue` enrich the document with the new metric.
The number of expired events is incremented every time a segment file is deleted; this happens opening the file and counting the `complete` and `start` records.
Updates DLQ writer's writeEvent method to clean the tail segments older then the duration period. This happens only if setting dead_letter_queue.retain.age is configured.
To read the age of a segment it extract the timestamp of the last (youngest) message in the segment.
The age is defined as a number followed by one of d, h, m, s that stands for days, hours, minutes and seconds. If nothing is used then assumes seconds as default measure entity.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
<artifact_path> need to be hardcoded so it can be replaced properly by
the ubireleaser during the creation of the Ironbank merge request.
Relates to https://github.com/elastic/logstash/pull/14298/
* Forward port of 8.3.1 release notes
* Forward port of 8.3.0 release notes
* Forward port of 8.2.3 release notes
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* add `ca_trusted_fingerprint` to core features (monitoring/central-management)
* Rely on released ES output
* fix: ensure commented-out examples in logstash.yml are functionally correct
* add admonition for how to get a trusted CA's fingerprint
devtools/label: Add initial script to auto-label issues/PRs.
The first iteration of this script only allows users to add a version label based on the local branch version.yml.
This commit adds a rake task `rake artifact:dockerfile_ironbank` to generate ironbank docker build context for automatic release.
The output can be found in build/logstash-ironbank-$VERSION-docker-build-context.tar.gz
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
without these changes doing (an in-process) `bundle install`
did not generate a Gemfile.lock
this is than problematic, when using git sourced gem (locally) those gems do not not get resolved properly with the *qa/integration/rspec.rb* script
Uses the FileLock mechanism, already used in PQ, to force the single consumer pattern in case clean_consumed is enabled. This avoid concurrency problem of multiple readers trying to delete each other consumed tail segments.
* Timestamp#toString(): ensure a minimum of 3 decimal places
Logstash 8 introduced internals for nano-precise timestamps, and began relying
on `java.time.format.DateTimeFormatter#ISO_INSTANT` to produce ISO8601-format
strings via `java.time.Instant#toString()`, resulting in a _variable length_
serialization that only includes sub-second digits when the Instant represents
a moment in time with fractional seconds.
By ensuring that a timestamp's serialization has a minimum of 3 decimal places,
we ensure that our output is backward-compatible with the equivalent timestamp
produced by Logstash 7.
* timestamp serialization-related specs fixup
During upgrade, Logstash packaging removes /etc/systemd/system/logstash.service because Logstahs used to install the systemd unit there. However, this could have been modified by the user by overriding the systemd file. As such, the upgrade process should only remove if it is certain it wasn't installed by Logstash.
Unfortunately, there's no particular way to determine if Logstash installation process installed the file because it was done on the fly, by a script, on a postinst script of the package.
As such, the best way to address this is to check if both the new and the old file exists. If the new one doesn't exist, we can assume we are upgrading (or installing afresh), and the old file should be removed. If the new one exists and the user-create one (old location) also exists, we assume it is user created, and as such, we don't remove it just to be safe.
When an external repository reaches IO error or generates network timeouts the build fails in resolving external dependencies (plugins or libraries used by the project).
This commits increase a little bit those limits.
the use of ranges (e.g. {0..5}) or seq (e.g. $(seq 0 5)) may not
correctly in some systems, so let's just have a plain list of elements
for the loop to go through.
* Logstash checks different file system if each pipeline has a symlink to other filesystem.
* Apply suggestions from code review
* FileAlreadyExistsException case handling when queue path is symlinked.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit replaces the use of a block with a lambda as an argument for Stream.forEach.
This is to work around the jruby issue identified in https://github.com/jruby/jruby/issues/7246.
This commit also updates the multiple_pipeline_spec to update the test case for pipeline->pipeline
communication to trigger the issue - it only occurs with Streams with more than one event in it.
This commit changes the behavior of PQ size checking.
When it checks the size usage, instead of throwing exception that stops the pipeline,
it gives warning msg in every converge state if it fails the check
Fixed: #14257
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Updates the logstash.yml with the description of dead_letter_queue.storage_policy which was missed in PR #13923
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
With JDK 18, the Javac lint checking was expanded to raise an error on serializable subclasses that contains instance fields which are not serializable themselves.
This could be solved suppressing the warning or marking the field as transient. The majority of classes with this problem inherit transitively from Serializable but are not intended to be serialized (from Java's serialization mechanism) because has the serialVersionUID = 1L which is just a fix to make linter happy, but are not effectively used in a serialization context.
This commit also removes a finalize method, that could be safely be removed.
Introduce a method (markForDelete) in the DeadLetterQueueReader class as way to get acknowledged when events, read by the input plugin, could be considered eligible for deletion. This mechanism is used to delete the tail segments once they result completely consumed.
Other than this, exposes a listener interface, that the client of this class has to register to get notified when the deletion of a segment happened. This is useful for the DLQ input plugin to understand when it could flush the current position to the sinceDB.
This commit also delete the, eventually present, segments files that are older than the current position, and it happens during the setCurrentReaderAndPosition execution.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Make generated input plugin immediately buildable
* Apply the same changes to codec, filter, and output templates
* Uncomment TODOs in example gemspecs
Proposes a code cleanup of the FileLockFactory.obtainLock method. It removes the nesting of ifs using a "precondition style" check, lifting up the error condition, and checking on top of the code flow.
It removed a throw exception from the clean-up finally block.
If an exception is raised from that point it hides the original cause of the problem.
This commit switches the finally block, used in obtaining a file lock, to a catch and re-throw.
since JRuby 9.3 a useful inspect is provided out of the box
LS' inspect: <Java::JavaUtil::ArrayList:3536147 ["some"]>
JRuby 9.3's: "#<Java::JavaUtil::ArrayList: [\"some\"]>"
* Refactor: require treetop/runtime - avoids loading polyglot
* Build: instruct Bundler not to auto-load polyglot/treetop
+ Build: these deps are properly required as needed
all of them only used in one place (outside of normal bootstrap)
In jackson-databind 2.10, enabling Default Typing requires having a type validator, and while there's an "allow all" validator called LaissezFaireSubTypeValidator, this commit also tightens the validation a bit by narrowing down the allowed classes.
The default typing validator is only applied to the ObjectMapper for CBOR, which is used in the DLQ, leaving the one for JSON as-is.
Other changes:
* make ingest-converter use versions.yml for jackson-databind
* update jrjackson
* Fix deprecation logging of password policy.
Give users a guide to not only upgrading but also keep current behavior if they really want.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit adds a checkpoint for a fully acked page before purging to keep the checkpoint up-to-date
Fixed: #6592
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* add failing tests for Event.new with field that look like field references
* fix: correctly handle FieldReference-special characters in field names.
Keys passed to most methods of `ConvertedMap`, based on `IdentityHashMap`
depend on identity and not equivalence, and therefore rely on the keys being
_interned_ strings. In order to avoid hitting the JVM's global String intern
pool (which can have performance problems), operations to normalize a string
to its interned counterpart have traditionally relied on the behaviour of
`FieldReference#from` returning a likely-cached `FieldReference`, that had
an interned `key` and an empty `path`.
This is problematic on two points.
First, when `ConvertedMap` was given data with keys that _were_ valid string
field references representing a nested field (such as `[host][geo][location]`),
the implementation of `ConvertedMap#put` effectively silently discarded the
path components because it assumed them to be empty, and only the key was
kept (`location`).
Second, when `ConvertedMap` was given a map whose keys contained what the
field reference parser considered special characters but _were NOT_
valid field references, the resulting `FieldReference.IllegalSyntaxException`
caused the operation to abort.
Instead of using the `FieldReference` cache, which sits on top of objects whose
`key` and `path`-components are known to have been interned, we introduce an
internment helper on our `ConvertedMap` that is also backed by the global string
intern pool, and ensure that our field references are primed through this pool.
In addition to fixing the `ConvertedMap#newFromMap` functionality, this has
three net effects:
- Our ConvertedMap operations still use strings
from the global intern pool
- We have a new, smaller cache of individual field
names, improving lookup performance
- Our FieldReference cache no longer is flooded
with fragments and therefore is more likely to
remain performant
NOTE: this does NOT create isolated intern pools, as doing so would require
a careful audit of the possible code-paths to `ConvertedMap#putInterned`.
The new cache is limited to 10k strings, and when more are used only
the FIRST 10k strings will be primed into the cache, leaving the
remainder to always hit the global String intern pool.
NOTE: by fixing this bug, we alow events to be created whose fields _CANNOT_
be referenced with the existing FieldReference implementation.
Resolves: https://github.com/elastic/logstash/issues/13606
Resolves: https://github.com/elastic/logstash/issues/11608
* field_reference: support escape sequences
Adds a `config.field_reference.escape_style` option and a companion
command-line flag `--field-reference-escape-style` allowing a user
to opt into one of two proposed escape-sequence implementations for field
reference parsing:
- `PERCENT`: URI-style `%`+`HH` hexadecimal encoding of UTF-8 bytes
- `AMPERSAND`: HTML-style `&#`+`DD`+`;` encoding of decimal Unicode code-points
The default is `NONE`, which does _not_ proccess escape sequences.
With this setting a user effectively cannot reference a field whose name
contains FieldReference-reserved characters.
| ESCAPE STYLE | `[` | `]` |
| ------------ | ------- | ------- |
| `NONE` | _N/A_ | _N/A_ |
| `PERCENT` | `%5B` | `%5D` |
| `AMPERSAND` | `[` | `]` |
* fixup: no need to double-escape HTML-ish escape sequences in docs
* Apply suggestions from code review
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
* field-reference: load escape style in runner
* docs: sentences over semiciolons
* field-reference: faster shortcut for PERCENT escape mode
* field-reference: escape mode control downcase
* field_reference: more s/experimental/technical preview/
* field_reference: still more s/experimental/technical preview/
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
* Switch adoptopenjdk url to adoptium
Newer versions of the JDK are only available from api.adoptium.net, and not dual hosted on api.adoptopenjdk.net
This commit allows the use of adoptium versions of the JDK.
Relates: #14072
* Add support for ca_trusted_fingerprint in Apache HTTP and Manticore
Adds a module `LogStash::Plugins::CATrustedFingerprintSupport`, which can be
included in a plugin class to add a `ca_trusted_fingerprint` option to create
an Apache SSL TrustStrategy that can be used to bypass the TrustManager when
a matching certificate is found on the chain.
When run in debug mode, #invoke was returning an instance of UI::Shell rather
than a string, causing the plugin to crash when `<<` was called on.
This commit ensures that a string is returned regardless of whether debug is set
Fixes: #14131
When PQ is full, workers wake up writer thread in every read.
However, without removing a fully acked page, queue is still full.
This commit changes the condition of notFull signal.
Fixed: #6801
Updates the SignAliasDefinitionsTask Gradle task to express as input dependency the Aliases registry yaml file, so that when the input changes then a new signed registry is created.
* Add troubleshooting information around using certain versions of the JDK on Windows
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This commit updates the version of jruby used in Logstash to `9.3.4.0`.
* Updates the references of `jruby` from `9.2.20.1` to `9.3.4.0`
* Updates references/locations of ruby from `2.5.0` to `2.6.0`
* Updates java imports including `org.logstash.util` to be quoted
* Without quoting the name of the import, the following error is observed in tests:
* `java.lang.NoClassDefFoundError: org/logstash/Util (wrong name: org/logstash/util)`
* Maybe an instance of https://github.com/jruby/jruby/issues/4861
* Adds a monkey patch to `require` to resolve compatibility issue between latest `jruby` and `polyglot` gem
* The addition of https://github.com/jruby/jruby/pull/7145 to disallow circular
causes, will throw when `polyglot` is thrown into the mix, and stop logstash from
starting and building - any gems that use an exception to determine whether or not
to load the native gem, will trigger the code added in that commit.
* This commit adds a monkey patch of `require` to rollback the circular cause exception
back to the original cause.
* Removes the use of the deprecated `JavaClass`
* Adds additional `require time` in `generate_build_metadata`
* Rewrites a test helper to avoid potentially calling `~>` on `FalseClass`
Co-authored-by: Joao Duarte <jsvduarte@gmail.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit moves the JvmOptionParser into its own gradle project.
This enables the JvmOptionParser to remain compatible with Java 1.8 to present a helpful error message to a user attempting to start Logstash using older versions of Java, while allowing the main Logstash code base to freely use idiomatic Java 11 features.
* [Central Pipeline Management] Fetch pipelines with wildcard IDs from ES and apply. #14076
* URL encoding applied, code review feedbacks taken: unit test case for getting es version API and separate method for ES wildcard support.
* Pin json to avoid 2.6.2 until upstream jruby issue is solved
Version 2.6.2 of json will not install under jruby - https://github.com/flori/json/issues/497
This commit pins to only allow previous versions until the issue is resolved.
Adds some tests to proof that the offset position retrieved from the `RecordIOReader` effectively point to the start of the next available event and not to the last channel position retrieved.
Adds fixes for the problem, moving the concept of `channelPosition` to `streamPosition` differentiating the position on the stream from the position on the channel. However the already published interface (method getChannelPosition) is not renamed to avoid the introduction of a breaking change.
The recent commit #14079 to the DLQ would cause the `testSeekByTimestampWhenAllSegmentsAreDeleted`
to intermittently hang. This is due to files being deleted from the file system in the test,
but not from the internal list of segments. This would cause `pollEntryBytes` to loop indefinitely
looking for a deleted segments file. This commit also removes the segment path from the internal
segment list when a File is not found initially, and not just when a `NoFileException` is thrown
This PR is a refactoring of `open` invocation inside `RecordIOReader` of DLQ segments from the reader side.
After PR #13923, which when `storage_policy` `drop_older` mode is active delete segment files from the tail, it could happen that, also after a file existence check, the opening operation throw a `NoSuchFileException`.
It could happen in unlucky concidences, where the file existence check return a positive response, but in the meanwhile the upstream process decides to delete the file. This event has to be recognized as a "deletion of a segment", and managed.
If #stop is called on Puma before #run is finished - for example, when an
incorrect configuration is specified, Puma can be left with threads hanging
around.
This is due to a check/notify pipe used to signal state changes not being created
until halfway through the #run method, enabling a window where #stop can be called
before the pipe has been created, leading to the run thread not exiting. Prior to
jruby-9.3.x, this would tend to be benign and Logstash would exit normally. However,
jruby-9.3.x introduced a tear down as part of the shutdown mechanism, which will join
active threads on shutdown. This puma issue would cause an indefinite hang on exit
when this condition is triggered.
* Add mandatory option to jvm configuration to handle logstash startup issues
OpenJDK versions 11.0.15+10, 17.0.3+7 introduced new functionality to allow Java to enable
strict path checking. This included disallowing the use of `:` in any place other than directly
after the drive letter. Unfortunately, this check had the side effect of breaking compatibility
with special device paths, such as `NUL:`, which in turn, prevents Logstash from starting.
This feature was gated by the use of the `jdk.io.File.enableADS` property with a value of `true`
disabling the check.
This property was introduced with the default value of `false`, which prevents logstash
from starting in a Windows environment. While the next release is anticipated to set this
value to `true`, this commit explicitly sets that value to enable Logstash to be able
to start correctly.
Relates: #14042
Adds some metrics to the dead_letter_queue part of the monitoring endpoint _node/stats/pipelines/, precisely under the path pipelines.<pipeline name>.dead_letter_queue. The metrics added are:
- dropped_events: count the number of dropped events caused by "queue full condition", when drop_newer storage policy is enabled, happened to this DLQ since the last restart of Logstash process.
- last_error: a string reporting the last error registered for DLQ dropping condition.
- max_queue_size: like for PQ it's the maximum size that the DLQ can reach.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Updates the `internalReceive` method implementation in Pipeline Input to catch exception error and return the position where the stream was interrupted. Modify the EventBus's batch forwarding logic to handle errors from Pipeline Input and apply retry logic only from last error position in the batch of events.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
This reverts commit 478eb6834a.
AdoptOpenJdk is lagging behind Adoptium and we should move the
downloadJdk gradle task to use Adoptium instead.
Until then, we're reverting this change to use a version available from
AdoptOpenJdk.
Prior to this commit, attempting to use a custom java plugin installed from rubygems would
fail as the `files` field used to find jar files in the gem is not populated for such gems
This commit uses the `full_require_path` to find the jar file for a native jar plugin,
which should work regardless of where the gem was installed from, locally or via rubygems
Additionally, this commit will only attempt to locate the jar file inside the gem when it
is required (only when the experimental separate plugin class loader is used)
This commit builds on the work done in #13888
Relates: #13888, #11305
Co-authored-by: twosom <two_somang@icloud.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Exposes dead_letter_queue.storage_policy configuration setting to explicitly enable the drop_older behavior in DLQs.
Moving from a drop_newer to a drop_older behavior has impact both on the writer side and to the reader side.
The implementation leverage the fact that a complete DLQ segment can be removed to free up space; on the writer side when the dead_letter_queue.max_bytes limit is reached it has to remove old segments.
On the reader side, the consuming has to be adapted to don't expect a continuous flow of segments, it could face an hole due to removal of tail segments.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
v1 seems to have started failing with:
```
Run thollander/actions-comment-pull-request@v1
with:
message: 📃 **DOCS PREVIEW** ✨https://logstash_14019.docs-preview.app.elstc.co/diff
Note: if you get a "404!" please wait until the elasticsearch ci job finishes.
GITHUB_TOKEN: ***
Error: Resource not accessible by integration
```
Prior to the change, pipeline `stop` and `delete` happen in two converge cycles, which
has a gap letting the stopped pipeline compare with the same pipeline definition
in central pipeline management, hence Logstash see the stopped pipeline as graceful finish
and not to delete in registry
This commit creates StopAndDelete action to delete running pipeline in one converge cycle
Fixed: #14017
* Make AliasRegistry a singleton
The current implementation of AliasRegistry will create an instance of the Alias Registry for each
pipeline, which appears to potentially result in situations, such as in #13996, where multiple pipelines
are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with
the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when
reading the yaml file as the JarFile has been closed after the initial thread completed accessing.
This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry.
Relates: #13996, #13666
* Update reflections library to 0.9.12 to avoid multi threading bug
Earlier versions of the reflections library used in the plugin registry would
use caches on JarUrlConnection, which when closed would also close the jar file
for other resources using it, such as the AliasRegistry. This, combined with the
fact that the AliasRegistry could be created simultaneously by many threads/pipelines
could cause issues during AliasRegistry creation leading to failure to create a
pipeline.
* Avoid use of URLConnection caching when getting alias yaml resource
* Use idiomatic ruby when accessing Java getInstance method
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Updates the `ConfigVariableExpander.expand` to selectively create `SecretVariable` instances for SecretStore resolved environment variables.
`SecretVariable` instances in if statements are decrypted during `eq` `EventCondition` compilation; bringing the secret value and using in the comparator.
* A structural change for aliased plugin declarations to support doc header replaces.
* YAML file structure made to be more readable, make AliasPlugin class immutable.
* Do not expose alias doc header replaces since we are not using them. Unit tests failure fix.
As pointed out in this post merge review comment, there is a window
where we could miss a pipeline transitioning from 'loading' to 'running'
in the original fix, as separate calls are made to the pipeline registry.
This commit fixes that by making a single call to the pipeline registry which
allows for also returning pipelines in the `loading` state.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
When logstash is run without automatic reloading, it is still possible to reload configurations
by using 'SIGHUP'. This functionality was broken in #12444, which split non-terminated pipelines
into "loading" and "running" states. The call `no_pipelines?` in agent#execute would no longer
find pipelines in a "loading" state, causing the loop to exit, and logstash to shutdown. This
commit tests for pipelines in a "loading" state to restore functionality
Certain errors during pipeline converge will emit an IllegalStateException - for example attempting
to reference an environment variable that does not exist. Attempting to add a java exception to the
converge result would result in an uncaught exception in a pipeline thread leading to an unclean
shutdown.
This had the effect of,prior to this commit, Logstash behaviour varying depending on the class of
error that caused a pipeline not to start - an invalid pipeline configuration would still enable
logstash to start other pipelines in a multiple pipeline configuration, or automatic reloading
enabling changes to the configuration to allow the pipeline to start(in multi- and single pipeline
configurations).
However, a missing environment variable would cause Logstash to crash hard no matter what.
This was discovered when updating to jruby-9.3.3.0, when new clean up code joining existing
threads to perform a graceful shutdown was stuck indefinitely, due to the webserver shutdown
code not being called, due to the unclean shutdown.
Fix the class reference `Instrument::MetricStore::MetricNotFound` to be `LogStash::Instrument::MetricStore::MetricNotFound` and do not raise a constant not found reference when the point "/_node/stats" is queried and no pipelines are defined.
Removes the installation of sys-v init.d scripts from .deb and .rpms.
All supported operating systems use systemd as the init system, as such, it is no longer necessary to continue to ship Sys-v Init.d scripts.
The conservative flag is preventing dependences, (both from Elastic and 3rd party) to be updated.
This causes unnecessary lag behind new dependency releases.
Sets the LS_JAVA_HOME environment variable for the environments used to spawn Logstash process in integration tests.
The JDK matrix testing is based on selecting the desired JDK to run the tests, through the BUILD_JAVA_HOME.
However, when the integration tests spawn a Logstash process this setting was missed.
This commit guards against reading a value from a stat that may not be
populated before an attempt is made to retrieve its value, potentially causing
a `undefined method `[]' for nil:NilClass`.
This commit is a forward port of #13925.
Covered all calls to ES with retryable when accessing the central management to query the pipelines configurations.
Adds http-output to http-input as a Logstash-Logstash configuration option
Expands info on considerations for both http-http and lumberjack-beats to help users make the best choice
Co-authored-by: Andres Rodriguez <andreserl@ubuntu.com>
* [Java17] Add `--add-export` settings to restore JDK17 compatibility
After #13700 updated google-java-format dependency, it is now required to add a number
of `--add-export` flags in order to run on JDK17. This commit adds these flags to the
jvm options for a running logstash, and to the tests running on gradle to enable tests
to still work
* Move mandatory JVM options out of `jvm.options` and into JvmOptionsParser
Certain values for the JVM are mandatory for Logstash to function correctly. Rather than
leave them in config/jvm.options where they can be updated, and can cause upgrade issues
for users when we add new mandatory options, move them into code where they cannot be
changed
This commit changes `queue.checkpoint.retry` to `true` by default allowing retry of checkpoint write failure.
Add exponential backoff retry to checkpoint write to mitigate AccessDeniedExcpetion in Windows.
Fixed: #12345
Wraps the calls to the central management Elasticsearch cluster with the utility class Stud::Try to handle the remote host error when the client used to connect hit a not available node.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes the CentOS package installation to ensure that if both the systemd unit and sys-v init script are installed, the “service” command a prefers systemd.
* Update release notes for 8.1.0
* Forward ported release notes for 8.0.1 to 8.1
Forward port PR #13828 to main branch
Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit 31e06c0b1c)
Co-authored-by: Logstash Machine <43502315+logstashmachine@users.noreply.github.com>
Ruby allows methods to have default values in arguments if they're not
passed. However if a nil is passed then the default value isn't used.
The artifact:archives tasks were passing nil values to the exclusion
argument, causing all files to be included in the package.
This commit cleans the naming of the path lists and ensures the default
exclusion list is always used.
Extracts the bundled JDK's version into a one-line text file which could easily read and print from bash/batch scripts.
Updates Logstash's bash/batch launcher scripts to print the bundled JDK version when warn the user about his override.
Updates the rake script used to create Logstash artifacts (rakelib/artifacts.rake) to include JDK related stuff only if packages that requires it.
This change removes the usage of 'pleaserun', which
was leveraged create the init.d scripts or systemd units.
Now, we ship init.d scripts and systemd units to make
the maintenance easier and following the lead of the
other Elastic projects.
Fixes: #10519, Fixes: #12439, Fixes: #8253
1. Explicitly deny content changes in the workflow
2. Move comment body to env var
3. Restrict workflow to created comments
4. Restrict workflow to allow list of users
currently the artifact tasks compute the file listing from a list of
include regexes and exclude regexes. However this is done by hand,
taking each include regex and running it through each exclude regex.
This is quite slow as we add more exclude regexes. This PR changes to
totally relying on Rake::FileList, by feeding it the include and exclude
lists. This speeds up file listing from 150 seconds to 1 second.
Some Logstash tools invokes directly the JRuby intepreter. The interpreter uses the JVM pointed by two environment variables:
- JAVACMD
- JAVA_HOME\bin\java.exe
The setup.bat script exported the selected JVM under the env var named JAVA, which isn't recognized by vendored jruby.
This commit fixes it renaming to JAVACMD.
* artifacts: omit openssl_pkcs8_pure specs from built artifacts
* Exclude _all_ top-level spec and test directories from built artifacts
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
When a pipeline isn't fully initialized, we run the risk of attempting to
format pipeline info that isn't yet fully-shaped. By using safe-fallback
methods like `Hash#dig` and conditional-chaining, we can avoid the spurious
`NoMethodError` caused by sending `[]` to nil.
* api: avoid 5xx when stats/events not yet populated
* Add catching of metrics exception also for retrieve of :queue metrics. No pipelines means no queues created
* Added empty result when no pipelines is present and query for node info API
Co-authored-by: andsel <selva.andre@gmail.com>
A number of plugins reach into Logstash's i18n translations to "helpfully"
communicate certain configuration errors, but rely on translations that were
moved in 00a99c19e5 from logstash.agent to
logstash.runner. Since then, it is possible to hit obtuse error messages about
failing to load a translation instead of the intended helpful message:
~~~
translation missing: en.logstash.agent.configuration.invalid_plugin_register
~~~
By moving the `logstash.agent` definition to _after_ the `logstash.runner`
definition, we can use YAML tooling to name the `logstash.runner.configuration`
node and then merge its contents into `logstash.agent.configuration`. This
effectively allows us to keep a single definition of those translations while
making them available at both addresses.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* action: add docs preview link when docs change.
When a pull-request is opened, or when the HEAD of an existing pull-request is changed, and the changeset modifies the docs, add a comment to the PR with a link to our docs preview.
* Apply suggestions from code review
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Refactor: logstash boot to know what to do
* Refactor: reduce LOAD_PATH - load jars directly
* Refactor: unused requires in runner
* Refactor: boot - require libs when to be used
Fix gem installer tests to enable unpinning the version of bundler
This commit removes changes the gem installer to use real gems, rather than
use `allow_instance_of` during testing, which appears to be problematic with the
latest version of bundler
This commit deleted the corrupted zero byte PQ file and recreated the head checkpoint file to get rid of page file size is too small exception
Fixed: #10855
* docs: avoid promoting homebrew installation
As of Stack 8.0, Elastic is no longer maintaining a separate homebrew cask
with Elastic-licensed artifacts for the MacOS package manager homebrew, due to
low usage, difficulty in maintaining multiple versions, lack of team expertise,
and an existing Apache-licensed formula.
As such, we are removing instructions about setting up and running Logstash
from these formulae that are no longer available.
* docs: fix typo in installation instructions
The acceptance tests are used to test a variety of scenarios installing, running and updating
logstash on a variety of Linux distributions.
The vagrant distributions we test on are old, and use java8 by default. This commit removes
the java8 tests and changes the tests to only test the jdk included distribution.
During the time Logstash switched from a set of Ruby files plus one fat jar to multiple jars, with configuration files
and shell scripts that also needs to be updated. It's not anymore sufficient to suggest to unpack a new distribution
over an existing one. This is problematic when we add jars from version to version because drive to the pollution
of the classpath.
This commit redefines to steps to do this, in a safer way, backing up data and config folders.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com
Use same technique for Unix system to extract the CPU percentage and publish to Logstash's metrics collector.
It doesn't retrieve the full set of metric of a Unix system, but the ones that are available from internal JDK class com.sun.management.OperatingSystemMXBean
This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env.
The substitution happens after syntax parsing and before graph execution.
Fixed: #5115
Fixes an integration test that verifies the capabilities of CLI tool to install a not bundled plugin.
Move away from logstash-input-google_cloud_storage which depends indirectly to OS's package named shared-mime-info, which is not always available.
* logging: move init into environment's settings post-processor
Ensures that the non-runner command line utilities like `bin/logstash-keystore`
correctly initialize the logger as-configured.
* fixup: ensure we get ruby stdlib URI & File
* Remove separate JavaVersionChecker executable
The JavaVersionChecker is unnecessary as we already run the JvmOptionsParser before logstash,
providing a convenient location to test the Jvm version before running logstash
* Remove JavaVersionChecker class
* Check exit code from java options parser
* Add integration test for Elasticsearch using defaults
Currently, the elasticsearch integration test turns off `data_streams`
and `ecs_compatibility`, hiding the out of the box experience for
ingesting into elasticsearch
* Removed debugging statement
* field-reference: cap RUBY_CACHE to 10k entries
Reduces the scope of a memory leak that can be caused by using UUIDs or other
high-cardinality field names by preventing the ruby string _keys_ from being
held by the cache indefinitely.
Note: this may not solve the problem entirely, but certainly limits its impact.
Because ConvertedMap requires individual field names to be interned into
the global String intern pool, their eligibility for GC is JVM-specific
and high-cardinality field names should still be avoided.
* noop: field-reference test refactor to consolodate reflection
Cleanly teardown an integration test that made fall other integration tests.
In some cases the CI integration tests fails because the launched Logstash can't find a gem named `mimemagic`. This gem is installed during a CLI plugin test (install of `logstash-input-google_cloud_storage` plugins kicks in that `mimemagic`).
* Replace generated cleaner code with concrete implementation
No need to to perform codegen to create JDK specific compatible classes to
call the ByteBuffer cleaner, as we are dropping support for Java 8.
* Remove dead/redundant java 8 code from LogstashJavaCompat
Removes setupByteBuffer, specific test for Java9 and above and updates memory monitor
unit test to only test for values relevant to Java>1.8
* Renamed version utility class to reflect use after removal of methods
Also fixed some dead imports
When the Bash script executes the vendored Ruby it has to use proper `GEM_HOME` to avoid the overwrite that happens inside the logstash.lib.sh
3064f7d0c3/bin/logstash.lib.sh (L161-L165)
* Add pipeline.ecs_compatibility to the list
* Update docs/static/running-logstash-command-line.asciidoc
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* fix: respect LS_JAVA_OPTS environment even when optionsfile missing
* Fixed integration tests
* Added unit test to cover the fix
* Wipe commented code
* Removed redundant log in a path that could never be reached
* Moved jvm.options checks into only one place
* javaopts: provide injection point for environment string
Co-authored-by: andsel <selva.andre@gmail.com>
we've seen in the wild a situation where a keystore had a single invalid secret that couldn't be removed since that action did a retrieve before purging. The retrieve would fail causing the purge to never happen.
This commit skips the retrieval but checks if the secret exists to preserve the scenario of reporting an error when removing a secret that doesn't exist.
This commit fixes 2 tests
- Set queue.drain to true in pipeline pq test
- Under certain conditions the pipeline_pq_file_spec test would fail as the pipeline would exit once the generator had generated all of its events, but before the events were processed, leading to the test hanging. This commit adds `queue.drain:true` to the settings to ensure that all of the events are processed before the pipeline is shut down
- Increase the flush delay in dead letter quest testFlushAfterDelay test
- Under certain conditions, the flush delay of 1 second was insufficient, and invalidated a pre-condition assertion that no events had been flushed before the expiry of that delay.
Fixes the issue #8752 in event.out counter. When a pipeline contains a drop filter the total out events counter should count only the events that reached the out stage.
This PR changes CompiledExecution.compute() interface to return the number of events that effectively reached the end of the pipeline. This change is used in WorkerLoop to update correctly the event.out metric, instead of relying on the batch's size.
* add product origin header to license checks
* add origin header to Central Management config fetcher
* add origin header to ES output for Monitoring pipeline
Sometime the deep_replace could be invoked by plugins, using the LogStash::Config::Mixin#validate.
This method receives a Ruby hash which could contains Java ArrayList instead of Ruby Array.
The iteration method `each_index` is not available for ArrayList, so resort to some form of "plain old way".
The reason why an ArrayList is recognized as a Ruby Array is due to the override classes, like RubyJavaIntegration.JavaCollectionOverride that monkey patches Ruby Array, so that a Java Collection could be seen as a RubyArray but it doesn't implement all the abstractions, like `each_index`.
Co-authored-by: Karol Bucek <kares@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Forward port of #13598 to main branch. Original message:
* Update release notes for 8.0.0
* Updated release notes for 8.0.0-rc1
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Logstash Machine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Add notes to the Event sprintf docs about timestamp formatting to call out the
UTC nature of the Timestamp object.
Resolves: elastic/logstash#13112Closes: elastic/logstash#13571
After the fix of unlocking ecs_compatibility_support version in plugin update (#13218), `logstash-plugin install` has a problem of installing non default plugin.
This commit removes `Bundler.setup` in install path to avoid Gemfile froze by bundler
Fixed: #13404
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Provides a "friendly" error message when running Logstash with an incompatible
version of Java
Readded from #13356, incorporated with use of Comparable from #13358
* Update logstash docker to use ubuntu 20.04 base image
* Correctly set locale for ubuntu docker image
* tiny typo fix: ubunto -> ubuntu
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Docker integration tests stability improvements
This commit contains numerous fixes to improve the stability of the docker integration tests
* Patch Excon::UnixSocket
Socket.new running on arm64 on Ubuntu 18.04, causes an immediate SIGSEGV error and crash on
that OS, and, as far as I can tell, only that OS. `TCPSocket.new`,`UDPSocket.new` and
`UNIXSocket.new` do not. This commit patches the UnixSocket of the Excon library to
do the absolute simplest thing possible to avoid this error.
* Ensure that container is deleted even if #kill fails
* Add extra waits to handle the incremental way the payload returned by the monitoring
API increases as logstash starts up and pipelines load.
* Use pyenv to ensure the same version of python is used across different jenkins workers
* Add container logs to help diagnose failed test.
* Update the pipeline definition on multi-pipeline integration test
This was causing a pipeline to halt after startup causing intermittent test failures.
* Remove `;` to ensure failures are propagated appropriately
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This update incorporates:
* Latest logstash release on `7.15` branch
* Release of 8.0.0-beta1
* Update of 8.x snapshot to 8.1.0-SNAPSHOT
This follows the precedence of #10627 to update snapshot release of 8.x to 8.1 following pre-GA releases
of major branch.
* ecs: report pipeline's ECS-compatibility with INFO at startup
Because the pipeline-level setting `pipeline.ecs_compatibility` affects the
default behaviour of nearly every plugin in the pipeline, an INFO-level log
message will provide useful hints, especially to our users who upgrade to
Logstash 8 without first reading the breaking changes docs.
For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages:
> ~~~
> [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
> [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
> ~~~
* ecs: make v8 the default for 8.0
* ecs: `pipeline.ecs_compatibility` defaults to `v8`
Related: elastic/logstash#11623
* doc: temporarily remove deep link from breaking changes doc to fix build
With #13308 configuration namespace that started with `http.` was renamed to `api.`, this commit fix a usage left behind.
Use the new `api.enabled` setting in one place instead of the deprecated `http.enable`.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Add ability to pull the version used to build java from the logstash repo, rather
than rely on system Java. Previously, builds would use JAVA_HOME, now this setting
is ignored in Logstash (and by extension, parts of the Logstash build), which was causing
variations in the version of Java used to build Logstash, including the use of Java 8,
which the Logstash team would like to remove support for.
Relates: https://github.com/elastic/infra/pull/32818
* Forwardport #13358 to master: Add warnings for JAVA_HOME/older versions of Java
Forwardport PR #13358 to master branch. Original message:
* Add deprecation warnings for JAVA_HOME/older versions of Java
Logstash 8.0 will remove support for java versions before java 11, this commit
adds entries to the deprecation log warning against this.
Also adds use of `JAVA_HOME` to the deprecation log.
* Change deprecation log entries to warnings to indicate imminent issue
* Add JavaVersionChecker to check Java version for compatibility before running
Provides a "friendly" error message when running Logstash with an incompatible
version of Java
* Add version check to Windows
* Improvements
Improve readability of `JavaVersion`
Fix logstash bash script to exit on JavaVersionChecker error
* Fix bundler handling of 'without'
Prior to this change, the values set in `set_local` are ignored when invoking
bundler via the command line, as is used with `invoke!`. This commit sets those
values in `ENV` variables instead, fixing the functionality to not install
development gems.
* Update bundler spec to check ENV variable
* Added test to ensure kramdown gem not vendored
* Re-add set_local setting to play nice with `expand_logstash_mixin_dependencies`
* logstash service needs to be installed
* gem_vendored? needs to use full path to vendor files
* use `stdout` from `cat` command to generate spec temporary file
* Removed unnecessary support for supplying a block from #gem_vendored?
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* Update documentation around JVM usage to reflect changes
For Logstash `8.0`, java 8 will no longer be supported, and the documentation should
reflect that.
Update troubleshooting documentation - changes made in #13349 should remove the warnings
at startup for fresh installs of Logstash, but upgrading may still have the same issues.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This includes removing invalid GC logging options, instead using the same parameters
as Elasticsearch.
This also sets the `add-opens` to remove warnings for Java 11 and 17
* settings: add "deprecated alias" support
A deprecated alias provides a path for renaming a setting.
- When a deprecated alias is set on its own, a deprecation notice is emitted
but fetching the canonical setting value will reflect the value set with the
deprecated alias.
- When both the canonical setting (new name) and the deprecated alias (old
name) are specified, it is an error condition.
- When the value of the deprecated alias is queried, a warning is emitted to
the logger and only the value explicitly set to the deprecated alias is
returned.
Additionally, some relevant cleanup is also included:
- Starting Logstash with invalid settings no longer results in the obtuse "An
unexpected error occurred" with backtrace and exception data obscuring the
issue. Instead, a simple message is emitted indicating that the settings are
invalid along with the originating exception's message.
- The various settings implementations share a common logger, instead of each
implementation class providing its own. This is aimed to reduce noise from
the logs and to ensure specs validating logging do not need to tie so
closely to implementation details.
* settings: add password-wrapped setting
* settings: make any setting type capable of being nullable
* settings: add `Settings#names` to power programatic iteration
* cli: route CLI-flag deprecations in to deprecation logger
* settings: group API-related settings under `api.*`
retains deprecated aliases, and is fully backward-compatible.
* webserver: cleanup orphaned attr accessors for never-set ivars
* api: pull settings extraction down from agent
This net-no-change refactor introduces a new method `WebServer#from_settings`
that bridges the gap between Logstash settings and Puma-related options, so
that future additions to the API settings don't add complexity to the Agent.
It also has the benefit of initializing the API Rack App and just ONCE, instead
of once per attempted HTTP port.
* api: add optional TLS/SSL
* docs: reference API security settings
* api: when configured securely, bind to all available interfaces by default
* cleanup: remove unused cert artifacts
* tests: generate fresh webserver certificates
* certs: actually add the binary keystores 🤦
This commit applies all the changes needed to run Logstash on JDK 17:
- opens access to module java.base for packages sun.nio.ch and java.io to run the application and to execute the tests
- removes SecurityManager classes used during Logstash startup
- fix exception type catched in JavaKeyStore tampering test
Related to meta issue #13306
Sets `LS_JAVA_HOME` of the spawned logstash to use the same `java.home`
that the test is running under, rather than default to the system JDK, which
would result in the spawned logstash running under a different JDK to that
intended in the test
* add nanoseconds support
Migrates internals of `org.logstash.Timestamp` from legacy `org.joda.time.*`
which is limited to millisecond-precision to modern `java.time.Instant`,
allowing us to retain nanosecond granularity of `@timestamp` values.
Timestamps that are generated by Logstash (such as when creating an event that
does _not_ have a `@timestamp` field) will be generated at the highest precision
available to the JVM and/or platform (in many cases, this is microseconds).
Timestamps that are _parsed_ from user input will capture the entire provided
precision, up to and including nanosecond granularity.
Throughout the flow in the pipeline, including serialization to PQ, DLQ, and
JSON, will retain all available precision.
BREAKING: This produces an effectively-breaking change to the serialization
format of both the persistent queue (PQ) and dead-letter queue (DLQ),
as the serialized format this changeset contains a higher granularity
of timestamp than previous releases of Logstash were capable of
parsing without error.
As such, it _MUST NOT_ be back-ported to the 7.x series.
This commit fixes the `logstash-plugin update` command which fail to update plugin
that depends on a new version of logstash-mixin-ecs_compatibility_support.
It resolves logstash-* dependencies and puts them in bundler update command.
Fixed: #13181
update golang image to 1.17.1 to get rid of expired DST Root CA X3
disable download manager test cases to silent Faraday::SSLError
Fixed: #13261
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This PR integrates Elasticsearch bootstrap script to help users keep Logstah geoip plugin run without online update check.
Add `xpack.geoip.download.endpoint` option to config geoip database service endpoint.
Users can point to `http://localhost:8080/overview.json` when using the script to bootstrap nginx docker
Removes the usage of JAVA_HOME completely which is not anymore used for JDK path resolution.
Updated all the Logstash launching scripts to use only LS_JAVA_HOME as environment variable to
determine the JDK to use to launch Logstash. JAVA_HOME is abandoned for this job.
This commit:
- Updates the Gradle wrapper to version 7.2
- Remove the deprecated jcenter and where it was used to retrieve Gradle's plugins it switches to gradlePluginPortal
- Insert an explicit dependency from test task to the log4j.properties manipulation task ("copyProductionLog4jConfiguration") used in integration
This commit modifies the launch scripts to take care of the LS_JAVA_HOME giving precedence over the JAVA_HOME, which is still used it the first is not found.
Gradle 7 introduced a check for implicit task dependencies.
This PR moves to a fine grained copyAlias tasks to tighten output dependencies and to make those explicit.
Remove usage of plugin Gradle-License-Report to generate CSV report of used dependencies.
This commit replaces the removed plugin with the custom `ListProjectDependencies` task that inspects Gradle's project dependencies and produces a CSV file containing the list.
Update releases list, aligning with 7.14.0 and 6.8.18. With ES 7.14.0 and security on by default, this PR alsoswithed off security on ES in integration tests
This PR enables the upgrade of bundler to the latest version.
Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.
This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.
This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg
```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```
to facilitate quick and simple plugin testing.
This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.
This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.
## Release notes
* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment
As we close in on the availability of 8.0.0 alphas, we are reassessing which
breaking changes are _necessary_, and which are merely _desired_. And while
we would love to be in a world where ECS was on by default, and have put
substantial effort into designing an upgrade path that would be as simple as
possible, we have determined that the time may not be right to change the
default value out of under our users.
This change restores the default value for `pipeline.ecs_compatibility` to
`disabled`, ensuring pipelines will continue running in Logstash 8 as they have
in Logstash 7 without modification. We will still encourage our users to be
explicit about which behaviour they desire, and will revisit making ECS on by
default at a later date.
This PR contains commits attempting to fix the broken acceptance tests:
* Fix the set of test platforms used to run unix acceptance tests
Modernizes the list of OS's used in acceptance tests, to the most modern OS's available at https://app.vagrantup.com/elastic,. This removes the centos-6 platform from the build, which is past end-of-life and fails vagrant bootstrapping, causing the build to fail.
This is more of band-aid than anything - in the longer term, we should remove these vagrant based tests completely, and rely
on the build infrastructure to perform OS-based acceptance tests.
* Fix regexes for plugin list tests. …
Fixes tests to support the plugin alias feature. This introduced a new format for
entries emitted by `bin/logstash-plugin list`:
eg
```
└── logstash-input-elastic_agent (alias)
```
This commit fixes the test to account for this change, and whitespace variances.
The Gradle's configuration of task should be as fast as possible and don't break the build.
This commit moves retrieval of Elastic Stack version from the remote registry to the execution phase of the tasks.
Also the tasks that depends on this has received the same change (downloadEs and check EsSHA), moving from configuration to execution phase.
Close#13030
Starting with version 7.10.0 the name of LS packages changed, adding os and CPU architecture in the name. This change broke the downloading of those from the benchmarking tool. This commit fixes it, composing correctly the name, based on the version it has to download.
This reverts commit 149ee41a8b.
The Gradle "runIntegrationTests" has to download the necessary ES and Fielbeats and then launch the task :logstash-integration-tests:integrationTests.
:logstash-integration-tests:integrationTests is not intented to be launched directly
otherwise this caused a LogStash::ConfigurationError:
GeoIP Filter in ECS-Compatiblity mode requires a `target` when `source` is not an `ip` sub-field, eg. [client][ip]
* noop: avoid declaring default value in config file
* docs: ecs compatibility from 7.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
* ecs: on by default
We know that ECS version 8 will release along-side Logstash 8.0, but its scope
is still coming into focus. In this changeset, we change the default value
of `pipeline.ecs_compatibility` from `disabled` to `v1`, which is a
significantly closer approximation to what will eventually ship in Logstash
8.0.0.
* docs: ecs from 8.x perspective
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Because the "Fatal Error" specs specifically inject fatal errors during
execution, and do so by reacting to a "poison" event, the fatal error prevents
the poison event from being ACK'd in the underlying queue.
By specifying a one-off temporary data directory in these specs and cleaning up
after ourselves, we ensure that a PQ containing un-ACK'd events isn't leaked to
the next spec to run.
* fpm to 1.13.0 which allows building packages with java 11 + jruby 9.2
* childprocess to 4.x + remove monkey patches
* clamp to 1.x to unlock fpm 1.13.0
Loads the production plugin_aliases.yml definition file and check that every alias has
a properly published gem on RubyGems.
Adds clean up of plugin_aliases.yml files
Fixed task dependency for copyPluginAlias
This PR makes the Windows logstash.bat exit with the last %ERRORLEVEL% at the end, so that any error in running Logstash will get propagated back to the command line.
Before this change, logstash.bat would always exit with code 0 - success (when doing cmd.exe /C logstash.bat), even if the java.exe process exited with a non-zero code (e.g. due to Logstash throwing an error at runtime).
Uses the OS defined path separator in Rake script to invoke the gradlew command. Without this the sh('./gradlew assemble') results in error when running .\gradlew clean installDefaultGems.
* spec: noop refactor of xpack central management
* spec: validate central management settings loading
* central management: allow pipeline.ordered and pipeline.ecs_compatibility settings
Adds a filter to Reflections library initialization so that when it scan "org.logstash.plugins" it includes only .class files and avoid to load and process AliasRegistry.yml and plugin_aliases.yml
Fixes#12992
Avoid the creation of log4j routing appender for log events without the `pipeline.id` fishtag.
In this way no spurious log file named "pipeline_${ctx:pipeline.id}.log" and logs are not duplicated with main Logstash log file.
Added test to cover the installation of aliased plugins when exists a gem with same name but that's not a Logstash plugin.
In this case the alias is resolved to the original, skipping the gem retrieved from RubyGems.
Remove an useful dynamic creation of appender's log file which leverages the `log.format` property
also when it's explicit by the appender itself.
Log4j configuration leverages the placeholder `${sys:ls.log.format}` to compose the name of the log file.
This generates some not evident conflicts in log4j internals, these conflicts became evident when enabling the `pipeline.separate_logs` feature is enabled and the log4j appender definitions contains both json and plain format.
The problem is that under those circumstances the rollover of the log file doesn't happen.
This commit also add a test against the production log4j configuration, to avoid future regressions.
This commit avoid to check for existence of jar files to decide if run or not Gradle assemble,
basically because the outputs of assemble task are not only jars but also others files, for example plugin-aliases.yml.
In this way the decision to execute or not is left the Gradle logic.
The module LogStash::PluginManager requires the file `lib/pluginmanager/plugin_aliases.yml` is created,
which happend during the Gradle's 'copyPluginAlias' executed as part of Rake's 'bootstrap'.
Remove hard coded alias definitions in favor of yaml descriptor file.
Introduce a single point of aliases definition (logstash-core/src/main/resources/org/logstash/plugins/AliasRegistry.yml), checksum and copy it around to be used by Logstash and by Logstash's plugin management tool.
The descriptor yml file contains a checksum to verify it's not changed accidentally in a deployment of Logstash, if the verification phase fail Logstash avoid to start and plugin management tool avoid to operate.
The signing and copying around is managed by a specific Gradle task invoked during the build.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes#12831
In the List a Plugin docs, we suggest users to create an issue to request we list community plugins. This commit improves that page by adding a template to create an issue.
logstash-keystore integration tests spawn a Java process, which by default uses the system JDK generally exposed with JAVA_HOME environment variable. It could be that this JDK is not the one selected with the build system variable BUILD_JAVA_HOME.
This commit uses the JDK defined in BUILD_JAVA_HOME if present.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Since we default to data streams in 8.0.0, the internal collection for monitoring stops working due to the ES output trying to send data to the logs- data stream instead of the monitoring endpoint.
* Add logstash-integration-elastic_enterprise_search to plugins-metadata.json
* Remove old elastic_app_search plugin and set integration as default
* Add license information for workplace search gem
ubi8 image uses microdnf as a package manager, and microdnf does
not support the "yum clean metadata" command. This commit adds
the logic to skip this command if the image_flavor is ubi8
On aarch64, yum does not pick the correct 'bind-license' package,
this commit installs a specific noarch RPM
This commit also adds retry to the yum installs and updates.
This pinning made sense when logstash-mixin-aws would depend on aws-sdk-v1 which forced json ~> 1.
The aws mixin v5.0.0 only depends on v2 now, so we can remove this.
Adapted install/uninstall/list PluginManager's CLI commands to respect the alised plugins
- adapt install plugin to resolve an alias giving precedence on a real plugin
- changed list to mark alised plugins
- uninstall avoid to remove the alias and ask the user to remove the original plugin
- update update the original plugin in case of alias, else fallback on usual behavior
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Introcuce the concept of alias for a plugin.
Creates an AliasRegistry to map plugin aliases to original plugins.
If a real plugin with same name of the an alias is present in the system, then the real plugin take precedence during the
instantiation of the pipeline.
Simplified the error handling in class lookup
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This PR changes the behavior of copying license files from .tgz
Originally, only two files, MaxMind LICENSE.txt and COPYRIGHT.txt, are required
Now more files, README.txt and Elastic ToC, are potentially required
Instead of targeting the files, this change copies all content in .tgz
Removed static part from PluginRegistry to avoid static initializer, transforming it into a singleton. Reflections library is not thread safe in jar archives scan, so there must be only one instance of PluginRegistry
This commit contains two fixes
* Fix Date class clash when used in pipelines with Date filter and GeoIP
* Pinned jruby-openssl version 0.10.5 to avoid SSL errors
* Fix: missing password dependency require
which causes `bin/logstash-keystore` to fail with an error:
```
ERROR: Failed to load settings file from "path.settings". Aborting...
path.setting=/logstash-7.12.0/config, exception=NameError,
message=>uninitialized constant LogStash::Util::Password
```
* Fix: review all LS parts depending on Password
* Test: bin/logstash-keystore create/list
Upstream `ElasticsearchOptions#es_options_from_settings` already uses the
setting `elasticsearch.ssl.verification_mode` to produce an appropriate
boolean-valued `ssl_certificate_verification` in our `es_settings` hash, so
we can rely on it instead of re-checking equality with a string.
Since the introduction of this block:
```
"pipeline" : {
"workers" : 16,
"batch_size" : 125,
"batch_delay" : 50
},
```
to the node stats API, the benchmarking tool has been broken. This commit fixes the
tool, and updates the payload in the tests to reflect the current payload.
Prior to this release a VERSION_QUALIFER env set to an empty string
would create versions looking like `8.0.0--SNAPSHOT` instead of
`8.0.0-SNAPSHOT`, causing the release manager builds to fail.
Ensure we always prefix the version number with 'v', so it is added to both, the original and the backport PR.
Co-authored-by: Andres Rodriguez <andres.lazo@elastic.co>
JRuby had a few releases where it shipped with bundler,
creating some difficulty in working with newer versions.
This no longer happens so we can remove these exclusions
from the jruby unzipping task.
This commit fixes up some IT flakiness which has been presenting mostly
in recent DLQ test failures, it includes the following improvements:
* A recent change to Elasticsearch has required the cluster setting
`action.destructive_requires_name` to be set to `false` to enable the use
of destruction actions with wildcards. This commit sets this before
tests on Elasticsearch and DLQ tests
* Adds some extra safety to the `have_hits` rspec matcher
This commit avoid an error in gathering monitoring information when webserver is disabled or is not yet started;
which could happen with slow loading pipelines or no pipelines defined from the central management UI.
This change fix the behavior of considering as "running" also pipelines that are still in "loading", both "loading" and "running" is considered as "not terminated".
Fixed a flakyness in tests due to different ways to looks at the same thing: pipeline status.
The pipeline status is determined by both `pipeline.running?` and by `agent.pipelines_running`.
The first checks for an atomic boolean in pipeline object, the second check for the status in PipelineRegistry.
Fixes#12190
Avoid the deletion of downaloaded ES artifact during copyEs task and download a new version only if the SHA512 of the local copy differs from the what is retrived from remote repository.
This avoid unusefull download of the same ES artifact when run integrationTests task multiple times.
This commit updates the dockerfile template to support environment
variables being used to retrieve the architecture appropriate logstash
build, in the same way as is currently done for the Elasticsearch docker build.
This is required to support the official dockerhub builds of Logstash.
Relates #12578
Docker container integration tests relating to the java process were
failing due to the introduction of the new JVM option parser. This
commit waits for logstash to start before testing that the logstash
java process is being run as expected
* This PR allows the agent to stop pipeline by pipeline_id instead of fetching the full set of pipelines from elasticsearch and compute the pipeline actions internally
Fixed: #12560
* Fix sub second `config.reload.interval`
Due to a change in #11803, using `to_seconds` to normalize values of `config.reload.interval`
would resolve to a value of 0 causing issues in tests where short reload intervals were desired.
This commit uses the `to_nanos` method to preserve the previous functionality.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
- moved parsing of jvm.options file into Java code
- chnaged the parsing code to consider conditional notation to bind the applicability of certain JVM flags to specific JVM versions
- changed the launch scripts (.sh and .bat) to use the options string composition
- binded CMS flags to JVM specifications 8-14
- replaces all scripted filters with custom Java implementation
- implemented routing appender per pipeline in Java
- adapted log4j configuration shipped with Logstash
- exit the Logstash process if it detects an scripted log4j configuration
When the PQ creates a new page and allocates a memory-mapped buffer, the
underlying file is zero'd out to full page capacity and the version byte is
written to the buffer.
If Logstash crashes or is shut down before any elements have been pushed into
the queue page, we have no guarantees that the version marker has been
persisted to the storage device. A subsequent attempt to load an all-zeros
queue page will result in an obscure error message and failure to load:
~~~
AbstractPipelineExt - Logstash failed to create queue.
org.logstash.ackedqueue.io.MmapPageIOV2$PageIOInvalidVersionException: Expected page version=2 but found version=0
~~~
By sending `MappedByteBuffer#force()` immediately after the version has been
added to the buffer, we can shrink the window in which a crash can leave the
queue on disk in a corrupt state.
Currently, LS does not respect fatal errors such as java.lang.OutOfMemoryError and continues executing.
This is dangerous since JVM errors are a legitimate reason to halt the process and not continue processing.
Additionally:
- make sure we log the full stack-trace on fatal errors
- halt the JVM wout executing finalizers/hook (scissors on how ES handles uncaught exceptions)
- also, we should now be aware of a potentially unexpectedly dying thread
This commit updates the license information for the license dependency report.
Specifically, this adds a notice for `racc`, a different version of which is now
pulled in by nokogiri from the version included with jruby.
* plugin: adds `:validate => :field_reference`
Provide plugins a way of validating that an input is a literal field-reference.
This is useful for input plugins that implement a `target` or other
non-interpolated directive, and allows these plugins to reject invalid
configuration before start-up instead of at run-time.
Plugins should not use this named validator directly, as doing so would cause
validation to fail with "Unknown validator" when the plugin is run on older
releases of Logstash. Instead, plugins should use the `validator_support`
adapter mixin that provides back-ports when necessary.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
these labels are required for redhat openshift certification.
These commit reintroduces the labels for the ubi8 image only, and adds
acceptance tests to ensure these labels are correct and not inherited
The ModulesSettingArray is responsible to obfuscate password in arrays of settings.
The test are still in Ruby to proove the interoperability with Ruby code that used the previous version.
Added method to mimic .first and .last methods of Ruby Array
There is two Password classes that almost does the same thing. One in Ruby (LogStash::Util::Password) and one in Java (co.elastic.logstash.api.Password).
This commit drop the the Ruby implementation to import the Java version in the LogStash::Util so that existing Ruby code haven't to be changed, works as it is.
Removing the freeform description labels left the container metadata
without a description label. This commit adds a description under the
"org.opencontainers.image.description" label
This commit fixes two issues with the docker metadata:
Removes non-OCI compliant freeform metadata labels
Uses a consistent build date for all the docker images and dockerfiles
Additionally, this commit adds a `build_docker_ubi8` rake task to enable
`ci/docker_acceptance_tests.sh` to run with no options to build all
docker images for the architecture.
Where available, this commit addS information from getSourceWithMetadata to the
error message of UnexpectedTypeException, dropping down to `toString`
if not, giving more context to find where the issue is caused in the configuration.
This commit adds context to the pipeline to pipeline input and output
plugins by adding a string containing the `address` field to the input
plugin, and an array containing the `send_to` field to the output plugin.
This helps gain a picture of how pipeline to pipeline enabled configurations
are communicating with each other, without having to refer back to the pipeline
definition
add wildcard support in xpack pipeline id
do the pattern matching with glob
add warning msg to wildcard with legacy api
check invalid pipeline in bootstrap
test cases for invalid checking
Restructures troubleshooting docs in preparation for expanding content.
Adds info for plugin tracing to help users track down plugins that might be causing problems.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes: #12228
Moves Cloud info to Configuration section to make it more obvious and easier to find
Expands content for using cloud id and cloud auth outside of modules
Moves module-specific info into modules section
to avoid gems being resolved from the usual LS GEM_HOME
this is problematic for gems such as jruby-openssl which are loaded
during boot (by RGs/Bundler) and thus activated in Bundler from a
different GEM_HOME. if such gem is updated it won't end up being
install-ed in the --path location as it's found on the GEM_HOME!
+ Fix: gem conflict 1.3.6 required by core
this is due now isolating GEM_HOME on `bundle install --path`
+ Refactor: we do not need LS_GEM_HOME/PATH
+ avoid pinning jruby-openssl to 0.10.4
resolves GH-12299 (reverting GH-12300)
* Add optional sourceURL to license report CSV
This commit adds an extra optional column 'sourceURL' to the license report. This
column contains a pointer to the source code, which is optional for most dependencies,
but a requirement for some, such as the Red Hat Universal Base Image.
This commit also populates the 'copyright' field, which previously was an used
column in the CSV definition
Relates #12297
This addresses an incomplete fix in #12019 starting in 7.8.1 where upon catching a worker exception (to avoid crashing the whole logstash per #12306) the input plugin(s) are not terminated prior to closing the pipeline leading to the input plugin(s) continuing execution and failing with IllegalStateException & Tried to write to a closed queue since closing the pipeline also correctly closes the queue.
When deleting temporary files created by the DLQ writer to store data before moving to their
final location, Windows may leave these files in a "delete pending" state, where the files
are somewhat in a state of limbo, where they result of `Files.exist(filename)` is `false`,
but the result of `filename.toFile().exists()` is true. When files are in this state, a new
file with the same name cannot be created, which causes the DLQ test used to ensure that
closing and reopening the DLQ (in such events as a pipeline restart) to fail.
This commit moves the temporary file to an alternative location before deletion, ensuring that
the "pending delete" status does not interrupt with the DLQ startup
Add devtools: backport and create_pr
Adds two scripts to help developers automatically create a PR:
* backport: Allows back porting commits to any other branch. It uses cherry-pick.
* create_pr: Automatically creates a PR from your local branch.
* Write DLQ entries to temp file first
This commit changes the DLQ writer to write to a temporary file
which will be renamed on "completion", to avoid the possibility
of the DLQ reader reading an incomplete DLQ segment. The temp file
will be renamed and made available, either when the capacity of this
segment is reached, or if a configurable 'flush interval' has elapsed
since the last event reached the dead letter queue.
This commit fixes#8022, #10275, #10967
This commit replaces #11127
The `shouldRunAfter` specified in the main script body was causing the runIntegrationTests
task to be evaluated even when it should not have been, causing unnecessary failures
when artifacts required only for integration tests are unavailable.
This can be removed, because the `shouldRunAfter` relationship for the `runIntegrationTests`
task is already defined in the task body.
* replace direct hidden indices access with system indices api
* fulfill backward compatibility
* fix log msg, rename class, simplify response handling
* modularise fetcher
Implements a plugin `ecs_compatibility` option, whose default value is powered
by the pipeline-level setting `pipeline.ecs_compatibility`, in line with the
proposal in elastic/logstash#11623:
In order to increase the confidence a user has when upgrading Logstash, this
implementation uses the deprecation logger to warn when `ecs_compatibility` is
used without an explicit directive.
For now, as we continue to add ECS Compatibility Modes, an opting into a
specific ECS Compatibility mode at a pipeline level is considered a BETA
feature. All plugins using the [ECS Compatibility Support][] adapter will
use the setting correctly, but pipelines configured in this way do not
guarantee consistent behaviour across minor versions of Logstash or the
plugins it bundles (e.g., upgraded plugins that have newly-implemented an ECS
Compatibility mode will use the pipeline-level setting as a default, causing
them to potentially behave differently after the upgrade).
This change-set also includes a significant amount of work within the
`PluginFactory`, which allows us to ensure that pipeline-level settings are
available to a Logstash plugin _before_ its `initialize` is executed,
including the maintaining of context for codecs that are routinely cloned.
* JEE: instantiate codecs only once
* PluginFactory: use passed FilterDelegator class
* PluginFactory: require engine name in init
* NOOP: remove useless secondary plugin factory interface
* PluginFactory: simplify, compute java args only when necessary
* PluginFactory: accept explicit id when vertex unavailable
* PluginFactory: make source optional, args required
* PluginFactory: threadsafe refactor of id duplicate tracking
* PluginFactory: make id extraction/geration more abstract/understandable
* PluginFactory: extract or generate ID when source not available
* PluginFactory: inject ExecutionContext before initializing plugins
* Codec: propagate execution_context and metric to clones
* Plugin: intercept string-specified codecs and propagate execution_context
* Plugin: implement `ecs_compatibility` for all plugins
* Plugin: deprecate use of `Config::Mixin::DSL::validate_value(String, :codec)`
This commit adds the ability for the docker build to build artifacts for multiple architectures.
By default, the target architecture is inferred from the architecture of the machine the build is being
run from - running the build from an aarch64 machine will build an aarch64 docker image, while building
from an x86_64 machine will build an x86_64 docker image.
This can be overridden by setting the environment variable DOCKER_ARCHITECTURE to either `x86_64` or
`aarch64`.
This commit also updates the integration tests to test against the architecture from the machine the test
is being run on, and includes the target architecture in the test description.
Changed Linux creation artifacts (tar.gz/deb/rpm) to include the ARM JDK.
Extracted common parts of artifact.rake into functions to be shared between ARM and Intel bundling tasks
Elevates visibility of Offline Plugin Management section so that air gapped users
don't have to struggle through instructions that require an internet connection.
Related: #12280
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
This commit includes the required changes to pass RedHat docker image certification.
This includes:
Moving license files to /licenses folder
Adding required base labels for name, description, vendor and summary
Relates: https://github.com/elastic/dev/issues/1287
* Fix keystore thread safety
This commit is intended to fix thread safety issues with the JavaKeystore implementation of the secret store.
From reading the code, it appears that thread safety for the keystore was intended to be provided by
a ReentrantReadWriteLock, a read lock for accessing secrets from the keystore, and a write lock for updating
secrets in the keystore.
In practice, this was insufficient, the act of accessing a secret from the keystore involved the mutation of
a shared keyStore object - the keyStore is `load`ed every time a secret is retrieved from the store.
Previous to https://github.com/elastic/logstash/pull/10794, this did not matter, each pipeline held its
own instance of the secret store, effectively meaning that only a single thread would ever access a key
store at any one time. This PR moved to using a shared keystore instance for substitution variables,
exposing the lack of thread safety in the JavaKeystore class.
This commit is intended to be the simplest change to fix the underlying issue, and does not address whether
we *need* to reload the secrets every time they are read.
Relates #12229
when running a pipeline with ordered execution, flushes on the pipeline
were no longer being called when compute is called with an empty batch, causing
issues with the aggregate filter, for example, not being able to push events on
timeout.
Expose the proxy xpack management proxy setting in docker (xpack.management.elasticsearch.proxy).
Also surface the same proxy setting in the sample config.
This commit adds the rake docker_ubi8 rake task, and associated
changes to the docker template and makefiles.
This commit also refactors the acceptance tests to extract xpack tests
into a helper class to allow the same tests to be used in both 'full'
and 'ubi8' docker image tests
Our internal representation of the composite config file needs only to inject
newline delimiters if they are missing, and to avoid doing so if they are
present. This allows `PipelineConfig#sourceReferences()`, used to map back from
the composite line/column to source file/column, to correctly track an offset
using the source fragments `SourceWithMetadata#getLinesCount()`.
Fixes: #12155
Accidentally succeeding at connecting to an HTTP resource that is not a real,
live Elasticsearch (such as an Elastic Cloud instance that has been shut down
and reaped) can cause client initialization to fail.
* Introduce integration tests for docker
This commit adds integration tests for the Logstash docker images. Previous
integration tests were removed in https://github.com/elastic/logstash/pull/10693,
due to the tests being non functional.
The commit adds image and container tests. The image tests check the contents and the
metadata of the image; the container tests check the logstash process, and includes tests
ensuring that logstash runs, and is configurable.
This test also adds a ci script to allow the tests to be run on jenkins, and to split the
running of these tests up based on the image type and includes updates to the rake tasks to
support this.
During the development of PR #11541 to direct ship monitoring data to an monitoring ES cluster without hopping through a production ES cluster, the settings for elasticsearch ouput was cloned into a version without the `xpack` prefix.
Since that feature has been removed the settings should also be removed from the Docker image
In PR #11799 we missed to add the exposure of proxy also as docker env variable so that uses can connect the dockerzied Logstash to a proxed monitoring cluster
Prior to this commit, the value of `org.label-schema.license` and
the values in `org.opencontainers.image.*` were not set, and therefore
would be inherited from the base OS image.
Although fossa has a fossa init tool to auto discover dependencies,
it doesn't work well for Logstash.
The mix of JRuby and Java allows for correct gradle detection but for
Ruby we tell FOSSA to look at the lockfile, which we generate using
ci/bootstrap_dependencies.sh
This is a work in progress and covers 99% of our dependencies.
As we get comfortable we'll have to uncomment a few ruby subprojects
contained in the logstash source tree.
Integration plugins need a different header. For example, the plugin docs should
to point to the integration repo rather than the input, output, filter, or codec
repo. The new header also includes boilerplate text to indicate that the individual
plugin is part of an integration rather than stand-alone. This work implements needed
changes.
This is a temporary fix.
Currently the check task depends on the integrationt tests task,
which means all dependant tasks will be resolved even if they're just
registered instead of created.
This resolution is a problem because the downloadES task will fail
if, for the version we're building, Elasticsearch doesn't yet have a
build we can download.
So for now we'll remove this to unblock builds, but finding a way
to compartimentalize failures is needed going forward
fixes a regression introduced with the api_key support for xpack monitoring and management in #11864 which disabled the possibility to not use any authentication by relying on the default options and only enabling monitoring for example. It now ignores the default username option when no password is explicitly set.
A pipeline in the process of being created was not marked as such in the pipeline registry resulting in a situation where a slow to initialize pipeline could be recreated on state convergence resulting in a PQ LockException because that pipeline was already existing and held the PQ lock. Replace native Java concurency with Ruby Mutex for simpler and straighforward implementation.
The worker threads were not correctly monitored for a worker loop exception resulting in a complete logstash crash upon any exception even when multiple pipelines are running. Now only the failed pipeline is terminated. If pipeline reloading is enabled, it is possible to edit the config and have that failed pipeline reloaded.
Release Manager builds were failing as `downloadEs` task was being
needlessly run during `rake artifact:all` task. When run with
`RELEASE=1`. this was causing build failures due to the non-availability
of Elasticsearch release artifacts. This commit aims to avoid running
the `downloadES` task when it is not needed, continuing the work done
in #11914
This commit also removes code that was repeated in different parts of
the build script.
This commit updates the kafka setup scripts to ensure that the kafka setup is clean between builds, by
setting an explicit zookeeper data directory to be cleaned each time, and correctly overriding `log.dirs`
instead of `log.dir` to ensure that the kafka logs are written and wiped in a consistent place each time,
which helps when using the non-immutable images used in arm64 tests.
This commit clears the `JAVA_HOME` variable when starting Elasticsearch
to force it to use the bundled version of the JDK, rather than the
default `JAVA_HOME` from the machine Logstash integration tests are being
run on, and removes the likelihood of tests failing to run due to `JAVA_HOME`
being set to a non-compliant JDK.
Many DLQ and PQ tests were run with default settings for their size,
or otherwise set to values such as 1GB.
This meant that the container/machine the test ran on needed a lot of disk space
(i.e. 1GB for the test + 1GB for OS and Logstash + some more free space).
This commit drops the disk space requirement overall by a factor of 10 (e.g. 1GB to 100MB)
this test only called "pipeline.start" and expected an output to receive
the "do_close" method call. Therefore it relies on a race condition that
the pipeline shuts down quickly enough, which could fail.
This change ensures the pipeline fully terminated before making the
assertion
This commit also removes an unused let from the time when logstash
output plugins had workers.
This test relies on the generator input sending 10 events and observing
the output. Calling shutdown immediately after start can cause the
plugin to abort earily.
This change waits for the execution to finish before performing the
test assertions.
The creation of a Ruby thread from Java seems to be a trigger
for jruby/jruby#6207.
Pipeline#shutdown now blocks on the ShutdownWatcher#start, which will wait for
pipeline.finished_execution? to be true.
This removes the need for the pattern:
`pipeline.shutdown { block } && pipeline.thread.join`
And can be replaced with just `pipeline.shutdown`
To avoid having `shutdown` blocked waiting for ready? when pipeline crashes too quickly,
this method returns immediately if finished_execution? is true.
Most uses of pipeline#run have also been replaced by pipeline#start
since the latter will block until the pipeline is ready, again avoiding
the pattern:
`pipeline.run && sleep 0.1 until pipeline.ready?`
Pipeline tests have been changed according to these two changes.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
The reduction from 500 to 100 is based on observations where 06d7f01fd
reduced the number of generated classes by about an order of magnitude
especially on very large pipelines (e.g. from ~600 to ~30).
Generated implementations of `Dataset` often have fields referencing
other specifically-generated `Dataset`, despite only using public methods
(`compute` and `clear`) defined on the `Dataset` interface.
By allowing the code generation to reference the interface instead of
the specific implementation, we eliminate the need for the compiler to chain
parent class loaders indefinitely, thereby eliminating the need for a global
mutual exclusion when compiling.
This change moves the locking semantics from the compiler to the non-evicting
cache itself, relying on tried-and-true `ConcurrentHashMap#computeIfAbsent`
to minimize synchronization and cache-priming stampedes.
This also vastly reduces the scope of `Dataset` implementations that we need
to generate, because datasets will no longer need to reference the specific
implementation details of all "downstream" datasets and will therefore be more
likely to match an implementation that has already been compiled and cached.
Integration tests may fail during elasticsearch teardown, as currently
the stop_es function sends a `SIGTERM` to Elasticsearch, but does not
wait for the process to exit. That can lead to issues when deleting
data directories from a still running process. This commit adds
wait functionality to `stop_es` to wait for a short period of time,
sending a `SIGKILL` if Elasticsearch does not terminate in time.
* plugin config: support space-deliminated URIs on list-type params
Since whitespace is illegal in URIs, we can safely use it as a delimiter when
validating `list`-type `URI` params, enabling the expansion of an arbitrary
list of URIs from a single Environment- or Keystore-variable.
Resolves: https://github.com/elastic/logstash/issues/8157
Resolves: https://github.com/elastic/logstash/issues/6366
* Doc: Create section for cross-plugin functionality and add space delimiters
Co-authored-by: Karen Metts <karen.metts@elastic.co>
Changed ComputeStepSyntaxElement to generate Java code to retrieve the plugin's id by a method instead of hardcoding the value in the generated code.
This permit to share more compiled classes, that differs only by plugin.id and speed up the pipeline compilation.
The change has been secured by future regression with unit test that track pipeline compilations times.
Co-authored-by: Andrea Selva <andsel@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes: #12031
Having the jar around would allow us to fine tune logging for libraries
such as manticore's http-client (4.5) using LS's `log4j2.properties`
e.g.
```
logger.apache_http_headers.name = org.apache.http.headers
logger.apache_http_headers.level = DEBUG
```
... to log http headers for each request
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Enable filebeat and elasticsearch downloads to pull different
architectures, Filebeat and Elasticsearch use different suffixes
to denote their aarch64 architectures, with beats using arm64 and
elasticsearch aarch64
Carrying on from the work done in #11958, update the gradle build to download
the same version of Elasticsearch as is specified in the logstash version.yml file.
This commit updates the standard integration tests to use the same version of
Elasticsearch that is already downloaded for x-pack integration tests, and also
fixes integration tests to allow for the different responses around hits generated
by different versions of Elasticsearch.
JSON.load allows the creation of complex objects, and should not
be given untrusted input. This commit changes the only three uses
of JSON.load in the codebase, which aren't user facing or present
in bundled product, so not really an attact vector.
This commit changes the download to pull the version of beats based on the version pulled from the branch rather than from an environment variable, or 6.5.4.
This commit also moves the download logic of Filebeat fromfilebeat_setup.sh to build.gradle in order to use the artifacts API in the same way as the downloadEs task, and does some refactoring to DRY up the artifact download tasks.
This commit also fixes the beats integration test to replace the use of a removed setting.
This commit also sets retries to 3 for the download tasks, using 'retries' functionality from gradle download task plugin
This work breaks out the JVM setting info into a new section, and
expands and updates the content. It adds new subheadings to make
scanning the content easier.
The LS reference needs a place to put conceptual info that transcends details for individual settings. This work creates and links the section, and adds content for pipeline ordering and Java pipeline initialization time.
ruby produces a LocalJumpError: unexpected return
error if there's a return in a block so this changes just uses
the value of the last expression as the value of the block
Try system ruby, then LS_HOME/bin/ruby, then relative path from
script to LS_HOME/bin/ruby. Use LS_RUBY_HOME variable to avoid
testing again on subsequent attempts to wait for port.
This adds the .ci/matrix-runtime-javas.yml file that defines all
the JDKs logstash could be tested against. This is meant to be
used for the Matrix Combinations Jenkins plugin to be able to
select which JDK to test against dynamically.
Some QA tests reads the FEATURE_FLAG environment variable, for example to test PQ functionality.
This PR passthrough the environement variable inside the Docker instance.
Avoid to reassing the subdocument for queue metrics preferring a merge
With PR #10576 the PluginsStats.report(stats) overwrites the subsection related to queue instead of merge with newly created entries.
When running filebeats integration tests on centos-7, the tests
fail due to permsisions checks on the temporary configuration file
created for the test. This commit sets strict permissions checks
to false in order for the tests to be able to succeed.
A previous commit attempted to fix this issue by adding Shellwords.escape to setup_script and teardown_script locations, but File.exists? returns false when called against a filename escaped by Shellwords.escape. This commit localizes the escaping to where the
file is executed.
This commit also adds Shellwords.escape to teardown script runner and the method used to execute logstash to retrieve version. This is to enable tests to run correctly when Jenkins creates execution environments with folders named with &&, eg centos-7&&immutable
Fixture test scripts use `nc` to wait for the port to determine
whether a test fixture is up and running. This commit adds a fall
back option to use ruby if `nc` is not available - it is not installed
on Jenkins centos worker nodes.
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
The 'prepare_offline_spec.rb' is failing due to a change in the warning message
from JDK11 to JDK14, and JAVA_TOOL_OPTIONS being passed in as an environment
variable by Jenkins, which was not happening before due to the dockerized
environment.
Escape test fixture service scripts to avoid test failures when run in
Jenkins using multiple yaml configuration files, which causes directories
to be constructed like `centos-7&&immutable` which cause issues with
the service runners cutting off directory locations before '&&'
To be unlinked from infra and made the selection of OSes that Logstash is tested against, the list is moved .ci specific yaml files.
By now the OSes are distinguished by Windows platform and by Linux distributions.
Separated selection of OSes in CI for compatibility and normal tests
Kafka teardown script can exit with failure, typically when trying to
stop the broker. This commit logs the error code if the scripts fail
rather than crash out causing build failure.
Fixes#11905
Conversion to seconds of values under 1_000_000_000 nanoseconds translates to value 0, and this led to NaN when used as denominator in a division.
A value of 996_920_400 nanoseconds once converted to seconds is not rounded to 1 second by to 0, this manifest on Windows OS
`RubyString#gsub` requires a (Ruby) frame to be present.
The method attempts to set a backref for the current caller's frame.
When the frame stack is empty there isn't really a place to set $~.
This can happen when a LogStash::Util::Loggable#logger is retrieved,
from the input worker thread while not being nested in any block.
Fixes#11874
for Strings with copy-on-write semantics when deep cloning.
motivated by "big" events reaching plugins such as split,
which might produce several new events out of a single one.
Fixes#11794
https://github.com/elastic/elasticsearch/pull/56211 removed the ability
to turn off certain features in Elasticsearch. This commit removes
the setting of `xpack.monitoring.enabled`, as this is now
obsolete.
Often when browsing logstash logs for debugging purposes we miss the
information about the Java version and platform being used.
Printing the global RUBY_PLATFORM gives us all of this information
plus the JRuby version as well.
Fixes#11852
* Fail fast when integration test fixtures cannot be setup.
Currently when integration test fixture setup fails, the tests proceed as usual,
resulting in noisy logs where the source of the error is not easy to discover.
This commit will raise an exception and log the exit status if the script fails to
setup or teardown.
In time comparison of LocalDateTime the isBefore is strict, so in case two instants has the same millisecond, it fails in test (happens in Windows tests)
Close: 11862
logstash_admin role is not enough.
As the ls-security page mentions correctly:
"The user you specify here must have the built-in logstash_admin role as well as the logstash_writer role that you created earlier"
Updates static settings for extra role needed
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
* Avoid using deprecated `getSystemCpuLoad` method with JDK14
JDK14 prefers the use of `getCpuLoad` over `getSystemCpuLoad`. This commit
reworks the call to use reflection to use the appropriate method call
depending on the version of the JDK being used.
* Added warning message if OperatingSystemMXBean not available
* Also corrected spacing after code review comments.
In some workflows such as simple file manipulation, starting a webserver is
unnecessary overhead, and we should be able to avoid it.
Here we introduce a new parameter `http.enabled`, which defaults to `true` to
maintain the existing functionality.
Resolves: elastic/logstash#9408
Closes: elastic/logstash#11525
Co-authored-by: Benoit Dupont <benoit.dupont@gmail.com>
Fixes#11533
We have "required" units for a variety of `TimeValue` settings when they are
provided as a `String`, but unquoted values in YAML have been passed through as
Integers, where we long assumed nanosecond units. This frequently leads to
surprise (e.g., when `config.reload.interval` is set to `60`, we consume 100%
of CPU in a tight loop trying to reload and re-parse the configs every 60
nanoseconds).
By making the setting retain the TimeValue object for the entirety of its
lifecycle, we can issue a deprecation notice the first time an Integer value is
encountered. As a secondary benefit, our usage of the setting value in code
becomes more clear since we are empowered to ask `TimeValue` for a numeric
value in a specific scale.
Fixes#11803
* [Doc] added description of xpack.monitoring.collection.write_direct.enabled setting
* Added page to mark as deprecated the legacy internal collector and fixed all the `xpack.monitoring.*` references
* Included legacy collector file into monitoring overview
* Restructure monitoring docs
* Incorporate review comments
Co-authored-by: andsel <selva.andre@gmail.com>
The separator vertices are an implementation detail of the serialized
output of the LIR, and are not meaningful to the pipeline viewer.
This commit removes the separator vertices, and reworks the edges to
account for this.
Fixes#11779
In the docs templating, the plugin name is used to autogenerate a code example of how
to configure specific plugin. As such, if a plugin name is different from how you
configure it, this results in an example of how to configure this plugin with an
incorrect name.
This changes the java_sink plugin name to sink to correctly autogenerate the example.
Fixes: #11675, Fixes: #11214
Loading a Java Keystore can take anywhere from ~0.3s to upwards of 3s, so the
pattern of loading one per variable we need to replace adds a significant
amount of overhead on pipelines that use these variables, whether or not they
are provided by the keystore.
By providing a private, constant, lazy singleton, we ensure that we don't
incur the cost of repeatedly building the keystore.
Fixes#10794
* Update gradle version to 6.3
Gradle versions prior to 6.3 cannot run under JDK14.
This commit upgrades the version of Gradle to 6.3, and removes all deprecation warnings that can currently be removed.
Changes include:
* Increase gradle memory to 2g
* Increase gradle memory in the license check job to 2g
* Replace use of `testCompile`
* Replace `runtime` with `runtimeOnly`
* Remove`compile` depedencies from gradle files
* Replace deprecated archive methods
* Fix dependencies report build
* Make jruby dependencies 'api', fix archiveVersion
* Set `duplicatesStrategy` for all tasks of type Copy
* Use `configureEach` for global 'withType' calls
** Use the recommended Tasks API calls
(https://blog.gradle.org/preview-avoiding-task-configuration-time)
* Run `./gradlew wrapper` earlier to improve caching
* Use copy with chown for resources that need to be run during `./gradlew wrapper`
cleanup RubyArray "rawtypes"
remove all LinkedHashSet from batch and queue classes
avoid processing empty batches in Java worker loop
cleanup AckedReadBatch and MemoryReadBatch
refactor Ruby worker loop similar to Java Execution to not use batch merge
remove QueueBatch merge and replace LinkedHashSet with ArrayList
while also making the array case cleaner & effective
(JRuby uses specialized array holder for 1 / 2 values)
+ Refactor: minor - use true/false constants directly
+ Refactor: do not allocate empty array
Fixes#11732
- changed `2 seconds` to `2s` for consistency
- exchanged *trace* with *debug* time values and vice versa to be referable to the example above
Fixes#11671
.ruby-version is used to select the external jruby
(for package building + acceptance tests on infra)
reverts the upgraded JRuby version from #11647Fixes#11663
there's no need for this and makes code base inconsistent
... also the original intent seems no longer relevant :
was introduced at 57e7a8a56b
> allows for a massive simplification for the invocation of filters and
outputs from the Java execution
Fixes#11587
reuse rubyArray for single element batches
rename preserveBatchOrder to preserveEventOrder
allow boolean and string values for the pipeline.ordered setting, reorg validation
update docs
yml typo
Update docs/static/running-logstash-command-line.asciidoc
Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>
Update docs/static/running-logstash-command-line.asciidoc
Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>
java execution specs and spec support
docs corrections per review
typo
close not shutdown
Ruby pipeline spec
aligns the Ruby/Java returns as they happen in scripted Java
e.g. as `java.lang.Thread.new` returns a JavaProxy instance
there's really no reason to use JavaObject which always needs `to_java`
conversion to be useful (and is considered legacy in JRuby).
considered breaking change e.g. `LogStash::MemoryReadClient#read_batch`
will now return a proper JavaProxy instead of the JavaObject
Fixes#11391
Since Logstash may not be cloned into a “logstash” directory (e.g. on Windows ci we mount it as A:/) gradle needs to be told the name of the top level project.
this is fairly recent - since 7.4 (added in GH-11075)
there's a risk plugins would assume ThreadContext to
exist or collide the 'global' constant - usually best
to import where the Java class actually gets used ...
Fixes#11356
Currently, the version of the elasticsearch gem bundled with Logstash
requires the type field to be set, which is incompatible with
Elasticsearch 8.x after types were removed.
use "127.0.0.1" instead of "localhost" to avoid binding to ipv4 and ipv6
don't assume port 10006 will be open in the machine
rely on the ranges and the actual bound port for the assertions
Fixes#11263
Certain malformed field reference literals (e.g., those containing a series of
multiple open-brackets `[[`) were propagated undetected by the parser, only to
create a crashing error when used.
Starting Logstash with the `--config.test_and_exit` flag (or `-t` shorthand)
would validate the config, even though it could not be used in practice.
By updating the grammar(s) to exclude the use of an open square bracket (`[`),
we more closely match the formal grammar and ensure these malformed literals
are rejected closer to the source.
NOTE: this PR only affects field reference _literals_, and does not resolve
a similar issue with field references in quoted format strings.
Resolves: https://github.com/elastic/logstash/issues/11022Fixes#11195
YAML.parse returns Psych nodes that then need to be converted to plain ruby objects.
Calling YAML.safe_load outputs basic ruby objects already and also increases security as it greatly restricts the classes it deserializes.
Fixes#11208
Quite often we see log entries that are truncated by this limit since java stack traces can be very verbose.
This prevents us from seeing the real issue and require us to ask for users to remove the limitation and trigger the issue again so we can see the full problem.
This commit removes this truncation.
Fixes#11206
This commit clarifies that Logstash monitoring metrics should not be
routed through master-only nodes, and should instead prefer coordinating
nodes.
Fixes#11194
Previously we'd only give a pipeline the settings related to pipelines
The PipelineSettings class was used for this.
However a pipeline may need other settings like the keystore location.
For this we instead clone the settings object and merge all the pipeline
specific settings. This is accomplished with a new method that ensures
that only pipeline level settings are overwritten in the clone.
Fixes#11076
Fixes formatting in a table cell in `logstash-monitoring-overview.html`.
A `+` which was required by AsciiDoc was leaking into the output when
the doc is built with Asciidoctor.
In the "methods" sections of the how to develop a plugin docs
Asciidoctor as incorrectly passing backticks into the output when it
should have marked the words surrounded by backticks as code. I'm not
100% sure why it did that. The fix is to force macro evaluation
immediately on attribute assignment.
Users following our documentation are frustrated to discover that they get 403 errors from Logstash, even when following the instructions to the letter. The problem is that the `create` privilege is missing. With this in place, it works as designed.
These changes may need to be back ported to previous branches, too.
Fixes#11013
* Starting to audit tests
* Additional field checking in stats
* Add epehemeral id
* More tests
* Test new structure of pipeline report
* Add default_metadata testing
* Add node command tests
* add jvm
* test no mutate
* Add check for graph flag
* Break apart test per review suggestion
* Remove test that doesn't test much
By removing the default plugins from the Gemfile.template
the current task that modified the template was not working correctly.
This commit either replaces the dependency entry if it exists or
otherwise creates it.
- have `bootstrap` task do as little as possible: install gems in Gemfile.template that don't belong to groups
- have test tasks depend on the `installTestGems` task instead of `bootstrap`
- logstash es output is now a dependency because of license checking
- fix out of memory problem in SharedHelpers.trap
Also use release lockfile during installDefaultGems
The release lockfile is only copied to Gemfile.lock if
it doesn't exist. During the `installDefaultGems` task other
plugin installation tasks already occurred, generating a lock file.
This commit removes it before running the plugin installation.
* Create running-logstash-windows.asciidoc
Initial commit for #4005
* Update running-logstash-windows
1. Added section to validate JVM pre-requisites and shell sections for nssm, task scheduler, and PowerShell
2. Updated options to run Logstash on Windows, update section headers
3. Clarified JVM pre-requisites and included example to add environmental variables using SETX
4. Added example Logstash configuration, added steps for running Logstash manually with PowerShell
5. Removed `WIP` from the PowerShell section; updated the example to include output to Elasticsearch; Added notes for running Logstash as a service with NSSM
6. Removed `WIP` from the NSSM section; Added notes for running Logstash as a Scheduled Task; Added notes to stopping Logstash for each section; Removed `WIP` from the Scheduled Task section; Removed `WIP` from the page header
7. Updated initial section; moved the running manually section as the first configuration; added notes to the NSSM and Schedule Task sections.
8. Push headings down one level
9. Clarify this document contains examples for running Logstash on Windows. Updated which NSSM file should be extracted for use.
10. Updated formatting for the example Logstash configuration
11. Update formatting for the command examples
12. Update the instructions in the Task Scheduler section
13. Update the instructions in the run Logstash manually section, the NSSM section, and update formatting
14. Update formatting
15. Add note regarding support for running multiple pipelines
16. Clarify use of command line options. Re-state what is mentioned in the `Running Logstash from the Command Line` doc that: "Specifying command line options is useful when you are testing Logstash. However, in a production environment, we recommend that you use [logstash-settings-file] to control Logstash execution."
17. Clarify steps to accessing the Windows Environmental Variables window (i.e., link to Microsoft docs).
18. Remove unnecessary plus signs
19. Updated source types for examples, updated documents for specific Logstash versions with `{logstash_version}`
* Update running-logstash-command-line
1. Add note for running Logstash on Windows with `bin\logstash.bat`
2. Update formatting for running Logstash from the Windows command line
* parent 8c5697c748
author Guy Boertje <guy@elastic.co> 1556806171 +0100
committer Mike Place <mike.place@elastic.co> 1557234770 +0200
Bump JrJackson to 0.4.8
Fixes#10748
LIR serializer refactor
Remove commented code
Remove more commented code
Remove license and add encoding
Style change to make code more vertical.
eid and hash
Use pipelines_info to construct the stats
Add tests for new fields
Add queue stats
* bad merge resolution
* bad merge resolution
* Don't merge if nil
* Better merge strategy
* add vertex gate
* Guard against nil
* Use extended queue stats in pipeline report
* Add cluster uuids to Elasticsearch outputters in pipeline output
* move uuid
* remove old uuid lookup
* Only populate cluster_uuids when present
* remove print
* cluster_uuids -> cluster_uuid
* Update logstash-core/lib/logstash/api/commands/stats.rb
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Update logstash-core/lib/logstash/api/commands/stats.rb
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Update logstash-core/lib/logstash/api/commands/stats.rb
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Make var singular
* Match singular var name
* Remove unnecessary nil check
* Pass in the matching pipeline for the report
* Remove old way of inserting cluster_uuids
* Update logstash-core/lib/logstash/api/commands/stats.rb
I like this much better and in testing it seems to work correctly.
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Remove unreferenced code that was part of debugging
* Remove events var which was unused
* Don't try to remove before insert
* Update logstash-core/lib/logstash/api/commands/stats.rb
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Make pipeline extended stats generation more efficient
* Implement suggestion to improve readability
* Cleaner merging per review recommendation
* Only generate extended_stats once
* remove unneeded comments
* Add cluster_uuid to node vertex
* remove top-level cluster_uuids
* Update logstash-core/lib/logstash/api/commands/stats.rb
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Implement change to make logic more simple suggested in review
* Rely on options gate to insert graph
Resolves concern here:
https://github.com/elastic/logstash/pull/10576#issuecomment-501774635
* Update logstash-core/lib/logstash/api/commands/stats.rb
Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Move UUID lookup to API layer
* Move private method to bottom per review recommandation
No Elastic Search dependency is mentioned in the guide for Logstash up to this point. This would be good for those who are getting started to Log Stash without knowing much about Elastic Search and unaware that it isn't already packages along with the install of Logstash.
Fixes#10852
When using the Jruby event API, re-cast java exceptions produced by illegal
field references to ruby `RuntimeError`s, which can be caught by the ruby-based
plugins.
This is similar to what we already do in the Jruby event API when directly
handling field references, but catches a case where the `Valuifier` encounters
an illegal reference when creating a `ConvertedMap`.
Fixes#10839
now that default plugins are read only from the json metadata file and
not from the lock file, the plugin version manifesto needs to be adapted
so the status of "default" plugin is read from the json file.
Fixes#10824
Previously the `do_close` method would never be called on a failing
plugin, because the retry call stops the code under `ensure` from
ever being called.
Fixes#10691
We need a way for a plugin to register simple metadata about external
resources it connects to in order to implement a Monitoring feature in which
an Elasticsearch Output Plugin can store the connected cluster's uuid (#10602)
Here, we add a generic `LogStash::PluginMetadata` along with a registry, and
expose an accessor on `LogStash::Plugin#plugin_metadata` so that instances
can access their own metadata object.
Fixes#10691
In filebeat prospectors settings have been renamed to inputs. When
prospectors is used a deprecation warning is printed. With 7.0
`filebeat.prospectors` will be removed.
This change updates all uses of prospectors with inputs. For now
filebeat events report `prospector.type` and `input.type` for
compatibility reasons.
* dont include docker tasks in artifact:all
* don't rebuild tar/zip if source hasn't changed
* allow SKIP_PREPARE to avoid tar creation if no modifications
* don't need a tarball to generate the dockerfile
* remove docker tests as they weren't working anymore
This commit adds a task to produce all necessary files to generate a docker image.
```
% RELEASE=1 rake artifact:dockerfile
....
Dockerfile created in /tmp/elastic/logstash/build/docker
% tree /tmp/elastic/logstash/build/docker
/tmp/elastic/logstash/build/docker
├── Dockerfile
├── bin
│ └── docker-entrypoint
├── config
│ ├── log4j2.properties
│ ├── logstash-full.yml
│ └── pipelines.yml
├── env2yaml
│ └── env2yaml
└── pipeline
└── default.conf
% docker build --rm .
.....
Step 19/20 : LABEL org.label-schema.schema-version="1.0" org.label-schema.vendor="Elastic" org.label-schema.name="logstash" org.label-schema.version="7.0.0" org.label-schema.url="https://www.elastic.co/products/logstash" org.label-schema.vcs-url="https://github.com/elastic/logstash" license="Elastic License"
---> Using cache
---> f622d7555220
Step 20/20 : ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
---> Using cache
---> b6feba7f4934
Successfully built b6feba7f4934
```
This task works only for releases (not snapshots).
This commit also adds a few tweaks to the artifacts building:
Using `SKIP_PREPARE=1` in `rake artifact:tar` or `rake artifact:tar_oss` will make a check to not rebuild the tarball if there are no code modifications.
These two changes are made since docker image build is new and we want to keep it out of artifact:all for a while. And if we're running these separately, we want to ensure the tarball built is used in the docker image (versus building a new one for each `rake artifact:tar` )
This means that, to generate all artifacts including docker images and dockerfile, it's necessary to run:
```
RELEASE=1 rake artifact:all
SKIP_PREPARE=1 RELEASE=1 rake artifact:docker
SKIP_PREPARE=1 RELEASE=1 rake artifact:docker_oss
RELEASE=1 rake artifact:dockerfile
```
* Clean up release notes for master
* update version number in links
* Update docs/static/releasenotes.asciidoc
Comment out future links
Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>
* Update releasenotes.asciidoc
Comment out future links
Revert "work around jruby-5642 during package installation on jdk11 (#10658)"
This reverts commit 033c896330.
skip the bundler-1.16.6 files when unpacking jruby
Fixes#10674
* Update breaking changes doc for 7.0
Update structure to allow for previous and current changes
* Update docs/static/breaking-changes.asciidoc
Adds info about field reference parser
Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>
* Populate content for plugin changes
Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>
* Fix asciidoc formatting
* Incorporate review comments and new content
* Minor change for clarity
* add anchors for linking
* Add anchor for ecs-beats
* Incorporate review comments
* [DOCS] Adds tagged region for notable breaking changes
* Incorporate review comments
Remove link
introduces two rake tasks: `rake artifact:docker_oss` and `rake artifact:docker`, which will create the docker images of the OSS and non OSS packages. These tasks depend on the tar artifacts being built.
Also `rake artifact:all` has been modified to also call these two tasks.
most code was moved from https://github.com/elastic/logstash-docker/
A tiny change suggestion due to:
- `-SIGHUP` was mentioned in the previous text
- `-SIGHUP` is easier to read than `-1`
- `-1` can be easily mixed up with `-l`
Fixes#10592
* simplify the plugins-metadata.json file
* sort and update the plugin list in the rakelib/plugins-metadata.json
* remove dependency on twitter input for testing
* sorted Gemfile.template (grouped by group)
* remove default plugins from Gemfile.template
Fixes#10509
This commit fixes a ClassCastException which happens when
a plugin has the `enable_metric` setting set to false - a
NullMetricExt is assumed, but that is only created when
'metric.collect' is set to 'false' in the Logstash configuration,
not when an individual plugin disables its metrics.
Fixes#10538
There are several scenarios in which we can trigger concurrent convergence in
the agent, resulting in two or more threads working to perform interleaved and
potentially conflicting or overlapping pipeline actions. Notably, our trap on
`SIGHUP` will be resolved in its own thread, so if we are sent `SIGHUP` while
in the process of converging, the second in-flight convergence may get its
starting state before, during, or after the effects of the first convergence.
By mutually excluding execution of the convergence cycle, we eliminate the
class of bugs in which one convergence acquires actions that cannot succeed due
to the prior success of actions given to the other convergence.
Fixes#10537
Fixes a crash that occurs on pipeline load and/or reload when using both the
java keystore and the multi-pipeline feature, when more than one pipeline
references `${}`-style variables.
Fixes#10408
## Steps to set up GCP instance to run benchmark script
- Create an instance "n2-standard-16" with Ubuntu image
- Install docker
- `sudo snap install docker`
- `sudo usermod -a -G docker $USER`
- Install jq
- Install vault
- `sudo snap install vault`
- `vault login --method github`
- `vault kv get -format json secret/ci/elastic-logstash/benchmark`
- Setup Elasticsearch index mapping and alias with `setup/*`
- Import Kibana dashboard with `save-objects/*`
- Run the benchmark script
- Send data to your own Elasticsearch. Customise `VAULT_PATH="secret/ci/elastic-logstash/your/path"`
- Run the script `main.sh`
- or run in background `nohup bash -x main.sh > log.log 2>&1 &`
## Notes
- Benchmarks should only be compared using the same hardware setup.
- Please do not send the test metrics to the benchmark cluster. You can set `VAULT_PATH` to send data and metrics to your own server.
- Run `all.sh` as calibration which gives you a baseline of performance in different versions.
- [#16586](https://github.com/elastic/logstash/pull/16586) allows legacy monitoring using the configuration `xpack.monitoring.allow_legacy_collection: true`, which is not recognized in version 8. To run benchmarks in version 8, use the script of the corresponding branch (e.g. `8.16`) instead of `main` in buildkite.
error "Workflow (WORKFLOW_TYPE variable) is not set, exiting..."
;;
esac
info "Uploading artifacts for ${WORKFLOW_TYPE} workflow on branch: ${RELEASE_BRANCH} for version: ${STACK_VERSION} with version_qualifier: ${VERSION_QUALIFIER}"
if["$RELEASE_VER" !="7.17"];then
# Version 7.17.x doesn't generates ARM artifacts for Darwin
symptom:"1 indicator is unhealthy (`abnormally-terminated-pp-multipipeline`) and 2 indicators are concerning (`slow-start-pp-multipipeline`, `normally-terminated-pp-multipipeline`)"
indicators:
slow-start-pp-multipipeline:
status:"yellow"
symptom:"The pipeline is concerning; 1 area is impacted and 1 diagnosis is available"
diagnosis:
- cause:"pipeline is loading"
action:"if pipeline does not come up quickly, you may need to check the logs to see if it is stalled"
impacts:
- impact_areas:["pipeline_execution"]
details:
status:
state:"LOADING"
normally-terminated-pp-multipipeline:
status:"yellow"
symptom:"The pipeline is concerning; 1 area is impacted and 1 diagnosis is available"
diagnosis:
- cause:"pipeline has finished running because its inputs have been closed and events have been processed"
action:"if you expect this pipeline to run indefinitely, you will need to configure its inputs to continue receiving or fetching events"
impacts:
- impact_areas:["pipeline_execution"]
details:
status:
state:"FINISHED"
abnormally-terminated-pp-multipipeline:
status:"red"
symptom:"The pipeline is unhealthy; 1 area is impacted and 1 diagnosis is available"
diagnosis:
- cause:"pipeline is not running, likely because it has encountered an error"
action:"view logs to determine the cause of abnormal pipeline shutdown"
impacts:
- description:"the pipeline is not currently processing"
Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.
For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.
Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.
For all general issues, please provide the following details for fast resolution:
- Version:
- Operating System:
- Config File (if you have sensitive info, please remove it):
about: "Report a confirmed bug. For unconfirmed bugs please
visit https://discuss.elastic.co/c/logstash"
labels: "bug,status:needs-triage"
---
<!--
GitHub is reserved for bug reports and feature requests; it is not the place
for general questions. If you have a question or an unconfirmed bug , please
visit the [forums](https://discuss.elastic.co/c/logstash). Please also
check your OS is [supported](https://www.elastic.co/support/matrix#show_os).
If it is not, the issue is likely to be closed.
Logstash Plugins are located in a different organization: [logstash-plugins](https://github.com/logstash-plugins). For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.
For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.
Please fill in the following details to help us reproduce the bug:
-->
**Logstash information**:
Please include the following information:
1. Logstash version (e.g. `bin/logstash --version`)
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
**JVM** (e.g. `java -version`):
If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.
**OS version** (`uname -a` if on a Unix-like system):
**Description of the problem including expected versus actual behavior**:
**Steps to reproduce**:
Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
about:Send security vulnerability reports to security@elastic.co.
- name:Plugin Issue
url:https://github.com/logstash-plugins
about:Plugins for Logstash do the heavy lifting when it comes to integrating with a variety of external technologies and services. If your issue has to do with a specific plugin, please file it in the appropriate repository.
Please label this PR with the release version and one of the following labels, depending on the scope of your change:
- bug
- enhancement
- breaking change
- doc
-->
## Release notes
<!-- Add content to appear in [Release Notes](https://www.elastic.co/guide/en/logstash/current/releasenotes.html), or add [rn:skip] to leave this PR out of release notes -->
## What does this PR do?
<!-- Mandatory
Explain here the changes you made on the PR. Please explain the WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
Example:
Expose 'xpack.monitoring.elasticsearch.proxy' in the docker environment variables and update logstash.yml to surface this config option.
This commit exposes the 'xpack.monitoring.elasticsearch.proxy' variable in the docker by adding it in env2yaml.go, which translates from
being an environment variable to a proper yaml config.
Additionally, this PR exposes this setting for both xpack monitoring & management to the logstash.yml file.
-->
## Why is it important/What is the impact to the user?
<!-- Mandatory
Explain here the WHY or the IMPACT to the user, or the rationale/motivation for the changes.
Example:
This PR fixes an issue that was preventing the docker image from using the proxy setting when sending xpack monitoring information.
and/or
This PR now allows the user to define the xpack monitoring proxy setting in the docker container.
-->
## Checklist
<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to have the PR approved
List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->
- [ ] My code follows the style guidelines of this project
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
- [ ] I have added tests that prove my fix is effective or that my feature works
## Author's Checklist
<!-- Recommended
Add a checklist of things that are required to be reviewed in order to have the PR approved
-->
- [ ]
## How to test this PR locally
<!-- Recommended
Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
-->
## Related issues
<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.
- Closes #123
- Relates #123
- Requires #123
- Superseeds #123
-->
-
## Use cases
<!-- Recommended
Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.
If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
-->
## Screenshots
<!-- Optional
Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
-->
## Logs
<!-- Recommended
Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
All contributions are welcome: ideas, patches, documentation, bug reports,
complaints, etc!
If you want to be rewarded for your contributions, sign up for the [Elastic Contributor Program](https://www.elastic.co/community/contributor). Each time you make a valid contribution, you’ll earn points that increase your chances of winning prizes and being recognized as a top contributor.
Programming is not a required skill, and there are many ways to help out!
It is more important to us that you are able to contribute.
@ -12,7 +14,7 @@ That said, some basic guidelines, which you are free to ignore :)
Want to lurk about and see what others are doing with Logstash?
* The irc channel (#logstash on irc.freenode.org) is a good place for this
* The #logstash channel on Elastic Stack Community slack (https://elasticstack.slack.com/channels/logstash) is a good place to start.
* The [forum](https://discuss.elastic.co/c/logstash) is also
great for learning from others.
@ -21,12 +23,11 @@ Want to lurk about and see what others are doing with Logstash?
Have a problem you want Logstash to solve for you?
* You can ask a question in the [forum](https://discuss.elastic.co/c/logstash)
* Alternately, you are welcome to join the IRC channel #logstash on
irc.freenode.org and ask for help there!
* You are welcome to join Elastic Stack Community slack (https://elasticstack.slack.com) and ask for help on the #logstash channel.
## Have an Idea or Feature Request?
* File a ticket on [GitHub](https://github.com/elastic/logstash/issues). Please remember that GitHub is used only for issues and feature requests. If you have a general question, the [forum](https://discuss.elastic.co/c/logstash) or IRC would be the best place to ask.
* File a ticket on [GitHub](https://github.com/elastic/logstash/issues). Please remember that GitHub is used only for issues and feature requests. If you have a general question, the [forum](https://discuss.elastic.co/c/logstash) or Elastic Stack Community slack (https://elasticstack.slack.com) is the best place to ask.
## Something Not Working? Found a Bug?
@ -49,29 +50,69 @@ get in touch with our security team [here](https://www.elastic.co/community/secu
If you have a bugfix or new feature that you would like to contribute to Logstash, and you think it will take
more than a few minutes to produce the fix (ie; write code), it is worth discussing the change with the Logstash
users and developers first. You can reach us via [GitHub](https://github.com/elastic/logstash/issues), the [forum](https://discuss.elastic.co/c/logstash), or via IRC (#logstash on freenode irc)
users and developers first. You can reach us via [GitHub](https://github.com/elastic/logstash/issues), the [forum](https://discuss.elastic.co/c/logstash), or Elastic Stack Community slack (https://elasticstack.slack.com).
Please note that Pull Requests without tests and documentation may not be merged. If you would like to contribute but do not have
experience with writing tests, please ping us on IRC/forum or create a PR and ask our help.
experience with writing tests, please ping us on the forum or create a PR and ask for our help.
If you would like to contribute to Logstash, but don't know where to start, you can use the GitHub labels "adoptme"
and "low hanging fruit". Issues marked with these labels are relatively easy, and provides a good starting
If you would like to contribute to Logstash, but don't know where to start, you can use the GitHub labels "adoptme",
"low hanging fruit" and "good first issue". Issues marked with these labels are relatively easy, and provide a good starting
Or go directly here for an exhaustive list: https://github.com/elastic/logstash/contribute
Sometimes during the development of Logstash core or plugins, configuration parameters for sensitive data such as password, API keys or SSL keyphrases need to be provided to the user.
To protect sensitive information leaking into the logs, follow the best practices below:
- Logstash core and plugin should flag any settings that may contain secrets. If your source changes are in Ruby, apply `:password` validation or wrap the object with `Logstash::Util::Password` object.
- A setting marked as secret should not disclose its value unless retrieved in a non-obvious way. If you are introducing a new class for sensitive object (check `Password.java` and `SecretVariable.java` classes before doing so), make sure to mask the sensitive info by overriding get value (or `toString()` of Java class) default methods.
- Logstash should not log secrets on any log level by default. Make sure you double-check the loggers if they are touching the objects carrying sensitive info.
- Logstash has a dedicated flag, disabled by default, that the raw configuration text be logged for debugging purposes only. Use of `config.debug` will log/display sensitive info in the debug logs, so beware of using it in Produciton.
As an example, suppose you have `my_auth` config which carries the sensitive key. Defining with `:password` validated protects `my_auth` being leaked in Logstash core logics.
```
:my_auth => { :validate => :password },
```
In the plugin level, make sure to wrap `my_auth` with `Password` object.
```
::LogStash::Util::Password.new(my_auth)
```
Using IntelliJ? See a detailed getting started guide [here](https://docs.google.com/document/d/1kqunARvYMrlfTEOgMpYHig0U-ZqCcMJfhvTtGt09iZg/pub).
## Breaking Changes
When introducing new behaviour, we favor implementing it in a non-breaking way that users can opt into, meaning users' existing configurations continue to work as they expect them to after upgrading Logstash.
But sometimes it is necessary to introduce "breaking changes," whether that is removing an option that doesn't make sense anymore, changing the default value of a setting, or reimplementing a major component differently for performance or safety reasons.
When we do so, we need to acknowledge the work we are placing on the rest of our users the next time they upgrade, and work to ensure that they can upgrade confidently.
Where possible, we:
1. first implement new behaviour in a way that users can explicitly opt into (MINOR),
2. commuicate the pending change in behaviour, including the introduction of deprecation warnings when old behaviour is used (MINOR, potentially along-side #1),
3. change the default to be new behaviour, communicate the breaking change, optionally allow users to opt out in favor of the old behaviour (MAJOR), and eventually
4. remove the old behaviour's implementation from the code-base (MAJOR, potentially along-side #3).
After a pull request is marked as a "breaking change," it becomes necessary to either:
- refactor into a non-breaking change targeted at next minor; OR
- split into non-breaking change targeted at next minor, plus breaking change targeted at next major
## Contributing to plugins
Check our [documentation](https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html) on how to contribute to plugins or write your own!
Check the _Contributing Documentation and Code Changes_ section to prevent plugin sensitive info from leaking in the debug logs.
### Logstash Plugin Changelog Guidelines
This document provides guidelines on editing a logstash plugin's CHANGELOG file.
#### What's a CHANGELOG file?
### What's a CHANGELOG file?
According to [keepachangelog.com](https://keepachangelog.com/en/1.0.0/):
@ -107,14 +148,24 @@ then you should still create an entry in the changelog, using the word `Unreleas
Most code in logstash-plugins comes from self-contained changes in the form of pull requests, so each entry should:
1. be a summary of the Pull Request and contain a markdown link to it.
2. start with [BREAKING], when it denotes a breaking change.
* Note: Breaking changes should warrant a major version bump.
3. after [BREAKING], start with one of the following keywords:
- [DOC] Fixed incorrect formatting of code sample [#85](http://example.org)
## 3.3.2
- Fixed incorrect serialization of input data when encoding was `Emacs-Mule` [#84](http://example.org)
@ -196,4 +250,3 @@ Keep these in mind as both authors and reviewers of PRs:
* If no, ask for clarifications on the PR. This will usually lead to changes in the code such as renaming of variables/functions or extracting of functions or simply adding "why" inline comments. But first ask the author for clarifications before assuming any intent on their part.
* I must not focus on personal preferences or nitpicks. If I understand the code in the PR but simply would've implemented the same solution a different way that's great but its not feedback that belongs in the PR. Such feedback only serves to slow down progress for little to no gain.
