* Update logstash_releases.json after 8.17.5
* Update 9.next post 9.0.0 release
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* change base images to ubi9-minimal
* do all env2yaml related copying in 1 COPY
* use -trimpath in go build
* move other COPY to end of dockerfile
* don't run package manager upgrade
* FROM and AS with same case
* ENV x=y instead of ENV x y
* remove indirect config folder
* Disabling some of metricsets may cause dashboards partially display some metrics. This change recommends collecting metricsets to fully visualize metrics on dashboards.
* Paraphrasing sentences and grammar correction.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Update docs/reference/serverless-monitoring-with-elastic-agent.md
* Make recommendation as a tip.
---------
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444
reporting the following error
```
File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
def integration_tests(self, part: int, parts: int) -> JobRetValues:
^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)
This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Fix standalone agent access for agent-driven monitoring
Change incorrect dedicated instructions, and base them on those from running Elastic Agent
in standalone mode.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
The rubocop-ast gem just introduced a new dependency on prism.
- https://rubygems.org/gems/rubocop-ast/versions/1.43.0
In our install default gem rake task we are seeing issues trying to build native
extensions. I see that in upstream jruby they are seeing a similar problem (at
least it is the same failure mode https://github.com/jruby/jruby/pull/8415
This commit pins rubocop-ast to 1.42.0 which is the last version that did not
have an explicit prism dependency.
Updates only test code to be able to run a test that consumes big memory if:
- the physical memory is bigger than the requested Java heap
- JDK version is greater than or equal to 21.
The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error.
* tests: make integration split quantity configurable
Refactors shared splitter bash function to take a list of files on stdin
and split into a configurable number of partitions, emitting only those from
the currently-selected partition to stdout.
Also refactors the only caller in the integration_tests launcher script to
accept an optional partition_count parameter (defaulting to `2` for backward-
compatibility), to provide the list of specs to the function's stdin, and to
output relevant information about the quantity of partition splits and which
was selected.
* ci: run integration tests in 3 parts
Use long instead of int type to keep the length of the first token.
The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test 9c0e50faac/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java (L123).
To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.
This commit migrated API doc to OpenAPI spec
- expand Node Info `/_node/<types>`
- expand Node Stats
- add health report
- break down the share part to components
- add examples
- update authentication
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
The https://github.com/elastic/logstash/pull/17310 PR changed the rake task for
artifact creation to use shellwords from standard library. The acceptance tests
need to explitily load that library. This commit updates the rake file to handle
loading the required code.
Previously when rake would shell out the output would be lost. This
made debugging CI logs difficult. This commit updates the stack with
improved message surfacing on error.
Reimplements `LogStash::Setting::Numeric` Ruby setting class into the `org.logstash.settings.NumericSetting` and exposes it through `java_import` as `LogStash::Setting::NumericSetting`.
Updates the rspec tests:
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.
This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue
Fixes: #17209
* tests: ls2ls delay checking until events have been processed
* Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec.
* Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable.
---------
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit migrates gradle task to the new artifacts-api
- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
with priority given in that order.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This commit refactors the `PersistedQueueConfigValidator` class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
https://github.com/elastic/logstash/issues/14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.
* plugin manager: add --no-expand flag for list command
Allows us to avoid expanding aliases and integration plugins
* spec: escape expected output in regexp
In https://github.com/elastic/logstash/pull/17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
Removal works in a single pass by finding plugins that would have unmet
dependencies if all of the specified plugins were to be removed, and
proceeding with the removal only if no conflicts were created.
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-remove-multiple ✘) }
> ╰─● bin/logstash-plugin remove logstash-input-syslog logstash-filter-grok
> Using system java: /Users/rye/.jenv/shims/java
> Resolving dependencies......
> Successfully removed logstash-input-syslog
> Successfully removed logstash-filter-grok
> [success (00:00:05)]
~~~
* spec: improve ls2ls spec
- fixes upstream/downstream convention
- upstream: the sending logstash (has an LS output)
- downstream: the receiving logstash (has an LS input)
- helper `run_logstash_instance` yields the `LogstashService` instance
and handles the teardown.
- pass the pipeline id and node name to the LS instances via command line
flags to make logging easier to differentiate
- use the generator input's sequence id to ensure that the _actual_ events
generated are received by the downstream pipeline
* start with port-offset 100
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Currently the deserialization is behind the readBatch's lock, so any large batch will take time deserializing, causing any other Queue writer (e.g. netty executor threads) and any other Queue reader (pipeline worker) to block.
This commit moves the deserialization out of the lock, allowing multiple pipeline workers to deserialize batches concurrently.
- add intermediate batch-holder from `Queue` methods
- make the intermediate batch-holder a private inner class of `Queue` with a descriptive name `SerializedBatchHolder`
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Starting from es-output 12.0.2, a 404 response is treated as an error. Previously, central pipeline management considered 404 as an empty pipeline, not an error.
This commit restores the expected behavior by handling 404 gracefully and logs a user-friendly message.
It also removes the redundant cache of pipeline in CPM
Fixes: #17035
* qa: use clean expansion of LS tarball per fixture instance
Because QA tests can _modify_ the Logstash installation (e.g. those that
invoke the plugin manager), it is important that the service wrapper
begins with a clean expansion of the logstash tarball.
* qa: enable safe reuse of ls_home in ls_to_ls tests
This commit simplifies the DRA process in Logstash by removing the need to maintain a separate file for the active branches, and instead rely on a centrally maintained file containing source of truth.
While at it, we refactor/simplify the creation of an array with the versions in `.buildkite/scripts/snyk/resolve_stack_version.sh`.
This commit preserves spaces in values, ensuring that multi-word strings are stored as intended.
Prior to this change, `logstash-keystore` incorrectly handled values containing spaces,
causing only the first word to be stored.
PR #17006 revealed that the `VERSION_QUALIFIER` env var gets honored in
various scripts when present but empty.
This shouldn't be the case as the DRA process is designed to gracefully
ignore empty values for this variable.
This commit changes various ruby scripts to not treat "" as truthy.
Bash scripts (used by CI etc.) are already ok with this as part of
refactorings done in #16907.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Permit to use effectively the tokenizer also in context where a line is bigger than a limit.
Fixes an issues related to token size limit error, when the offending token was bigger than the input fragment in happened that the tokenzer wasn't unable to recover the token stream from the first delimiter after the offending token but messed things, loosing part of tokens.
## How solve the problem
This is a second take to fix the processing of tokens from the tokenizer after a buffer full error. The first try #16482 was rollbacked to the encoding error #16694.
The first try failed on returning the tokens in the same encoding of the input.
This PR does a couple of things:
- accumulates the tokens, so that after a full condition can resume with the next tokens after the offending one.
- respect the encoding of the input string. Use `concat` method instead of `addAll`, which avoid to convert RubyString to String and back to RubyString. When return the head `StringBuilder` it enforce the encoding with the input charset.
Log input has been deprecated in filebeat 9.0.0 and throws an error if it's present in the configuration.
This commit switches the configuration to the "filestream" input.
This commit removed irrelevant warning for internal pipeline, such as monitoring pipeline.
Monitoring pipeline is expected to be one worker. The warning is not useful
Fixes: #13298
Reimplements `LogStash::Setting::String` Ruby setting class into the `org.logstash.settings.SettingString` and exposes it through `java_import` as `LogStash::Setting::SettingString`.
Updates the rspec tests in two ways:
- logging mock is now converted to real Log4J appender that spy log line that are later verified
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
handle maven output that can carry "garbage" information after the jar's name. this patch deletes that extra information, also pins psych to 5.2.2 until jruby ships with snakeyaml-engine 2.9 and jar-dependencies 0.5.2
Related to: https://github.com/jruby/jruby/issues/8579
This commit uses the new --qualifier parameter in the release manager
for publishing dra artifacts. Additionally, simplifies the expected
variables to rely on a simple `VERSION_QUALIFIER`.
Snapshot builds are skipped when VERSION_QUALIFIER is set.
Finally, for helping to test DRA PRs, we also allow passing the `DRA_BRANCH` option/env var
to override BUILDKITE_BRANCH.
Closes https://github.com/elastic/ingest-dev/issues/4856
* Add a new pattern %{{TIME_NOW}} to `event.sprintf` to generate a fresh timestamp.
The timestamp is represented as a string in the default ISO 8601 format
For example,
```
input {
heartbeat {
add_field => { "heartbeat_time" => "%{{TIME_NOW}}" }
}
}
```
VERSION_QUALIFIER was already observed in rake artifacts task but only to influence the name of the artifact.
This commit ensure that the qualifier is also displayed in the cli and in the http api.
The preferred channel for communication about LS is the elastic discussion
forum, this commit updates the source code and readme files to reflect that.
* Doc: Environment variables cannot be in config.string comments due to substitution logic.
* Apply suggestions from code review
Suggestion to revise the statements.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Make sentences present tense.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Revert "Apply Jackson stream read constraints defaults at runtime (#16832)"
This reverts commit cc608eb88b.
* jackson stream read constraints: code-based defaults
refactors stream read constraints to couple default values with their
associated overrides, which allows us to have more descriptive logging
that includes provenance of the value that has been applied.
Previously when the `:deprecated` modifier was used in the plugin config DSL a
log message was sent at `:warn` level to the main logger. This commit updates
that message to be routed *only* to the deprecation logger.
JvmOptionsParser adds support for ${VAR:default} syntax when parsing jvm.options
- allow dynamic resolution of environment variables in the jvm.options file
- enables fallback to default value when the environment variable is not set
When Logstash 8.12.0 added increased Jackson stream read constraints to
jvm.options, assumptions about the existence of that file's contents
were invalidated. This led to issues like #16683.
This change ensures Logstash applies defaults from config at runtime:
- MAX_STRING_LENGTH: 200_000_000
- MAX_NUMBER_LENGTH: 10_000
- MAX_NESTING_DEPTH: 1_000
These match the jvm.options defaults and are applied even when config
is missing. Config values still override these defaults when present.
Because a `break` escapes a `begin`...`end` block, we must not use a `break` in order to ensure that the explicitly set value gets memoized to avoid lock contention.
> ~~~ ruby
> def fake_sync(&block)
> puts "FAKE_SYNC:enter"
> val = yield
> puts "FAKE_SYNC:return(#{val})"
> return val
> ensure
> puts "FAKE_SYNC:ensure"
> end
>
> fake_sync do
> @ivar = begin
> puts("BE:begin")
> break :break
>
> val = :ret
> puts("BE:return(#{val})")
> val
> ensure
> puts("BE:ensure")
> end
> end
> ~~~
Note: no `FAKE_SYNC:return`:
> ~~~
> ╭─{ rye@perhaps:~/src/elastic/logstash (main ✔) }
> ╰─● ruby break-esc.rb
> FAKE_SYNC:enter
> BE:begin
> BE:ensure
> FAKE_SYNC:ensure
> [success]
> ~~~
This commit updates the curl scripts that interact with Kibana's
`api/logstash/pipeline/*` endpoin. Additionally this adds the header to any curl
that interacts with elasticsearch API as well.
Moves the application of jackson defaults overrides into pure java, and
applies them statically _before_ the `org.logstash.ObjectMappers` has a chance
to start initializing object mappers that rely on the defaults.
We replace the runner's invocation (which was too late to be fully applied) with
a _verification_ that the configured defaults have been applied.
* pipeline bus: add deadlock test for unlisten/unregisterSender
* pipeline bus: eliminate deadlock
Moves the sync-to-notify out of the `AddressStateMapping#mutate`'s effective
synchronous block to eliminate a race condition where unlistening to an address
and unregistering a sender could deadlock.
It is safe to notify an AddressState's attached input without exclusive access
to the AddressState, because notifying an input that has since been disconnected
is net zero harm.
(cherry picked from commit 8af6343a26)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Pip install is unnecessary and sometimes hangs with prompt dialog. This commit aligns on agent python version which has a default pip installed.
* Update .buildkite/scripts/health-report-tests/main.sh
Improve readability.
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
---------
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
In https://github.com/elastic/logstash/pull/16586 the module include was
removed. This causes failures in the script when the module is referenced. This
commit re-enables the include for the yaml module.
Introduce a new flag setting `xpack.monitoring.allow_legacy_collection` which eventually enable the legacy monitoring collector.
Update the method to test if monitoring is enabled so that consider also `xpack.monitoring.allow_legacy_collection` to determine if `monitoring.*` settings are valid or not.
By default it's false, the user has to intentionally enable it to continue to use the legacy monitoring settings.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Switch the default value of `pipeline.buffer.type` to use the heap memory instead of direct one.
Change the default value of the setting `pipeline.buffer.type` from direct to heap and update consequently the documentation.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
See naming rational in https://github.com/logstash-plugins/.ci/pull/63#issue-2597373955
After 8.16 is GA, the "releases" entry should become:
```json
"releases": {
"7.current": "7.17.24",
"8.current": "8.16.0",
"8.previous": "8.15.3"
},
```
For snapshots we'll also test against "main", "8.next", and "8.future". The labels are:
- `main`: main branch
- `8.future`: the future 8.x release, i.e. current version of the 8.x branch
- `8.next`: the short lived period between a minor's FF - when the new branch is cut from 8.x - and GA
- `8.current`: the most recent 8.x release
- `8.previous`: the previous, but still supported, 8.x release
This commit extends the gradle and rake tasks to pass through a list of files
for rubocop to lint. This allows more specificity and fine grained control for
linting when the consumer of the tasks only wishes to lint a select few files.
Logstash Docker images, full and OSS, now use UBI image as its base, replacing the previous Ubuntu base.
- change the base image of `full` and `oss` to ubi
- Set locale to C.UTF-8
- remove ubi flavour
- use go image to build env2yaml
- remove redundant and refactor steps
- add support to build image in mac aarch64
- allow customizing ELASTIC_VERSION and LOCAL_ARTIFACTS for test purpose
there's no need to build jdk-less and windows tarballs for docker images
so this change simplifies the build process.
It should reduce the time spent needed to build docker images.
Switch execution agent of JDK availability check pipeline from vm-agent to container-agent.
Moves the schedule definition from the `Logstash Pipeline Scheduler` pipeline into the pipeline definition, adding a schedule also for `8.x` branch.
Updates the existing `createElasticCatalogDownloadUrl` method to use the precise version retrieved `versions.yml` to download the JDK instead of using the latest of major version. This makes the build reproducible again.
Defines a new Gradle `checkNewJdkVersion` task to check if there is a new JDK version available from JVM catalog matching the same major of the current branch.
Creates a new Buildkite pipeline to execute a `bash` script to run the Gradle task; plus it also update the `catalog-info.yaml` with the new pipeline and a trigger to execute every week.
The current max inflight error message focuses on a single pipeline and on a maximum amount of 10k events regardless of the heap size.
The new warning will take into account all loaded pipelines and also consider the heap size, giving a warning if the total number of events consumes 10% or more of the total heap.
For the purpose of the warning events are assumed to be 2KB as it a normal size for a small log entry.
Adds a JMH benchmark to measure the peformances of BufferedTokenizerExt.
Update also Gradle build script to remove CMS GC flags and fix deprecations for Gradle 9.0.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Fixes the behaviour of the tokenizer to be able to work properly when buffer full conditions are met.
Updates BufferedTokenizerExt so that can accumulate token fragments coming from different data segments. When a "buffer full" condition is matched, it record this state in a local field so that on next data segment it can consume all the token fragments till the next token delimiter.
Updated the accumulation variable from RubyArray containing strings to a StringBuilder which contains the head token, plus the remaining token fragments are stored in the input array.
Furthermore it translates the `buftok_spec` tests into JUnit tests.
* Health API: Add 1min 5min backpressure cases and improve Logstash temination logic.
* Apply suggestions from code review
Uncomment accidentally commented sources.
* Update .buildkite/scripts/health-report-tests/tests/slow-start.yaml
No need to wait for LS startup when using slow start scenario.
* Apply suggestions from code review
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Standardize YAML structure and rename wait time to wait_seconds
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Fixes a potential flaky test, due to shared (LogStash:SETTINGS) fixture across the test base.
Forward port the commit 609155a61b used to fix the non clean backport PR #16531 of #16525 to 8.x.
LogStash:SETTINGS is used in the constructor of LogStash::Inputs::Metrics::StatsEventFactory to query the value of api.enabled. This PR keeps updated the value for the setting provided to the Agent constructor and to the StatsEventFactory.
* Utilize default agent for Health API CI. Call python scripts from directly CI step.
* Change BK agent to support both Java and python. Install pip manually and send env vars to subprocess.
* [health] bootstrap HealthObserver from agent to API (#16141)
* [health] bootstrap HealthObserver from agent to API
* specs: mocked agent needs health observer
* add license headers
* Merge `main` into `feature/health-report-api` (#16397)
* Add GH vault plugin bot to allowed list (#16301)
* regenerate webserver test certificates (#16331)
* correctly handle stack overflow errors during pipeline compilation (#16323)
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* Doc: Reposition worker-utilization in doc (#16335)
* settings: add support for observing settings after post-process hooks (#16339)
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
* fix line used to determine ES is up (#16349)
* add retries to snyk buildkite job (#16343)
* Fix 8.13.1 release notes (#16363)
make a note of the fix that went to 8.13.1: #16026
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16347)
* [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Doc: Add SNMP integration to breaking changes (#16374)
* deprecate java less-than 17 (#16370)
* Exclude substitution refinement on pipelines.yml (#16375)
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Doc: Forwardport 8.15.0 release notes to main (#16388)
* Removing 8.14 from ci/branches.json as we have 8.15. (#16390)
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Squashed merge from 8.x
* Failure injector plugin implementation. (#16466)
* Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes.
* Fix the degrate method name typo.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Add explanation to the config params and rebuild plugin gem.
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Health report integration tests bootstrapper and initial tests implementation (#16467)
* Health Report integration tests bootstrapper and initial slow start scenario implementation.
* Apply suggestions from code review
Renaming expectation check method name.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Changed to branch concept, YAML structure simplified as changed to Dict.
* Apply suggestions from code review
Reflect `help_url` to the integration test.
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* health api: expose `GET /_health_report` with pipelines/*/status probe (#16398)
Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the
resulting report status into the other API responses, replacing their hard-coded `green`
with a meaningful status indication.
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* docs: health report API, and diagnosis links (feature-targeted) (#16518)
* docs: health report API, and diagnosis links
* Remove plus-for-passthrough markers
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* merge 8.x into feature branch... (#16519)
* Add GH vault plugin bot to allowed list (#16301)
* regenerate webserver test certificates (#16331)
* correctly handle stack overflow errors during pipeline compilation (#16323)
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* Doc: Reposition worker-utilization in doc (#16335)
* settings: add support for observing settings after post-process hooks (#16339)
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
* fix line used to determine ES is up (#16349)
* add retries to snyk buildkite job (#16343)
* Fix 8.13.1 release notes (#16363)
make a note of the fix that went to 8.13.1: #16026
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16347)
* [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365)
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Doc: Add SNMP integration to breaking changes (#16374)
* deprecate java less-than 17 (#16370)
* Exclude substitution refinement on pipelines.yml (#16375)
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Doc: Forwardport 8.15.0 release notes to main (#16388)
* Removing 8.14 from ci/branches.json as we have 8.15. (#16390)
* Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408)
Fix: #16406
* Generate Dataset code with meaningful fields names (#16386)
This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields.
Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`.
Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`.
Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields.
* Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322)
This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it.
This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch.
Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node.
Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* Update logstash_releases.json (#16426)
* Release notes for 8.15.1 (#16405) (#16427)
* Update release notes for 8.15.1
* update release note
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co>
(cherry picked from commit 2fca7e39e8)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Fix ConditionalEvaluationError to do not include the event that errored in its serialiaxed form, because it's not expected that this class is ever serialized. (#16429) (#16430)
Make inner field of ConditionalEvaluationError transient to be avoided during serialization.
(cherry picked from commit bb7ecc203f)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* use gnu tar compatible minitar to generate tar artifact (#16432) (#16434)
Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters.
For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem.
(cherry picked from commit 69f0fa54ca)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* account for the 8.x in DRA publishing task (#16436) (#16440)
the current DRA publishing task computes the branch from the version
contained in the version.yml
This is done by taking the major.minor and confirming that a branch
exists with that name.
However this pattern won't be applicable for 8.x, as that branch
currently points to 8.16.0 and there is no 8.16 branch.
This commit falls back to reading the buildkite injected
BUILDKITE_BRANCH variable.
(cherry picked from commit 17dba9f829)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Fixes the issue where LS wipes out all quotes from docker env variables. (#16456) (#16459)
* Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.
* Add a docker acceptance integration test.
(cherry picked from commit 7c64c7394b)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Known issue for 8.15.1 related to env vars references (#16455) (#16469)
(cherry picked from commit b54caf3fd8)
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
* bump .ruby_version to jruby-9.4.8.0 (#16477) (#16480)
(cherry picked from commit 51cca7320e)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Release notes for 8.15.2 (#16471) (#16478)
Co-authored-by: andsel <selva.andre@gmail.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit 01dc76f3b5)
* Change LogStash::Util::SubstitutionVariables#replace_placeholders refine argument to optional (#16485) (#16488)
(cherry picked from commit 8368c00367)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Use jruby-9.4.8.0 in exhaustive CIs. (#16489) (#16491)
(cherry picked from commit fd1de39005)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Don't use an older JRuby with oraclelinux-7 (#16499) (#16501)
A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images
and removed JRuby 9.4.5.0 and some older versions.
This ended up breaking exhaustive test on Oracle Linux 7 that hard coded
JRuby 9.4.5.0.
PR https://github.com/elastic/logstash/pull/16489 worked around the
problem by pinning to the new JRuby, but actually we don't
need the conditional anymore since the original issue
https://github.com/jruby/jruby/issues/7579#issuecomment-1425885324 has
been resolved and none of our releasable branches (apart from 7.17 which
uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`.
Therefore, this commit removes conditional setting of JRuby for
OracleLinux 7 agents in exhaustive tests (and relies on whatever
`/.ruby-version` defines).
(cherry picked from commit 07c01f8231)
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
* Improve pipeline bootstrap error logs (#16495) (#16504)
This PR adds the cause errors details on the pipeline converge state error logs
(cherry picked from commit e84fb458ce)
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
* Logstash Health Report Tests Buildkite pipeline setup. (#16416) (#16511)
(cherry picked from commit 5195332bc6)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Make health report test runner script executable. (#16446) (#16512)
(cherry picked from commit 2ebf2658ff)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* Backport PR #16423 to 8.x: DLQ-ing events that trigger an conditional evaluation error. (#16493)
* DLQ-ing events that trigger an conditional evaluation error. (#16423)
When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline.
This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it.
(cherry picked from commit b69d993d71)
* Fixed warning about non serializable field DeadLetterQueueWriter in serializable AbstractPipelineExt
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* add deprecation log for `--event_api.tags.illegal` (#16507) (#16515)
- move `--event_api.tags.illegal` from option to deprecated_option
- add deprecation log when the flag is explicitly used
relates: #16356
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit a4eddb8a2a)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
---------
Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co>
Co-authored-by: Edmo Vamerlatti Costa <11836452+edmocosta@users.noreply.github.com>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
(cherry picked from commit 7eb5185b4e)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Set the 'api.enabled' setting to reflect the flag webserver_enabled and consequently test for http_address presence in settings iff the web server is enabled.
* Update rspec dependency of the QA package.
* Update qa/Gemfile
Align on rspec 3.13.x
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Fix the QA test failure caused after reflecting Health Report status to the Node stats.
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit 1e5105fcd8)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
- move `--event_api.tags.illegal` from option to deprecated_option
- add deprecation log when the flag is explicitly used
relates: #16356
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Update Settings to_hash method to also skip Java DeprecatedAlias and not just the Ruby one.
With PR #15679 was introduced org.logstash.settings.DeprecatedAlias which mirrors the behaviour of Ruby class Setting::DeprecatedAlias. The equality check at Logstash::Settings, as descibed in #16505 (comment), is implemented comparing the maps.
The conversion of Settings to the corresponding maps filtered out the Ruby implementation of DeprecatedAlias but not the Java one.
This PR adds also the Java one to the list of filter.
A recent PR (elastic/ci-agent-images/pull/932) modernized the VM images
and removed JRuby 9.4.5.0 and some older versions.
This ended up breaking exhaustive test on Oracle Linux 7 that hard coded
JRuby 9.4.5.0.
PR https://github.com/elastic/logstash/pull/16489 worked around the
problem by pinning to the new JRuby, but actually we don't
need the conditional anymore since the original issue
https://github.com/jruby/jruby/issues/7579#issuecomment-1425885324 has
been resolved and none of our releasable branches (apart from 7.17 which
uses `9.2.20.1`) specify `9.3.x.y` in `/.ruby-version`.
Therefore, this commit removes conditional setting of JRuby for
OracleLinux 7 agents in exhaustive tests (and relies on whatever
`/.ruby-version` defines).
Suppress some warnings compared with JDK 21
- this-escape uses this before it is completely initialised.
- avoid a non serialisable DeadLetterQueueWriter field from serialisable instance.
When a conditional evaluation encounter an error in the expression the event that triggered the issue is sent to pipeline's DLQ, if enabled for the executing pipeline.
This PR engage with the work done in #16322, the `ConditionalEvaluationListener` that is receives notifications about if-statements evaluation failure, is improved to also send the event to DLQ (if enabled in the pipeline) and not just logging it.
Reimplement the root Ruby Setting class in Java and use it from the Ruby one moving the original Ruby class to a shell wrapping the Java instance.
In particular create a new symmetric hierarchy (at the time just for `Setting`, `Coercible` and `Boolean` classes) to the Ruby one, moving also the feature for setting deprecation. In this way the new `org.logstash.settings.Boolean` is syntactically and semantically equivalent to the old Ruby Boolean class, which replaces.
- split main.sh to core.sh and main.sh
- rename all.sh to marathon.sh
- fix vault expiry issue for long running task
- fix unable to call main function
- update save object runtime field `release` to return true when `version` contains "SNAPSHOT"
* Fixes the issue where LS wipes out all quotes from docker env variables. This is an issue when running LS on docker with CONFIG_STRING, needs to keep quotes with env variable.
* Add a docker acceptance integration test.
If a user runs Logstash with a hosted JDK and not the one bundled with Logstash distribution, like setting a specific LS_JAVA_HOME, which is minor than JDK 17 then Logstash refuses to start. Has to provide at least a JDK 17 or unset the LS_JAVA_HOME and let Logstash uses the bundled JDK.
Updates the jvm.options and JvmOptionsParser to remove support for JDK 11. If the options parser identifies that the running JVM is less than 17, it refuses to start.
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
- add becnhmark dashboard and related save objects
- add one buildkite pipeline to test against multiple versions
- remove null field in json
- add `FLOG_FILE_CNT`, `VAULT_PATH`, `TAGS`
the current DRA publishing task computes the branch from the version
contained in the version.yml
This is done by taking the major.minor and confirming that a branch
exists with that name.
However this pattern won't be applicable for 8.x, as that branch
currently points to 8.16.0 and there is no 8.16 branch.
This commit falls back to reading the buildkite injected
BUILDKITE_BRANCH variable.
Using VERSION_QUALIFIER when building the tarball distribution will fail since Ruby's TarWriter implements the older POSIX88 version of tar and paths will be longer than 100 characters.
For the long paths being used in Logstash's plugins, mainly due to nested folders from jar-dependencies, we need the tarball to follow either the 2001 ustar format or gnu tar, which is implemented by the minitar gem.
This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it.
This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch.
Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node.
Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event.
---------
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields.
Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`.
Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`.
Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields.
* Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars)
* Safety integration test for pipeline config.string contains ENV .
* Properly resolve the values from ENV vars if literal array string provided with ENV var.
* Docker acceptance test for persisting keys and use actual values in docker container.
* Review suggestion.
Simplify the code by stripping whitespace before `gsub`, no need to check comma and split.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Because logging configuration occurs after loading the `logstash.yml`
settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are
effectively emitted to a null logger and lost.
By re-emitting after the post-process hooks, we can ensure that they make
their way to the deprecation log. This change adds support for any setting
that responds to `Object#observe_post_process` to receive it after all
post-processing hooks have been executed.
Resolves: elastic/logstash#16332
This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened.
A couple of thoughts on the way this is implemented:
* There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens.
* The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause.
Solves #16320
* licenses: allow elv2, standard abbreviation for Elastic License version 2
* json-dump: reduce unicode normalization cost
Since the underlying JrJackson now properly (and efficiently) encodes the
UTF-8 transcode of whichever strings it is given, we no longer need to
pre-normalize to UTF-8 in ruby _except_ when the string is flagged as BINARY
because we have alternate behaviour to preserve valid UTF-8 sequences.
By emitting a _copy_ of binary-flagged strings that have been re-flagged as
UTF-8, we allow the downstream (efficient) encoding operation in jrjackson
to produce equivalent behaviour at much lower cost.
* cleanup: remove orphan unicode normalizer
- fixes the CI benchmark script to always runs against the latest snapshot version
- uses `/v1/versions/$VERSION/builds/latest` to get the latest build id
Fixes: #16307
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
This commit fixed the configuration reload process to clean up the pipeline's metric store, so it does not retain references to failed pipelines components.
* Add RubyEvent#dup support and unit test case to keep Json#dump(Event) safe.
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
---------
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* fix: restore support for unicode pipeline- and plugin-id's
JRuby's `Ruby#newSymbol(String)` throws an exception when provided a `String`
that contains characters outside of lower-ASCII because JRuby internals expect
"the incoming String to be one of our mangled ISO-8859-1 strings" as noted in
a comment on jruby/jruby#6217.
Instead, we use `Ruby#newString(String)` to create a new `RubyString` (which
works properly), and then rely on `RubyString#intern` to get our `RubySymbol`.
This fixes a regression introduced in the 8.7 series in which pipeline id's
are consistently represented as ruby symbols in the metrics store, and ensures
similar issue does not exist when specifying a plugin id that contains
characters above the lower-ASCII plane.
* fix: use properly-encoded RubySymbol in PipelineConfig
We cannot rely on `RubySymbol#toString` to produce a properly-encoded `String`
whe the string contains characters above the lower-ASCII plane because the
result is effectively a binary ruby-internal marshal of the bytes that only
holds when the symbol contains lower-ASCII.
Instead, we can use the internally-memoizing `RubySymbol#name` to get a
properly-encoded `RubyString`, and `RubyString#asJavaString()` to get a
properly-encoded java-`String`.
* fix: properly serialize unicode pipeline names in API output
Jackson's JSON serializer leaks the JRuby-internal byte structure of Symbols,
which only aligns with the byte-structure of the symbol's actual string when
that string is wholly-comprised of lower-ASCII characters.
By pre-converting Symbols to Strings, we ensure that the result is readable
and useful.
* spec: bypass monitoring specs for unicode pipeline ids when PQ enabled
Add a buildkite pipeline to do benchmark.
The script does benchmark by running Filebeats (docker) -> Logstash (docker) -> ES Cloud.
Logstash metrics and benchmark results are sent to the same ES Cloud.
- Secrets store in vault `secret/ci/elastic-logstash/benchmark`
- Use flog (docker) to generate ~2GB logs
- Pull the snapshot docker image of the main branch every day
- Logstash runs two pipelines, main and node_stats
- The main pipeline handles beats ingestion, sending data to the data stream `logs-generic-default`
- It runs for all combinations. (pq + mq) x worker x batch size
- Each test runs for ~7 minutes
- The node_stats pipeline retrieves /_node/stats API every 30s and sends it to the data stream `metrics-nodestats-logstash`
- The script sends a summary of EPS and resource usage to index `benchmark_summary`
The buildkite pipeline accepts ENV variables to customize the test
| Variable Name | Default Value | Comment |
|-----------------|---------------------|----------------------------------------------------|
| FB_VERSION | 8.13.4 | docker tag |
| LS_VERSION | | docker tag |
| LS_JAVA_OPTS | -Xmx2g | by default, Xmx is set to half of memory |
| MULTIPLIERS | 2,4,6 | determine the number of workers (cpu * multiplier) |
| BATCH_SIZES | 125,1000 | |
| CPU | 4 | number of cpu for Logstash container |
| MEM | 4 | number of GB for Logstash container |
| QTYPE | memory | queue type to test -- persisted; memory; all |
| FB_CNT | 4 | number of filebeats to use in benchmark |
To check the result
- `vault read secret/ci/elastic-logstash/benchmark` to get the host and credentials
- `curl -u "$ES_USER:$ES_PW" "$ES_HOST/benchmark_summary/_search"`
Fixes: https://github.com/elastic/ingest-dev/issues/3377
* Rework the logic to delete DLQ eldest segments to be more resilient on file not found errors and avoid to log warn messages that there isn't any action the user can do to solve.
* Fixed test case, when path point to a file that doesn't exist, rely always on path name comparator. Reworked the code to simplify, not needing anymore the tri-state variable
This a refactoring of test fixture.
Avoid mocking the value returned in global SETTINGS constant. Use instead the local setting map instance used in subject creation.
* Add wolfi as an option to the build process
* Add docker acceptance tests for the wolfi image
* Change how tests are done on the java process, due to "ps -C" not being available on wolfi
replaces and closes https://github.com/elastic/logstash/pull/16116
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
* p2p: extract interface from v1 pipeline bus
* p2p: extract pipeline push to abstract
* p2p: add opt-in unblocked "v2" implementation
Adds a v2 implementation that does not synchronize on the sender so that
multiple workers can send events through a common `pipeline` output instance
simultaneously.
In this implementation, an `AddressStateMapping` provides synchronized
mutation and cleanup of the underlying `AddressState`, and allows only
queryable mutable views (`AddressState.ReadOnly`) to escape encapsulation.
The implementation also holds indentity-keyed mapping from `PipelineOutput`s
to the set of `AddressState.ReadOnly`s it is regested as a sender for so
that they can be quickly resolved at runtime.
* p2p: more tests for pipeline restart behaviour
* p2p: make v2 pipeline bus the default
Updates the DLQ reader to create a notification file (`.deleted_segment`) which signal when a segment is deleted in consequence of `clean_consumed` set. Updates the DLQ writer to have a filesystem watch so that can receive the reader's signal and update the exposed metric, loading the size by listing FS segments occupation.
The PR was created to skip resolving environment variable references in comments present in the “config.string” pipelines defined in the pipelines.yml file.
However it introduced a bug that no longer resolves env var references in values of settings like pipeline.batch.size or queue.max_bytes.
For now we’ll revert this PR and create a fix that handles both problems.
Updates the plain, json and pipeline appenders in default config/log4j2.properties to define a delete rule executed during the rollover strategy, which deletes compressed log archives older than 7 days.
Updates the documentation that describe the logging configuration to explain how the rollover file works, how to configure the strategy, in particular how to update to setup space limitation condition on the rollover.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* pq: avoid blocking writer when queue is precisely full
A PQ is considered full (and therefore needs to block before releasing the
writer) when its persisted size on disk _exceeds_ its `queue.max_bytes`
capacity.
This removes an edge-case preemptive block when the persisted size after
writing an event _meets_ its `queue.max_bytes` precisely AND its current
head page has insufficient room to also accept a hypothetical future event.
Fixes: elastic/logstash#16172
* docs: PQ `queue.max_bytes` cannot be less than `queue.page_capacity`
* A logic to handle non-unicode payload in Logstash.
* Well tested and code organized version of the logic.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Upgrade jrjackson to 0.4.20
* Code review: simplify the logic with a standard String#encode interface with replace option.
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Adds section to describe the intended usage of and impact on memory sizing.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
In the grammar definitions for hashes, `whitespace` was replaced with `cs` to allow either whitespace _or_ comments.
Additionally, the grammar definition for comments was previously required to end with a newline, now it can end with a newline _or_ EOF, using the "not anything" treetop rule `!.`.
Co-authored-by: Jonas Lundholm Bertelsen <jonas.lundholm.bertelsen@beumer.com>
Adds log.format.json.fix_duplicate_message_fields feature flag to rename the clashing fields when json logging format (log.format) is selected.
In case two message fields clashes on structured log message, then the second is renamed attaching _1 suffix to the field name.
By default the feature is disabled and requires user to explicitly enable the behaviour.
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Bugfix to parse correctly Java options when the environment variable LS_JAVA_OPTS contains multiple definitions separated by space character.
Adapt the parsing of `LS_JAVA_OPTS` environment variable to split by space various definitions it can contains.
* Wipe out comment lines if config comment contains.
* Remove substitution var process when loading the YAML, instead align on the generic approach which LSCL happens during the pipeline compile.
* Update logstash-core/src/main/java/org/logstash/config/ir/PipelineConfig.java
Put the logging config back as it is being used with composed configs.
Introduce a new setting named `pipeline.buffer.type` which could be valued direct or heap to enable the allocation on Java heap.
The processing of the setting is done in `LogStash::Runner#execute` and sets the Java properties considered by Netty to disable the direct allocation: `io.netty.noPreferDirect`.
However, if that system property is already configured explicitly by the user (because set in `jvm.options`or `LS_JAVA_OPTS`) the setting doesn't take place and warning log is reported, respecting the user's will.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Replace 'git show-ref' with 'git rev-parse' to fix the issue where show-ref is not working as expected.
* Use git checkout instead 'git rev-parse'.
* Apply prune dependencies recommended for big projects (like we have multi gradle projects) by Snyk.
* Apply prune repeated dependency option directly to snyk monitor.
* Avoid the exit, continue scanning to the end.
* Remove the debugging.
This commit adds logic to copy the appropriate env2yaml file to the Docker image
* Clean up env2yaml folder
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Change trigger from pull_request_target to pull_request, as the former uses the base branch instead of the PR source code.
This allows simplification of the checkout action (also took the opportunity to bump from v2 to v4).
* Save name came through ENV vars to let Logstash decide using either keystore or ENV value.
* Apply suggestions from code review to simplify array declaration.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
---------
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Given that JRuby comes with ruby-maven-libs 3.3.9 this commit upgrades the gem to 3.8.9 and ensures files from 3.3.9 are not included in the distribution.
As a follow up to #15861 this commit splits the current unit tests step
for the Windows JDK matrix pipeline to two that run
Java and Ruby unit tests separately.
Closes https://github.com/elastic/logstash/issues/15566
* Add arm64 support for env2yaml
This commit builds env2yaml in arm64 and amd64 flavors, and uses
$TARGETARCH in the Dockerfile to ensure that the correct version is used
when building for alternative architectures
Fixes: #15913
* Add env2yaml executables to build context
* Split `COPY_FILES` for readability
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Remove unused method createFlushScheduler and add some debug logs to the finalizeSegment method to better follow what happens during its execution in case of problems.
This PR adds a shutdown method to the SchedulerService class used to handle actions to be executed on a certain cadence. In particular is used to execute scheduled finalization of DLQ head segment.
Updates the close method of the DLQ writer to invoke this additional shutdown on the service instance.
Adds a burning of time condition to avoid a collision of time which, under certain circumstances, would fail the test.
The sealing of a segment happens if the segment is considered as stale, which requires 2 conditions:
- the segment must have received a write.
- the time of the last write must exceed the flush interval.
In this failing test, the flush interval is set to ZERO because of the synchronicity of the test, to avoid time dependency. However, with coarse grain timer resolution, could happen that the last write coincide with the time of the stale check, so fail the seal condition.
Adds a global new task named javaTests which groups both :logstash-core:javaTests and :jvm-options-parser:test to include the JvmOptionParser unit test in javaTests phase.
PR#15900 missed a few more places where Logstash is installed but
a working minimal pipeline config is added.
This commit fixes that and stabilizes all acceptance tests, thus
minizing the need for time consuming BK retries of corresponding
steps.
Relates #15900
Relates https://github.com/elastic/logstash/issues/15784
This commit tightens the checks for the status
output of the Logstash OS service to specifically
scan for `org.logstash.Logstash` rather than
only the jdk path.
The reason is that the startup script first runs
an options parser, and then the logstash process
itself, both referencing the JDK path.
Closes https://github.com/elastic/ingest-dev/issues/2950
* Exclude plugins feature in OSS distributions.
* Set elastic_integration plugin default.
* Remove non-OSS plugins after installing default plugins.
* Testing local can't find gem bundler (= 2.3.26) issue.
* Include extract non-OSS plugins logic indocker build operations.
* Only default plugins can be excluded from OSS distros.
* Simplification: instead conditional check, use intersection to make OSS exlucluded plugin list.
* Gem and specification files still stay after removing the plugin. This change removes the stayed files.
* Rename oss-exclude to skip-oss to align namings with other params.
* Make intersection method simpler.
* [Test] Temporary excluding elastic integration plugin from default plugin list.
* Sets elastic_integration plugin default back. When removing locally installed gems, Gem::Specification doesn't recognize the gem. We have Bundle::setup in the removal logic but it is causing an issue when we re-use the bundle.
* Test the build order, remove plugin from cache logic seems invalid since we don't pack the cache.
Our shared CI infrastructure relies on this file being here with a specific
format, expanding a given `ELASTIC_STACK_VERSION` environment variable and
an optional `SNAPSHOT` environment variable into the information necessary
to target either the most recent published release or a snapshot representing
a candidate for the _next_ possible release in the series.
When we cut a new minor branch from `main` (at feature freeze), the tip of
that branch contains unreleased code that is in-flight for the next release.
We need to have a way to reference artifacts from this in-flight release,
instead of skipping over them while a minor release is in flight.
This change affects the _semantics_ of what is tested during a minor stack
release's feature-freeze period, ensuring that automated tests throughout
the Logstash ecosystem continue to run against the pending release with NO
changes required in those plugins.
It also introduces a new `snapshots.main` entry, which refers to snapshot
artifacts built from the un-feature-frozen `main` branch, so that plugins
can _optionally_ test against the _next_ minor in advance of its feature
freeze, as they currently do unintentionally in builds that target
`ELASTIC_STACK_VERSION=8.x SNAPSHOT=true`.
Logstash on ECK requires openssl command to build TLS keystore.
This commit adds `microdnf install -y openssl` to ensure the command exists in ubi image.
Updates Netty's configuration of maxOrder to a previously proven value, if not already customised by the user.
Adds a step to the JvmOption parsing tool, which is used to compose the JVM options string to pass down to Logstash at startup.
The added step rework the parsed options to set the allocator max order -Dio.netty.allocator.maxOrder=11 so that the maximum pooled buffer is up to 16MB and not 4MB.
This option is added iff it's not yet specified by the user
Update the env2yaml to have a go.mod instead of relying on disabling go modules, otherwise building with golang 1.22 will fail in the future.
This change also directly uses the golang image to build the binary removing the need for an intermediate image.
* Fix the latest 8.12 version, to the actually latest released
* Update version fo 7.17
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
This commit fixes the startup of the Logstash service during packaging
tests by adding a minimal pipeline config. Without it, the service was
flapping from start to start and vice versa causing test flakiness.
Relates https://github.com/elastic/logstash/issues/15784
This commit allows separate running of Java and Ruby tests on Windows i.e. the same way as we currently do on unix (unit_tests.sh) via a cli argument.
If no argument has been supplied, both tests are run (as it does now).
The wrapper script is also rewritten from old batch style script to Powershell.
This work allows us to split the existing Windows CI job in a subsequent PR to separate steps, as we currently do on Linux.
Relates: https://github.com/elastic/logstash/issues/15566
Currently for pipelines that have steps with auto retries, if there
is a failure of a step, even if the subsequent retry succeeds, we
received Slack alerts.
This commits uses new functionality available in the Buildkite PR Bot
to avoid Slack notifications in those cases and changes the settings
for all pipelines where we have, or expect to have, retries.
This commit adds a schedule to run the exhaustive pipeline
(biweekly, every other Wednesday @2AM UTC) and the aarch64
(weekly, every Monday@2AM UTC).
Closes https://github.com/elastic/ingest-dev/issues/2852
This commit adds (up to 3) retries for all steps of the `serverless-integration-testing`
pipeline as a stop-gap measure to prevent network related transient failures.
* Describe how to troubleshoot connectivity problems at HTTP level with Azure Blob Storage
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
This commit fixes IT failures that frequently occur after
version bumps due to missing unified release snapshot builds for
the new version.
This commit uses project specific DRA snapshot URLs for ES and Filebeat
in all cases apart from release builds.
Closes#2825
The current mechanism of discovering the latest released version per
branch (via ARTIFACTS_API) isn't foolproof near the time of a new
release, as it may be pick a version that hasn't been released
yet. This leads to failures[^1] of the packaging upgrade tests, as we
attempt to download a package file that doesn't exist yet.
This commit switches to an API that that is more up to date regarding
the release version truth.
[^1]: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/125#018d319b-9a33-4306-b7f2-5b41937a8881/1033-1125
This commit fixes the flaky IT test:
`install non bundle plugin successfully installs the plugin with debug enabled`
by being a bit more lenient with the output which can get garbled by Bundler.
Closes#15801
* Ruby code coverage with SimpleCov json formatter
* [CI] Send Java and ruby tests to sonarqube simultaneously
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Added compiled classes to artifacts
* Test change
* Removed test changes
* Returned back ENABLE_SONARQUBE condition
* Removed debug line
* Diable Ruby coverage if ENABLE_SONARQUBE is not true
* Run sonar scan on pull requests and onn push to main
* Run sonar can on release branches
Having enabled branch pushes as triggers for the exhaustive test
pipeline, we have an unwanted sideeffect, that when using
PR automation (@logstashmachine) to create backports, the temporary
branches also triggered exhaustive tests.
This commit skips triggering this pipeline when upstream branches
starting with `backport` get pushed.
There is occasional flakiness mainly with IT tests requiring us to
manually retry such failures when we raise PR (or the first
group of the exhaustive suite, which runs the same steps).
This commit adds up to 3 retries for all the steps of the PR
pipeline.
This commit enables running the exhaustive tests Buildkite pipeline
(i.e. the equivalent to the `main` Jenkins tests) ; the trigger is
code events, i.e. direct pushes, merge commits and creation of new branches.
CI is skipped if changes are only related to files under `docs/`.
This commit added a few jvm.options properties to configure the Jackson read constraints defaults (Maximum Number value length, Maximum String value length, and Maximum Nesting depth).
This commit modernizes the qa/acceptance (packaging) test framework by
moving away from Vagrant and having the tests operate locally.
As we are migrating to Buildkite, the expectation is that those tests
will run on dedicated vms thus removing the necessity of vagrant.
Relates: https://github.com/elastic/ingest-dev/issues/1722
Add the `logstash-filter-elasticsearch_integration` plugin to the list of test_plugins pipeline.
In doing so, reworked the definition of plugins, by not just specifying a plugin name but also adding support leve, type and organization. This necessary for plugins which repository is not part of `logstash-plugins` Github organization but at the same time permit an easier management of the different axes that partitions the plugin landscape.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes how the keystore tool handle the command's options, including validation for unknown options, and adding the --stdin flag to the add command.
- Re-enabled logstash-filter-json and logstash-codec-netflow plugins into the test_plugins BK pipeline.
- Moved codecs tier2 to be executed in separata VM.
- Commented plugin logstash-codec-collectd till is fixed
PR#15729 missed the input step. As a result when the job is triggered
the steps are executed, but the pause icon still shows in the job
requiring manual unblock[^1]
This commit also skips the input step when the job is triggered from
the scheduler pipeline.
[^1] https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/86
The recent PRs #15668 and #15705 refactored jobs with a custom schedule
to leverage a centralized trigger pipeline.
An unexpected sideffect of this is that the conditional for the wait
step doesn't work anymore.
This commit skips the wait step when the JDK matrix pipelines get triggered
from another pipeline.
As a follow up to #15705, this commit shushes Slack alerts for the
common Buildkite scheduling pipeline. This is because the pipeline
scheduler just triggers other pipelines which have their own dedicated
alerts setup, therefore we want to avoid duplicate alerts when
there is a failure with one of the triggered pipelines.
This commit is a pre-requisite for adding unit + IT tests in a
dedicated phase of the Exhaustive tests pipeline.
It refactors the tests currently used by PR jobs, so that they become
reusable.
As an improvement from #15668 / #15700, rather than having one
dedicated side-car scheduling job per pipeline, we move to a single
scheduling job. Various pipelines that need triggering with different
schedules now live under each schedule in the new pipeline.
This reduces the amount of jobs we have to maintain in yaml.
This commit enhances the functionality introduced in #15668 and #15700
by allowing a single Buildkite scheduling job to trigger several
pipelines, in addition to multiple branches which it already does.
We rename the env var PIPELINE_TO_TRIGGER to PIPELINES_TO_TRIGGER
which now supports comma separate values.
This enhancement can be useful for pipelines like JDK matrix which
have variants (Linux and Windows) that we want to trigger with a single
scheduling job, thus reducing unnecessary entries in catalog-info.
This commit fixes a few bugs introduced in #15668 related to paths for
the calling script. We also stop limiting the execution only from the
main branch (to facilitate e.g. tests from PRs) and, finally, remove
the async clause, which is not needed, since by default BK steps are
run in parallel.
This commit is the first making use of #15627 to remove hard coded
branches for the DRA Snapshot build schedule.
With this pattern, we will only need to keep `ci/branches.json` up to date,
as versions evolve, and not need to update/maintain hard coded branches
in `catalog-info.yaml` anymore.
Once this is verified working, we'll add a corresponding schedule
pipeline (in `catalog.info`) for the JDK matrix job.
Relates: https://github.com/elastic/ingest-dev/issues/2664
Now that CI VM images are pre-provisioned with various flavors of
Java 21, we add the option for the corresponding CI job.
Adoptium 17 remains the default pre-selected option.
Relates https://github.com/elastic/ci-agent-images/pull/463
Introduces a new interface named SchedulerService to abstract from the ScheduledExecutorService to execute the DLQ flushes of segments. Abstracting from time provides a benefit in testing, where the test doesn't have to wait for things to happen, but those things could happen synchronously.
Update the test_plugins pipeline script to execute only the unit tests.
Use the vendored JRuby in every Ruby related duty, such as running `bundler` and `gem`.
Temporarily comments plugins that has needs to be fixed and already fails on their Travis CI.
Executes the testing of input tier1 plugins in VM instead of Buildkite agent.
In DLQ unit testing sometime the DLQ writer is started explicitly without starting the segments flushers. In such cases the test 's logs contains exceptions which could lead to think that the test fails silently.
Avoid to invoke scheduledFlusher's shutdown when it's not started (such behaviour is present only in tests).
This commit adds the compatibility tier for the Exhaustive tests suite.
Specifically, we introduce two new groups (running in parallel) for Linux and Windows compat tests.
Linux picks one OS per family from [^] and likewise Windows one of the three available choices from the same file.
We also support manual override, if user chooses to, by setting `LINUX_OS` or `WINDOWS_OS` as env vars in the Buildkite build prompt (in this case there is no randomization, and only one OS can be defined for Linux and Windows respectively).
For example:
```
LINUX_OS=rhel-9
WINDOWS_OS=windows=216
```
Relates:
- https://github.com/elastic/ingest-dev/issues/1722
[^1]: 4d6bd955e6/.buildkite/scripts/common/vm-images.json
This commit added support to add and remove multiple keystore keys in a single operation. It also fixed the empty value validation for editing existing key values and added ASCII validation for values.
The last remaining Jenkins job prior to BK migration is for
exhaustive tests. The compatibility phase seems to be failing
since 57dc14c92
with Java 17.0.2
This commit switches from OpenJDK 17 (whose last release was 17.0.2)
to AdoptiumJDK 17 which actively receives updates and is bundled
in the custom images used by Jenkins.
* Use Java installed BK agent and remove unnecessary git clone operation since repo is already cloned.
* Switch back to normal VM since Logstash BK agent doesn't support docker operations.
This commit adds a maximum default (global) timeout for every pipeline
definition (now that it's possible to define this programmatically in
an RRE).
The default values have been chosen arbitrarily based on intuition
about how much (in the worst case) we should wait for each job to
run until we consider them stuck/failed.
While at it, we update the yaml schema for RREs to point to the
latest commit (rather than a pinned commit that doesn't reflect the
latest changes, e.g. `maximum_default_timeout`).
Relates #15380
This removes the dependency on jackson's dataformat-yaml. Since there's only a single place where this library is used in core: to load the plugin alias definition, the code can be replaced by the underlying snakeyaml.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 93d8a9da32)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes the issue with Jackson > 2.15 and `log.format=json`: "Can not write a field name, expecting a value", by adding a default serializers for JRuby's objects.
This commit enables scheduled runs of the JDK matrix pipelines for
both Windows and Linux, once a week, Tuesdays at 1am UTC, using the
pipeline defaults for OS and JDK.
The last part of the Logstash JDK matrix CI migration from Jenkins to
Buildkite is AmazonLinux 2023.
While we have a working image[^1], this is the only step that requires
a agent that runs on AWS.
This commit refactors the builder to support GCP or AWS agents depending
on the OS.
[^1]: https://github.com/elastic/ci-agent-images/pull/441
Add missing yaml-language-server definition to Buildkite pipeline files
(static and dynamic generated) for consistency and to ease spotting
errors with editors.
This commit adds a global max timeout of 90min for the supported plugins
Buildkite pipeline. This prevents hanging builds (for 24hrs, which is
the default).
Relates: #15380
Build's maximum_timeout_in_minutes and default_timeout_in_minutes are available now through the catalog-info.yaml file.
As this change was made manually before we implemented this in RRE/Terrazzo, they got reverted to the default value (0); thus, I am raising this PR to get it as it was specified before the upgrade (120 mins.)
This commit is the first part of the migration of JDK matrix tests
from Jenkins to Buildkite. There will be two separate pipelines, for
Linux and Windows.
Linux is currently limited to Ubuntu 22.04 and 20.04, but
additional operating systems will be added outside of the Logstash
repository seamlessly through additional VM images.
Steps are created dynamically and the underlying script is meant to be
common for Linux and Windows. Windows is currently a stub and
will be added in a follow up PR.
Relates:
- https://github.com/elastic/ingest-dev/issues/1725
- https://github.com/elastic/ci-agent-images/pull/424
* Update the Logstash to Logstash Native doc to reflect the multiple hosts usage.
* Logstash to Logstash comm page, adding LS-to-LS native HA support.
* Apply suggestions from code review
Refining the context.
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
---------
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
* geoip: extract database manager to stand-alone feature
Introduces an Elastic-licensed GeoipDatabaseManagement tool that can be used
by ANY plugin running on Elastic-licensed Logstash to retrieve a subscription
to a GeoIP database that ensures EULA-compliance and frequent updates, and
migrates the previous Elastic-licensed code-in-Logstash-core extension to
the Geoip Filter to use this new tool, requiring ZERO changes to in-the-wild
versions of the plugin.
The implementation of the new tool follows the previous implementation as
closely as possible, but presents a new interface that ensures that a
consumer can ATOMICALLY subscribe to a database path without risk that the
subscriber will receive an update or expiry before it is finished applying
the initial value:
~~~ ruby
geoip_manager = LogStash::GeoipDatabaseManagement::Manager.instance
subscription = geoip_manager.subscribe('City')
subscription.observe(construct: ->(initial_dbinfo){ },
on_update: ->(updated_dbinfo){ },
on_expire: ->( _ ){ })
subscription.release!
~~~
* docs: link in geoip database manager docs
* docs: reorganize pending 'geoip database management' feature
* docs: link to geoip pages from feature index
* geoip: add SubscriptionObserver "interface"
simplifies using Subscription#observe from Java
* geoip: fixup SubscriptionObserver after rename
* geoip: quacking like a SubscriptionObserver is enough
* geoip: simplify constants of legacy geoip filter extension
* geoip: bump logging level to debug for non-actionable log
* geoip: refine log message to omit non-actionable info
* re-enable invokedynamic (was disabled to avoid upstream bug)
* geoip: resolve testing fall-out from filter extension's "private" constants removal
* geoip: consistently use `DataPath#resolve` internally, too
* Adapted the JDK's download URL creation to intereact with Elastic catalog to get metadata, and return the catalog download link instead of directly pointing to Adoptium API
* Silenced the Download task of JDK to print the full url
Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861
Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message.
PR#15466 skipped the Java unit tests as on the `main` and `8.11`
branches they attempted to run sonar scans (which are only meant to
run for PRs).
This commit re-enables the Java unit tests, taking advantage of #15486,
disabling the sonar scan part of the test suite.
This commit introduces a way to optionally disable SonarQube scanning
and coverage reports when running Java unit tests. The integration
was introduced in #15279, however, there are cases (e.g. running Java
unit tests outside of PRs) where we don't want this integration.
Disabling can be achieved by setting the env var `ENABLE_SONARQUBE`
to `false`.
This commit is the first part of migrating away the aarch64 Jenkins
jobs to Buildkite. It adds a group of exhaustive test steps in the
aarch64 pipeline.
The java unit tests are temporarily disabled as they run SonarQube
scans which need to be associated with pull requests.
Relates:
https://github.com/elastic/ingest-dev/issues/1724
This commit enables automatically updated comments from Buildkite for
PRs. It posts comments to PRs whenever a Buildkite build finishes or
encounters its first failing step and keeps the comment updated.
This commit enables the snapshot job schedule for Buildkite jobs.
They are set to run after Jenkins (scheduled @midnight) so that any
remote/unknown chance of conflict with the release manager is limited.
While at it, we also enable slack notifications for failures to the
same channels as Jenkins.
The new psych 5.1.1 gem seems to not work when installed in JRuby 9.4 (used in main).
This change pins the version back to 5.1.0 until ruby/psych#655 is sorted.
Defines a Buildkite pipeline to run specs tests on tier1 and tier2 plugins, using container images. The tasks are divided by plugin type for each tier, to run them in parallel. Once all tier1 testing is completed then starts with tier2.
This commit adds a Buildkite resource for pull requests and
the two simple jobs, license checking and ruby unit tests that are
already part of the Jenkins PR multi job setup.
As this is WiP, slack notifications aren't enabled.
* Add known imap and email plugin issues section to Logstash 8.10+ versions.
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit a88f82e77f)
DRA artifact builds support two optional parameters
`VERSION_QUALIFIER_OPT` and `DRA_DRY_RUN`. The most important is
`VERSION_QUALIFIER_OPT` which should be provided when `alpha1` or
similar versions need to be built.
Currently, after clicking new build, the pipeline takes ~20s to
assemble the steps and then pauses the job waiting for these options to be
filled (or just accept the empty defaults) and press continue.
I feel that this could be trappy behavior because the majority of the
use cases don't need it, and it's likely that a user manually clicks
build and forgets that they'd need to confirm these parameters later
on, left with a hanging build.
This commit makes the parameters optional. If needed, they should be
defined explicitly as Environment Variables in the New Build prompt,
after expanding the Options section.
The downside of this approach is that when needed, users need to
consult the documentation about the environment variable names.
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
2023-10-03 14:32:28 -07:00
2177 changed files with 40273 additions and 62179 deletions
## Steps to set up GCP instance to run benchmark script
- Create an instance "n2-standard-16" with Ubuntu image
- Install docker
- `sudo snap install docker`
- `sudo usermod -a -G docker $USER`
- Install jq
- Install vault
- `sudo snap install vault`
- `vault login --method github`
- `vault kv get -format json secret/ci/elastic-logstash/benchmark`
- Setup Elasticsearch index mapping and alias with `setup/*`
- Import Kibana dashboard with `save-objects/*`
- Run the benchmark script
- Send data to your own Elasticsearch. Customise `VAULT_PATH="secret/ci/elastic-logstash/your/path"`
- Run the script `main.sh`
- or run in background `nohup bash -x main.sh > log.log 2>&1 &`
## Notes
- Benchmarks should only be compared using the same hardware setup.
- Please do not send the test metrics to the benchmark cluster. You can set `VAULT_PATH` to send data and metrics to your own server.
- Run `all.sh` as calibration which gives you a baseline of performance in different versions.
- [#16586](https://github.com/elastic/logstash/pull/16586) allows legacy monitoring using the configuration `xpack.monitoring.allow_legacy_collection: true`, which is not recognized in version 8. To run benchmarks in version 8, use the script of the corresponding branch (e.g. `8.16`) instead of `main` in buildkite.
error "Worklflow (WORKFLOW_TYPE variable) is not set, exiting..."
error "Workflow (WORKFLOW_TYPE variable) is not set, exiting..."
;;
esac
info "Uploading artifacts for ${WORKFLOW_TYPE} workflow on branch: ${RELEASE_BRANCH}"
info "Uploading artifacts for ${WORKFLOW_TYPE} workflow on branch: ${RELEASE_BRANCH} for version: ${STACK_VERSION} with version_qualifier: ${VERSION_QUALIFIER}"
if["$RELEASE_VER" !="7.17"];then
# Version 7.17.x doesn't generates ARM artifacts for Darwin
@ -45,17 +42,12 @@ if [ "$RELEASE_VER" != "7.17" ]; then
:
fi
# Deleting ubi8 for aarch64 for the time being. This image itself is not being built, and it is not expected
# by the release manager.
# See https://github.com/elastic/infra/blob/master/cd/release/release-manager/project-configs/8.5/logstash.gradle
# for more details.
# TODO filter it out when uploading artifacts instead
symptom:"1 indicator is unhealthy (`abnormally-terminated-pp-multipipeline`) and 2 indicators are concerning (`slow-start-pp-multipipeline`, `normally-terminated-pp-multipipeline`)"
indicators:
slow-start-pp-multipipeline:
status:"yellow"
symptom:"The pipeline is concerning; 1 area is impacted and 1 diagnosis is available"
diagnosis:
- cause:"pipeline is loading"
action:"if pipeline does not come up quickly, you may need to check the logs to see if it is stalled"
impacts:
- impact_areas:["pipeline_execution"]
details:
status:
state:"LOADING"
normally-terminated-pp-multipipeline:
status:"yellow"
symptom:"The pipeline is concerning; 1 area is impacted and 1 diagnosis is available"
diagnosis:
- cause:"pipeline has finished running because its inputs have been closed and events have been processed"
action:"if you expect this pipeline to run indefinitely, you will need to configure its inputs to continue receiving or fetching events"
impacts:
- impact_areas:["pipeline_execution"]
details:
status:
state:"FINISHED"
abnormally-terminated-pp-multipipeline:
status:"red"
symptom:"The pipeline is unhealthy; 1 area is impacted and 1 diagnosis is available"
diagnosis:
- cause:"pipeline is not running, likely because it has encountered an error"
action:"view logs to determine the cause of abnormal pipeline shutdown"
impacts:
- description:"the pipeline is not currently processing"
RUN for iter in `seq 1 10`; do ./gradlew wrapper --warning-mode all && exit_code=0 && break || exit_code=$? && echo "gradlew error: retry $iter in 10s" && sleep 10; done; exit $exit_code
@ -146,7 +145,7 @@ Run the doc build script from within the `docs` repo. For example:
## Testing
Most of the unit tests in Logstash are written using [rspec](http://rspec.info/) for the Ruby parts. For the Java parts, we use junit. For testing you can use the *test*`rake` tasks and the `bin/rspec` command, see instructions below:
Most of the unit tests in Logstash are written using [rspec](http://rspec.info/) for the Ruby parts. For the Java parts, we use [junit](https://junit.org). For testing you can use the *test*`rake` tasks and the `bin/rspec` command, see instructions below:
### Core tests
@ -253,10 +252,10 @@ All contributions are welcome: ideas, patches, documentation, bug reports,
complaints, and even something you drew up on a napkin.
Programming is not a required skill. Whatever you've seen about open source and
maintainers or community members saying "send patches or die" - you will not
maintainers or community members saying "send patches or die" - you will not
see that here.
It is more important to me that you are able to contribute.
It is more important that you are able to contribute.
# The acceptance test in our CI infrastructure doesn't clear the workspace between run
# this mean the lock of the Gemfile can be sticky from a previous run, before generating any package
# we will clear them out to make sure we use the latest version of theses files
# If we don't do this we will run into gem Conflict error.
[ -f Gemfile ]&& rm Gemfile
[ -f Gemfile.lock ]&& rm Gemfile.lock
# When running these tests in a Jenkins matrix, in parallel, once one Vagrant job is done, the Jenkins ProcessTreeKiller will kill any other Vagrant processes with the same
# BUILD_ID unless you set this magic flag: https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller
exportBUILD_ID=dontKillMe
LS_HOME="$PWD"
QA_DIR="$PWD/qa"
# Always run the halt, even if the test times out or an exit is sent
cleanup(){
cd$LS_HOME
cd$QA_DIR
bundle check || bundle install
bundle exec rake qa:vm:halt
}
trap cleanup EXIT
get_package_type
# Cleanup any stale VMs from old jobs first
cleanup
if[[$SELECTED_TEST_SUITE==$"redhat"]];then
echo"Generating the RPM, make sure you start with a clean environment before generating other packages."
cd$LS_HOME
# in CI (Buildkite), packaging artifacts are pre-built from a previous step
if[[$BUILDKITE==true]];then
exportLS_ARTIFACTS_PATH="$HOME/build"
echo"--- Downloading artifacts from \"build/*${PACKAGE_TYPE}\" to $LS_ARTIFACTS_PATH"
