mirror of
https://github.com/elastic/logstash.git
synced 2025-04-22 13:47:21 -04:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ry Biesemeyer
|
7eb5185b4e
|
Feature: health report api (#16520)
* [health] bootstrap HealthObserver from agent to API (#16141) * [health] bootstrap HealthObserver from agent to API * specs: mocked agent needs health observer * add license headers * Merge `main` into `feature/health-report-api` (#16397) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: elastic/logstash#16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) --------- Co-authored-by: ev1yehor <146825775+ev1yehor@users.noreply.github.com> Co-authored-by: João Duarte <jsvd@users.noreply.github.com> Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> Co-authored-by: Andrea Selva <selva.andre@gmail.com> Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * Squashed merge from 8.x * Failure injector plugin implementation. (#16466) * Test purpose only failure injector integration (filter and output) plugins implementation. Add unit tests and include license notes. * Fix the degrate method name typo. Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Add explanation to the config params and rebuild plugin gem. --------- Co-authored-by: Andrea Selva <selva.andre@gmail.com> * Health report integration tests bootstrapper and initial tests implementation (#16467) * Health Report integration tests bootstrapper and initial slow start scenario implementation. * Apply suggestions from code review Renaming expectation check method name. Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * Changed to branch concept, YAML structure simplified as changed to Dict. * Apply suggestions from code review Reflect `help_url` to the integration test. --------- Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com> * health api: expose `GET /_health_report` with pipelines/*/status probe (#16398) Adds a `GET /_health_report` endpoint with per-pipeline status probes, and wires the resulting report status into the other API responses, replacing their hard-coded `green` with a meaningful status indication. --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * docs: health report API, and diagnosis links (feature-targeted) (#16518) * docs: health report API, and diagnosis links * Remove plus-for-passthrough markers Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> --------- Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com> * merge 8.x into feature branch... (#16519) * Add GH vault plugin bot to allowed list (#16301) * regenerate webserver test certificates (#16331) * correctly handle stack overflow errors during pipeline compilation (#16323) This commit improves error handling when pipelines that are too big hit the Xss limit and throw a StackOverflowError. Currently the exception is printed outside of the logger, and doesn’t even show if log.format is json, leaving the user to wonder what happened. A couple of thoughts on the way this is implemented: * There should be a first barrier to handle pipelines that are too large based on the PipelineIR compilation. The barrier would use the detection of Xss to determine how big a pipeline could be. This however doesn't reduce the need to still handle a StackOverflow if it happens. * The catching of StackOverflowError could also be done on the WorkerLoop. However I'd suggest that this is unrelated to the Worker initialization itself, it just so happens that compiledPipeline.buildExecution is computed inside the WorkerLoop class for performance reasons. So I'd prefer logging to not come from the existing catch, but from a dedicated catch clause. Solves #16320 * Doc: Reposition worker-utilization in doc (#16335) * settings: add support for observing settings after post-process hooks (#16339) Because logging configuration occurs after loading the `logstash.yml` settings, deprecation logs from `LogStash::Settings::DeprecatedAlias#set` are effectively emitted to a null logger and lost. By re-emitting after the post-process hooks, we can ensure that they make their way to the deprecation log. This change adds support for any setting that responds to `Object#observe_post_process` to receive it after all post-processing hooks have been executed. Resolves: elastic/logstash#16332 * fix line used to determine ES is up (#16349) * add retries to snyk buildkite job (#16343) * Fix 8.13.1 release notes (#16363) make a note of the fix that went to 8.13.1: #16026 Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16347) * [Bugfix] Resolve the array and char (single | double quote) escaped values of ${ENV} (#16365) * Properly resolve the values from ENV vars if literal array string provided with ENV var. * Docker acceptance test for persisting keys and use actual values in docker container. * Review suggestion. Simplify the code by stripping whitespace before `gsub`, no need to check comma and split. Co-authored-by: João Duarte <jsvd@users.noreply.github.com> --------- Co-authored-by: João Duarte <jsvd@users.noreply.github.com> * Doc: Add SNMP integration to breaking changes (#16374) * deprecate java less-than 17 (#16370) * Exclude substitution refinement on pipelines.yml (#16375) * Exclude substitution refinement on pipelines.yml (applies on ENV vars and logstash.yml where env2yaml saves vars) * Safety integration test for pipeline config.string contains ENV . * Doc: Forwardport 8.15.0 release notes to main (#16388) * Removing 8.14 from ci/branches.json as we have 8.15. (#16390) * Increase Jruby -Xmx to avoid OOM during zip task in DRA (#16408) Fix: #16406 * Generate Dataset code with meaningful fields names (#16386) This PR is intended to help Logstash developers or users that want to better understand the code that's autogenerated to model a pipeline, assigning more meaningful names to the Datasets subclasses' fields. Updates `FieldDefinition` to receive the name of the field from construction methods, so that it can be used during the code generation phase, instead of the existing incremental `field%n`. Updates `ClassFields` to propagate the explicit field name down to the `FieldDefinitions`. Update the `DatasetCompiler` that add fields to `ClassFields` to assign a proper name to generated Dataset's fields. * Implements safe evaluation of conditional expressions, logging the error without killing the pipeline (#16322) This PR protects the if statements against expression evaluation errors, cancel the event under processing and log it. This avoids to crash the pipeline which encounter a runtime error during event condition evaluation, permitting to debug the root cause reporting the offending event and removing from the current processing batch. Translates the `org.jruby.exceptions.TypeError`, `IllegalArgumentException`, `org.jruby.exceptions.ArgumentError` that could happen during `EventCodition` evaluation into a custom `ConditionalEvaluationError` which bubbles up on AST tree nodes. It's catched in the `SplitDataset` node. Updates the generation of the `SplitDataset `so that the execution of `filterEvents` method inside the compute body is try-catch guarded and defer the execution to an instance of `AbstractPipelineExt.ConditionalEvaluationListener` to handle such error. In this particular case the error management consist in just logging the offending Event. --------- Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> * Update logstash_releases.json (#16426) * Release notes for 8.15.1 (#16405) (#16427) * Update release notes for 8.15.1 * update release note --------- Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com> Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co> (cherry picked from commit |
||
Andres Rodriguez
|
4255a8fd1c
|
Rubocop: Enable SpaceAround cops (#15196)
* Enable SpaceARoundBlockParameters * Enable SpaceAroundEqualsInParameterDefault * Enable SpaceAroundKeyword * Enable SpaceAroundOperators * Enable SpaceBeforeBlockBraces, which yields no changes |
||
|
Andres Rodriguez
|
26ff2f7db9
|
Enable Style/TrailingEmptyLines and fix ocurrences (#15179) | ||
Pier-Hugues Pellerin
|
fdda39964d |
Add bin/logstash-plugin prepare-offline-pack command
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins. The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip. And this zip can be installed with the same flow as the pack. Definition: Source Logstash: Where you run the prepare-offline-pack. Target Logstash: Where you install the offline package. PROS: - If you install a .gem in the source logstash, the .gem and his dependencies will be bundled. - The install flow doesn't need to have access to the internet. - Nothing special need to be setup in the target logstash environment. CONS: - The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins. - The source Logstash need to have access to the internet when you install plugins before packaging them. Usage examples: bin/logstash-plugin prepare-offline-pack logstash-input-beats bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats bin/logstash-plugin prepare-offline-pack logstash-filter-* bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats How to install: bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip Fixes #6404 |
||
|
Pier-Hugues Pellerin
|
7b38e798c2 |
Migration of the CLI script to run under Vagrant VMs.
This is the First pass to add the `integration` test for the CLI, the Logstash and the logstash-plugin command. Fixes #5259 Fixes #5350 |