* Update failing policy in Central Management fetcher and license checker if hit ES down node (#13689)
Wraps the calls to the central management Elasticsearch cluster with the utility class Stud::Try to handle the remote host error when the client used to connect hit a not available node.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit c544ecb380)
* Covered all calls to ES with retryable
* Mocked logger interaction in test after wrapping the client calls with retryable
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Backport PR #12198 to 7.17 branch
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 682f07b703)
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit changes `queue.checkpoint.retry` to `true` by default allowing retry of checkpoint write failure.
Add exponential backoff retry to checkpoint write to mitigate AccessDeniedExcpetion in Windows.
Fixed: #12345
(cherry picked from commit 1a5030bd63)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Ruby allows methods to have default values in arguments if they're not
passed. However if a nil is passed then the default value isn't used.
The artifact:archives tasks were passing nil values to the exclusion
argument, causing all files to be included in the package.
This commit cleans the naming of the path lists and ensures the default
exclusion list is always used.
Updates all Windows batch scripts used as CLI tools to quotes the %JAVACMD% to avoid path problems when the path contains spaces.
(cherry picked from commit a8bd90c22d)
A number of plugins reach into Logstash's i18n translations to "helpfully"
communicate certain configuration errors, but rely on translations that were
moved in 00a99c19e5 from logstash.agent to
logstash.runner. Since then, it is possible to hit obtuse error messages about
failing to load a translation instead of the intended helpful message:
~~~
translation missing: en.logstash.agent.configuration.invalid_plugin_register
~~~
By moving the `logstash.agent` definition to _after_ the `logstash.runner`
definition, we can use YAML tooling to name the `logstash.runner.configuration`
node and then merge its contents into `logstash.agent.configuration`. This
effectively allows us to keep a single definition of those translations while
making them available at both addresses.
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
(cherry picked from commit 8bec0e658a)
* api: avoid 5xx when stats/events not yet populated
* Add catching of metrics exception also for retrieve of :queue metrics. No pipelines means no queues created
* Added empty result when no pipelines is present and query for node info API
Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit a6c0e75b53)
When a pipeline isn't fully initialized, we run the risk of attempting to
format pipeline info that isn't yet fully-shaped. By using safe-fallback
methods like `Hash#dig` and conditional-chaining, we can avoid the spurious
`NoMethodError` caused by sending `[]` to nil.
(cherry picked from commit e9455ca81e)
currently the artifact tasks compute the file listing from a list of
include regexes and exclude regexes. However this is done by hand,
taking each include regex and running it through each exclude regex.
This is quite slow as we add more exclude regexes. This PR changes to
totally relying on Rake::FileList, by feeding it the include and exclude
lists. This speeds up file listing from 150 seconds to 1 second.
(cherry picked from commit edfbabf2fc)
* Fix Logstash cli tools to use the selected JDK under Windows (#13839)
Some Logstash tools invokes directly the JRuby intepreter. The interpreter uses the JVM pointed by two environment variables:
- JAVACMD
- JAVA_HOME\bin\java.exe
The setup.bat script exported the selected JVM under the env var named JAVA, which isn't recognized by vendored jruby.
This commit fixes it renaming to JAVACMD.
(cherry picked from commit 0084492494)
* Fixed the case for JAVA_HOME selection path
* artifacts: omit openssl_pkcs8_pure specs from built artifacts (#13715)
* artifacts: omit openssl_pkcs8_pure specs from built artifacts
* Exclude _all_ top-level spec and test directories from built artifacts
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit 0369ba208d)
* Update releasenotes.asciidoc (#13701)
Fixed OS name. Ubuntu instead of Ununtu.
Co-authored-by: Cris da Rocha <cdarocha.astro@gmail.com>
Fix gem installer tests to enable unpinning the version of bundler
This commit removes changes the gem installer to use real gems, rather than
use `allow_instance_of` during testing, which appears to be problematic with the
latest version of bundler
# Conflicts:
# build.gradle
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
This commit deleted the corrupted zero byte PQ file and recreated the head checkpoint file to get rid of page file size is too small exception
Fixed: #10855
Clean backport of #13727 to branch `7.17.`
----
Original comment:
Use same technique for Unix system to extract the CPU percentage and publish to Logstash's metrics collector.
It doesn't retrieve the full set of metric of a Unix system, but the ones that are available from internal JDK class com.sun.management.OperatingSystemMXBean
(cherry picked from commit d8f4784d69)
This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env.
The substitution happens after syntax parsing and before graph execution.
Fixed: #5115
Clean backport of #13672 to 7.17 branch
----
Fixes an integration test that verifies the capabilities of CLI tool to install a not bundled plugin.
Move away from logstash-input-google_cloud_storage which depends indirectly to OS's package named shared-mime-info, which is not always available.
(cherry picked from commit 7bb56e46dd)
* logging: move init into environment's settings post-processor
Ensures that the non-runner command line utilities like `bin/logstash-keystore`
correctly initialize the logger as-configured.
* fixup: ensure we get ruby stdlib URI & File
(cherry picked from commit 2a5e54cd21)
Backport #13656 to branch 7.17
----
Use the System.lineSeparator instead of "\n" to make the test portable across platforms
(cherry picked from commit 3fdc4c3aa7)
Co-authored-by: Andrea Selva <andrea.selva@elastic.co>
* [Docs] Add pipeline.ecs_compatibility to the list (#13612)
* Add pipeline.ecs_compatibility to the list
* Update docs/static/running-logstash-command-line.asciidoc
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit e11d0364d4)
* Rephrase docs for --pipeline.ecs_compatibility flag for 7.x perspective
Co-authored-by: Toby Sutor <55087308+toby-sutor@users.noreply.github.com>
* field-reference: cap RUBY_CACHE to 10k entries
Reduces the scope of a memory leak that can be caused by using UUIDs or other
high-cardinality field names by preventing the ruby string _keys_ from being
held by the cache indefinitely.
Note: this may not solve the problem entirely, but certainly limits its impact.
Because ConvertedMap requires individual field names to be interned into
the global String intern pool, their eligibility for GC is JVM-specific
and high-cardinality field names should still be avoided.
* noop: field-reference test refactor to consolodate reflection
(cherry picked from commit ca501acdcf)
Clean backport of #13631 to 7.17
Original message:
When the Bash script executes the vendored Ruby it has to use proper `GEM_HOME` to avoid the overwrite that happens inside the logstash.lib.sh
3064f7d0c3/bin/logstash.lib.sh (L161-L165)
(cherry picked from commit 93f37b9609)
Clean backport of #13641 to 7.17
Original message:
Cleanly teardown an integration test that made fall other integration tests.
In some cases the CI integration tests fails because the launched Logstash can't find a gem named `mimemagic`. This gem is installed during a CLI plugin test (install of `logstash-input-google_cloud_storage` plugins kicks in that `mimemagic`).
(cherry picked from commit 640ba8489f)
