* Update failing policy in Central Management fetcher and license checker if hit ES down node (#13689)
Wraps the calls to the central management Elasticsearch cluster with the utility class Stud::Try to handle the remote host error when the client used to connect hit a not available node.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit c544ecb380)
* Covered all calls to ES with retryable
* Mocked logger interaction in test after wrapping the client calls with retryable
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env.
The substitution happens after syntax parsing and before graph execution.
Fixed: #5115
* add product origin header to license checks
* add origin header to Central Management config fetcher
* add origin header to ES output for Monitoring pipeline
(cherry picked from commit 2892964ba1)
With #13308 configuration namespace that started with `http.` was renamed to `api.`, this commit fix a usage left behind.
Use the new `api.enabled` setting in one place instead of the deprecated `http.enable`.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
(cherry picked from commit 88c80ebb19)
* Soften deprecation language and point module deprecations to agent integrations
* Remove extra `and`
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
* settings: add "deprecated alias" support
A deprecated alias provides a path for renaming a setting.
- When a deprecated alias is set on its own, a deprecation notice is emitted
but fetching the canonical setting value will reflect the value set with the
deprecated alias.
- When both the canonical setting (new name) and the deprecated alias (old
name) are specified, it is an error condition.
- When the value of the deprecated alias is queried, a warning is emitted to
the logger and only the value explicitly set to the deprecated alias is
returned.
Additionally, some relevant cleanup is also included:
- Starting Logstash with invalid settings no longer results in the obtuse "An
unexpected error occurred" with backtrace and exception data obscuring the
issue. Instead, a simple message is emitted indicating that the settings are
invalid along with the originating exception's message.
- The various settings implementations share a common logger, instead of each
implementation class providing its own. This is aimed to reduce noise from
the logs and to ensure specs validating logging do not need to tie so
closely to implementation details.
* settings: add password-wrapped setting
* settings: make any setting type capable of being nullable
* settings: add `Settings#names` to power programatic iteration
* cli: route CLI-flag deprecations in to deprecation logger
* settings: group API-related settings under `api.*`
retains deprecated aliases, and is fully backward-compatible.
* webserver: cleanup orphaned attr accessors for never-set ivars
* api: pull settings extraction down from agent
This net-no-change refactor introduces a new method `WebServer#from_settings`
that bridges the gap between Logstash settings and Puma-related options, so
that future additions to the API settings don't add complexity to the Agent.
It also has the benefit of initializing the API Rack App and just ONCE, instead
of once per attempted HTTP port.
* api: add optional TLS/SSL
* docs: reference API security settings
* api: when configured securely, bind to all available interfaces by default
* cleanup: remove unused cert artifacts
* tests: generate fresh webserver certificates
* certs: actually add the binary keystores 🤦
Backport #13306 to branch 7.x
(cherry picked from commit 7395641a43)
----
This commit applies all the changes needed to run Logstash on JDK 17:
- opens access to module java.base for packages sun.nio.ch and java.io to run the application and to execute the tests
- removes SecurityManager classes used during Logstash startup
- fix exception type catched in JavaKeyStore tampering test
Related to meta issue #13306
update golang image to 1.17.1 to get rid of expired DST Root CA X3
disable download manager test cases to silent Faraday::SSLError
Fixed: #13261
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This PR integrates Elasticsearch bootstrap script to help users keep Logstah geoip plugin run without online update check.
Add `xpack.geoip.download.endpoint` option to config geoip database service endpoint.
Users can point to `http://localhost:8080/overview.json` when using the script to bootstrap nginx docker
This PR changes the behavior of copying license files from .tgz
Originally, only two files, MaxMind LICENSE.txt and COPYRIGHT.txt, are required
Now more files, README.txt and Elastic ToC, are potentially required
Instead of targeting the files, this change copies all content in .tgz
This commit contains two fixes
* Fix Date class clash when used in pipelines with Date filter and GeoIP
* Pinned jruby-openssl version 0.10.5 to avoid SSL errors
(cherry picked from commit 6f55066b17)
Upstream `ElasticsearchOptions#es_options_from_settings` already uses the
setting `elasticsearch.ssl.verification_mode` to produce an appropriate
boolean-valued `ssl_certificate_verification` in our `es_settings` hash, so
we can rely on it instead of re-checking equality with a string.
(cherry picked from commit d5becc0082)
This commit avoid an error in gathering monitoring information when webserver is disabled or is not yet started;
which could happen with slow loading pipelines or no pipelines defined from the central management UI.
(cherry picked from commit 91996cf2a2)
add wildcard support in xpack pipeline id
do the pattern matching with glob
add warning msg to wildcard with legacy api
check invalid pipeline in bootstrap
test cases for invalid checking
Fixed: #10558
* replace direct hidden indices access with system indices api
* fulfill backward compatibility
* fix log msg, rename class, simplify response handling
* modularise fetcher
Accidentally succeeding at connecting to an HTTP resource that is not a real,
live Elasticsearch (such as an Elastic Cloud instance that has been shut down
and reaped) can cause client initialization to fail.
fixes a regression introduced with the api_key support for xpack monitoring and management in #11864 which disabled the possibility to not use any authentication by relying on the default options and only enabling monitoring for example. It now ignores the default username option when no password is explicitly set.
Release Manager builds were failing as `downloadEs` task was being
needlessly run during `rake artifact:all` task. When run with
`RELEASE=1`. this was causing build failures due to the non-availability
of Elasticsearch release artifacts. This commit aims to avoid running
the `downloadES` task when it is not needed, continuing the work done
in #11914
This commit also removes code that was repeated in different parts of
the build script.
Refactor: move PipelineConfig from Ruby to Java
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
We have "required" units for a variety of `TimeValue` settings when they are
provided as a `String`, but unquoted values in YAML have been passed through as
Integers, where we long assumed nanosecond units. This frequently leads to
surprise (e.g., when `config.reload.interval` is set to `60`, we consume 100%
of CPU in a tight loop trying to reload and re-parse the configs every 60
nanoseconds).
By making the setting retain the TimeValue object for the entirety of its
lifecycle, we can issue a deprecation notice the first time an Integer value is
encountered. As a secondary benefit, our usage of the setting value in code
becomes more clear since we are empowered to ask `TimeValue` for a numeric
value in a specific scale.
Fixes#11803
