Implements a plugin `ecs_compatibility` option, whose default value is powered
by the pipeline-level setting `pipeline.ecs_compatibility`, in line with the
proposal in elastic/logstash#11623:
In order to increase the confidence a user has when upgrading Logstash, this
implementation uses the deprecation logger to warn when `ecs_compatibility` is
used without an explicit directive.
For now, as we continue to add ECS Compatibility Modes, an opting into a
specific ECS Compatibility mode at a pipeline level is considered a BETA
feature. All plugins using the [ECS Compatibility Support][] adapter will
use the setting correctly, but pipelines configured in this way do not
guarantee consistent behaviour across minor versions of Logstash or the
plugins it bundles (e.g., upgraded plugins that have newly-implemented an ECS
Compatibility mode will use the pipeline-level setting as a default, causing
them to potentially behave differently after the upgrade).
This change-set also includes a significant amount of work within the
`PluginFactory`, which allows us to ensure that pipeline-level settings are
available to a Logstash plugin _before_ its `initialize` is executed,
including the maintaining of context for codecs that are routinely cloned.
* JEE: instantiate codecs only once
* PluginFactory: use passed FilterDelegator class
* PluginFactory: require engine name in init
* NOOP: remove useless secondary plugin factory interface
* PluginFactory: simplify, compute java args only when necessary
* PluginFactory: accept explicit id when vertex unavailable
* PluginFactory: make source optional, args required
* PluginFactory: threadsafe refactor of id duplicate tracking
* PluginFactory: make id extraction/geration more abstract/understandable
* PluginFactory: extract or generate ID when source not available
* PluginFactory: inject ExecutionContext before initializing plugins
* Codec: propagate execution_context and metric to clones
* Plugin: intercept string-specified codecs and propagate execution_context
* Plugin: implement `ecs_compatibility` for all plugins
* Plugin: deprecate use of `Config::Mixin::DSL::validate_value(String, :codec)`
Clean backport of #12302
This commit adds the ability for the docker build to build artifacts for multiple architectures.
By default, the target architecture is inferred from the architecture of the machine the build is being
run from - running the build from an aarch64 machine will build an aarch64 docker image, while building
from an x86_64 machine will build an x86_64 docker image.
This can be overridden by setting the environment variable DOCKER_ARCHITECTURE to either `x86_64` or
`aarch64`.
This commit also updates the integration tests to test against the architecture from the machine the test
is being run on, and includes the target architecture in the test description.
Changed Linux creation artifacts (tar.gz/deb/rpm) to include the ARM JDK.
Extracted common parts of artifact.rake into functions to be shared between ARM and Intel bundling tasks
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
Elevates visibility of Offline Plugin Management section so that air gapped users
don't have to struggle through instructions that require an internet connection.
Backports: #12283
Related: #12280
Clean backport of #12242
This commit includes the required changes to pass RedHat docker image certification.
This includes:
Moving license files to /licenses folder
Adding required base labels for name, description, vendor and summary
Relates: https://github.com/elastic/dev/issues/1287
Clean backport of #12233
This commit is intended to fix thread safety issues with the JavaKeystore implementation of the secret store.
From reading the code, it appears that thread safety for the keystore was intended to be provided by
a ReentrantReadWriteLock, a read lock for accessing secrets from the keystore, and a write lock for updating
secrets in the keystore.
In practice, this was insufficient, the act of accessing a secret from the keystore involved the mutation of
a shared keyStore object - the keyStore is `load`ed every time a secret is retrieved from the store.
Previous to https://github.com/elastic/logstash/pull/10794, this did not matter, each pipeline held its
own instance of the secret store, effectively meaning that only a single thread would ever access a key
store at any one time. This PR moved to using a shared keystore instance for substitution variables,
exposing the lack of thread safety in the JavaKeystore class.
This commit is intended to be the simplest change to fix the underlying issue, and does not address whether
we *need* to reload the secrets every time they are read.
Relates #12229
when running a pipeline with ordered execution, flushes on the pipeline
were no longer being called when compute is called with an empty batch, causing
issues with the aggregate filter, for example, not being able to push events on
timeout.
Expose the proxy xpack management proxy setting in docker (xpack.management.elasticsearch.proxy).
Also surface the same proxy setting in the sample config.
This commit adds the rake docker_ubi8 rake task, and associated
changes to the docker template and makefiles.
This commit also refactors the acceptance tests to extract xpack tests
into a helper class to allow the same tests to be used in both 'full'
and 'ubi8' docker image tests
Our internal representation of the composite config file needs only to inject
newline delimiters if they are missing, and to avoid doing so if they are
present. This allows `PipelineConfig#sourceReferences()`, used to map back from
the composite line/column to source file/column, to correctly track an offset
using the source fragments `SourceWithMetadata#getLinesCount()`.
Fixes: #12155
Accidentally succeeding at connecting to an HTTP resource that is not a real,
live Elasticsearch (such as an Elastic Cloud instance that has been shut down
and reaped) can cause client initialization to fail.
Clean backport of #12135
This commit adds integration tests for the Logstash docker images. Previous
integration tests were removed in https://github.com/elastic/logstash/pull/10693,
due to the tests being non functional.
The commit adds image and container tests. The image tests check the contents and the
metadata of the image; the container tests check the logstash process, and includes tests
ensuring that logstash runs, and is configurable.
This test also adds a ci script to allow the tests to be run on jenkins, and to split the
running of these tests up based on the image type and includes updates to the rake tasks to
support this.
During the development of PR #11541 to direct ship monitoring data to an monitoring ES cluster without hopping through a production ES cluster, the settings for elasticsearch ouput was cloned into a version without the `xpack` prefix.
Since that feature has been removed the settings should also be removed from the Docker image
In PR #11799 we missed to add the exposure of proxy also as docker env variable so that uses can connect the dockerzied Logstash to a proxed monitoring cluster
Prior to this commit, the value of `org.label-schema.license` and
the values in `org.opencontainers.image.*` were not set, and therefore
would be inherited from the base OS image.
Although fossa has a fossa init tool to auto discover dependencies,
it doesn't work well for Logstash.
The mix of JRuby and Java allows for correct gradle detection but for
Ruby we tell FOSSA to look at the lockfile, which we generate using
ci/bootstrap_dependencies.sh
This is a work in progress and covers 99% of our dependencies.
As we get comfortable we'll have to uncomment a few ruby subprojects
contained in the logstash source tree.
Integration plugins need a different header. For example, the plugin docs should
to point to the integration repo rather than the input, output, filter, or codec
repo. The new header also includes boilerplate text to indicate that the individual
plugin is part of an integration rather than stand-alone. This work implements needed
changes.
