* Provision automatic test runs for ruby/java unit tests and integration tests with fips mode (#17029)
* Run ruby unit tests under FIPS mode
This commit shows a proposed pattern for running automated tests for logstash in
FIPS mode. It uses a new identifier in gradle for conditionally setting
properties to configure fips mode. The tests are run in a container
representative of the base image the final artifacts will be built from.
* Move everything from qa/fips -> x-pack
This commit moves test setup/config under x-pack dir.
* Extend test pipelines for fips mode to java unit tests and integration
* Add git to container for gradle
* move fips-mode gradle hooks to x-pack
* Skip license check for now
---------
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
* Split fips integration tests into two steps (#17038)
* Split fips integration tests into two steps
The integration tests suite takes about 40 minutes. This is far too slow for
reasonable feedback on a PR. This commit follows the pattern for the non-fips
integration tests whereby the tests are split into two sections that can run in
parallel across two steps. This should halve the feedback time.
The logic for getting a list of specs files to run has been extracted to a
shared shell script for use here and in the integration tests shell script.
* Use shared function for splitting integration tests
The logic for getting a list of specs to run has been extracted so that it can
be shared across fips and non fips integration test modes. This commit updates
the non fips integration tests to use the shared function.
* fix typo in helper name (kebab case, not snake)
* Escape $ so buildkite upload does not try to interpolate
* Wrap integration tests in shell script to avoid BK interpolation
* Move entrypoint for running integration tests inside docker
* Skip offline pack manager tests when running in fips mode (#17160)
This commit introduces a pattern for skipping tests we do not want to run in
fips mode. In this case the plugin manager tests rely on using
bundler/net-http/openssl which is not configured to be run with bouncycastle
fips providers.
* Get tests running in FIPS environment (#17096)
* Modify FIPS test runner environment for integration tests
This commit makes two small changes to the dockerfile used to define the fips
test environment. Specifically it adds curl (which is required by integration
tests), make (which is required by test setup), adds a c compiler (gcc and glibc
for integration tests which compile a small c program) and turns off debug ssl
logging as it is extremely noisy in logs and breaking some assumptions in
tests about logfile content.
Closes https://github.com/elastic/ingest-dev/issues/5074
* Do not run test env as root
The elastic stack is not meant to be run as root. This commit updates the test
environment to provision a non root user and have the container context execute
under that providioned user.
Closes https://github.com/elastic/ingest-dev/issues/5088
* Skip unit tests that reach out to rubygems for fips mode
The `update` test setup reaches out to rubygems with net/http which is
incompatible with our use of openssl in fips mode. This commit skips those tests
when running under fips.
See https://github.com/elastic/ingest-dev/issues/5071
* Work around random data request limits in BCFIPS
This commit changes test setup to make chunked calls to random data generation
in order to work around a limit in fips mode.
See https://github.com/elastic/ingest-dev/issues/5072 for details.
* Skip tests validating openssl defaults
Openssl will not be used when running under FIPS mode. The test setup and tests
themselves were failing when running in FIPS mode. This commit skips the tests
that are covering behavior that will be disabled.
See https://github.com/elastic/ingest-dev/issues/5069
* Skip tests that require pluginmanager to install plugins
This commit skips tests that rely on using the pluginmanager to install plugins
during tests which require reaching out to rubygems.
See https://github.com/elastic/ingest-dev/issues/5108
* Skip prepare offline pack integration tests in fips mode
The offline pack tests require on pluginmanager to use net-http library for
resolving deps. This will not operate under fips mode. Skip when running in fips
mode.
See https://github.com/elastic/ingest-dev/issues/5109
* Ensure a gem executible is on path for test setup
This commit modifies the generate-gems script to ensure that a `gem` executable
is on the path. If there is not one on the test runner, then use the one bundled
with vendored jruby.
* Skip webserver specs when running in FIPS mode
This commit skips the existing webserver tests. We have some options and need to
understand some requirements for the webserver functionality for fips mode. The
https://github.com/elastic/ingest-dev/issues/5110 issue has a ton of details.
* Skip cli `remove` integration tests for FIPS
This commit skips tests that are running `remove` action for the pluginmanager.
These require reaching out to rubygems which is not available in FIPS mode.
These tests were added post initial integration tests scoping work but are
clearly requiring skips for FIPS mode.
* Add openssl package to FIPS testing env container
The setup script for filebeats requires an openssl executable. This commit
updates the testing container with this tool.
See https://github.com/elastic/ingest-dev/issues/5107
* Re-introduce retries for FIPS tests now that we are in a passing state
* Backport 17203 and 17267 fedramp8x (#17271)
* Pluginmanager clean after mutate (#17203)
* pluginmanager: always clean after mutate
* pluginmanager: don't skip updating plugins installed with --version
* pr feedback
(cherry picked from commit 8c96913807)
* Pluginmanager install preserve (#17267)
* tests: integration tests for pluginmanager install --preserve
* fix regression where pluginmanager's install --preserve flag didn't
* Add :skip_fips to update_spec.rb
* Run x-pack tests under FIPS mode (#17254)
This commit adds two new CI cells to cover x-pack tests running in FIPS mode.
This ensures we have coverage of these features when running existing x-pack
tests.
* observabilitySRE: docker rake tasks (#17272)
* observabilitySRE: docker rake tasks
* Apply suggestions from code review
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* Update rakelib/plugin.rake
* Update rakelib/plugin.rake
* Update docker/Makefile
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
---------
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* Ensure env2yaml dep is properly expressed in observabilitySRE task (#17305)
The `build-from-local-observability-sre-artifacts` task depends on the `env2yaml`
task. This was easy to miss in local development if other images had been built.
This commit updates the makefile to properly define that dependency.
* Add a smoke test for observability SRE container (#17298)
* Add a smoke test for observability SRE container
Add a CI cell to ensure the observability contater is building successfully. In
order to show success run a quick smoke test to point out any glaring issues.
This adds some general, low risk plugins for doing quick testing. This will help
developers in debugging as we work on this image.
* Show what is happening when rake fails
* Debug deeper in the stack
Show the stdout/stderr when shelling out fails.
* Debug layers of build tooling
Open3 is not capturing stdout for some reason. Capture it and print to see what is wrong in CI.
* Actually run ls command in docker container 🤦
* Update safe_system based on code review suggestion
* Dynamically generate version for container invocation
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Refactor smoke test setup to script
Avoid interpolation backflips with buildkite by extracting to a script.
* Split out message surfacing improvment to separate PR.
Moved to: https://github.com/elastic/logstash/pull/17310
* Extract version qualifier into standalone script
* Wait for version-qualifier.sh script to land upstream
Use https://github.com/elastic/logstash/pull/17311 once it lands and gets
backported to 8.x. For now just hard code version.
---------
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Configure observability SRE container for FIPS (#17297)
This commit establishes a pattern for configuring the container to run in fips mode.
- Use chainguard-fips
- Copy over java properties from ls tar archive
- Convert default jks to BC keystore
- Configure logstash to use java properties and FIPS config
NOTE: this assumes bouncycastle jars are in the tarball. The
https://github.com/elastic/ingest-dev/issues/5049 ticket will address that.
* Exclude plugin manager and keystore cli from observabilitySRE artifact (#17375)
* Conditionally install bcfips jars when building/testing observabilitySRE (#17359)
* Conditionally install bcfips jars when building for observabilitySRE
This commit implements a pattern for performing specific gradle tasks based on a
newly named "fedrampHighMode" option. This option is used to configure tests to
run with additional configuration specific to the observabilitySRE use case.
Similarly the additional jar dependencies for bouncycastle fips providers are
conditionally installed gated on the "fedrampHighMode" option.
In order to ensure the the "fedrampHighMode" option persists through the layers
of sub-processes spawned between gradle and rake we store and respect an
environment variable FEDRAMP_HIGH_MODE. This may be useful generally in building
the docker image.
Try codereview suggestion
* Use gradle pattern for setting properties with env vars
Gradle has a mechanism for setting properties with environment variables
prefixed with `ORG_GRADLE_PROJECT`. This commit updates the gradle tasks to use
that pattern.
See
https://docs.gradle.org/current/userguide/build_environment.html#setting_a_project_property
for details.
* Pull in latests commits from 8.x and update based on new patterns (#17385)
* Fix empty node stats pipelines (#17185) (#17197)
Fixed an issue where the `/_node/stats` API displayed empty pipeline metrics
when X-Pack monitoring was enabled
(cherry picked from commit 86785815bd)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Update z_rubycheck.rake to no longer inject Xmx1g (#17211)
This allows the environment variable JRUBY_OPTS to be used for setting properties like Xmx
original pr: #16420
(cherry picked from commit f562f37df2)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Improve warning for insufficient file resources for PQ max_bytes (#16656) (#17222)
This commit refactors the `PersistedQueueConfigValidator` class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
https://github.com/elastic/logstash/issues/14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.
(cherry picked from commit 062154494a)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* gradle task migrate to the new artifacts-api (#17232) (#17236)
This commit migrates gradle task to the new artifacts-api
- remove dependency on staging artifacts
- all builds use snapshot artifacts
- resolve version from current branch, major.x, previous minor,
with priority given in that order.
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit 0a745686f6)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* tests: ls2ls delay checking until events have been processed (#17167) (#17252)
* tests: ls2ls delay checking until events have been processed
* Make sure upstream sends expected number of events before checking the expectation with downstream. Remove unnecessary or duplicated logics from the spec.
* Add exception handling in `wait_for_rest_api` to make wait for LS REST API retriable.
---------
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
(cherry picked from commit 73ffa243bf)
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
* Additional cleanify changes to ls2ls integ tests (#17246) (#17255)
* Additional cleanify changes to ls2ls integ tests: replace heartbeat-input with reload option, set queue drain to get consistent result.
(cherry picked from commit 1e06eea86e)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
* [8.x] Reimplement LogStash::Numeric setting in Java (backport #17127) (#17273)
This is an automatic backport of pull request #17127 done by [Mergify](https://mergify.com).
----
* Reimplement LogStash::Numeric setting in Java (#17127)
Reimplements `LogStash::Setting::Numeric` Ruby setting class into the `org.logstash.settings.NumericSetting` and exposes it through `java_import` as `LogStash::Setting::NumericSetting`.
Updates the rspec tests:
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
(cherry picked from commit 07a3c8e73b)
* Fixed reference of SettingNumeric class (on main modules were removed)
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [CI] Health report integration tests use the new artifacts-api (#17274) (#17277)
migrate to the new artifacts-api
(cherry picked from commit feb2b92ba2)
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
* Backport 17203 and 17267 8.x (#17270)
* Pluginmanager clean after mutate (#17203)
* pluginmanager: always clean after mutate
* pluginmanager: don't skip updating plugins installed with --version
* pr feedback
(cherry picked from commit 8c96913807)
* Pluginmanager install preserve (#17267)
* tests: integration tests for pluginmanager install --preserve
* fix regression where pluginmanager's install --preserve flag didn't
* [Backport 8.x] benchmark script (#17283)
This commit cherry-picked the missing becnhmark script PRs
The deprecated artifacts-api is removed
[CI] benchmark uses the new artifacts-api (#17224)
[CI] benchmark readme (#16783)
Introduce a new flag to explicitly permit legacy monitoring (#16586) (Only take the benchmark script)
[ci] fix wrong queue type in benchmark marathon (#16465)
[CI] fix benchmark marathon (#16447)
[CI] benchmark dashboard and pipeline for testing against multiple versions (#16421)
* Fix pqcheck and pqrepair on Windows (#17210) (#17259)
A recent change to pqheck, attempted to address an issue where the
pqcheck would not on Windows mahcines when located in a folder containing
a space, such as "C:\program files\elastic\logstash". While this fixed an
issue with spaces in folders, it introduced a new issue related to Java options,
and the pqcheck was still unable to run on Windows.
This PR attempts to address the issue, by removing the quotes around the Java options,
which caused the option parsing to fail, and instead removes the explicit setting of
the classpath - the use of `set CLASSPATH=` in the `:concat` function is sufficient
to set the classpath, and should also fix the spaces issue
Fixes: #17209
(cherry picked from commit ba5f21576c)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Shareable function for partitioning integration tests (#17223) (#17303)
For the fedramp high work https://github.com/elastic/logstash/pull/17038/files a
use case for multiple scripts consuming the partitioning functionality emerged.
As we look to more advanced partitioning we want to ensure that the
functionality will be consumable from multiple scripts.
See https://github.com/elastic/logstash/pull/17219#issuecomment-2698650296
(cherry picked from commit d916972877)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* [8.x] Surface failures from nested rake/shell tasks (backport #17310) (#17317)
* Surface failures from nested rake/shell tasks (#17310)
Previously when rake would shell out the output would be lost. This
made debugging CI logs difficult. This commit updates the stack with
improved message surfacing on error.
(cherry picked from commit 0d931a502a)
# Conflicts:
# rubyUtils.gradle
* Extend ruby linting tasks to handle file inputs (#16660)
This commit extends the gradle and rake tasks to pass through a list of files
for rubocop to lint. This allows more specificity and fine grained control for
linting when the consumer of the tasks only wishes to lint a select few files.
* Ensure shellwords library is loaded
Without this depending on task load order `Shellwords` may not be available.
---------
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* Forward Port of Release notes for `8.16.5` and `8.17.3` (#17187), (#17188) (#17266) (#17321)
* Forward Port of Release notes for 8.17.3 (#17187)
* Update release notes for 8.17.3
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Forward Port of Release notes for 8.16.5 (#17188)
* Update release notes for 8.16.5
---------
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com>
(cherry picked from commit 63e8fd1d21)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
* Add Deprecation tag to arcsight module (#17331)
* [8.x] Upgrade elasticsearch-ruby client. (backport #17161) (#17306)
* Upgrade elasticsearch-ruby client. (#17161)
* Fix Faraday removed basic auth option and apply the ES client module name change.
(cherry picked from commit e748488e4a)
* Apply the required changes in elasticsearch_client.rb after upgrading the elasticsearch-ruby client to 8.x
* Swallow the exception and make non-connectable client when ES client raises connection refuses exception.
---------
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
* Removed unused configHash computation that can be replaced by PipelineConfig.configHash() (#17336) (#17345)
Removed unused configHash computation happening in AbstractPipeline and used only in tests replaced by PipelineConfig.configHash() invocation
(cherry picked from commit 787fd2c62f)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Use org.logstash.common.Util to hashing by default to SHA256 (#17346) (#17352)
Removes the usage fo Apache Commons Codec MessgeDigest to use internal Util class with embodies hashing methods.
(cherry picked from commit 9c0e50faac)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* Added test to verify the int overflow happen (#17353) (#17354)
Use long instead of int type to keep the length of the first token.
The size limit validation requires to sum two integers, one with the length of the accumulated chars till now plus the next fragment head part. If any of the two sizes is close to the max integer it generates an overflow and could successfully fail the test 9c0e50faac/logstash-core/src/main/java/org/logstash/common/BufferedTokenizerExt.java (L123).
To fall in this case it's required that sizeLimit is bigger then 2^32 bytes (2GB) and data fragments without any line delimiter is pushed to the tokenizer with a total size close to 2^32 bytes.
(cherry picked from commit afde43f918)
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
* [8.x] add ci shared qualified-version script (backport #17311) (#17348)
* add ci shared qualified-version script (#17311)
* ci: add shareable script for generating qualified version
* ci: use shared script to generate qualified version
(cherry picked from commit 10b5a84f84)
# Conflicts:
# .buildkite/scripts/dra/build_docker.sh
* resolve merge conflict
---------
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* tests: make integration split quantity configurable (#17219) (#17367)
* tests: make integration split quantity configurable
Refactors shared splitter bash function to take a list of files on stdin
and split into a configurable number of partitions, emitting only those from
the currently-selected partition to stdout.
Also refactors the only caller in the integration_tests launcher script to
accept an optional partition_count parameter (defaulting to `2` for backward-
compatibility), to provide the list of specs to the function's stdin, and to
output relevant information about the quantity of partition splits and which
was selected.
* ci: run integration tests in 3 parts
(cherry picked from commit 3e0f488df2)
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* Update buildkite with new patterns from 8.x
This commit updates the buildkite definitions to be compatible with the
upstream 8.x branch. Specificially:
- Split integration tests for fips into 3 runners.
- Use the new shared bash helper for computing QUALIFIED_VERSION
It also continues standardization of using a "fedrampHighMode" for indicating
the tests should be running in the context of our custom image for the SRE team.
* Bug fix: Actually use shared integration_tests.sh file
After refactoring to use the same script, I forgot to actually use it
in the buildkite definition...
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
* Pin rubocop-ast development gem due to new dep on prism (#17407) (#17433)
The rubocop-ast gem just introduced a new dependency on prism.
- https://rubygems.org/gems/rubocop-ast/versions/1.43.0
In our install default gem rake task we are seeing issues trying to build native
extensions. I see that in upstream jruby they are seeing a similar problem (at
least it is the same failure mode https://github.com/jruby/jruby/pull/8415
This commit pins rubocop-ast to 1.42.0 which is the last version that did not
have an explicit prism dependency.
(cherry picked from commit 6de59f2c02)
Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
* Add age filter fedramp (#17434)
* net-zero-change refactor
* add logstash-filter-age to observabilitySRE artifact
* Add licenses for bouncycastle fips jars (#17406)
This commit adds licences for bouncycastle jars that are added for the
observability SRE container artifact. It re-enables the previously disabled
license check and adds a new one running in fips mode.
* Publish Observability SRE images to internal container registry (#17401)
* POC for publishing observability SRE images
This commit adds a step to the pull_request_pipeline buildkite definition to
push a docker image to the elastic container registry. It is added here to show
that we have the proper creds etc in CI to push the container where it needs to
go. We will likely move this into the DRA pipeline once we are confident it is
pushing to the correct place with a naming convention that works for all
consumers/producers.
The general idea is to build the container with our gradle task, then once we
have that image we can tag it with the git sha and a "latest" identifier. This
would allow consumers to choose between an exact sha for a stream like 8.19.0 or
the "latest". I will also need to factor in the case where we have the tag
*without* the sha postfix. Obviously we will want to fold this in to the existing DRA
pipeline for building/staging images but for now it seems reasonable to handle
this separately.
* check variable resolution
* Move POC code into DRA pipeline
This commit takes the POC from the pull_request_pipeline and adds it to the DRA
pipeline. Noteably, we take care to not disrupt anything about the existing DRA
pipeline by making this wait until after the artifacts are published and we set
a soft_fail. While this is being introduced and stabilized we want to ensure the
existing DRA pipeline continues to work without interruption. As we get more
stability we can look at a tigther integration.
* Disambiguate architectures
Eventually we will want to do proper annotations with manifests but for now
just add arch to the tag.
* Use docker manifest for multi-architecture builds
This commit refactors the POC pipeline for pushing observabilty SRE containers
to handle conflicts for tags based on target architectures. Cells with
respective architectures build containers and push to the container registry
with a unique identifier. Once those exist we introduce a separate step to use
the docker manifest command to annotate those images such that a container
client can download the correct image based on architecture. As a result for
every artifact there will be 2 images pushed (one for each arch) and N manifests
pushed. The manifests will handle the final naming that the consumer would
expect.
* Refactor docker naming scheme
In order to follow more closely the existing tagging scheme this commit
refactors the naming for images to include the build sha BEFORE the SNAPSHOT
identifier. WHile this does not exactly follow the whole system that exists
today for container images in DRA it follows a pattern that is more similar.
Ideally we can iterate to fold handling of this container into DRA and in that
case consumers would not need to update their patterns for identifying images.
* Code review refactor
Rename INCLUDE_SHA to INCLUDE_COMMIT_ID in qualified-version script.
Confine use of this argument to individual invocations instead at top level in scripts.
* Build observabilitySRE containers after DRA is published
This gates build/push for observability SRE containers on success of DRA pipeline.
* x-pack: add fips validation plugin from x-pack (#16940)
* x-pack: add fips_validation plugin to be included in fips builds
The `logstash-integration-fips_validation` plugin provides no runtime
pipeline plugins, but instead provides hooks to ensure that the logstash
process is correctly configured for compliance with FIPS 140-3.
It is installed while building the observabilitySRE artifacts.
* fips validation: ensure BCFIPS,BCJSSE,SUN are first 3 security providers
* remove re-injection of BCFIPS jars
* Update lib/bootstrap/rubygems.rb
* add integration spec for fips_validation plugin
* add missing logstash_plugin helper
* fixup
* skip non-fips spec on fips-configured artifact, add spec details
* Improve smoke tests for observability SRE image (#17486)
* Improve smoke tests for observability SRE image
This commit adds a new rspec test to run the observability SRE container in a
docker compose network with filebeat and elasticsearch. It uses some simple test
data through a pipeline with plugins we expect to be used in production. The
rspec tests will ensure the test data is flowing from filebeat to logstash to
elasticsearch by querying elasticsearch for expected transformed data.
* REVERT ME: debug whats goig on in CI :(
* Run filebeat container as root
* Work around strict file ownership perms for filebeat
We add the filebeat config in a volume, the permissions checks fail due test
runner not being a root user. This commit disables that check in filebeat as
seems to be the consensus solution online for example: https://event-driven.io/en/tricks_on_how_to_set_up_related_docker_images/
* Dynaimcally generate PKI instead of checking it in
Instead of checking in PKI, dynamically generate it with gradle task for
starting containers and running the tests. This improvement avoids github
warning of checked in keys and avoid expiration headaches. Generation is very
fast and does not add any significant overhead to test setup.
* Remove use of "should" in rspec docstrings
see https://github.com/rubocop/rspec-style-guide?tab=readme-ov-file#should-in-example-docstrings
* Ensure permissions readable for volume
Now that certs are dynamically generated, ensure they are able to be read in container
* Use elasticsearch-fips image for smoke testing
* Add git ignore for temp certs
* Fix naming convention for integration tests
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* Use parameter expansion for FEDRAMP_HIGH_MODE
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* Use parameter expansion for FEDRAMP_HIGH_MODE
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
* Use parameter expansion for FEDRAMP_HIGH_MODE
Co-authored-by: Rye Biesemeyer <yaauie@users.noreply.github.com>
---------
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Mashhur <mashhur.sattorov@elastic.co>
NOTE: we decided to squash these commits as the feature branch had cherry-picks (and squshed change sets 182f15ebde ) from 8.x which would potentially make the commit history confusing. We determined that the benefit of having individual commits from the feature branch was outweighed by the potentially confusing git history. This will also make porting this bit of work to other streams more simple.
Non clean backport of #16576
----
Reimplements `LogStash::Setting::String` Ruby setting class into the `org.logstash.settings.SettingString` and exposes it through `java_import` as `LogStash::Setting::SettingString`.
Updates the rspec tests in two ways:
- logging mock is now converted to real Log4J appender that spy log line that are later verified
- verifies `java.lang.IllegalArgumentException` instead of `ArgumentError` is thrown because the kind of exception thrown by Java code, during verification.
* Fixed the rename of NullableString to SettingNullableString
* Fixed runner test to use real spy logger from Java Settings instead of mock test double
This fixes the offline install problem of the logstash-input-java_filter_example off-line install.
(cherry picked from commit 113585d4a5)
Co-authored-by: Nicole Albee <2642763+a03nikki@users.noreply.github.com>
* licenses: allow elv2, standard abbreviation for Elastic License version 2
* json-dump: reduce unicode normalization cost
Since the underlying JrJackson now properly (and efficiently) encodes the
UTF-8 transcode of whichever strings it is given, we no longer need to
pre-normalize to UTF-8 in ruby _except_ when the string is flagged as BINARY
because we have alternate behaviour to preserve valid UTF-8 sequences.
By emitting a _copy_ of binary-flagged strings that have been re-flagged as
UTF-8, we allow the downstream (efficient) encoding operation in jrjackson
to produce equivalent behaviour at much lower cost.
* cleanup: remove orphan unicode normalizer
* Ruby code coverage with SimpleCov json formatter
* [CI] Send Java and ruby tests to sonarqube simultaneously
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Enabled COVERAGE for ruby tests
* Added compiled classes to artifacts
* Test change
* Removed test changes
* Returned back ENABLE_SONARQUBE condition
* Removed debug line
* Diable Ruby coverage if ENABLE_SONARQUBE is not true
* Run sonar scan on pull requests and onn push to main
* Run sonar can on release branches
* deps: downgrade jruby, keep updated default-gem dependencies (#15283)
forward-ports non-release-branch components of #15283 to `main`
* deps: downgrade jruby, keep updated default-gem dependencies
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.
Upstream bug: https://github.com/jruby/jruby/issues/7904
Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.
We also remove a pinned-dependency on `racc` that is no longer relevant.
Resolves: https://github.com/elastic/logstash/issues/15282
* Imported the licenses for some gems
- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri
* specs: avoid mocking global ::Gem::Dependency::new
* build: remove redundanct dependsOn declaration
* deps: notice use of ffi-binary-libfixposix via Ruby license
this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available
---------
Co-authored-by: andsel <selva.andre@gmail.com>
* deps: add license notices for gems moved from default to bundled
---------
Co-authored-by: andsel <selva.andre@gmail.com>
Set of changes to make Logstash compatible to JRuby 9.4.
Bundle JRuby 9.4.3.0
- Redefine space token in `LSCL` and `grammar` treetop from `_` which would generated methods in the form `def _0` (deprecated since `2.7`) to `sc`.
- `I18n.t` method doesn't accept hash as second argument
- `URI.encode` has been replaced with same functionality with `URI::Parser.new.escape`
- `YAML.load` needs explicit `fallback: false` to return false when the yaml string is empty (or contains only comments)
- JRuby's `JavaClass` has been removed, now it can use `java.lang.Class` directly
- explicitly require gem `thwait` to satisfy `require "thwait"` (In `Gemfile.template` and `logstash-core/logstash-core.gemspec`)
- fix not args `clone` to be `def clone(*args)`
- fix `Enumeration.each_slice` which from `Ruby 3.1` is [chainable](https://rubyreferences.github.io/rubychanges/3.1.html#enumerableeach_cons-and-each_slice-return-a-receiver) and doesn't return `nil`. JRuby fixed in https://github.com/jruby/jruby/issues/7015
- Expanded `Down.download` arguments map ca16bbed3c302006967413eb9d3862f2da81f7ae
- Avoid to pass `nil` in the list of couples used in `Hash[ <list of couples> ]` which from Ruby `3.0` generates an `ArgumentError`
- Removed space not allowed between method name and parentheses `initialize (` is forbidden. 29b607dcdef98f81a73ad171639fd13aaa65e243
- With [Ruby 2.7 the `Kernel#open`](https://rubyreferences.github.io/rubychanges/2.7.html#network-and-web) doesn't fallback to `URI#open`, fixed test code that used that to verify open port. e5b70de54c5301f51a767da67294092af0cfafdc
- Avoid to drop `rdoc/` folder from vendored JRuby else `bin/logstash -i irb` would crash, commit b71f73e9c6edb81a7b7ae1305047e506f61c6e8c
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Revert "Revert "Unpin bundler to allow 2.4 (#14894)" (#14942)"
This reverts commit 5e3038a3d3.
* prevent bundler 2.4 from blowing up memory during ./gradlew generatePluginsVersion
for some reason, instances of Bundler::Dsl are retained in
org.jruby.MetaClass. Each Bundler::Dsl will retain Bundler::SourceList
which is > 150MB. This commit sets the internal state of each Dsl's
source list so the large objects can be collected.
The proper fix would be to either reuse a single Dsl object or fix the
code upstream to avoid having Dsl objects retained by org.jruby.MetaClass.
* skip input cloudwatch during generatePluginsVersion
the new bundler 2.4's resolution algorithm seems to struggle in certain
situations, like resolving combination of "logstash-input-cloudwatch" and
"logstash-integration-aws".
Until the issue is solved let's skip it.
This reverts commit 6d08a7c1cd.
Bundler 2.4 is accumulating memory during the `generateDocsVersion` task
outside of the known Bundler 2.3 DepProxy class.
Until the memory mitigation patch is adapted to Bundler 2.4 it's best to revert 6d08a7c1cd.
* A structural change for aliased plugin declarations to support doc header replaces.
* YAML file structure made to be more readable, make AliasPlugin class immutable.
* Do not expose alias doc header replaces since we are not using them. Unit tests failure fix.
Fix gem installer tests to enable unpinning the version of bundler
This commit removes changes the gem installer to use real gems, rather than
use `allow_instance_of` during testing, which appears to be problematic with the
latest version of bundler
* Fix bundler handling of 'without'
Prior to this change, the values set in `set_local` are ignored when invoking
bundler via the command line, as is used with `invoke!`. This commit sets those
values in `ENV` variables instead, fixing the functionality to not install
development gems.
* Update bundler spec to check ENV variable
* Added test to ensure kramdown gem not vendored
* Re-add set_local setting to play nice with `expand_logstash_mixin_dependencies`
* logstash service needs to be installed
* gem_vendored? needs to use full path to vendor files
* use `stdout` from `cat` command to generate spec temporary file
* Removed unnecessary support for supplying a block from #gem_vendored?
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
This commit fixes the `logstash-plugin update` command which fail to update plugin
that depends on a new version of logstash-mixin-ecs_compatibility_support.
It resolves logstash-* dependencies and puts them in bundler update command.
Fixed: #13181
Added test to cover the installation of aliased plugins when exists a gem with same name but that's not a Logstash plugin.
In this case the alias is resolved to the original, skipping the gem retrieved from RubyGems.
Remove hard coded alias definitions in favor of yaml descriptor file.
Introduce a single point of aliases definition (logstash-core/src/main/resources/org/logstash/plugins/AliasRegistry.yml), checksum and copy it around to be used by Logstash and by Logstash's plugin management tool.
The descriptor yml file contains a checksum to verify it's not changed accidentally in a deployment of Logstash, if the verification phase fail Logstash avoid to start and plugin management tool avoid to operate.
The signing and copying around is managed by a specific Gradle task invoked during the build.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes#12831
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
* update bundler to 1.17.1
This commit required some tweaking of how we setup Bundler
due to changes in reset behaviour, an internal variable name change,
and the Bundler::Settings api changing.
In an effort to stabilise our specs and reduce the amount of overflow from
each spec to the next, this patch temporarily sets the default queue type
to be the memory queue.
Fixes#9601
As reported in #9132, if one sets `http_proxy=myproxy:8080` the URL
gets parsed to scheme=myproxy and host=nil. This results in a
NullPointerException.
To solve it, we check that the URL is valid as HTTP/HTTPs and raise a
slightly better error message.
Fixes#9132.
Fixes#9230
* allow installing a plugin using proxy without having to use the `--no-verify` flag
* allow the use of proxies that have basic authentication
* add tests for proxy handling in plugin manager
Fixes#8435
use rm_rf to delete dir and shutdown pipeline after run
avoid the use of rescue nil, bad practice
do not mock close as it prevents closing the file an prevents removing it on Windows
cleanup temporary files and relax file name check for Windows
fix paths handling for Windows
flag the read file string as UTF-8 which is what we expect
fix Windows issues
ignore load_average on windows
windows safe URI
cleanup and fix file handling for windows
wait for pipeline shutdown to complete
revert to original puts
cleanups
use environment for windows platform check
fix hash path
wait for pipeline thread to complete after shutdown
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
When building an offline pack we do a real network call to rubygems to
test the structure of the created package, something the network can be
a bit unreliable and SSLError can be raised, when theses errors happen
we will retry them.
Fix: #7251Fixes#7254
Instead of downloading all the filter plugins we reduce the scope to
just download the `logstash-filter-x*` files, this will make the test
faster and reduce the risk of running into IOError.
Ref: #6904Fixes#7211
We have seen unreliable run in the past on travis or jenkins, in that
test we actually test the creation of a physical package we require
internet access to correctly download the gems, this commit make sure we
retries IOErrors.
Fixes: #6904Fixes#7211
This is a big chang, it:
1. Moves API specs out of their special hierarchy
2. Removes the API spec spec_helper
3. Reactivates that stats command spec (that was accidentally not being
run before due to it not having _spec as a suffix
This was required to fix the preceeding commit, where we added a
before(:each) hook to the spec_helper that wasn't being picked up in
some cases due to the existence of two spec helpers and a $LOAD_PATH
that could change.
Fixes#7132
