Backport PR #13071 to 7.x branch. Original message:
This PR contains commits attempting to fix the broken acceptance tests:
* Fix the set of test platforms used to run unix acceptance tests
Modernizes the list of OS's used in acceptance tests, to the most modern OS's available at https://app.vagrantup.com/elastic,. This removes the centos-6 platform from the build, which is past end-of-life and fails vagrant bootstrapping, causing the build to fail.
This is more of band-aid than anything - in the longer term, we should remove these vagrant based tests completely, and rely
on the build infrastructure to perform OS-based acceptance tests.
* Fix regexes for plugin list tests. …
Fixes tests to support the plugin alias feature. This introduced a new format for
entries emitted by `bin/logstash-plugin list`:
eg
```
└── logstash-input-elastic_agent (alias)
```
This commit fixes the test to account for this change, and whitespace variances.
The Gradle's configuration of task should be as fast as possible and don't break the build.
This commit moves retrieval of Elastic Stack version from the remote registry to the execution phase of the tasks.
Also the tasks that depends on this has received the same change (downloadEs and check EsSHA), moving from configuration to execution phase.
Close#13030
(cherry picked from commit cef339ce57)
Update getting-started-with-logstash.asciidoc (#12706)
Single quotations cause errors, should be double quotes for Windows
Co-authored-by: Megan Humphreys <catonice@gmail.com>
Adds framework for showing both windows and unix examples.
Doc: Add unix command for running basic pipeline (#12714)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Starting with version 7.10.0 the name of LS packages changed, adding os and CPU architecture in the name. This change broke the downloading of those from the benchmarking tool. This commit fixes it, composing correctly the name, based on the version it has to download.
(cherry picked from commit b722360ebd)
Loads the production plugin_aliases.yml definition file and check that every alias has
a properly published gem on RubyGems.
Adds clean up of plugin_aliases.yml files
Fixed task dependency for copyPluginAlias
(cherry picked from commit a5f3153a8f)
Because the "Fatal Error" specs specifically inject fatal errors during
execution, and do so by reacting to a "poison" event, the fatal error prevents
the poison event from being ACK'd in the underlying queue.
By specifying a one-off temporary data directory in these specs and cleaning up
after ourselves, we ensure that a PQ containing un-ACK'd events isn't leaked to
the next spec to run.
(cherry picked from commit 6032e5ff64)
This PR makes the Windows logstash.bat exit with the last %ERRORLEVEL% at the end, so that any error in running Logstash will get propagated back to the command line.
Before this change, logstash.bat would always exit with code 0 - success (when doing cmd.exe /C logstash.bat), even if the java.exe process exited with a non-zero code (e.g. due to Logstash throwing an error at runtime).
(cherry picked from commit 1f9ef97836)
Co-authored-by: Dion Williams <dionrhys1@gmail.com>
Uses the OS defined path separator in Rake script to invoke the gradlew command. Without this the sh('./gradlew assemble') results in error when running .\gradlew clean installDefaultGems.
(cherry picked from commit d2c68fc0f9)
Adds a filter to Reflections library initialization so that when it scan "org.logstash.plugins" it includes only .class files and avoid to load and process AliasRegistry.yml and plugin_aliases.yml
Fixes#12992
(cherry picked from commit a6e9a6bcfd)
Avoid the creation of log4j routing appender for log events without the `pipeline.id` fishtag.
In this way no spurious log file named "pipeline_${ctx:pipeline.id}.log" and logs are not duplicated with main Logstash log file.
(cherry picked from commit 1d6a3e4bb3)
Added test to cover the installation of aliased plugins when exists a gem with same name but that's not a Logstash plugin.
In this case the alias is resolved to the original, skipping the gem retrieved from RubyGems.
(cherry picked from commit cafbf03158)
Remove an useful dynamic creation of appender's log file which leverages the `log.format` property
also when it's explicit by the appender itself.
Log4j configuration leverages the placeholder `${sys:ls.log.format}` to compose the name of the log file.
This generates some not evident conflicts in log4j internals, these conflicts became evident when enabling the `pipeline.separate_logs` feature is enabled and the log4j appender definitions contains both json and plain format.
The problem is that under those circumstances the rollover of the log file doesn't happen.
This commit also add a test against the production log4j configuration, to avoid future regressions.
(cherry picked from commit a0774c4e76)
This commit avoid to check for existence of jar files to decide if run or not Gradle assemble,
basically because the outputs of assemble task are not only jars but also others files, for example plugin-aliases.yml.
In this way the decision to execute or not is left the Gradle logic.
(cherry picked from commit 3eaff3612d)
Remove hard coded alias definitions in favor of yaml descriptor file.
Introduce a single point of aliases definition (logstash-core/src/main/resources/org/logstash/plugins/AliasRegistry.yml), checksum and copy it around to be used by Logstash and by Logstash's plugin management tool.
The descriptor yml file contains a checksum to verify it's not changed accidentally in a deployment of Logstash, if the verification phase fail Logstash avoid to start and plugin management tool avoid to operate.
The signing and copying around is managed by a specific Gradle task invoked during the build.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Fixes#12831
(cherry picked from commit 446dc7d906)
Backport PR #12791 to 7.x branch. Original message:
Version bump of JRuby.
+ Fix: a missing require in bootstrap
(cherry picked from commit ee6038afec)
The module LogStash::PluginManager requires the file `lib/pluginmanager/plugin_aliases.yml` is created,
which happend during the Gradle's 'copyPluginAlias' executed as part of Rake's 'bootstrap'.
(cherry picked from commit 8e62e8a01c)
logstash-keystore integration tests spawn a Java process, which by default uses the system JDK generally exposed with JAVA_HOME environment variable. It could be that this JDK is not the one selected with the build system variable BUILD_JAVA_HOME.
This commit uses the JDK defined in BUILD_JAVA_HOME if present.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
(cherry picked from commit fa9adb4b86)
Backport PR #12925 to 7.x. Original Message:
* Add logstash-integration-elastic_enterprise_search to plugins-metadata.json
* Remove old elastic_app_search plugin and set integration as default
* Add license information for workplace search gem
(cherry picked from commit a935261eeb)
ubi8 image uses microdnf as a package manager, and microdnf does
not support the "yum clean metadata" command. This commit adds
the logic to skip this command if the image_flavor is ubi8
(cherry picked from commit d1b12ded1d)
Backport PR #12891 to 7.x branch. Original message:
On aarch64, yum does not pick the correct 'bind-license' package,
this commit installs a specific noarch RPM
This commit also adds retry to the yum installs and updates.
Update Elasticsearch minimum version requirement to ~>7 also on the 7.x branch.
Relates to #11258
(cherry picked from commit ef9b0d2db5)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
The integrationTests start instances of Elasticsearch so they need it to be present and unpacked in build/ folder before start.
(cherry picked from commit 149ee41a8b)
