Release Manager builds were failing as `downloadEs` task was being
needlessly run during `rake artifact:all` task. When run with
`RELEASE=1`. this was causing build failures due to the non-availability
of Elasticsearch release artifacts. This commit aims to avoid running
the `downloadES` task when it is not needed, continuing the work done
in #11914
This commit also removes code that was repeated in different parts of
the build script.
Enable filebeat and elasticsearch downloads to pull different
architectures, Filebeat and Elasticsearch use different suffixes
to denote their aarch64 architectures, with beats using arm64 and
elasticsearch aarch64
Carrying on from the work done in #11958, update the gradle build to download
the same version of Elasticsearch as is specified in the logstash version.yml file.
This commit updates the standard integration tests to use the same version of
Elasticsearch that is already downloaded for x-pack integration tests, and also
fixes integration tests to allow for the different responses around hits generated
by different versions of Elasticsearch.
This commit changes the download to pull the version of beats based on the version pulled from the branch rather than from an environment variable, or 6.5.4.
This commit also moves the download logic of Filebeat fromfilebeat_setup.sh to build.gradle in order to use the artifacts API in the same way as the downloadEs task, and does some refactoring to DRY up the artifact download tasks.
This commit also fixes the beats integration test to replace the use of a removed setting.
This commit also sets retries to 3 for the download tasks, using 'retries' functionality from gradle download task plugin
* Use task avoidance API in gradle scripts
This commit uses the task avoidance api (tasks.register vs task.create/
task DSL), as recommended since Gradle 5.1
This should reduce the execution of unnecessary tasks in build jobs, and
hopefully improve build resiliency and execution time.
* Update gradle version to 6.3
Gradle versions prior to 6.3 cannot run under JDK14.
This commit upgrades the version of Gradle to 6.3, and removes all deprecation warnings that can currently be removed.
Changes include:
* Increase gradle memory to 2g
* Increase gradle memory in the license check job to 2g
* Replace use of `testCompile`
* Replace `runtime` with `runtimeOnly`
* Remove`compile` depedencies from gradle files
* Replace deprecated archive methods
* Fix dependencies report build
* Make jruby dependencies 'api', fix archiveVersion
* Set `duplicatesStrategy` for all tasks of type Copy
* Use `configureEach` for global 'withType' calls
** Use the recommended Tasks API calls
(https://blog.gradle.org/preview-avoiding-task-configuration-time)
* Run `./gradlew wrapper` earlier to improve caching
* Use copy with chown for resources that need to be run during `./gradlew wrapper`
- have `bootstrap` task do as little as possible: install gems in Gemfile.template that don't belong to groups
- have test tasks depend on the `installTestGems` task instead of `bootstrap`
- logstash es output is now a dependency because of license checking
- fix out of memory problem in SharedHelpers.trap
Also use release lockfile during installDefaultGems
The release lockfile is only copied to Gemfile.lock if
it doesn't exist. During the `installDefaultGems` task other
plugin installation tasks already occurred, generating a lock file.
This commit removes it before running the plugin installation.
* simplify the plugins-metadata.json file
* sort and update the plugin list in the rakelib/plugins-metadata.json
* remove dependency on twitter input for testing
* sorted Gemfile.template (grouped by group)
* remove default plugins from Gemfile.template
Fixes#10509
fpm requires json 1.x but the artifact rake tasks
will activate the default gem json 2
This PR ensures json 1 is activated early and is present right after
jruby is bootstrapped.
Also downgrades fpm to 1.3 as 1.11.0 wasn't building correctly
Downgrades .ruby-version to a version we have available in CI
Fixes#10396
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
* update bundler to 1.17.1
This commit required some tweaking of how we setup Bundler
due to changes in reset behaviour, an internal variable name change,
and the Bundler::Settings api changing.
* Add VERSION_QUALIFIER support for use by release manager
* Make the gem build processes aware of the version qualifier
* Try debugging xpack ci test failure.
* Try to use the artifacts-api.elastic.co api for ES download.
* It builds/tests locally now.
* add some comments explaining the artifacts-api and the version string
* cahnges requested in review.
Fixes#9956
This does a few small things:
1. Prefer invoking rake tasks via ./gradlew vs rake. This prevents Java from segfaulting on docker for unknown reasons
2. Remove use of COPYING.csv which has already been removed
Fixes#9583
