This commit adds environment variable LOG_STYLE to control log behavior of docker container
`console` - this is the default to output to standout
`file` - log to disk.
Fixed: #14941
* Skip dependency Gemfile.lock during the artifact creation. This will help vulnerability scanners not to confuse while scanning.
* Apply suggestions from code review
Add to exclude `Gemfile`, apply pattern to include shared gems as well.
Exclude Jruby's bundler and rake from the built artifacts. The artifacts don't need to ship with such dependencies. Also, Logstash will bundle its own bundler for plugin management but it is not the one shipped with jruby.
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* Adds tasks to add bundled JDK to tar file used to run integration tests
* Uses `RUNTIME_JAVA_HOME` environment variable to control whether bundled JDK or
alternative is to be used
* Updates logstash service helper to respect value of `RUNTIME_JAVA_HOME`
* Requires updates to jenkins repo to set `RUNTIME_JAVA_HOME` correctly only for
integration tests that expect to use a custom version of Java, such as the JDK
matrix tests.
This commit adds a rake task `rake artifact:dockerfile_ironbank` to generate ironbank docker build context for automatic release.
The output can be found in build/logstash-ironbank-$VERSION-docker-build-context.tar.gz
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
This commit updates the version of jruby used in Logstash to `9.3.4.0`.
* Updates the references of `jruby` from `9.2.20.1` to `9.3.4.0`
* Updates references/locations of ruby from `2.5.0` to `2.6.0`
* Updates java imports including `org.logstash.util` to be quoted
* Without quoting the name of the import, the following error is observed in tests:
* `java.lang.NoClassDefFoundError: org/logstash/Util (wrong name: org/logstash/util)`
* Maybe an instance of https://github.com/jruby/jruby/issues/4861
* Adds a monkey patch to `require` to resolve compatibility issue between latest `jruby` and `polyglot` gem
* The addition of https://github.com/jruby/jruby/pull/7145 to disallow circular
causes, will throw when `polyglot` is thrown into the mix, and stop logstash from
starting and building - any gems that use an exception to determine whether or not
to load the native gem, will trigger the code added in that commit.
* This commit adds a monkey patch of `require` to rollback the circular cause exception
back to the original cause.
* Removes the use of the deprecated `JavaClass`
* Adds additional `require time` in `generate_build_metadata`
* Rewrites a test helper to avoid potentially calling `~>` on `FalseClass`
Co-authored-by: Joao Duarte <jsvduarte@gmail.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Removes the installation of sys-v init.d scripts from .deb and .rpms.
All supported operating systems use systemd as the init system, as such, it is no longer necessary to continue to ship Sys-v Init.d scripts.
Ruby allows methods to have default values in arguments if they're not
passed. However if a nil is passed then the default value isn't used.
The artifact:archives tasks were passing nil values to the exclusion
argument, causing all files to be included in the package.
This commit cleans the naming of the path lists and ensures the default
exclusion list is always used.
Extracts the bundled JDK's version into a one-line text file which could easily read and print from bash/batch scripts.
Updates Logstash's bash/batch launcher scripts to print the bundled JDK version when warn the user about his override.
Updates the rake script used to create Logstash artifacts (rakelib/artifacts.rake) to include JDK related stuff only if packages that requires it.
This change removes the usage of 'pleaserun', which
was leveraged create the init.d scripts or systemd units.
Now, we ship init.d scripts and systemd units to make
the maintenance easier and following the lead of the
other Elastic projects.
Fixes: #10519, Fixes: #12439, Fixes: #8253
currently the artifact tasks compute the file listing from a list of
include regexes and exclude regexes. However this is done by hand,
taking each include regex and running it through each exclude regex.
This is quite slow as we add more exclude regexes. This PR changes to
totally relying on Rake::FileList, by feeding it the include and exclude
lists. This speeds up file listing from 150 seconds to 1 second.
* artifacts: omit openssl_pkcs8_pure specs from built artifacts
* Exclude _all_ top-level spec and test directories from built artifacts
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
* fpm to 1.13.0 which allows building packages with java 11 + jruby 9.2
* childprocess to 4.x + remove monkey patches
* clamp to 1.x to unlock fpm 1.13.0
This commit fixes two issues with the docker metadata:
Removes non-OCI compliant freeform metadata labels
Uses a consistent build date for all the docker images and dockerfiles
Additionally, this commit adds a `build_docker_ubi8` rake task to enable
`ci/docker_acceptance_tests.sh` to run with no options to build all
docker images for the architecture.
Changed Linux creation artifacts (tar.gz/deb/rpm) to include the ARM JDK.
Extracted common parts of artifact.rake into functions to be shared between ARM and Intel bundling tasks
Create new artifacts with bundled JDK for the supported platforms on x86_64. Download JDK packages from AdoptOpenJDK site, the selected version is loaded from `versions.yml`.
Changed also the launch scripts to give precedence to JAVA_HOME, then fallback on bundled JDK if present, as last resource go to the system Java.
New artifacts produced with bundled JDK are:
- tar.gz with JDK for Linux and Darwin
- zip file for Windows
- dep and rpm
- Docker image
All artifacts without JDK are now postfixed with '-no-jdk' while the ones with JDK included has the architecture extension.
Covered with tests the touched parts
Co-authored-by: Rob Bavey <robbavey@users.noreply.github.com>
This commit adds the rake docker_ubi8 rake task, and associated
changes to the docker template and makefiles.
This commit also refactors the acceptance tests to extract xpack tests
into a helper class to allow the same tests to be used in both 'full'
and 'ubi8' docker image tests
* Introduce integration tests for docker
This commit adds integration tests for the Logstash docker images. Previous
integration tests were removed in https://github.com/elastic/logstash/pull/10693,
due to the tests being non functional.
The commit adds image and container tests. The image tests check the contents and the
metadata of the image; the container tests check the logstash process, and includes tests
ensuring that logstash runs, and is configurable.
This test also adds a ci script to allow the tests to be run on jenkins, and to split the
running of these tests up based on the image type and includes updates to the rake tasks to
support this.
* dont include docker tasks in artifact:all
* don't rebuild tar/zip if source hasn't changed
* allow SKIP_PREPARE to avoid tar creation if no modifications
* don't need a tarball to generate the dockerfile
* remove docker tests as they weren't working anymore
This commit adds a task to produce all necessary files to generate a docker image.
```
% RELEASE=1 rake artifact:dockerfile
....
Dockerfile created in /tmp/elastic/logstash/build/docker
% tree /tmp/elastic/logstash/build/docker
/tmp/elastic/logstash/build/docker
├── Dockerfile
├── bin
│ └── docker-entrypoint
├── config
│ ├── log4j2.properties
│ ├── logstash-full.yml
│ └── pipelines.yml
├── env2yaml
│ └── env2yaml
└── pipeline
└── default.conf
% docker build --rm .
.....
Step 19/20 : LABEL org.label-schema.schema-version="1.0" org.label-schema.vendor="Elastic" org.label-schema.name="logstash" org.label-schema.version="7.0.0" org.label-schema.url="https://www.elastic.co/products/logstash" org.label-schema.vcs-url="https://github.com/elastic/logstash" license="Elastic License"
---> Using cache
---> f622d7555220
Step 20/20 : ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
---> Using cache
---> b6feba7f4934
Successfully built b6feba7f4934
```
This task works only for releases (not snapshots).
This commit also adds a few tweaks to the artifacts building:
Using `SKIP_PREPARE=1` in `rake artifact:tar` or `rake artifact:tar_oss` will make a check to not rebuild the tarball if there are no code modifications.
These two changes are made since docker image build is new and we want to keep it out of artifact:all for a while. And if we're running these separately, we want to ensure the tarball built is used in the docker image (versus building a new one for each `rake artifact:tar` )
This means that, to generate all artifacts including docker images and dockerfile, it's necessary to run:
```
RELEASE=1 rake artifact:all
SKIP_PREPARE=1 RELEASE=1 rake artifact:docker
SKIP_PREPARE=1 RELEASE=1 rake artifact:docker_oss
RELEASE=1 rake artifact:dockerfile
```
introduces two rake tasks: `rake artifact:docker_oss` and `rake artifact:docker`, which will create the docker images of the OSS and non OSS packages. These tasks depend on the tar artifacts being built.
Also `rake artifact:all` has been modified to also call these two tasks.
most code was moved from https://github.com/elastic/logstash-docker/
fpm requires json 1.x but the artifact rake tasks
will activate the default gem json 2
This PR ensures json 1 is activated early and is present right after
jruby is bootstrapped.
Also downgrades fpm to 1.3 as 1.11.0 wasn't building correctly
Downgrades .ruby-version to a version we have available in CI
Fixes#10396
* bump jruby to 9.2
* don't rely on logstash-base docker image
* work around webmock ruby 2.5 support
* ensure data folder exists in docker
* change fixnum and bignum to integer
* FileUtils.rmdir to rm_rf
this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.
* bump jruby to 9.2.5.0 and fix test
* make rake default task since prepare pack needs it
* Resolve compiler warnings (#10247)
There are 3 types of compiler warnings that are either resolved or suppressed:
1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
our codebase to the now "raw" type trigger warnings. In most cases we cannot
actually resolve the issue, since the JRuby-provided methods for creating
`RubyArray`s still return the raw type, so these have been suppressed.
2. Deprecations:
- `RubyString#intern19()` -> `RubyString#intern()`
- `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
- `NativeException`: remove import & reference directly; suppress usage
warnings
- `RaiseException()`: migrate to equivalent non-deprecated methods wherever
possible; in some cases where we are using this in conjunction with the
also-deprecated `NativeException` to preserve java stacktraces, there
seems to be no non-deprecated path forward, so these cases have been
suppressed.
3. Redundant Casts
- Resolved
* JRuby 9.2 bundler shenanigans (#10266)
* Revert "Revert "remove forced dependency on old bundler (#9395)""
This reverts commit bef984143d.
* plugin management: update internal bundler to 1.17.x APIs
* deps: update dev dependency webmock to version compatible with JRuby 9.2
* spec: update Pack fixture to include manticore version that doesn't conflict
* build: update gradle to version that has Java 11 support
* java11: resolve or suppress deprecation warnings
* Remove superfluous flag opting into ParNew GC implementation
When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.
https://bugs.openjdk.java.net/browse/JDK-8006478https://openjdk.java.net/jeps/214
* spec: set thread name to example description for easier debugging
* spec: prevent errors in testing specs by checking against skip list before using
* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code
> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)
* qa: by default, run integration against Elastic Stack 6.5.x
To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.
* qa: ignore deprecation warnings when comparing offline pack output
* qa: add Java 9+ support to ChildProcess dev dependency
this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141
* qa: allow connections to localhost in webmock
* bump jrjackson version
* fix filebeat integration tests
* spec: ensure license compliance spec runs first
The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.
Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.
Also updates the rake invocation, although I'm not sure it is used
any more.
* update bundler to 1.17.1
This commit required some tweaking of how we setup Bundler
due to changes in reset behaviour, an internal variable name change,
and the Bundler::Settings api changing.
* Add VERSION_QUALIFIER support for use by release manager
* Make the gem build processes aware of the version qualifier
* Try debugging xpack ci test failure.
* Try to use the artifacts-api.elastic.co api for ES download.
* It builds/tests locally now.
* add some comments explaining the artifacts-api and the version string
* cahnges requested in review.
Fixes#9956
This adds a simple Logstash config sample that works as a standard
Beats -> Logstash -> Elasticsearch starting point.
It's being deployed in packages, typically under /etc/logstash/logstash-sample.conf
This also adds a Dockerfile that can be used to build the artifacts.
Fixes#9588
