Logstash Docker images, full and OSS, now use UBI image as its base, replacing the previous Ubuntu base.
- change the base image of `full` and `oss` to ubi
- Set locale to C.UTF-8
- remove ubi flavour
- use go image to build env2yaml
- remove redundant and refactor steps
- add support to build image in mac aarch64
- allow customizing ELASTIC_VERSION and LOCAL_ARTIFACTS for test purpose
* Add wolfi as an option to the build process
* Add docker acceptance tests for the wolfi image
* Change how tests are done on the java process, due to "ps -C" not being available on wolfi
replaces and closes https://github.com/elastic/logstash/pull/16116
Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
* Add arm64 support for env2yaml
This commit builds env2yaml in arm64 and amd64 flavors, and uses
$TARGETARCH in the Dockerfile to ensure that the correct version is used
when building for alternative architectures
Fixes: #15913
* Add env2yaml executables to build context
* Split `COPY_FILES` for readability
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
---------
Co-authored-by: Andrea Selva <selva.andre@gmail.com>
Logstash on ECK requires openssl command to build TLS keystore.
This commit adds `microdnf install -y openssl` to ensure the command exists in ubi image.
Update the env2yaml to have a go.mod instead of relying on disabling go modules, otherwise building with golang 1.22 will fail in the future.
This change also directly uses the golang image to build the binary removing the need for an intermediate image.
This commit adds environment variable LOG_STYLE to control log behavior of docker container
`console` - this is the default to output to standout
`file` - log to disk.
Fixed: #14941
This commit adds a rake task `rake artifact:dockerfile_ironbank` to generate ironbank docker build context for automatic release.
The output can be found in build/logstash-ironbank-$VERSION-docker-build-context.tar.gz
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
the use of ranges (e.g. {0..5}) or seq (e.g. $(seq 0 5)) may not
correctly in some systems, so let's just have a plain list of elements
for the loop to go through.
* Docker integration tests stability improvements
This commit contains numerous fixes to improve the stability of the docker integration tests
* Patch Excon::UnixSocket
Socket.new running on arm64 on Ubuntu 18.04, causes an immediate SIGSEGV error and crash on
that OS, and, as far as I can tell, only that OS. `TCPSocket.new`,`UDPSocket.new` and
`UNIXSocket.new` do not. This commit patches the UnixSocket of the Excon library to
do the absolute simplest thing possible to avoid this error.
* Ensure that container is deleted even if #kill fails
* Add extra waits to handle the incremental way the payload returned by the monitoring
API increases as logstash starts up and pipelines load.
* Use pyenv to ensure the same version of python is used across different jenkins workers
* Add container logs to help diagnose failed test.
* Update the pipeline definition on multi-pipeline integration test
This was causing a pipeline to halt after startup causing intermittent test failures.
* Remove `;` to ensure failures are propagated appropriately
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
This commit fixes two issues with the docker metadata:
Removes non-OCI compliant freeform metadata labels
Uses a consistent build date for all the docker images and dockerfiles
Additionally, this commit adds a `build_docker_ubi8` rake task to enable
`ci/docker_acceptance_tests.sh` to run with no options to build all
docker images for the architecture.
This commit adds the ability for the docker build to build artifacts for multiple architectures.
By default, the target architecture is inferred from the architecture of the machine the build is being
run from - running the build from an aarch64 machine will build an aarch64 docker image, while building
from an x86_64 machine will build an x86_64 docker image.
This can be overridden by setting the environment variable DOCKER_ARCHITECTURE to either `x86_64` or
`aarch64`.
This commit also updates the integration tests to test against the architecture from the machine the test
is being run on, and includes the target architecture in the test description.
This commit adds the rake docker_ubi8 rake task, and associated
changes to the docker template and makefiles.
This commit also refactors the acceptance tests to extract xpack tests
into a helper class to allow the same tests to be used in both 'full'
and 'ubi8' docker image tests
* Introduce integration tests for docker
This commit adds integration tests for the Logstash docker images. Previous
integration tests were removed in https://github.com/elastic/logstash/pull/10693,
due to the tests being non functional.
The commit adds image and container tests. The image tests check the contents and the
metadata of the image; the container tests check the logstash process, and includes tests
ensuring that logstash runs, and is configurable.
This test also adds a ci script to allow the tests to be run on jenkins, and to split the
running of these tests up based on the image type and includes updates to the rake tasks to
support this.
Prior to this commit, the value of `org.label-schema.license` and
the values in `org.opencontainers.image.*` were not set, and therefore
would be inherited from the base OS image.
* dont include docker tasks in artifact:all
* don't rebuild tar/zip if source hasn't changed
* allow SKIP_PREPARE to avoid tar creation if no modifications
* don't need a tarball to generate the dockerfile
* remove docker tests as they weren't working anymore
This commit adds a task to produce all necessary files to generate a docker image.
```
% RELEASE=1 rake artifact:dockerfile
....
Dockerfile created in /tmp/elastic/logstash/build/docker
% tree /tmp/elastic/logstash/build/docker
/tmp/elastic/logstash/build/docker
├── Dockerfile
├── bin
│ └── docker-entrypoint
├── config
│ ├── log4j2.properties
│ ├── logstash-full.yml
│ └── pipelines.yml
├── env2yaml
│ └── env2yaml
└── pipeline
└── default.conf
% docker build --rm .
.....
Step 19/20 : LABEL org.label-schema.schema-version="1.0" org.label-schema.vendor="Elastic" org.label-schema.name="logstash" org.label-schema.version="7.0.0" org.label-schema.url="https://www.elastic.co/products/logstash" org.label-schema.vcs-url="https://github.com/elastic/logstash" license="Elastic License"
---> Using cache
---> f622d7555220
Step 20/20 : ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
---> Using cache
---> b6feba7f4934
Successfully built b6feba7f4934
```
This task works only for releases (not snapshots).
This commit also adds a few tweaks to the artifacts building:
Using `SKIP_PREPARE=1` in `rake artifact:tar` or `rake artifact:tar_oss` will make a check to not rebuild the tarball if there are no code modifications.
These two changes are made since docker image build is new and we want to keep it out of artifact:all for a while. And if we're running these separately, we want to ensure the tarball built is used in the docker image (versus building a new one for each `rake artifact:tar` )
This means that, to generate all artifacts including docker images and dockerfile, it's necessary to run:
```
RELEASE=1 rake artifact:all
SKIP_PREPARE=1 RELEASE=1 rake artifact:docker
SKIP_PREPARE=1 RELEASE=1 rake artifact:docker_oss
RELEASE=1 rake artifact:dockerfile
```
introduces two rake tasks: `rake artifact:docker_oss` and `rake artifact:docker`, which will create the docker images of the OSS and non OSS packages. These tasks depend on the tar artifacts being built.
Also `rake artifact:all` has been modified to also call these two tasks.
most code was moved from https://github.com/elastic/logstash-docker/
