* [ci] Add debian-12 to matrix in packaging and platform jobs (#116172)
Lintian test has been changed to parse the result instead of using exit
code. This was required, because now `mismatched-override` is
non-erasable tag which cannot be ignored for exit code.
Lintian introduced non-backward-compatible format change for overrides
file. Because of that, some overrides are now duplicated in a format for
older versions.
Additionally, Lintian overrides file has been cleaned up to remove the
tags which are no longer failing.
(cherry picked from commit 2ac267de3a)
# Conflicts:
# distribution/packages/src/deb/lintian/elasticsearch
# qa/os/src/test/java/org/elasticsearch/packaging/util/LintianResultParser.java
# qa/packaging/src/test/java/org/elasticsearch/packaging/test/DebMetadataTests.java
* Spotless fix
* Make docker packaging test more resilient (#111205)
Wrap check for container shutdown log message in an `assertBusy()` to
deal with race conditions.
Closes#111132
# Conflicts:
# muted-tests.yml
# qa/os/src/test/java/org/elasticsearch/packaging/test/DockerTests.java
* Fix compile issue
* [CI] Do not cache any es distros when creating ci images (#110742)
(cherry picked from commit 816cedc217)
# Conflicts:
# qa/os/build.gradle
* Update build.gradle
The `systemd-sysv-install` utility is now provided by the
`systemd-sysvcompat` package. Ensure it's installed when running
packaging tests.
Closes#109831
We continue to have CI failures for open files when trying to cleanup on
Windows. This commit tries to account for one of those cases, where the
out/err redirects are cleaned up, opting to retry once after a delay.
Improve InnocuousThread permission checks handling (6a3855112c)
on shutdown, the jdk's InnocuousThread can try to change a thread name. This requires "java.lang.RuntimePermission" "modifyThread" permission. However InnocuousThread doe not inherit any Access Control Context and therefore have no permissions. This results in AccessControlException.
This commit fixes this by skipping a check for modify thread permission if a thread is innocuous.
relates #91658 and #91650
When previously described AccessControlException is thrown, it is not being catched anywhere in the Elasticsearch, hence it ends up being handled by ElasticsearchUncaughtExceptionHandler#onNonFatalUncaught
This is being again being run by the thread [process reaper] which is an innocuous thread (jdk specific) and has no permissions. onNonFatalUncaught is trying to log a message, but this in turn requires java.lang.RuntimePermission" "getenv." permission. which is does not have. This again results in AccessControlException java.lang.RuntimePermission" "getenv."
We can fix this by executing with doPrivileged in ElasticsearchUncaughtExceptionHandler#onNonFatalUncaught and this will stop the Security Manager's walk and will have ES's global grant permissions.
backport(#91704)
closes#91650
Linux packages (deb and rpm) for Elasticsearch 6.x don't include a Java
runtime. Our packaging tests are designed to use an explicit JDK for
packages that do not bundle a JDK. This is not working as intended when
running upgrade tests from 6.x versions and we were instead assuming
a JDK was bundled. The side effect of this is that those versions of
Elasticsearch are using the system JDK, which is different for every
Linux distribution. Specifically, RHEL-based distributions have
recently introduced an update to their Java 8 packages that are
incompatible with the 6.x security manager. This change fixes the logic
that determines if a distribution includes a bundled JDK to ensure that
all versions prior to 7.0 are assumed to not include a bundled JDK.
Closes#88651. When using an `alpine` image to perform container
fiddling, first explicitly pull the image and do so in a loop, in an
attempt to make things more robust.
Closes#88218.
Docker image scans have flagged up the `USER` that the UBI Docker image
runs with. Switch to `elasticsearch:root`, which is what the Iron Bank
image also uses, and is what we use for all images from
8.0 onwards.
Closes#84204 again. When running archive packging tests with a keystore
password and the -d option, there does actually need to be an `eof`
expectation or else we don't capture the error when the keystore
password is incorrect.
Closes#82433. If the environment variable `RESTART_ON_UPGRADE` is true,
then ensure that we delay restarting Elasticseach until after the
keystore is upgraded, or else we can run into permissions problems.
Hopefully fixes#84204. When we build an expect script in the packaging
tests, we insert `expect eof` after supplying the password. This makes
expect wait until the output of ES is closed, which isn't actaully what
we want, as we then go on to expect on a number of other outputs. So,
remove this `eof`, and let the other expectations do their work.
As a result of changing the base Docker to Ubuntu in #80820, the default shell
i.e. `/bin/sh` changed to `dash`, rather than `bash`, which could impact anyone
invoking `/bin/sh` and expecting it to still propagate environment variables with
periods in their names.
Reconfigure the default shell back to `bash` so that this type of situation works
again.
Apply suggestions from Docker Inc about how to update the `cacerts` in
our Ubuntu-based Docker image. Instead of copying around files and
symlinking, instead install `ca-certificates` and `p11-kit`, and use the
latter to regenerate Java's `cacerts`, as well as ensuring it is
regenerated if the system ca certs are updated.
Closes#81208. Elasticsearch uses zlib for two purposes: *
Compression of stored fields with `index.codec: best_compression`,
which we use for observability and security data. * Request /
response compression. Historically, zlib was packaged within the JDK, so
that users wouldn't have to have zlib installed for basic usage of Java.
However, the original zlib optimizes for portability and misses a number
of important optimizations such as leveraging vectorization support for
x86 and ARM architectures. Several forks have been created in order to
address this. Since version 9, the JDK uses the system's zlib when
available and falls back to the zlib that is packaged within the JDK if
a system zlib cannot be found. This commit changes the Docker image to
install the Cloudflare fork of zlib, and run Java using the fork instead
of the original zlib, so that users of the Docker image can get better
performance. Other ES distribution types are out-of-scope, since
configuring the JVM to use an alternative zlib requires an environment
config as well as installed another zlib, and Docker is the only
distribution type where we can control both.
Backport of #80640.
Switch the ES base Docker image for the default and Cloud images to ubuntu:20.04,
as Ubuntu has a more favourable posture on security updates.
Closes#70219.
Introduce a declarative way for the Elasticsearch server to manage plugins,
which reads the `elasticsearch-plugins.yml` file and works which out
plugins need to be added and / or removed to match the configuration. Also
make it possible to configure a proxy in the config file, instead of
through the environment.
Most of the work of adding and removing is still done in the
`InstallPluginAction` and `RemovePluginAction` classes, so the
behaviour should be the same as with the `install` and `remove`
commands. However, these commands will now abort if the above config
file exists. The intent is to make it harder for the configuration
to drift.
This new method only applies to `docker` distribution types at the
moment.
Since this syncing mechanism declarative, rather than imperative,
the Cloud-specific plugin wrapper script is no longer required.
Instead, an environment variable informs `InstallPluginAction` to
install plugins from an archive directory instead of downloading
them, where possible.
* Set LIBFFI_TMPDIR at startup (#80651)
Today if `libffi` cannot allocate pages of memory which are both
writeable and executable then it will attempt to write code to a
temporary file. Elasticsearch configures itself a suitable temporary
directory for use by JNA but by default `libffi` won't find this
directory and will try various other places. In certain configurations,
none of the other places that `libffi` tries are suitable. With older
versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM
will exit with an exception.
With this commit we use the `LIBFFI_TMPDIR` environment variable to
configure `libffi` to use the same directory as JNA for its temporary
files if they are needed.
Closes#18272Closes#73309Closes#74545Closes#77014Closes#77053
Relates #77285
Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
* Fix incorrect SSL usage
Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Closes#76681. Our approach to using `scratch` for building Docker
images has caused problems at Docker Hub. Fix this situation by
removing the whole process of using scratch and instead bases the
default distribution on `almalinux:8.4-minimal`. Alma Linux is
binary-compatible with RHEL, and therefore very similar to UBI.
Systemd exit codes on RedHat "flavored" distributions behave a bit
differently and we have to account for this in our test assertions. This
adds RHEL to the list of distributions that behave similarly. Closes
#80327
Changing the default for deprecation log indexing to be true.
This commit also overrides this default to tests where a deprecation
data stream would interfere - because it uses index template, it would
not be possible to delete with _index_template/*.
The overrides should be removed when #78850 is done.
closes#76292
backport #78991
Firstly: we tag our Docker images with various pieces of information,
including a timestamp for when the image was built. However, this makes
it impossible completely cache the image. When developing the Docker
images, it's very tedious to completely rebuild an image for every
single change. Therefore, provided we're not building a proper release
build, we fix the build time to midnight so that the Docker build cache
is usable.
Secondly: the `DockerBuildTask` outputs a marker file to indicate that
an image has been built, but that isn't enough for a meaningful
up-to-date check by Gradle. Improve this by fetching the newly-built
image's hash, and writing that to the output file.
Thirdly: improve the Docker tests to make them more ergonomic, and also
disable `ingest.geoip.downloader.enabled` by default.
Fourthly: add missing test coverage for sourcing settings from env vars.
* Handle cgroups v2 in `OsProbe` (#77128)
Closes#76812. Closes#77126.
OsProbe was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.
Note that we have to open access to all of /sys/fs/cgroup because with
cgroups v2, the files we need are in an unpredictably location.
* Handle a max memory value of 'max' (#77289)
* Handle a max memory value of 'max'
* Update docs/changelog/77289.yaml
* Delete 77289.yaml
* Fixes to backport
* Fix
Closes#76812.
`OsProbe` was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.
* Introduce Cloud docker variant (#74980)
Closes#74795.
Introduce two Docker image variants for Cloud. The first bundles
(actually installs) the S3, Azure and GCS repository plugins. The
second bundles all official plugins, but only installs the repository
plugins.
Both images also bundle Filebeat and Metricbeat.
The testing utils have been refactored to introduce a `docker`
sub-package. This allows the static `Docker.containerId` to be
shared without needing all the code in one big class. The code for
checking file ownership / permissions has also been refactored to
a more Hamcrest style, using a custom Docker file matcher.
* Don't build Cloud docker images on assemble
* Backport fixes
* Tweak Dockerfile.cloud-ess
Some Docker builds are apparently complaining about the `COPY` syntax in
this file, so try and fix it as Docker suggests.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
