This removes the overhead of zipping up modules that are immediately unzipped again when packaging
the elasticsearch distribution.
We also move some logic for packaging the elasticsearch distribution into a plugin and remove some outdated
overhead dealing with 'meta plugins' when copying modules into a distribution.
Another follow up and related optimization out of scope of this PR is, to also not zip unzip modules declared
for usage in our test cluster setups.
This partially addresses #76726.
Plugin downloads can sometimes take unexpectedly
long because of slow random number generation on certain
operating system versions, e.g. Debian based OS
distros with kernels older than 5.6. This PR adds
a warning message for users explaining why plugin
installation from remote URLs is taking unexpectedly long.
This commit adds a jar separate from the test framework to provide
utilities for testing x-content related code. The first thing moved
there is the base schema validation test case, which also pulls along
the com.networknt dependency and jackson. For now these are direct
dependencies, though we could consider shading them in the future so as
not to expose downstream projects to them, which may have version
conflicts.
This change isolates the Jackson implementation of x-content parsers and generators to a separate classloader. The code is loaded dynamically upon accessing any x-content functionality.
The x-content implementation is embedded inside the x-content jar, as a hidden set of resource files. These are loaded through a special classloader created to initialize the XContentProvider through service loader. One caveat to this approach is that IDEs will no longer trigger building the x-content implementation when it changes. However, running any test from the command line, or running a full Build in IntelliJ will trigger the directory to be built.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
Replace any old references of master with main. These are mostly
cosmetic, as the name of the default branch doesn't actually affect the
build in any meaningful way.
The tools used in launching Elasticsearch have a separate jar file and
dependencies. This commit moves the launcher tools to its own lib
directory, so that it does not bleed into the main Elasticsearch
classpath.
Closes#82433. If the environment variable `RESTART_ON_UPGRADE` is true,
then ensure that we delay restarting Elasticseach until after the
keystore is upgraded, or else we can run into permissions problems.
This flag has already been removed by ES on Oct 21, 2019 after they created their own allocator: 547e399dbf
However, in parallel, on Oct 28, 2019 all command line flags were moved from jvm.options file into the launcher, and this option has been re-introduced: c4fbda3310
See original commit from Oct 21 for the reasoning on why this flag is not needed anymore.
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
2.4.0 Introduces support for linux and macOS aarch64. Without this
we cannot detect whether a terminal is attached to elasticsearch
on macoOS/linux aarch64 and so we do not generate and print the
elastic password and enrollment token on startup
In order to display formatted (bolded) autoconfiguration text on
Windows cmd prompt using JANSI, one needs to invoke the
SetConsoleMode Kernel32 API, via JANSI's AnsiConsole#install.
But we lack the testing infra to properly assert the formatting behavior,
so this PR disables ANSI-formatted output on Windows cmd prompt.
Fixes: #83316
This PR slightly improves the format of the security auto-configuration
information that is printed on the terminal when the initial cluster
node first starts up.
It uses eye-catching Unicode characters for bullet points.
It also uses Unicode to display a continuous border, for the whole
width of the terminal, before and after the information. In addition,
it uses ANSI escape sequences to render some of the information
in bold fonts.
It will fallback to using regular characters if the JVM is set-up with
a non-UTF encoding for the standard out.
Try to represent immutable data with Java records introduced in JEP 395
Convert only existing immutable classes, no "POJO with setters to a record" refactorings.
Fixes#81999.
Change `EnrollmentProcessTests` and the testing infrastructure so
that `ServerUtils` can send requests to the correct local port.
This is also a step towards addressing #79688.
Also change the Iron Bank build to that it works on aarch64. This
isn't currently a supported configuration, but it's useful for
testing.
Closes#81652.
Convert the `repository-azure`, `repository-gcs` and `repository-s3`
plugins into modules, so that they are always included in the
Elasticsearch distribution. Also change plugin installation, removal
and syncing so that attempting to add or remove these plugins still
succeeds but is now a no-op.
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.
Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
We (mostly I) were initially advocating for the auto-generated files to
use unique names (the name containing a timestamp particle), in order to
avoid that subsequent invocations of the config step conflict with
itself. Moreover, I was wishing that these files will not have to be
handled directly by admins (that the enrollment process was to be used).
However, experience proved us otherwise, admins have to manipulate these
files, and unique configuration names are hard to deal with in scripts
and docs, so this PR is all about using a fixed name for all the
generated files. _Labeling as a bug fix because the feedback is that it
very negatively impacts usabilty._ Closes
https://github.com/elastic/elasticsearch/issues/81057
For some reason, the way we generate `/app/elasticsearch.sh` for Cloud
images has broken. It may be the result of changing the default shell to
`bash`. Fix by adding `-e` to the `echo` command.
As a result of changing the base Docker to Ubuntu in #80820, the default shell
i.e. `/bin/sh` changed to `dash`, rather than `bash`, which could impact anyone
invoking `/bin/sh` and expecting it to still propagate environment variables with
periods in their names.
Reconfigure the default shell back to `bash` so that this type of situation works
again.
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
Apply suggestions from Docker Inc about how to update the `cacerts` in
our Ubuntu-based Docker image. Instead of copying around files and
symlinking, instead install `ca-certificates` and `p11-kit`, and use the
latter to regenerate Java's `cacerts`, as well as ensuring it is
regenerated if the system ca certs are updated.
Allow users to pass an ENROLLMENT_TOKEN environment variable that
would be passed as the `--enrollment-token` parameter to the
elasticsearch executable running in the container. This allows an
elasticsearch node running in docker to enroll itself to an
existing secured cluster.
Resolves: #81068
Closes#81208. Elasticsearch uses zlib for two purposes: *
Compression of stored fields with `index.codec: best_compression`,
which we use for observability and security data. * Request /
response compression. Historically, zlib was packaged within the JDK, so
that users wouldn't have to have zlib installed for basic usage of Java.
However, the original zlib optimizes for portability and misses a number
of important optimizations such as leveraging vectorization support for
x86 and ARM architectures. Several forks have been created in order to
address this. Since version 9, the JDK uses the system's zlib when
available and falls back to the zlib that is packaged within the JDK if
a system zlib cannot be found. This commit changes the Docker image to
install the Cloudflare fork of zlib, and run Java using the fork instead
of the original zlib, so that users of the Docker image can get better
performance. Other ES distribution types are out-of-scope, since
configuring the JVM to use an alternative zlib requires an environment
config as well as installed another zlib, and Docker is the only
distribution type where we can control both.
Part of #81150. Sometimes, building the Cloud Docker images fails due to
some issue with the `.tar.gz` files. Add extra checks and error messages
so that we have a better idea what the precise problem is.
Fix the split package org.elasticsearch.cli, between server and the cli library. Move the server org.elasticsearch.cli package to org.elasticsearch.common.cli. Removing split packages is a prerequisite to modularization.
Post JDK 17 the security manager is disabled by default - setSecurityManager throws UOE - see JEP 411.
This change adds a command line option to explicitly allow to set the security manager, which enables early testing with releases greater than JDK 17.
Closes#70219.
Introduce a declarative way for the Elasticsearch server to manage plugins,
which reads the `elasticsearch-plugins.yml` file and works which out
plugins need to be added and / or removed to match the configuration. Also
make it possible to configure a proxy in the config file, instead of
through the environment.
Most of the work of adding and removing is still done in the
`InstallPluginAction` and `RemovePluginAction` classes, so the
behaviour should be the same as with the `install` and `remove`
commands. However, these commands will now abort if the above config
file exists. The intent is to make it harder for the configuration
to drift.
This new method only applies to `docker` distribution types at the
moment.
Since this syncing mechanism declarative, rather than imperative,
the Cloud-specific plugin wrapper script is no longer required.
Instead, an environment variable informs `InstallPluginAction` to
install plugins from an archive directory instead of downloading
them, where possible.
Today if `libffi` cannot allocate pages of memory which are both
writeable and executable then it will attempt to write code to a
temporary file. Elasticsearch configures itself a suitable temporary
directory for use by JNA but by default `libffi` won't find this
directory and will try various other places. In certain configurations,
none of the other places that `libffi` tries are suitable. With older
versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM
will exit with an exception.
With this commit we use the `LIBFFI_TMPDIR` environment variable to
configure `libffi` to use the same directory as JNA for its temporary
files if they are needed.
Closes#18272Closes#73309Closes#74545Closes#77014Closes#77053
Relates #77285
Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Closes#76681. Our approach to using `scratch` for building Docker
images has caused problems at Docker Hub. Fix this situation by
removing the whole process of using scratch and instead bases the
default distribution on `almalinux:8.4-minimal`. Alma Linux is
binary-compatible with RHEL, and therefore very similar to UBI.
Adds a new "ConsoleLoader" that uses jANSI in a separate classloader
to determine whether standard output is a real console (that is, not
redirected to a file or /dev/null, etc)
Also updates security auto-configuration to only print out credentials
when there is a console.
This change introduces a CLI tool that can be run directly after
installation time in packaged installations, to allow for a node
that was auto-configured to be the initial node of a cluster during
installation ( default installation behavior) to be reconfigured
to join an existing cluster, using an enrollment token.
The use of this tool presumes that the user has the
appropriate permissions to read/write to the installation dirs and
that this node has not been yet started, i.e. this tool is run
directly after installation. It is destructive, as it removes
existing security auto-configuration, and as such it requires an
explicit verification from the user.
This is a follow-up to #7718.
The functionality to enroll a new node to a cluster was
introduced in #77292 as a CLI tool. This change replaces this
CLI tool with the option to trigger the enrollment functionality
on startup of elasticsearch via a named argument that can be
passed to the elasticsearch startup script (--enrollment-token)
so that the users that want to enroll a node to a cluster can do
this with one command instead of two.
In a followup PR we are introducing a CLI tool version of this
functionality, that can be used to reconfigure packaged
installations.
This change introduces a new CLI tool that can be used to set and
reset the password of all the built-in users and users in the native
realm in Elasticsearch. It depends on the file realm being enabled
(which it is, by default) and can (re)set one built-in user password at a time.
It removes the previously introduced elasticsearch-reset-elastic-password
and elasticsearch-reset-kibana-system-password as their functionality is
covered by this new tool.
