In `elasticsearch` script, we source `elasticsearch-env` that ends
with a `cd` command to $ES_HOME. As such we don't need to call
`dirname` to determine where `elasticsearch-cli`, but reference it
with the relevant path `bin/elastisearch-cli`
Since #65905 Elasticsearch has determined the Java heap settings
from node roles and total system memory.
This change allows the total system memory used in that calculation
to be overridden with a user-specified value. This is intended to
be used when Elasticsearch is running on a machine where some other
software that consumes a non-negligible amount of memory is running.
For example, a user could tell Elasticsearch to assume it was
running on a machine with 3GB of RAM when actually it was running
on a machine with 4GB of RAM.
The system property is `es.total_memory_bytes`, so, for example,
could be specified using `-Des.total_memory_bytes=3221225472`.
(It is specified in bytes rather than using a unit, because it
needs to be parsed by startup code that does not have access to
the utility classes that interpret byte size units.)
This commit ensures that for packaged installations
we will run the auto-configuration code on installation (but not upgrade) time.
This is needed because we expect elasticsearch to be run as a service.
By the time the service runs, the configuration directory is not writable by the
user that runs elasticsearch so we can't persist configuration and key/certificate
material on runtime. Running auto-configuration on installation time
allows us to print information to the user that they have better chance of seeing
(barring unattended installations). We don't have the option to show output to the
user when starting the service with systemctl.
During installation we:
- Generate TLS material, enable security and TLS and persist on disk
- Generate a password for the elastic user and store a hash of this
in the elasticsearch.keystore. This will be picked up by the node
starting and will be "promoted" to be the cluster wide elastic
password on first startup. (see #78306 )
- We notify the user in the output of the package installation about
whether we succeed and what the password of the elastic user is.
This commit introduces TLS auto-configuration for elasticsearch nodes, during
the first startup. A number of heuristics are performed in order to determine if
the node should get TLS auto-configuration which can also be explicitly
disallowed with the use of xpack.security.autoconfiguration.enabled setting.
This affects archive installations and docker. Packaged installations are
handled in #75144 and #75704 .
Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
Firstly: we tag our Docker images with various pieces of information,
including a timestamp for when the image was built. However, this makes
it impossible completely cache the image. When developing the Docker
images, it's very tedious to completely rebuild an image for every
single change. Therefore, provided we're not building a proper release
build, we fix the build time to midnight so that the Docker build cache
is usable.
Secondly: the `DockerBuildTask` outputs a marker file to indicate that
an image has been built, but that isn't enough for a meaningful
up-to-date check by Gradle. Improve this by fetching the newly-built
image's hash, and writing that to the output file.
Thirdly: improve the Docker tests to make them more ergonomic, and also
disable `ingest.geoip.downloader.enabled` by default.
Fourthly: add missing test coverage for sourcing settings from env vars.
Closes#77862.
Optimize Docker startup times in environments with a lot of environment variables by using `grep` for regular expressions instead of `bash`.
* Do not create unused testCluster
This avoids creating test clusters that are not required during the build.
We use lazy configuration here on testClusters and only instantiate them as theyre
* Do not fail on run task (debug)
* Create more test cluster lazy
* Make more test cluster lazy
* Avoid creating unused testcluster
* Fix PluginBuildPlugin
* Fix disabling geo db download
* Fix cluster setup in repository-multi-version
* Polishing
* Fix issue with irretic groovy ogic
* Fix bwc tests
* Fix more bwcTests
* Fix more bwc tests
* Fix more bwc tests
* Fix more bwc tests
* Fix typo
* Minor polishing
* Fix rolling upgrade tests
* Fix cluster config in sql qa mixedcluster project
* Fix more bwc tests
* Clean up before review
* Document test cluster usage
* Api polising after Review
provide useCluster(Provider) method to TestClusterAware
Ideally we take this a step further and realize those test clusters only on use.
But out of scope of this PR.
* Allow gradle provider as value for nonSystemProperties
* Some simplification on test configuration
* Fix typo in rest test config
* Fix more typos
* Fix another typo
* Fix more typos
This adds a new "elasticsearch-keystore show" command that displays
the value of a single secure setting from the keystore.
An optional `-o` (or `--output`) parameter can be used to direct
output to a file.
The `-o` option is required for binary keystore values
because the CLI `Terminal` class does not support writing binary data.
Hence this command:
elasticsearch-keystore show xpack.watcher.encryption_key > watcher.key
would not produce a file with the correct contents.
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
The keystore-cli contains cli command classes for reading and writing
the elasticsearch-keystore. These classes need access to the save
methods of the KeyStoreWrapper, but those are package private. This
change openes those methods to make them available. Note that setString
was already public, only setFile and a couple minor methods were
necessary to open up. Additionally, some methods from bootstrap for
reading the keystore needed to be accessible to tests. This change moves
those to a public utility class, as they were already static.
This change makes it so x-pack-core and x-pack-security are bundled
in the INTEG TEST distribution that we use for testClusters in our
tests. There are two reasons for this:
- In https://github.com/elastic/elasticsearch/pull/77231 where we
are looking into enabling and auto-configuring security by default
for all nodes, we need to call out to ConfigInitialNode to
determine whether we should do the auto-configuration or not.
- Since we are enabling security by default, we should be looking
into enabling security for all for our tests moving forward, or
at least make a conscious decision about which ones run without
security. This change is a step towards that direction.
* Fix split packages in plugin cli
The plugin CLI was born out of the server jar, to allow the code to be
independent, as well as have more isolated tests that use jimfs to test
any many filesystem implementations. Yet the cli classes continue to
reside in the same package as the plugin service. This commit renames
the package in the cli to be cli specific.
Most of the change is simple renames and adding imports. A few cases
required making plugin service specific classes public. In the future
these could be made scoped exports specific to a plugin-cli module, but
for now this fix the split packages. Also note the PluginSecurity class
was only used by the cli, but is also only tested in the evil-tests,
which are run without security manager. Since those will all eventually
go away, the test package is also renamed there.
* fix shell scripts
Closes#77414.
The generated shell logic in ShellRetry doesn't work in more basic shells, so instead
of using the {1..10} construct, expand the sequence.
This commit changes default deprecation logger level to CRITICAL, where default means deprecations emitted by DeprecationLogger#critical method.
It also introduces WARN deprecations which are emitted by DeprecationLogger#warn Those log lines emitted at WARN are meant to indicate that a functionality is deprecated but will not break at next major version.
relates #76754
* Only depend on beats when necessary
The unified build happens in stages, and for release builds the main
part of Elasticsearch will be built before the Beats distributions are
available. To work around this, infer in the Docker part of the build
whether to include beats as a dependency. In the future, we should find
a more robust mechanism.
* Move decision about including beats
Elasticsearch's keystore initial md5sum was added in #28928 with
the intention to allow us to remove the elasticsearch.keystore
file upon package removal, if this hadn't been altered after
installation. At that time this decision made perfect sense as
the elasticsearch keystore only contains transient data by
default ( keystore.seed ) that is meant to be useful for bootstrap
related actions, and doesn't need to survive re-installations.
With Security ON by default, we will be storing additional
settings in the keystore upon installation(namely, the passwords
for the PKCS#12 keystores used for TLS) and these have a more
persistent nature. Since `remove` doesn't delete the configuration
directories and files where said PKCS#12 keystores are stored, it
makes sense to also not delete the elasticsearch.keystore which
stores the passwords.
Our packaging tests set up dependencies so that the tests depend on the
artifact that is going to be tested. For Docker, the tests depend on an
export of the Docker image, partly to follow the pattern, but partly to
support loading the image when the tests are executed inside Vagrant.
There is a sub-project for each type of image that can be exported. The
main Docker build file adds an export task and an artifact to the
sub-project. However, it also makes the export task a dependency of the
sub-project's `assemble` task. This last step isn't necessary for the
tests to run, and causes extra and needless work to be done when running
`assemble` on the entire build. Therefore, removing the last bit of task
wiring.
Closes#74795.
Introduce two Docker image variants for Cloud. The first bundles
(actually installs) the S3, Azure and GCS repository plugins. The
second bundles all official plugins, but only installs the repository
plugins.
Both images also bundle Filebeat and Metricbeat.
The testing utils have been refactored to introduce a `docker`
sub-package. This allows the static `Docker.containerId` to be
shared without needing all the code in one big class. The code for
checking file ownership / permissions has also been refactored to
a more Hamcrest style, using a custom Docker file matcher.
* Update redline library to 1.2.10
The redline team just released version 1.2.10 of the redline
library which contains our fix of the rpm signatures / headers.
Also a PR to update that dependency in the ospackage plugin has been
raised at https://github.com/nebula-plugins/gradle-ospackage-plugin/pull/402
* Update common about enforcing redline 1.2.10
Second attempt after reverting #76087.
This PR fixes a number of issues with the Iron Bank Docker context. Most
notably it changes the generated `Dockerfile` in the build context to
`COPY` the Elasticsearch archive, instead of downloading it, since Iron
Bank images have a different process and will never be built on Docker Hub.
This requires that the context task be wired to `assemble` instead of the
build image task. The latter is only used for testing purposes.
Closes#76148. Previously, in #74327, we added support for setting
Elasticsearch settings in the Docker image via environment variables.
However there are scenarios e.g in Kubernetes where it is easy to
accidentally define environment variables that the ES startup process
rejects.
Work around this by using the more explicit env var prefix
`ES_SETTING_`.
A MultiCommand requires that a (sub)command is provided for each
execution. Previously, the error handling for MultiCommand would
simply state "ERROR: Missing command" but give no assistance about
what commands were accepted. The user was required to pass "-help" in
order to get the command list.
This commit changes the behaviour so that the list of commands is
printed after the error message.
The test cluster took slightly more than 10s to apply the cluster state for the mapping update
randomly which failed the run in #75941. This only ever failed once but since there's no real harm in doing so
increasing the timeout to 30s here like we did in other similar spots.
closes#75941
Explicitly set permissions for all files in the Elasticsearch home
directory to the minimum required set, and change ownership to
`root:root` where possible.
Part of #70219.
Split the install and remove plugin CLI commands into `Command` and
`Action` classes, so that the main implementations can be reused
in a later PR to create a `Sync` command.
This adds support for Sha256 header signature in our RPMs by
updating the dependency to the readline library to a version
we have patched until the provided PR (https://github.com/craigwblake/redline/pull/157)
got merged and released by the redline folks.
This work is related to #58257
The process for building `log4j2.properties` in the default distribution
is complicated, expensive and requires all modules and plugins to be
built in order to generate the file. Improve this by directly grabbing
the files and concatenating them.
Also remove code for building the OSS distro config.
