If xpack.ml.use_auto_machine_memory_percent is not explicitly set to true then
the default value (false) means ML will only use 30% of the available memory making
it impractical to run the ELSER model. This is useful for users wanting to get started
with semantic search.The single node docker instructions have been updated with a
command that gives the container enough memory to run the ELSER model and enables xpack.ml.use_auto_machine_memory_percent. For the multi-node guide the docker
compose file is updated to enable the ml setting for every node in the cluster.
* [DOCS] Add local dev setup instructions
- Replace existing Run ES in Docker locally page, with simpler no-security local dev setup
- Move this file into Quickstart folder, along with existing quickstart guide
- Update self-managed instructions in Quickstart guide to use local dev approach
**Problem:**
- The `elasticsearch-reset-password` commands in the ES Docker install docs are missing the required `-u` flag
- The `ifeval` blocks in the Kibana section of the ES Docker install docs aren't rendering correctly in released docs.
**Solution:**
- Add the `-u` flag to `elasticsearch-reset-password` examples
- Fix the Asciidoc syntax to correctly render the `ifeval` blocks. Example:
Rel: https://github.com/elastic/elasticsearch/pull/99112
**Problem:**
The [Kibana Docker install docs](https://www.elastic.co/guide/en/kibana/master/docker.html) duplicate the Elasticsearch Docker instructions. This makes the two doc sets harder to maintain. For example, the docs currently use different container names, which makes them incompatible.
**Solution:**
Adds Kibana setup instructions to the Elasticsearch Docker install docs. This will let us eventually merge the two doc sets.
**Issues:**
Rel: https://github.com/elastic/platform-docs-team/issues/182
- Deduplicates the JVM sizing instructions
- Links the `-m` tag tip section to the JVM sizing instructions
- Replaces the fully typed out container registry to {docker-image}
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Problem: The current install docs contain several `ifeval` statements that hide commands and code snippets based on the branch's release state. These statements make the Asciidoc hard to read and maintain. It also makes doc changes difficult to preview.
Solution: Remove `ifeval` statements that hide commands or code snippets. Leave in any `ifeval` statements used to add warnings.
Problem:
The current [single-node Docker instructions](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode) are overly verbose and contain a lot of unneeded info about security internals.
Solution:
- Restructure the above docs to focus primarily on actionable steps.
- Test the docs across operating systems (Mac, Linux, Windows) to ensure they work.
* [DOCS] Verifying Elasticsearch container image signatures with Cosign/Sigstore
This commit adds a step to verify the Elasticsearch container image signatures after pulling the image and before starting any cluster.
The goal is to introduce an easy and standard way for Elastic users to verify the provenance of the Elasticsearch container images before deploying them to any infrastructure and therefore protect against supply chain attacks.
* Update docker.asciidoc
* Update docs/reference/setup/install/docker.asciidoc
* Update docs/reference/setup/install/docker.asciidoc
* Update docs/reference/setup/install/docker.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Update docs/reference/setup/install/docker.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Remove JSON section from sample output
* Fix up command output format
* Change 'console' to 'sh'
---------
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
Co-authored-by: David Kilfoyle <david.kilfoyle@elastic.co>
The current command to start the first Elasticsearch node includes `-p 9200:9200` and `-p 9300:9300`. This PR removes the extra ports so that they're not open unnecessarily.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
ES_JAVA_OPTS is still the correct way to pass options to
the Elasticsearch process, CLI_JAVA_OPTS affects only the
command line tool. CLI_JAVA_OPTS is the correct way to pass
options for plugin installation or other tools.
In #84340 we adjusted the docs to show instructions on how to
retrieve TLS keystores passwords from the elasticsearch keystore
but the security-files-reference.asciidoc was not referenced in
all places where the instructions should be used. This commit
changes that and also removes references for beta versions.
Fixes an intro sentence for the Docker install instructions.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
(cherry picked from commit 472a7d8e91)
Co-authored-by: Alexander Reelsen <alexander@reelsen.net>
* Clarify commands shown for "permanently" setting max_map_count
The current text shows a grep, which of course does not set the value at all. I've clarified why it's offered (to CHECK the value).
I've also moved the command for setting the value temporarily to above that discussion, to make it more clear to readers that that's what it does.
* Reorder commands, add headings, and fix typos
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: Charlie Arehart <charlie@carehart.org>
Closes#69533.
The Docker docs mention bind-mounting the `config`, `data` and
`logs` directories when using an arbitrary UID / GID, but they fail
to mention that the `plugins` dir must also be mounted in order to
install plugins.
* [DOCS] Remove extraneous Elasticsearch Docker image information
In the step for starting Elasticsearch with the generated enrollment token, `docker.elastic.co/elasticsearch/elasticsearch:` was included in conjunction with the `{docker-image}` variable. This include led to a duplicate Docker image that displayed as `ocker.elastic.co/elasticsearch/elasticsearch:docker.elastic.co/elasticsearch/elasticsearch:8.0.0-rc1`. This PR removes the duplicate image information.
* Update ifeval statements and add sub-heading for setting JVM heap size
We (mostly I) were initially advocating for the auto-generated files to
use unique names (the name containing a timestamp particle), in order to
avoid that subsequent invocations of the config step conflict with
itself. Moreover, I was wishing that these files will not have to be
handled directly by admins (that the enrollment process was to be used).
However, experience proved us otherwise, admins have to manipulate these
files, and unique configuration names are hard to deal with in scripts
and docs, so this PR is all about using a fixed name for all the
generated files. _Labeling as a bug fix because the feedback is that it
very negatively impacts usabilty._ Closes
https://github.com/elastic/elasticsearch/issues/81057
* [DOCS] Enroll additional nodes on Docker
* Remove -p option for second node
Co-authored-by: Fabio Busatto <52658645+bytebilly@users.noreply.github.com>
* Rename nodes to align with other Docker docs
* Add elastic network to first node docker run command
* Remove hyphen from node names
Co-authored-by: Fabio Busatto <52658645+bytebilly@users.noreply.github.com>
Closes#76681. Our approach to using `scratch` for building Docker
images has caused problems at Docker Hub. Fix this situation by
removing the whole process of using scratch and instead bases the
default distribution on `almalinux:8.4-minimal`. Alma Linux is
binary-compatible with RHEL, and therefore very similar to UBI.
The `elasticsearch.yml` file that ships with our Docker image includes the
`network.host: 0.0.0.0` setting by default. If a user bind-mounts a custom
config file, it should include this setting to ensure Elasticsearch is reachable.
Closes#77937.
PR #77155 updated the keystore instructions for Docker. However, it removed an
example that included the `KEYSTORE_PASSWORD` env variable.
This replaces a docker compose example with the original example from PR #51123.
* [DOC] Update Persist Keystore via Docker
From feedback from ES Devs summarized in [^1], I believe this needs to reflect a directory mount rather than file mount to not error. Also adding in the two common mounting errors, but not sure if this is the right place for them.
[^1] https://discuss.elastic.co/t/persist-elasticsearch-kibana-keystores-with-docker/283099
* feedback
* Reorganize
* reword
* fix formatting
* address review feedback
* remove extra whitespace
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Closes#76148. Previously, in #74327, we added support for setting
Elasticsearch settings in the Docker image via environment variables.
However there are scenarios e.g in Kubernetes where it is easy to
accidentally define environment variables that the ES startup process
rejects.
Work around this by using the more explicit env var prefix
`ES_SETTING_`.
Closes#74036. Since some orchestration platforms forbid periods in
environment variable names, allow Docker users to pass settings to ES
using an alternative name scheme. For example:
bootstrap.memory_lock
...becomes:
ES_BOOTSTRAP_MEMORY__LOCK
The setting name is uppercased, prefixed, all underscores are converted
to double underscores, and all periods are converted to underscores.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Closes#62758.
Include the Stack log4j config in the Docker image, in order to
make it possible to write logs in a container environment in the
same way as for an archive or package deployment. This is useful
in situations where the user is bind-mounting the logs directory
and has their own arrangements for log shipping.
To use stack logging, set the environment variable `ES_LOG_STYLE`
to `file`. It can also be set to `console`, which is the same as
not specifying it at all.
The Docker logging config is now auto-generated at image build time,
by running the default config through a transformer program when
preparing the distribution in an image builder step.
Also, in the docker distribution `build.gradle`, I changed a helper
closure into a class with a static method in order to fix an
issue where the Docker image was always being rebuilt, even when
there were no changes.
