mirror of
https://github.com/elastic/logstash.git
synced 2025-04-23 14:17:58 -04:00
b831888318
* Add instructions to verify Docker install images * remove line breaks * Update docs/static/docker.asciidoc Co-authored-by: Maxime Gréau <maxime.greau@elastic.co> --------- Co-authored-by: Maxime Gréau <maxime.greau@elastic.co>
229 lines
9.3 KiB
Text
229 lines
9.3 KiB
Text
[[docker]]
|
|
=== Running Logstash on Docker
|
|
Docker images for Logstash are available from the Elastic Docker
|
|
registry. The base image is https://hub.docker.com/_/ubuntu/[ubuntu:20.04].
|
|
|
|
A list of all published Docker images and tags is available at
|
|
https://www.docker.elastic.co[www.docker.elastic.co]. The source code is in
|
|
https://github.com/elastic/logstash/tree/{branch}[GitHub].
|
|
|
|
These images are free to use under the Elastic license. They contain open source
|
|
and free commercial features and access to paid commercial features.
|
|
{kibana-ref}/managing-licenses.html[Start a 30-day trial] to try out all of the
|
|
paid commercial features. See the
|
|
https://www.elastic.co/subscriptions[Subscriptions] page for information about
|
|
Elastic license levels.
|
|
|
|
==== Pulling the image
|
|
|
|
Obtaining Logstash for Docker is as simple as issuing a +docker
|
|
pull+ command against the Elastic Docker registry.
|
|
|
|
ifeval::["{release-state}"=="unreleased"]
|
|
|
|
However, version {logstash_version} of Logstash has not yet been
|
|
released, so no Docker image is currently available for this version.
|
|
|
|
endif::[]
|
|
|
|
ifeval::["{release-state}"!="unreleased"]
|
|
|
|
["source","sh",subs="attributes"]
|
|
--------------------------------------------
|
|
docker pull {docker-image}
|
|
--------------------------------------------
|
|
|
|
Alternatively, you can download other Docker images that contain only features
|
|
available under the Apache 2.0 license. To download the images, go to
|
|
https://www.docker.elastic.co[www.docker.elastic.co].
|
|
|
|
endif::[]
|
|
|
|
==== Verifying the image
|
|
|
|
Although it's optional, we highly recommend verifying the signatures included with your downloaded Docker images to ensure that the images are valid.
|
|
|
|
Elastic images are signed with https://docs.sigstore.dev/cosign/overview/[Cosign] which is part of the https://www.sigstore.dev/[Sigstore] project. Cosign supports container signing, verification, and storage in an OCI registry. Install the appropriate https://docs.sigstore.dev/cosign/installation/[Cosign application]
|
|
for your operating system.
|
|
|
|
Run the following commands to verify the container image signature for {ls} v{version}:
|
|
|
|
ifeval::["{release-state}"=="unreleased"]
|
|
|
|
Version {logstash_version} of Logstash has not yet been
|
|
released, so no Docker image is currently available for this version.
|
|
|
|
endif::[]
|
|
|
|
ifeval::["{release-state}"!="unreleased"]
|
|
|
|
["source","sh",subs="attributes"]
|
|
--------------------------------------------
|
|
wget https://artifacts.elastic.co/cosign.pub <1>
|
|
cosign verify --key cosign.pub {docker-image} <2>
|
|
--------------------------------------------
|
|
<1> Download the Elastic public key to verify container signature
|
|
<2> Verify the container against the Elastic public key
|
|
|
|
The command prints the check results and the signature payload in JSON format, for example:
|
|
|
|
["source","sh",subs="attributes"]
|
|
--------------------------------------------
|
|
Verification for {docker-image} --
|
|
The following checks were performed on each of these signatures:
|
|
- The cosign claims were validated
|
|
- Existence of the claims in the transparency log was verified offline
|
|
- The signatures were verified against the specified public key
|
|
--------------------------------------------
|
|
|
|
endif::[]
|
|
|
|
[[docker-config]]
|
|
=== Configuring Logstash for Docker
|
|
|
|
Logstash differentiates between two types of configuration:
|
|
<<config-setting-files,Settings and Pipeline Configuration>>.
|
|
|
|
==== Pipeline Configuration
|
|
|
|
It is essential to place your pipeline configuration where it can be
|
|
found by Logstash. By default, the container will look in
|
|
+/usr/share/logstash/pipeline/+ for pipeline configuration files.
|
|
|
|
In this example we use a bind-mounted volume to provide the
|
|
configuration via the +docker run+ command:
|
|
|
|
["source","sh",subs="attributes"]
|
|
--------------------------------------------
|
|
docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ {docker-image}
|
|
--------------------------------------------
|
|
|
|
Every file in the host directory +~/pipeline/+ will then be parsed
|
|
by Logstash as pipeline configuration.
|
|
|
|
If you don't provide configuration to Logstash, it will run with a
|
|
minimal config that listens for messages from the
|
|
<<plugins-inputs-beats,Beats input plugin>> and echoes any that are
|
|
received to `stdout`. In this case, the startup logs will be similar
|
|
to the following:
|
|
|
|
["source","text"]
|
|
--------------------------------------------
|
|
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties.
|
|
[2016-10-26T05:11:34,992][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
|
|
[2016-10-26T05:11:35,068][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
|
|
[2016-10-26T05:11:35,078][INFO ][org.logstash.beats.Server] Starting server on port: 5044
|
|
[2016-10-26T05:11:35,078][INFO ][logstash.pipeline ] Pipeline main started
|
|
[2016-10-26T05:11:35,105][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
|
|
--------------------------------------------
|
|
|
|
This is the default configuration for the image, defined in
|
|
+/usr/share/logstash/pipeline/logstash.conf+. If this is the
|
|
behaviour that you are observing, ensure that your pipeline
|
|
configuration is being picked up correctly, and that you are replacing
|
|
either +logstash.conf+ or the entire +pipeline+ directory.
|
|
|
|
==== Settings
|
|
|
|
The image provides several methods for configuring settings. The conventional
|
|
approach is to provide a custom `logstash.yml` file, but it's
|
|
also possible to use environment variables to define settings.
|
|
|
|
[[docker-bind-mount-settings]]
|
|
===== Bind-mounted settings files
|
|
|
|
Settings files can also be provided through bind-mounts. Logstash
|
|
expects to find them at +/usr/share/logstash/config/+.
|
|
|
|
It's possible to provide an entire directory containing all needed
|
|
files:
|
|
|
|
["source","sh",subs="attributes"]
|
|
--------------------------------------------
|
|
docker run --rm -it -v ~/settings/:/usr/share/logstash/config/ {docker-image}
|
|
--------------------------------------------
|
|
|
|
Alternatively, a single file can be mounted:
|
|
|
|
["source","sh",subs="attributes"]
|
|
--------------------------------------------
|
|
docker run --rm -it -v ~/settings/logstash.yml:/usr/share/logstash/config/logstash.yml {docker-image}
|
|
--------------------------------------------
|
|
|
|
NOTE: Bind-mounted configuration files will retain the same permissions and
|
|
ownership within the container that they have on the host system. Be sure
|
|
to set permissions such that the files will be readable and, ideally, not
|
|
writeable by the container's +logstash+ user (UID 1000).
|
|
|
|
===== Custom Images
|
|
|
|
Bind-mounted configuration is not the only option, naturally. If you
|
|
prefer the _Immutable Infrastructure_ approach, you can prepare a
|
|
custom image containing your configuration by using a +Dockerfile+
|
|
like this one:
|
|
|
|
["source","dockerfile",subs="attributes"]
|
|
--------------------------------------------
|
|
FROM {docker-image}
|
|
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
|
|
COPY pipeline/ /usr/share/logstash/pipeline/
|
|
COPY config/ /usr/share/logstash/config/
|
|
--------------------------------------------
|
|
|
|
Be sure to replace or delete `logstash.conf` in your custom image, so
|
|
that you don't retain the example config from the base image.
|
|
|
|
[[docker-env-config]]
|
|
===== Environment variable configuration
|
|
|
|
Under Docker, Logstash settings can be configured via environment
|
|
variables. When the container starts, a helper process checks the environment
|
|
for variables that can be mapped to Logstash settings. Settings that are found
|
|
in the environment override those in the `logstash.yml` as the container starts up.
|
|
|
|
For compatibility with container orchestration systems, these environment
|
|
variables are written in all capitals, with underscores as word
|
|
separators.
|
|
|
|
Some example translations are shown here:
|
|
|
|
.Example Docker Environment Variables
|
|
[horizontal]
|
|
**Environment Variable**:: **Logstash Setting**
|
|
`PIPELINE_WORKERS`:: `pipeline.workers`
|
|
`LOG_LEVEL`:: `log.level`
|
|
`MONITORING_ENABLED`:: `monitoring.enabled`
|
|
|
|
In general, any setting listed in the <<logstash-settings-file, settings
|
|
documentation>> can be configured with this technique.
|
|
|
|
NOTE: Defining settings with environment variables causes `logstash.yml` to
|
|
be modified in place. This behaviour is likely undesirable if `logstash.yml` was
|
|
bind-mounted from the host system. Thus, it is not recommended to
|
|
combine the bind-mount technique with the environment variable technique. It
|
|
is best to choose a single method for defining Logstash settings.
|
|
|
|
==== Docker defaults
|
|
The following settings have different default values when using the Docker
|
|
images:
|
|
|
|
[horizontal]
|
|
`http.host`:: `0.0.0.0`
|
|
`monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200`
|
|
|
|
NOTE: The setting `monitoring.elasticsearch.hosts` is not
|
|
defined in the `-oss` image.
|
|
|
|
These settings are defined in the default `logstash.yml`. They can be overridden
|
|
with a <<docker-bind-mount-settings,custom `logstash.yml`>> or via
|
|
<<docker-env-config,environment variables>>.
|
|
|
|
IMPORTANT: If replacing `logstash.yml` with a custom version, be sure to copy the
|
|
above defaults to the custom file if you want to retain them. If not, they will
|
|
be "masked" by the new file.
|
|
|
|
==== Logging Configuration
|
|
|
|
Under Docker, Logstash logs go to standard output by default. To
|
|
change this behaviour, use any of the techniques above to replace the
|
|
file at +/usr/share/logstash/config/log4j2.properties+.
|