build docker images from logstash repo (#10603)

introduces two rake tasks: `rake artifact:docker_oss` and `rake artifact:docker`, which will create the docker images of the OSS and non OSS packages. These tasks depend on the tar artifacts being built. Also `rake artifact:all` has been modified to also call these two tasks. most code was moved from https://github.com/elastic/logstash-docker/
2025-06-27 17:08:55 -04:00 · 2019-04-04 11:27:06 +01:00 · 2019-04-04 11:27:06 +01:00 · dc5db673ee
commit dc5db673ee
parent bb8d4fbc19
33 changed files with 1088 additions and 2 deletions
--- a/.gitignore
+++ b/.gitignore
@ -17,7 +17,7 @@ local
 test/setup/elasticsearch/elasticsearch-*
 vendor
 .sass-cache
-data
+/data
 .buildpath
 .project
 .DS_Store
--- a/docker/LICENSE
+++ b/docker/LICENSE
@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright {yyyy} {name of copyright owner}
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
--- a/docker/Makefile
+++ b/docker/Makefile
@ -0,0 +1,133 @@
+SHELL=/bin/bash
+ELASTIC_REGISTRY ?= docker.elastic.co
+
+export PATH := ./bin:./venv/bin:$(PATH)
+
+# Determine the version to build. Override by setting ELASTIC_VERSION env var.
+ELASTIC_VERSION := $(shell ./bin/elastic-version)
+
+ifdef STAGING_BUILD_NUM
+  VERSION_TAG := $(ELASTIC_VERSION)-$(STAGING_BUILD_NUM)
+else
+  VERSION_TAG := $(ELASTIC_VERSION)
+endif
+
+IMAGE_FLAVORS ?= oss full
+DEFAULT_IMAGE_FLAVOR ?= full
+
+IMAGE_TAG := $(ELASTIC_REGISTRY)/logstash/logstash
+HTTPD ?= logstash-docker-artifact-server
+
+FIGLET := pyfiglet -w 160 -f puffy
+
+all: build-from-local-artifacts build-from-local-oss-artifacts
+
+test: lint docker-compose
+	$(foreach FLAVOR, $(IMAGE_FLAVORS), \
+	  $(FIGLET) "test: $(FLAVOR)"; \
+	  ./bin/pytest tests --image-flavor=$(FLAVOR); \
+	)
+
+test-snapshot:
+	ELASTIC_VERSION=$(ELASTIC_VERSION)-SNAPSHOT make test
+
+lint: venv
+	flake8 tests
+
+# Build from artifacts on the local filesystem, using an http server (running
+# in a container) to provide the artifacts to the Dockerfile.
+build-from-local-artifacts: venv dockerfile docker-compose env2yaml
+	docker run --rm -d --name=$(HTTPD) \
+	           -p 8000:8000 --expose=8000 -v $(ARTIFACTS_DIR):/mnt \
+	           python:3 bash -c 'cd /mnt && python3 -m http.server'
+	gtimeout 120 bash -c 'until curl -s localhost:8000 > /dev/null; do sleep 1; done'
+	pyfiglet -f puffy -w 160 "Building: full"; \
+	docker build --network=host -t $(IMAGE_TAG)-full:$(VERSION_TAG) -f $(ARTIFACTS_DIR)/Dockerfile-full data/logstash || \
+	  (docker kill $(HTTPD); false); \
+	docker tag $(IMAGE_TAG)-full:$(VERSION_TAG) $(IMAGE_TAG):$(VERSION_TAG);
+	docker kill $(HTTPD)
+
+build-from-local-oss-artifacts: venv dockerfile docker-compose env2yaml
+	docker run --rm -d --name=$(HTTPD) \
+	           -p 8000:8000 --expose=8000 -v $(ARTIFACTS_DIR):/mnt \
+	           python:3 bash -c 'cd /mnt && python3 -m http.server'
+	gtimeout 120 bash -c 'until curl -s localhost:8000 > /dev/null; do sleep 1; done'
+	pyfiglet -f puffy -w 160 "Building: oss"; \
+	docker build --network=host -t $(IMAGE_TAG)-oss:$(VERSION_TAG) -f $(ARTIFACTS_DIR)/Dockerfile-oss data/logstash || \
+	  (docker kill $(HTTPD); false);
+	-docker kill $(HTTPD)
+
+demo: docker-compose clean-demo
+	docker-compose up
+
+# Push the image to the dedicated push endpoint at "push.docker.elastic.co"
+push: test
+	$(foreach FLAVOR, $(IMAGE_FLAVORS), \
+	  docker tag $(IMAGE_TAG)-$(FLAVOR):$(VERSION_TAG) push.$(IMAGE_TAG)-$(FLAVOR):$(VERSION_TAG); \
+	  docker push push.$(IMAGE_TAG)-$(FLAVOR):$(VERSION_TAG); \
+	  docker rmi push.$(IMAGE_TAG)-$(FLAVOR):$(VERSION_TAG); \
+	)
+	# Also push the default version, with no suffix like '-oss' or '-full'
+	docker tag $(IMAGE_TAG):$(VERSION_TAG) push.$(IMAGE_TAG):$(VERSION_TAG);
+	docker push push.$(IMAGE_TAG):$(VERSION_TAG);
+	docker rmi push.$(IMAGE_TAG):$(VERSION_TAG);
+
+# The tests are written in Python. Make a virtualenv to handle the dependencies.
+venv: requirements.txt
+	@if [ -z $$PYTHON3 ]; then\
+	    PY3_MINOR_VER=`python3 --version 2>&1 | cut -d " " -f 2 | cut -d "." -f 2`;\
+	    if (( $$PY3_MINOR_VER < 5 )); then\
+		echo "Couldn't find python3 in \$PATH that is >=3.5";\
+		echo "Please install python3.5 or later or explicity define the python3 executable name with \$PYTHON3";\
+		echo "Exiting here";\
+		exit 1;\
+	    else\
+		export PYTHON3="python3.$$PY3_MINOR_VER";\
+	   fi;\
+	fi;\
+	test -d venv || virtualenv --python=$$PYTHON3 venv;\
+	pip install -r requirements.txt;\
+	touch venv;\
+
+# Make a Golang container that can compile our env2yaml tool.
+golang:
+	docker build -t golang:env2yaml data/golang
+
+# Compile "env2yaml", the helper for configuring logstash.yml via environment
+# variables.
+env2yaml: golang
+	docker run --rm -i \
+	  -v $(PWD)/data/logstash/env2yaml:/usr/local/src/env2yaml:Z \
+	  golang:env2yaml
+
+# Generate the Dockerfiles from Jinja2 templates.
+dockerfile: venv templates/Dockerfile.j2
+	$(foreach FLAVOR, $(IMAGE_FLAVORS), \
+	  jinja2 \
+	    -D elastic_version='$(ELASTIC_VERSION)' \
+	    -D version_tag='$(VERSION_TAG)' \
+	    -D image_flavor='$(FLAVOR)' \
+	    -D artifacts_dir='$(ARTIFACTS_DIR)' \
+	    templates/Dockerfile.j2 > $(ARTIFACTS_DIR)/Dockerfile-$(FLAVOR); \
+	)
+
+
+# Generate docker-compose files from Jinja2 templates.
+docker-compose: venv
+	$(foreach FLAVOR, $(IMAGE_FLAVORS), \
+	  jinja2 \
+	    -D version_tag='$(VERSION_TAG)' \
+	    -D image_flavor='$(FLAVOR)' \
+	    templates/docker-compose.yml.j2 > docker-compose-$(FLAVOR).yml; \
+	)
+	ln -sf docker-compose-$(DEFAULT_IMAGE_FLAVOR).yml docker-compose.yml
+
+clean: clean-demo
+	rm -f ${ARTIFACTS_DIR}/env2yaml/env2yaml ${ARTIFACTS_DIR}/Dockerfile
+	rm -rf venv
+
+clean-demo: docker-compose
+	docker-compose down
+	docker-compose rm --force
+
+.PHONY: clean clean-demo demo push test
--- a/docker/README.md
+++ b/docker/README.md
@ -0,0 +1,39 @@
+## Description
+
+This repository contains the official [Logstash][logstash] Docker image from
+[Elastic][elastic].
+
+Documentation can be found on the [Elastic website](https://www.elastic.co/guide/en/logstash/current/docker.html).
+
+[logstash]: https://www.elastic.co/products/logstash
+[elastic]: https://www.elastic.co/
+
+## Supported Docker versions
+
+The images have been tested on Docker version 18.09.2, build 6247962
+
+## Requirements
+A full build and test requires:
+* Docker
+* GNU Make
+* Python 3.5 with Virtualenv
+* JRuby 9.1+
+
+## Running a build
+To build an image check out the corresponding branch for the version and run the rake task
+Like this:
+```
+git checkout 7.0
+rake artifact:docker
+# and for the OSS package
+rake artifact:docker_oss
+```
+
+## Contributing, issues and testing
+
+Acceptance tests for the image are located in the `test` directory, and can
+be invoked with `make test`.
+
+This image is built on [Centos 7][centos-7].
+
+[centos-7]: https://github.com/CentOS/sig-cloud-instance-images/blob/50281d86d6ed5c61975971150adfd0ede86423bb/docker/Dockerfile
--- a/docker/bin/elastic-version
+++ b/docker/bin/elastic-version
@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+#
+# Print the Elastic Stack version for the current branch, as defined in
+# the 'version.json' file.
+#
+require 'yaml'
+
+def get_hard_coded_version
+  version_info = YAML::safe_load(IO.read('../versions.yml'))
+  version_info['logstash']
+end
+
+def qualify(version)
+  qualifier = ENV['VERSION_QUALIFIER']
+  qualifier ? [version, qualifier].join("-") : version
+end
+
+def get_version
+  version = get_hard_coded_version()
+  version = qualify(version)
+  ENV["RELEASE"] == "1" ? version : [version, "SNAPSHOT"].join("-")
+end
+
+puts get_version
--- a/docker/bin/pytest
+++ b/docker/bin/pytest
@ -0,0 +1,5 @@
+#!/bin/bash
+#
+# A wrapper for `pytest` to handle the appropriate Testinfra arguments.
+
+./venv/bin/pytest --verbose --connection=docker --hosts=logstash-test $@
--- a/docker/data/golang/Dockerfile
+++ b/docker/data/golang/Dockerfile
@ -0,0 +1,4 @@
+FROM golang:1.8
+RUN go get gopkg.in/yaml.v2
+WORKDIR /usr/local/src/env2yaml
+CMD ["go", "build"]
--- a/docker/data/logstash/bin/docker-entrypoint
+++ b/docker/data/logstash/bin/docker-entrypoint
@ -0,0 +1,15 @@
+#!/bin/bash -e
+
+# Map environment variables to entries in logstash.yml.
+# Note that this will mutate logstash.yml in place if any such settings are found.
+# This may be undesirable, especially if logstash.yml is bind-mounted from the
+# host system.
+env2yaml /usr/share/logstash/config/logstash.yml
+
+export LS_JAVA_OPTS="-Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ $LS_JAVA_OPTS"
+
+if [[ -z $1 ]] || [[ ${1:0:1} == '-' ]] ; then
+  exec logstash "$@"
+else
+  exec "$@"
+fi
--- a/docker/data/logstash/config/log4j2.properties
+++ b/docker/data/logstash/config/log4j2.properties
@ -0,0 +1,16 @@
+status = error
+name = LogstashPropertiesConfig
+
+appender.console.type = Console
+appender.console.name = plain_console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %m%n
+
+appender.json_console.type = Console
+appender.json_console.name = json_console
+appender.json_console.layout.type = JSONLayout
+appender.json_console.layout.compact = true
+appender.json_console.layout.eventEol = true
+
+rootLogger.level = ${sys:ls.log.level}
+rootLogger.appenderRef.console.ref = ${sys:ls.log.format}_console
--- a/docker/data/logstash/config/logstash-full.yml
+++ b/docker/data/logstash/config/logstash-full.yml
@ -0,0 +1,2 @@
+http.host: "0.0.0.0"
+xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
--- a/docker/data/logstash/config/logstash-oss.yml
+++ b/docker/data/logstash/config/logstash-oss.yml
@ -0,0 +1 @@
+http.host: "0.0.0.0"
--- a/docker/data/logstash/config/pipelines.yml
+++ b/docker/data/logstash/config/pipelines.yml
@ -0,0 +1,6 @@
+# This file is where you define your pipelines. You can define multiple.
+# For more information on multiple pipelines, see the documentation:
+#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
+
+- pipeline.id: main
+  path.config: "/usr/share/logstash/pipeline"
--- a/docker/data/logstash/env2yaml/env2yaml.go
+++ b/docker/data/logstash/env2yaml/env2yaml.go
@ -0,0 +1,167 @@
+// env2yaml
+//
+// Merge environment variables into logstash.yml.
+// For example, running Docker with:
+//
+//   docker run -e pipeline.workers=6
+//
+// or
+//
+//   docker run -e PIPELINE_WORKERS=6
+//
+// will cause logstash.yml to contain the line:
+//
+//   pipeline.workers: 6
+//
+package main
+
+import (
+	"errors"
+	"gopkg.in/yaml.v2"
+	"io/ioutil"
+	"log"
+	"os"
+	"strings"
+)
+
+// If the given string can be parsed as YAML, then do so and return the
+// resulting entity. Otherwise, return the string unmodified.
+func FromYamlIfPossible(str string) interface{} {
+	var entity interface{}
+	err := yaml.Unmarshal([]byte(str), &entity)
+	if err == nil {
+		return entity
+	} else {
+		return str
+	}
+}
+
+// Given a setting name, return a downcased version with delimiters removed.
+func squashSetting(setting string) string {
+	downcased := strings.ToLower(setting)
+	de_dotted := strings.Replace(downcased, ".", "", -1)
+	de_underscored := strings.Replace(de_dotted, "_", "", -1)
+	return de_underscored
+}
+
+// Given a setting name like "pipeline.workers" or "PIPELINE_UNSAFE_SHUTDOWN",
+// return the canonical setting name. eg. 'pipeline.unsafe_shutdown'
+func normalizeSetting(setting string) (string, error) {
+	valid_settings := []string{
+		"node.name",
+		"path.data",
+		"pipeline.id",
+		"pipeline.workers",
+		"pipeline.output.workers",
+		"pipeline.batch.size",
+		"pipeline.batch.delay",
+		"pipeline.unsafe_shutdown",
+		"pipeline.java_execution",
+		"path.config",
+		"config.string",
+		"config.test_and_exit",
+		"config.reload.automatic",
+		"config.reload.interval",
+		"config.debug",
+		"config.support_escapes",
+		"queue.type",
+		"path.queue",
+		"queue.page_capacity",
+		"queue.max_events",
+		"queue.max_bytes",
+		"queue.checkpoint.acks",
+		"queue.checkpoint.writes",
+		"queue.checkpoint.interval",
+		"queue.drain",
+		"dead_letter_queue.enable",
+		"dead_letter_queue.max_bytes",
+		"path.dead_letter_queue",
+		"http.host",
+		"http.port",
+		"log.level",
+		"log.format",
+		"modules",
+		"path.logs",
+		"path.plugins",
+		"xpack.monitoring.enabled",
+		"xpack.monitoring.collection.interval",
+		"xpack.monitoring.elasticsearch.hosts",
+		"xpack.monitoring.elasticsearch.username",
+		"xpack.monitoring.elasticsearch.password",
+		"xpack.monitoring.elasticsearch.ssl.certificate_authority",
+		"xpack.monitoring.elasticsearch.ssl.truststore.path",
+		"xpack.monitoring.elasticsearch.ssl.truststore.password",
+		"xpack.monitoring.elasticsearch.ssl.keystore.path",
+		"xpack.monitoring.elasticsearch.ssl.keystore.password",
+		"xpack.management.enabled",
+		"xpack.management.logstash.poll_interval",
+		"xpack.management.pipeline.id",
+		"xpack.management.elasticsearch.hosts",
+		"xpack.management.elasticsearch.username",
+		"xpack.management.elasticsearch.password",
+		"xpack.management.elasticsearch.ssl.certificate_authority",
+		"xpack.management.elasticsearch.ssl.truststore.path",
+		"xpack.management.elasticsearch.ssl.truststore.password",
+		"xpack.management.elasticsearch.ssl.keystore.path",
+		"xpack.management.elasticsearch.ssl.keystore.password",
+		"cloud.id",
+		"cloud.auth",
+	}
+
+	for _, valid_setting := range valid_settings {
+		if squashSetting(setting) == squashSetting(valid_setting) {
+			return valid_setting, nil
+		}
+	}
+	return "", errors.New("Invalid setting: " + setting)
+}
+
+func main() {
+	if len(os.Args) != 2 {
+		log.Fatalf("usage: env2yaml FILENAME")
+	}
+	settingsFilePath := os.Args[1]
+
+	settingsFile, err := ioutil.ReadFile(settingsFilePath)
+	if err != nil {
+		log.Fatalf("error: %v", err)
+	}
+
+	// Read the original settings file into a map.
+	settings := make(map[string]interface{})
+	err = yaml.Unmarshal(settingsFile, &settings)
+	if err != nil {
+		log.Fatalf("error: %v", err)
+	}
+
+	// Merge any valid settings found in the environment.
+	foundNewSettings := false
+	for _, line := range os.Environ() {
+		kv := strings.SplitN(line, "=", 2)
+		key := kv[0]
+		value := kv[1]
+		setting, err := normalizeSetting(key)
+		if err == nil {
+			foundNewSettings = true
+			log.Printf("Setting '%s' from environment.", setting)
+			settings[setting] = FromYamlIfPossible(value)
+		}
+	}
+
+	if foundNewSettings {
+		output, err := yaml.Marshal(&settings)
+		if err != nil {
+			log.Fatalf("error: %v", err)
+		}
+
+		stat, err := os.Stat(settingsFilePath)
+		if err != nil {
+			log.Fatalf("error: %v", err)
+		}
+
+		err = ioutil.WriteFile(settingsFilePath, output, stat.Mode())
+		if err != nil {
+			log.Fatalf("error: %v", err)
+		}
+	}
+}
--- a/docker/data/logstash/pipeline/default.conf
+++ b/docker/data/logstash/pipeline/default.conf
@ -0,0 +1,12 @@
+input {
+  beats {
+    port => 5044
+  }
+}
+
+output {
+  stdout {
+    codec => rubydebug
+  }
+}
+
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@ -0,0 +1 @@
+docker-compose-full.yml
--- a/docker/examples/logstash.conf
+++ b/docker/examples/logstash.conf
@ -0,0 +1,14 @@
+input {
+  heartbeat {
+    interval => 5
+    message  => 'Hello from Logstash 💓'
+  }
+}
+
+output {
+  elasticsearch {
+    hosts    => [ 'elasticsearch' ]
+    user     => 'elastic'
+    password => 'changeme'
+  }
+}
--- a/docker/requirements.txt
+++ b/docker/requirements.txt
@ -0,0 +1,7 @@
+docker-compose==1.11.2
+flake8==3.4.1
+jinja2-cli[yaml]==0.6.0
+jinja2==2.9.5
+retrying==1.3.3
+testinfra==1.6.0
+pyfiglet
--- a/docker/templates/Dockerfile.j2
+++ b/docker/templates/Dockerfile.j2
@ -0,0 +1,72 @@
+# This Dockerfile was generated from templates/Dockerfile.j2
+{% if image_flavor == 'oss' -%}
+  {% set tarball = 'logstash-oss-%s.tar.gz' % elastic_version -%}
+{% else -%}
+  {% set tarball = 'logstash-%s.tar.gz' % elastic_version -%}
+{% endif -%}
+
+
+FROM centos:7
+
+# Install Java and the "which" command, which is needed by Logstash's shell
+# scripts.
+RUN yum update -y && yum install -y java-1.8.0-openjdk-devel which && \
+    yum clean all
+
+# Provide a non-root user to run the process.
+RUN groupadd --gid 1000 logstash && \
+    adduser --uid 1000 --gid 1000 \
+      --home-dir /usr/share/logstash --no-create-home \
+      logstash
+
+# Add Logstash itself.
+RUN curl -Lo - 'http://localhost:8000/'{{ tarball }} | \
+    tar zxf - -C /usr/share && \
+    mv /usr/share/logstash-{{ elastic_version }} /usr/share/logstash && \
+    chown --recursive logstash:logstash /usr/share/logstash/ && \
+    chown -R logstash:root /usr/share/logstash && \
+    chmod -R g=u /usr/share/logstash && \
+    find /usr/share/logstash -type d -exec chmod g+s {} \; && \
+    ln -s /usr/share/logstash /opt/logstash
+
+WORKDIR /usr/share/logstash
+
+ENV ELASTIC_CONTAINER true
+ENV PATH=/usr/share/logstash/bin:$PATH
+
+# Provide a minimal configuration, so that simple invocations will provide
+# a good experience.
+ADD config/pipelines.yml config/pipelines.yml
+ADD config/logstash-{{ image_flavor }}.yml config/logstash.yml
+ADD config/log4j2.properties config/
+ADD pipeline/default.conf pipeline/logstash.conf
+RUN chown --recursive logstash:root config/ pipeline/
+
+# Ensure Logstash gets a UTF-8 locale by default.
+ENV LANG='en_US.UTF-8' LC_ALL='en_US.UTF-8'
+
+# Place the startup wrapper script.
+ADD bin/docker-entrypoint /usr/local/bin/
+RUN chmod 0755 /usr/local/bin/docker-entrypoint
+
+USER 1000
+
+ADD env2yaml/env2yaml /usr/local/bin/
+
+EXPOSE 9600 5044
+
+
+LABEL org.label-schema.schema-version="1.0" \
+  org.label-schema.vendor="Elastic" \
+  org.label-schema.name="logstash" \
+  org.label-schema.version="{{ elastic_version }}" \
+  org.label-schema.url="https://www.elastic.co/products/logstash" \
+  org.label-schema.vcs-url="https://github.com/elastic/logstash" \
+{% if image_flavor == 'oss' -%}
+  license="Apache-2.0"
+{% else -%}
+  license="Elastic License"
+{% endif -%}
+
+
+ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
--- a/docker/templates/docker-compose.yml.j2
+++ b/docker/templates/docker-compose.yml.j2
@ -0,0 +1,23 @@
+---
+version: '3.0'
+services:
+  logstash:
+    image: docker.elastic.co/logstash/logstash:{{ version_tag }}
+    volumes:
+      - ./examples/logstash.conf/:/usr/share/logstash/pipeline/logstash.conf
+    networks:
+      - elastic-stack
+
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch-platinum:{{ version_tag }}
+    networks:
+      - elastic-stack
+
+  kibana:
+    image: docker.elastic.co/kibana/kibana:{{ version_tag }}
+    ports: [ '5601:5601' ]
+    networks:
+      - elastic-stack
+
+networks:
+  elastic-stack:
--- a/docker/tests/init.py
+++ b/docker/tests/init.py
--- a/docker/tests/conftest.py
+++ b/docker/tests/conftest.py
@ -0,0 +1,25 @@
+from subprocess import run
+import pytest
+from .constants import container_name, version
+import docker
+
+docker_engine = docker.from_env()
+
+
+def pytest_addoption(parser):
+    """Customize testinfra with config options via cli args"""
+    # Let us specify which docker-compose-(image_flavor).yml file to use
+    parser.addoption('--image-flavor', action='store', default='full',
+                     help='Docker image flavor; the suffix used in docker-compose-<flavor>.yml')
+
+
+@pytest.fixture(scope='session', autouse=True)
+def start_container():
+    image = 'docker.elastic.co/logstash/logstash-%s:%s' % (pytest.config.getoption('--image-flavor'), version)
+    docker_engine.containers.run(image, name=container_name, detach=True, stdin_open=False)
+
+
+def pytest_unconfigure(config):
+    container = docker_engine.containers.get(container_name)
+    container.stop()
+    container.remove()
--- a/docker/tests/constants.py
+++ b/docker/tests/constants.py
@ -0,0 +1,16 @@
+import os
+import pytest
+from subprocess import run, PIPE
+
+version = run('./bin/elastic-version', stdout=PIPE).stdout.decode().strip()
+version_number = version.split('-')[0]  # '7.0.0-alpha1-SNAPSHOT' -> '7.0.0'
+logstash_version_string = 'logstash %s' % version_number  # eg. 'logstash 7.0.0'
+
+
+try:
+    if len(os.environ['STAGING_BUILD_NUM']) > 0:
+        version += '-%s' % os.environ['STAGING_BUILD_NUM']  # eg. '5.3.0-d5b30bd7'
+except KeyError:
+    pass
+
+container_name = 'logstash-test'
--- a/docker/tests/docker-compose.yml
+++ b/docker/tests/docker-compose.yml
@ -0,0 +1,5 @@
+---
+version: '3'
+services:
+  logstash:
+    container_name: logstash-test
--- a/docker/tests/fixtures.py
+++ b/docker/tests/fixtures.py
@ -0,0 +1,91 @@
+import json
+import os
+import yaml
+from pytest import config, fixture
+from .constants import container_name, version
+from retrying import retry
+from subprocess import run, PIPE
+from time import sleep
+
+retry_settings = {
+    'wait_fixed': 1000,
+    'stop_max_attempt_number': 60
+}
+
+
+@fixture
+def logstash(host):
+    class Logstash:
+        def __init__(self):
+            self.version = version
+            self.name = container_name
+            self.process = host.process.get(comm='java')
+            self.settings_file = host.file('/usr/share/logstash/config/logstash.yml')
+            self.image_flavor = config.getoption('--image-flavor')
+            self.image = 'docker.elastic.co/logstash/logstash-%s:%s' % (self.image_flavor, version)
+
+            if 'STAGING_BUILD_NUM' in os.environ:
+                self.tag = '%s-%s' % (self.version, os.environ['STAGING_BUILD_NUM'])
+            else:
+                self.tag = self.version
+
+            self.docker_metadata = json.loads(
+                run(['docker', 'inspect', self.image], stdout=PIPE).stdout.decode())[0]
+
+        def start(self, args=None):
+            if args:
+                arg_array = args.split(' ')
+            else:
+                arg_array = []
+            run(['docker', 'run', '-d', '--name', self.name] + arg_array + [self.image])
+
+        def stop(self):
+            run(['docker', 'kill', self.name])
+            run(['docker', 'rm', self.name])
+
+        def restart(self, args=None):
+            self.stop()
+            self.start(args)
+
+        @retry(**retry_settings)
+        def get_node_info(self):
+            """Return the contents of Logstash's node info API.
+
+            It retries for a while, since Logstash may still be coming up.
+            Refer: https://www.elastic.co/guide/en/logstash/master/node-info-api.html
+            """
+            result = json.loads(host.command.check_output('curl -s http://localhost:9600/_node'))
+            assert 'workers' in result['pipelines']['main']
+            return result
+
+        def get_settings(self):
+            return yaml.load(self.settings_file.content_string)
+
+        def run(self, command):
+            return host.run(command)
+
+        def stdout_of(self, command):
+            return host.run(command).stdout.strip()
+
+        def stderr_of(self, command):
+            return host.run(command).stderr.strip()
+
+        def environment(self, varname):
+            environ = {}
+            for line in self.run('env').stdout.strip().split("\n"):
+                var, value = line.split('=')
+                environ[var] = value
+            return environ[varname]
+
+        def get_docker_log(self):
+            return run(['docker', 'logs', self.name], stdout=PIPE).stdout.decode()
+
+        @retry(**retry_settings)
+        def assert_in_log(self, string):
+            assert string in self.get_docker_log()
+
+        @retry(**retry_settings)
+        def assert_not_in_log(self, string):
+            assert string not in self.get_docker_log()
+
+    return Logstash()
--- a/docker/tests/helpers.py
+++ b/docker/tests/helpers.py
@ -0,0 +1,8 @@
+import subprocess
+import os
+from .constants import image, version
+
+try:
+    version += '-%s' % os.environ['STAGING_BUILD_NUM']
+except KeyError:
+    pass
--- a/docker/tests/test_basics.py
+++ b/docker/tests/test_basics.py
@ -0,0 +1,50 @@
+from .fixtures import logstash
+from .constants import logstash_version_string
+
+
+def test_logstash_is_the_correct_version(logstash):
+    assert logstash_version_string in logstash.stdout_of('logstash --version')
+
+
+def test_the_default_user_is_logstash(logstash):
+    assert logstash.stdout_of('whoami') == 'logstash'
+
+
+def test_that_the_user_home_directory_is_usr_share_logstash(logstash):
+    assert logstash.environment('HOME') == '/usr/share/logstash'
+
+
+def test_locale_variables_are_set_correctly(logstash):
+    assert logstash.environment('LANG') == 'en_US.UTF-8'
+    assert logstash.environment('LC_ALL') == 'en_US.UTF-8'
+
+
+def test_opt_logstash_is_a_symlink_to_usr_share_logstash(logstash):
+    assert logstash.stdout_of('realpath /opt/logstash') == '/usr/share/logstash'
+
+
+def test_all_logstash_files_are_owned_by_logstash(logstash):
+    assert logstash.stdout_of('find /usr/share/logstash ! -user logstash') == ''
+
+
+def test_logstash_user_is_uid_1000(logstash):
+    assert logstash.stdout_of('id -u logstash') == '1000'
+
+
+def test_logstash_user_is_gid_1000(logstash):
+    assert logstash.stdout_of('id -g logstash') == '1000'
+
+
+def test_logging_config_does_not_log_to_files(logstash):
+    assert logstash.stdout_of('grep RollingFile /logstash/config/log4j2.properties') == ''
+
+
+# REF: https://docs.openshift.com/container-platform/3.5/creating_images/guidelines.html
+def test_all_files_in_logstash_directory_are_gid_zero(logstash):
+    bad_files = logstash.stdout_of('find /usr/share/logstash ! -gid 0').split()
+    assert len(bad_files) is 0
+
+
+def test_all_directories_in_logstash_directory_are_setgid(logstash):
+    bad_dirs = logstash.stdout_of('find /usr/share/logstash -type d ! -perm /g+s').split()
+    assert len(bad_dirs) is 0
--- a/docker/tests/test_entrypoint.py
+++ b/docker/tests/test_entrypoint.py
@ -0,0 +1,14 @@
+from .fixtures import logstash
+import pytest
+
+
+@pytest.mark.xfail
+def test_whitespace_in_config_string_cli_flag(logstash):
+    config = 'input{heartbeat{}}    output{stdout{}}'
+    assert logstash.run("-t -e '%s'" % config).rc == 0
+
+
+def test_running_an_arbitrary_command(logstash):
+    result = logstash.run('uname --all')
+    assert result.rc == 0
+    assert 'GNU/Linux' in str(result.stdout)
--- a/docker/tests/test_labels.py
+++ b/docker/tests/test_labels.py
@ -0,0 +1,15 @@
+from .fixtures import logstash
+
+
+def test_labels(logstash):
+    labels = logstash.docker_metadata['Config']['Labels']
+    assert labels['org.label-schema.name'] == 'logstash'
+    assert labels['org.label-schema.schema-version'] == '1.0'
+    assert labels['org.label-schema.url'] == 'https://www.elastic.co/products/logstash'
+    assert labels['org.label-schema.vcs-url'] == 'https://github.com/elastic/logstash'
+    assert labels['org.label-schema.vendor'] == 'Elastic'
+    assert labels['org.label-schema.version'] == logstash.tag
+    if logstash.image_flavor == 'oss':
+        assert labels['license'] == 'Apache-2.0'
+    else:
+        assert labels['license'] == 'Elastic License'
--- a/docker/tests/test_process.py
+++ b/docker/tests/test_process.py
@ -0,0 +1,15 @@
+from .fixtures import logstash
+
+
+def test_process_is_pid_1(logstash):
+    assert logstash.process.pid == 1
+
+
+def test_process_is_running_as_the_correct_user(logstash):
+    assert logstash.process.user == 'logstash'
+
+
+def test_process_is_running_with_cgroup_override_flags(logstash):
+    # REF: https://github.com/elastic/logstash-docker/pull/97
+    assert '-Dls.cgroup.cpu.path.override=/' in logstash.process.args
+    assert '-Dls.cgroup.cpuacct.path.override=/' in logstash.process.args
--- a/docker/tests/test_settings.py
+++ b/docker/tests/test_settings.py
@ -0,0 +1,71 @@
+from .fixtures import logstash
+from retrying import retry
+import time
+
+
+def test_setting_pipeline_workers_from_environment(logstash):
+    logstash.restart(args='-e pipeline.workers=6')
+    assert logstash.get_node_info()['pipelines']['main']['workers'] == 6
+
+
+def test_setting_pipeline_batch_size_from_environment(logstash):
+    logstash.restart(args='-e pipeline.batch.size=123')
+    assert logstash.get_node_info()['pipelines']['main']['batch_size'] == 123
+
+
+def test_setting_pipeline_batch_delay_from_environment(logstash):
+    logstash.restart(args='-e pipeline.batch.delay=36')
+    assert logstash.get_node_info()['pipelines']['main']['batch_delay'] == 36
+
+
+def test_setting_pipeline_unsafe_shutdown_from_environment(logstash):
+    logstash.restart(args='-e pipeline.unsafe_shutdown=true')
+    assert logstash.get_settings()['pipeline.unsafe_shutdown'] is True
+
+
+def test_setting_pipeline_unsafe_shutdown_with_shell_style_variable(logstash):
+    logstash.restart(args='-e PIPELINE_UNSAFE_SHUTDOWN=true')
+    assert logstash.get_settings()['pipeline.unsafe_shutdown'] is True
+
+
+def test_setting_things_with_upcased_and_underscored_env_vars(logstash):
+    logstash.restart(args='-e PIPELINE_BATCH_DELAY=24')
+    assert logstash.get_node_info()['pipelines']['main']['batch_delay'] == 24
+
+
+def test_disabling_xpack_monitoring_via_environment(logstash):
+    logstash.restart(args='-e xpack.monitoring.enabled=false')
+    assert logstash.get_settings()['xpack.monitoring.enabled'] is False
+
+
+def test_enabling_java_execution_via_environment(logstash):
+    logstash.restart(args='-e pipeline.java_execution=true')
+    logstash.assert_in_log('logstash.javapipeline')
+
+
+def test_disabling_java_execution_via_environment(logstash):
+    logstash.restart(args='-e pipeline.java_execution=true')
+    logstash.assert_not_in_log('logstash.javapipeline')
+
+
+def test_setting_elasticsearch_urls_as_an_array(logstash):
+    setting_string = '["http://node1:9200","http://node2:9200"]'
+    logstash.restart(args='-e xpack.monitoring.elasticsearch.hosts=%s' % setting_string)
+    live_setting = logstash.get_settings()['xpack.monitoring.elasticsearch.hosts']
+    assert type(live_setting) is list
+    assert 'http://node1:9200' in live_setting
+    assert 'http://node2:9200' in live_setting
+
+
+def test_invalid_settings_in_environment_are_ignored(logstash):
+    logstash.restart(args='-e cheese.ftw=true')
+    assert not logstash.settings_file.contains('cheese.ftw')
+
+
+def test_settings_file_is_untouched_when_no_settings_in_env(logstash):
+    original_timestamp = logstash.settings_file.mtime
+    original_hash = logstash.settings_file.sha256sum
+    logstash.restart()
+    time.sleep(1)  # since mtime() has one second resolution
+    assert logstash.settings_file.mtime == original_timestamp
+    assert logstash.settings_file.sha256sum == original_hash
--- a/docker/tox.ini
+++ b/docker/tox.ini
@ -0,0 +1,5 @@
+# pytest fixtures (which are wonderful) trigger false positives for these
+# pyflakes checks.
+[flake8]
+ignore = F401,F811
+max-line-length = 120
--- a/docs/static/docker.asciidoc
+++ b/docs/static/docker.asciidoc
@ -5,7 +5,7 @@ registry. The base image is https://hub.docker.com/_/centos/[centos:7].

 A list of all published Docker images and tags is available at
 https://www.docker.elastic.co[www.docker.elastic.co]. The source code is in
-https://github.com/elastic/logstash-docker/tree/{branch}[GitHub].
+https://github.com/elastic/logstash/tree/{branch}[GitHub].

 These images are free to use under the Elastic license. They contain open source 
 and free commercial features and access to paid commercial features.  
--- a/rakelib/artifacts.rake
+++ b/rakelib/artifacts.rake
@ -167,6 +167,18 @@ namespace "artifact" do
    build_tar('ELASTIC-LICENSE', "-all-plugins")
  end

+  desc "Build docker image"
+  task "docker" => ["prepare", "generate_build_metadata", "tar"] do
+    puts("[docker] Building docker image")
+    build_docker(false)
+  end
+
+  desc "Build OSS docker image"
+  task "docker_oss" => ["prepare", "generate_build_metadata", "tar_oss"] do
+    puts("[docker_oss] Building OSS docker image")
+    build_docker(true)
+  end
+
  # Auxiliary tasks
  task "build" => [:generate_build_metadata] do
    Rake::Task["artifact:gems"].invoke unless SNAPSHOT_BUILD
@ -178,6 +190,8 @@ namespace "artifact" do
    Rake::Task["artifact:zip_oss"].invoke
    Rake::Task["artifact:tar"].invoke
    Rake::Task["artifact:tar_oss"].invoke
+    Rake::Task["artifact:docker"].invoke
+    Rake::Task["artifact:docker_oss"].invoke
  end

  task "generate_build_metadata" do
@ -499,4 +513,19 @@ namespace "artifact" do
      out.cleanup
    end
  end # def package
+
+  def build_docker(oss = false)
+    env = {
+      "ARTIFACTS_DIR" => ::File.join(Dir.pwd, "build"),
+      "RELEASE" => ENV["RELEASE"],
+      "VERSION_QUALIFIER" => VERSION_QUALIFIER
+    }
+    Dir.chdir("docker") do |dir|
+      if oss
+        system(env, "make build-from-local-oss-artifacts")
+      else
+        system(env, "make build-from-local-artifacts")
+      end
+    end
+  end
 end