[[sample-configuration-files]] === Sample configuration files WARNING: This documentation is still in development and may be changed or removed in a future release. These configuration files are used in the <>. You can use them as templates when you configure Logstash together with the rest of the Elastic Stack in a Kubernetes environment. You can download the files together as a link:https://github.com/elastic/logstash/blob/main/docsk8s/sample-files/logstash-k8s-qs.zip[zip archive]. [[qs-setup-files]] ==== Setup files These files are used to create certificates and keys required for secure communication between {beats} and {ls}. They are included for illustration purposes only. For production environments, supply your own keys and certificates as appropriate. `cert/generate_cert.sh`:: Generates the `ca.crt`, `client.key`, `client.crt`, `server.key`, and `server.pkcs8.key` used to establish a secure connection between Filebeat and Logstash. The certificates and keys are all contained in the `001-secret.yaml` file that is generated when you run `generate_cert.sh`. `cert/openssl.conf`:: The OpenSSL Generated Server Certificate used for TLS communication between resources. This config file creates a secrets file `001-secret.yaml`. We will install the secrets file as we set up the {stack}. [[qs-logstash-configuration-files]] ==== Logstash configuration files [[qs-configmap]] `001-configmap.yaml`:: This file contains the Logstash settings and pipeline configuration: + [source,yaml] -- --- # ConfigMap for logstash pipeline definition data: logstash.conf: | <1> input { beats { port => "5044" ssl => true ssl_certificate_authorities => ["/usr/share/logstash/config/ca.crt"] ssl_certificate => "/usr/share/logstash/config/server.crt" ssl_key => "/usr/share/logstash/config/server.pkcs8.key" ssl_verify_mode => "force_peer" } } output { elasticsearch { hosts => ["https://demo-es-http:9200"] index => "kube-apiserver-%{+YYYY.MM.dd}" cacert => "/usr/share/logstash/config/es_ca.crt" user => 'elastic' password => '${ELASTICSEARCH_PASSWORD}' } } --- # ConfigMap for logstash.yml definition data: logstash.yml: | <2> api.http.host: "0.0.0.0" -- <1> Definition of {ls} configuration file. We will refer to this definition later in the deployment file, where we will define volumes. <2> Definition of {logstash-ref}/logstash-settings-file.html[logstash.yml] file Define each key/value pair to override defaults. We will refer to this definition later in the deployment file. [[qs-secrets]] `001-secrets.yaml`:: This secrets file includes certificates and key files required for secure communication between {ls} and the rest of the {stack}. This example was generated by the supplied script, but for your own configuration it should contain the base64 encoded representations of your own certificates and keys. + You can generate this file for your own certs and keys by using the `kubectl create secret generic` command: + [source,sh] -- kubectl create secret generic logstash-beats-tls --from-file=ca.crt --from-file=client.crt --from-file=client.key --from-file=server.crt --from-file=server.pkcs8.key --dry-run=client -o yaml | kubectl label -f- --dry-run=client -o yaml --local app=logstash-demo > ../001-secret.yaml -- + The command generates a secrets file that looks resembles this. + [source,yaml] -- apiVersion: v1 data: ca.crt: client.crt: client.key: server.crt: server.pkcs8.key: kind: Secret metadata: creationTimestamp: null labels: app: logstash-demo name: logstash-beats-tls -- [[qs-deployment]] `002-deployment.yaml`:: Contains the configuration definition for {ls}. + [source,yaml] -- spec: replicas: 1 selector: matchLabels: app: logstash-demo template: metadata: labels: app: logstash-demo spec: containers: - name: logstash securityContext: runAsNonRoot: true runAsUser: 1000 image: {docker-image} <1> env: - name: LS_JAVA_OPTS <2> value: "-Xmx1g -Xms1g" - name: ELASTICSEARCH_PASSWORD <11> valueFrom: secretKeyRef: name: demo-es-elastic-user key: elastic resources: limits: <3> cpu: 2000m memory: 2Gi requests: cpu: 1000m memory: 2Gi ports: <4> - containerPort: 9600 name: stats - containerPort: 5044 name: beats livenessProbe: <5> httpGet: path: / port: 9600 initialDelaySeconds: 60 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: <6> httpGet: path: / port: 9600 initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 volumeMounts: <7> - name: logstash-pipeline mountPath: /usr/share/logstash/pipeline - name: logstash-config <8> mountPath: /usr/share/logstash/config/logstash.yml subPath: logstash.yml - name: es-certs <9> mountPath: /usr/share/logstash/config/es_ca.crt subPath: ca.crt - name: logstash-beats-tls mountPath: /usr/share/logstash/config/ca.crt subPath: ca.crt - name: logstash-beats-tls mountPath: /usr/share/logstash/config/server.pkcs8.key subPath: server.pkcs8.key - name: logstash-beats-tls mountPath: /usr/share/logstash/config/server.crt subPath: server.crt volumes: - name: logstash-pipeline <7> configMap: name: logstash-pipeline - name: logstash-config <8> configMap: name: logstash-config - name: es-certs <9> secret: secretName: demo-es-http-certs-public - name: logstash-beats-tls <10> secret: secretName: logstash-beats-tls - name: es-user <11> secret: secretName: demo-es-elastic-user -- <1> {ls} {logstash-ref}/docker.html[docker image] <2> Set non-default JVM settings, such as memory allocation, here in the `LS_JAVA_OPTS` env variable to avoid the need to add a whole `jvm.options` file in a `ConfigMap` <3> Resource/memory limits for the pod. Refer to Kubernetes documentation to set resources appropriately for each pod. Ensure that each pod has sufficient memory to handle the heap specified in <2>, allowing enough memory to deal with direct memory. Check out {logstash-ref}/jvm-settings.html#heap-size[Logstash JVM settings] for details. <4> Expose the necessary ports on the container. Here we are exposing port `5044` for the beats input, and `9600` for the metricbeat instance to query the logstash metrics API for stack monitoring purposes. <5> Liveness probe to determine whether Logstash is running. Here we point to the Logstash Metrics API, an HTTP based API that will be ready shortly after logstash starts. Note that the endpoint shows no indication that Logstash is active, only that the API is available. <6> Readiness probe to determine whether Logstash is running. Here we point to the {ls} Metrics API, an HTTP based API that will be ready shortly after {ls} starts. Note that the endpoint shows no indication that {ls} is active, only that the API is available. <7> The pipeline configuration that we created in <> needs a `volume` and a `volumeMount`. The `volume` refers to the created <> and the `volumeMount` refers to the created `volume` and mounts in a location that logstash will read. Unless a separate `pipeline.yml` file is created by a further `ConfigMap` definition, the expected location of pipeline configurations is `/usr/share/logstash/pipelines` and the `mountPath` should be set accordingly. <8> Name of the <> we created earlier. This file should contain key/value pairs intended to override the default values in {logstash-ref}/logstash-settings-file.html[logstash.yml], using the `flat key syntax` described in that document. To setup, this needs a `volume` and a `volumeMount`. The `volume` refers to the created <> and the `volumeMount` refers to the created `volume` and mounts in a location that {ls} will read. The `mountPath` should be set to ` `/usr/share/logstash/logstash.yml`. <9> `Volume` and `VolumeMount` definitions for certificates to use with Elasticsearch. This contains the CA certificate to output data to {es}. Refer to link:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html[TLS certificates] in the {eck} Guide for details. <10> `Volume` and `VolumeMount` definitions for certificates to use with Beats. <11> The {es} password is taken from `demo-es-elastic-user` and passed to the Logstash pipeline as an `ELASTICSEARCH_PASSWORD` environment variable. Refer to link:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-request-elasticsearch-endpoint.html[Access the {es} endpoint] in the {eck} Guide for details. [[qs-service]] `003-service.yaml`:: + This file contains the Service definition, opening up ports on the logstash pods to the internal metricbeat (for stack monitoring) and filebeat in this instance. [source,yaml] -- spec: type: ClusterIP ports: - port: 9600 <1> name: "stats" protocol: TCP targetPort: 9600 <1> - port: 5044 <2> name: "beats" protocol: TCP targetPort: 5044 <2> selector: app: logstash-demo -- <1> Opens port `9600` for {metricbeat} to connect to the {ls} metrics API. <2> Opens port `5044` for {filebeat} to connect to the {beats} input defined in the <>. [[qs-additional-logstash-configuration]] [[qs-autoscaler]] `004-hpa.yml`:: + This file sets up a horizontal pod autoscaler to scale {ls} instances up and down, depending on the load on the {ls} instance(s). See link:https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[kubernetes autoscaler docs] for more details. [source,yaml] -- apiVersion: autoscaling/v2 <1> kind: HorizontalPodAutoscaler metadata: name: logstash labels: app: logstash-demo spec: minReplicas: 1 <2> maxReplicas: 2 behavior: scaleUp: stabilizationWindowSeconds: 60 <3> scaleDown: stabilizationWindowSeconds: 180 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: logstash <4> metrics: - type: Resource <5> resource: name: cpu target: type: Utilization averageUtilization: 80 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 80 -- <1> Requires {k8s} `1.23` and higher. <2> Specifies the maximum and minimum number of Logstashes desired for the cluster. <3> Specifies stabilization windows to avoid rapidly scaling nodes up and down unnecessarily. <4> `Deployment` created <> [[qs-stack-monitoring-files]] `006-metricbeat.yaml`:: Enables the {metricbeat} {ls} module and sets it to collect metrics data from `logstash:9600`: + [source,yaml] -- - module: logstash <1> metricsets: - node - node_stats period: 10s hosts: - logstash:9600 xpack.enabled: true -- <1> Definition for logstash module, defined under `spec.config.metricbeat.modules` [[qs-filebeat-configuration]] `005-filebeat.yaml`:: This file includes the configuration required for a beat to communicate with {ls}. It includes the {ls} output definition, and makes the generated certs and key files from <> available to the beat to enable secure communication with {ls}. + [source,yaml] -- volumes: <1> - name: logstash-beats-tls secret: secretName: logstash-beats-tls -- <1> Volume definition for certs/keys defined under `deployment.podTemplate.spec`. + [source,yaml] -- volumeMounts: <1> - name: logstash-beats-tls mountPath: /usr/share/filebeat/ca.crt subPath: ca.crt - name: logstash-beats-tls mountPath: /usr/share/filebeat/client.key subPath: client.key - name: logstash-beats-tls mountPath: /usr/share/filebeat/client.crt subPath: client.crt -- <1> Volume mount definition for certs/keys defined under `deployment.podTemplate.spec.containers`. + [source,yaml] -- output.logstash: <1> hosts: - "logstash:5044" ssl.certificate_authorities: ["/usr/share/filebeat/ca.crt"] ssl.certificate: "/usr/share/filebeat/client.crt" ssl.key: "/usr/share/filebeat/client.key" -- <1> Logstash output definition defined under `spec.config`. [[qs-stack-configuration-files]] `000-elasticsearch.yaml`:: Configures a single {es} instance to receive output data from {ls}. `007-kibana.yaml`:: Configures a single {kib} instance to visualize the logs and metrics data.