[[sample-configuration-files]]
=== Sample configuration files

WARNING: This documentation is still in development and may be changed or removed in a future release.

These configuration files are used in the <<ls-k8s-quick-start,{ls} and Kubernetes quick start>>. You can use them as templates when you configure Logstash together with the rest of the Elastic Stack in a Kubernetes environment.

You can download the files together as a link:https://github.com/elastic/logstash/blob/main/docsk8s/sample-files/logstash-k8s-qs.zip[zip archive].


[[qs-setup-files]]
==== Setup files

These files are used to create certificates and keys required for secure communication between {beats} and {ls}.
They are included for illustration purposes only.
For production environments, supply your own keys and certificates as appropriate.

`cert/generate_cert.sh`::
Generates the `ca.crt`, `client.key`, `client.crt`, `server.key`, and `server.pkcs8.key` used to establish a secure connection between Filebeat and Logstash. The certificates and keys are all contained in the `001-secret.yaml` file that is generated when you run `generate_cert.sh`.

`cert/openssl.conf`::
The OpenSSL Generated Server Certificate used for TLS communication between resources.

This config file creates a secrets file `001-secret.yaml`. 
We will install the secrets file as we set up the {stack}.

[[qs-logstash-configuration-files]]
==== Logstash configuration files


[[qs-configmap]]
`001-configmap.yaml`::
This file contains the Logstash settings and pipeline configuration:
+
[source,yaml]
--
  ---
# ConfigMap for logstash pipeline definition
data:
  logstash.conf: | <1>
    input {
      beats {
        port => "5044"
        ssl => true
        ssl_certificate_authorities => ["/usr/share/logstash/config/ca.crt"]
        ssl_certificate => "/usr/share/logstash/config/server.crt"
        ssl_key => "/usr/share/logstash/config/server.pkcs8.key"
        ssl_verify_mode => "force_peer"
      }
    }
    output {
      elasticsearch {
        hosts => ["https://demo-es-http:9200"]
        index => "kube-apiserver-%{+YYYY.MM.dd}"
        cacert => "/usr/share/logstash/config/es_ca.crt"
        user => 'elastic'
        password => '${ELASTICSEARCH_PASSWORD}'
      }
    }
---
# ConfigMap for logstash.yml definition
data:
  logstash.yml: | <2>
    api.http.host: "0.0.0.0"
--

<1> Definition of {ls} configuration file. 
We will refer to this definition later in the deployment file, where we will define volumes.
<2> Definition of {logstash-ref}/logstash-settings-file.html[logstash.yml] file
Define each key/value pair to override defaults. We will refer to this definition later in the deployment file.

[[qs-secrets]]
`001-secrets.yaml`::

This secrets file includes certificates and key files required for secure communication between {ls} and the rest of the {stack}. This example was generated by the supplied script, but for your own configuration it should contain the base64 encoded representations of your own certificates and keys.
+
You can generate this file for your own certs and keys by using the `kubectl create secret generic` command:
+
[source,sh]
--
kubectl create secret generic logstash-beats-tls --from-file=ca.crt --from-file=client.crt --from-file=client.key --from-file=server.crt --from-file=server.pkcs8.key --dry-run=client -o yaml | kubectl label -f- --dry-run=client -o yaml --local app=logstash-demo  > ../001-secret.yaml
--
+
The command generates a secrets file that looks resembles this.
+
[source,yaml]
--
apiVersion: v1
data:
  ca.crt: <BASE64 representation of ca cert, used in beats client and logstash beats input>
  client.crt: <BASE64 representation of beats client cert>
  client.key: <BASE64 representation of beats client key>
  server.crt: <BASE64 representation of server certificate, used in beats input>
  server.pkcs8.key: <BASE64 representation of PKCS8 server key, used in beats input>
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    app: logstash-demo
  name: logstash-beats-tls
--


[[qs-deployment]]
`002-deployment.yaml`::
Contains the configuration definition for {ls}.
+
[source,yaml]
--
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash-demo
  template:
    metadata:
      labels:
        app: logstash-demo
    spec:
      containers:
        - name: logstash
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
          image: {docker-image} <1>
          env:
            - name: LS_JAVA_OPTS <2>
              value: "-Xmx1g -Xms1g"
            - name: ELASTICSEARCH_PASSWORD <11>
              valueFrom:
                secretKeyRef:
                  name: demo-es-elastic-user
                  key: elastic
          resources:
            limits: <3>
              cpu: 2000m
              memory: 2Gi
            requests:
              cpu: 1000m
              memory: 2Gi
          ports: <4>
            - containerPort: 9600
              name: stats
            - containerPort: 5044
              name: beats
          livenessProbe: <5>
            httpGet:
              path: /
              port: 9600
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe: <6>
            httpGet:
              path: /
              port: 9600
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts: <7>
            - name: logstash-pipeline
              mountPath: /usr/share/logstash/pipeline
            - name: logstash-config <8>
              mountPath: /usr/share/logstash/config/logstash.yml
              subPath: logstash.yml
            - name: es-certs <9>
              mountPath: /usr/share/logstash/config/es_ca.crt
              subPath: ca.crt
            - name: logstash-beats-tls
              mountPath: /usr/share/logstash/config/ca.crt
              subPath: ca.crt
            - name: logstash-beats-tls
              mountPath: /usr/share/logstash/config/server.pkcs8.key
              subPath: server.pkcs8.key
            - name: logstash-beats-tls
              mountPath: /usr/share/logstash/config/server.crt
              subPath: server.crt
      volumes:
        - name: logstash-pipeline <7>
          configMap:
            name: logstash-pipeline
        - name: logstash-config <8>
          configMap:
            name: logstash-config
        - name: es-certs <9>
          secret:
            secretName: demo-es-http-certs-public
        - name: logstash-beats-tls <10>
          secret:
            secretName: logstash-beats-tls
        - name: es-user <11>
          secret:
            secretName: demo-es-elastic-user
--

<1> {ls} {logstash-ref}/docker.html[docker image]
<2> Set non-default JVM settings, such as memory allocation, here in the `LS_JAVA_OPTS` env variable to avoid the need to add a whole `jvm.options` file in a `ConfigMap`
<3> Resource/memory limits for the pod. Refer to Kubernetes documentation to set resources appropriately for each pod. Ensure that each pod has sufficient memory to handle the
heap specified in <2>, allowing enough memory to deal with direct memory. Check out {logstash-ref}/jvm-settings.html#heap-size[Logstash JVM settings] for details.
<4> Expose the necessary ports on the container. Here we are exposing port `5044` for the beats input, and `9600` for the metricbeat instance to query the logstash metrics API for stack monitoring purposes.
<5> Liveness probe to determine whether Logstash is running. Here we point to the Logstash Metrics API, an HTTP based API that will be ready shortly after logstash starts. Note that the endpoint shows no indication that Logstash is active, only that the API is available.
<6> Readiness probe to determine whether Logstash is running. Here we point to the {ls} Metrics API, an HTTP based API that will be ready shortly after {ls} starts. Note that the endpoint shows no indication that {ls} is active, only that the API is available.
<7> The pipeline configuration that we created in <<qs-configmap,the ConfigMap declaration>> needs a `volume` and a `volumeMount`. The `volume` refers to the created <<qs-configmap,config map>> and the `volumeMount` refers to the created `volume` and mounts in a location that logstash will read. Unless a separate `pipeline.yml` file is created by a further `ConfigMap` definition, the expected location of pipeline configurations is `/usr/share/logstash/pipelines` and the `mountPath` should be set accordingly.
<8> Name of the <<qs-configmap,Logstash configuration>> we created earlier. This file should contain key/value pairs intended to override the default values in {logstash-ref}/logstash-settings-file.html[logstash.yml], using the `flat key syntax` described in that document. To setup, this needs a `volume` and a `volumeMount`. The `volume` refers to the created <<qs-configmap,config map>> and the `volumeMount` refers to the created `volume` and mounts in a location that {ls} will read. The `mountPath` should be set to ` `/usr/share/logstash/logstash.yml`.
<9> `Volume` and `VolumeMount` definitions for certificates to use with Elasticsearch. This contains the CA certificate to output data to {es}. Refer to link:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html[TLS certificates] in the {eck} Guide for details.
<10> `Volume` and `VolumeMount` definitions for certificates to use with Beats.
<11> The {es} password is taken from `demo-es-elastic-user` and passed to the Logstash pipeline as an `ELASTICSEARCH_PASSWORD` environment variable. Refer to link:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-request-elasticsearch-endpoint.html[Access the {es} endpoint] in the {eck} Guide for details.

[[qs-service]]
`003-service.yaml`::
+
This file contains the Service definition, opening up ports on the logstash pods to the internal metricbeat (for stack monitoring) and filebeat in this instance.

[source,yaml]
--
spec:
  type: ClusterIP
  ports:
    - port: 9600 <1>
      name: "stats"
      protocol: TCP
      targetPort: 9600 <1>
    - port: 5044 <2>
      name: "beats"
      protocol: TCP
      targetPort: 5044 <2>
  selector:
    app: logstash-demo
--

<1> Opens port `9600` for {metricbeat} to connect to the {ls} metrics API.
<2> Opens port `5044` for {filebeat} to connect to the {beats} input defined in the <<qs-configmap,ConfigMap>>.

[[qs-additional-logstash-configuration]]

[[qs-autoscaler]]
`004-hpa.yml`::

+
This file sets up a horizontal pod autoscaler to scale {ls} instances up and down, depending on the load on the {ls} instance(s). See link:https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[kubernetes autoscaler docs] for more details.

[source,yaml]
--
apiVersion: autoscaling/v2 <1>
kind: HorizontalPodAutoscaler
metadata:
  name: logstash
  labels:
    app: logstash-demo
spec:
  minReplicas: 1 <2>
  maxReplicas: 2
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60 <3>
    scaleDown:
      stabilizationWindowSeconds: 180
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: logstash <4>
  metrics:
    - type: Resource <5>
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
--
<1> Requires {k8s} `1.23` and higher.
<2> Specifies the maximum and minimum number of Logstashes desired for the cluster.
<3> Specifies stabilization windows to avoid rapidly scaling nodes up and down unnecessarily.
<4> `Deployment` created <<qs-deployment, earlier>>


[[qs-stack-monitoring-files]]

`006-metricbeat.yaml`::
Enables the {metricbeat} {ls} module and sets it to collect metrics data from `logstash:9600`:
+
[source,yaml]
--
  - module: logstash <1>
    metricsets:
      - node
      - node_stats
    period: 10s
    hosts:
      - logstash:9600
    xpack.enabled: true
--
<1> Definition for logstash module, defined under `spec.config.metricbeat.modules`

[[qs-filebeat-configuration]]

`005-filebeat.yaml`::

This file includes the configuration required for a beat to communicate with {ls}.
It includes the {ls} output definition, and makes the generated certs and key files from <<qs-secrets, the secrets file>> available to the beat to enable secure communication with {ls}.
+
[source,yaml]
--
volumes: <1>
  - name: logstash-beats-tls
    secret:
      secretName: logstash-beats-tls
--
<1> Volume definition for certs/keys defined under `deployment.podTemplate.spec`.
+
[source,yaml]
--
volumeMounts: <1>
  - name: logstash-beats-tls
    mountPath: /usr/share/filebeat/ca.crt
    subPath: ca.crt
  - name: logstash-beats-tls
    mountPath: /usr/share/filebeat/client.key
    subPath: client.key
  - name: logstash-beats-tls
    mountPath: /usr/share/filebeat/client.crt
    subPath: client.crt
--
<1> Volume mount definition for certs/keys defined under `deployment.podTemplate.spec.containers`.
+
[source,yaml]
--
output.logstash: <1>
  hosts:
    - "logstash:5044"
  ssl.certificate_authorities: ["/usr/share/filebeat/ca.crt"]
  ssl.certificate: "/usr/share/filebeat/client.crt"
  ssl.key: "/usr/share/filebeat/client.key"
--
<1> Logstash output definition defined under `spec.config`.


[[qs-stack-configuration-files]]

`000-elasticsearch.yaml`::
Configures a single {es} instance to receive output data from {ls}.

`007-kibana.yaml`::
Configures a single {kib} instance to visualize the logs and metrics data.

