github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-19 04:15:23 -04:00
Code Issues Projects Releases Packages Wiki Activity
logstash/docsk8s/quick-start/sample-configuration-files.asciidoc

352 lines
13 KiB
Text
Raw Blame History

[[sample-configuration-files]]
=== Sample configuration files
WARNING: This documentation is still in development and may be changed or removed in a future release.
These configuration files are used in the <<ls-k8s-quick-start,{ls} and Kubernetes quick start>>. You can use them as templates when you configure Logstash together with the rest of the Elastic Stack in a Kubernetes environment.
You can download the files together as a link:https://github.com/elastic/logstash/blob/main/docsk8s/sample-files/logstash-k8s-qs.zip[zip archive].
[[qs-setup-files]]
==== Setup files
These files are used to create certificates and keys required for secure communication between {beats} and {ls}.
They are included for illustration purposes only.
For production environments, supply your own keys and certificates as appropriate.
`cert/generate_cert.sh`::
Generates the `ca.crt`, `client.key`, `client.crt`, `server.key`, and `server.pkcs8.key` used to establish a secure connection between Filebeat and Logstash. The certificates and keys are all contained in the `001-secret.yaml` file that is generated when you run `generate_cert.sh`.
`cert/openssl.conf`::
The OpenSSL Generated Server Certificate used for TLS communication between resources.
This config file creates a secrets file `001-secret.yaml`.
We will install the secrets file as we set up the {stack}.
[[qs-logstash-configuration-files]]
==== Logstash configuration files
[[qs-configmap]]
`001-configmap.yaml`::
This file contains the Logstash settings and pipeline configuration:
+
[source,yaml]
--
---
# ConfigMap for logstash pipeline definition
data:
logstash.conf: | <1>
input {
beats {
port => "5044"
ssl_enabled => true
ssl_certificate_authorities => ["/usr/share/logstash/config/ca.crt"]
ssl_certificate => "/usr/share/logstash/config/server.crt"
ssl_key => "/usr/share/logstash/config/server.pkcs8.key"
ssl_client_authentication => "required"
}
}
output {
elasticsearch {
hosts => ["https://demo-es-http:9200"]
index => "kube-apiserver-%{+YYYY.MM.dd}"
cacert => "/usr/share/logstash/config/es_ca.crt"
user => 'elastic'
password => '${ELASTICSEARCH_PASSWORD}'
}
}
---
# ConfigMap for logstash.yml definition
data:
logstash.yml: | <2>
api.http.host: "0.0.0.0"
--
<1> Definition of {ls} configuration file.
We will refer to this definition later in the deployment file, where we will define volumes.
<2> Definition of {logstash-ref}/logstash-settings-file.html[logstash.yml] file
Define each key/value pair to override defaults. We will refer to this definition later in the deployment file.
[[qs-secrets]]
`001-secrets.yaml`::
This secrets file includes certificates and key files required for secure communication between {ls} and the rest of the {stack}. This example was generated by the supplied script, but for your own configuration it should contain the base64 encoded representations of your own certificates and keys.
+
You can generate this file for your own certs and keys by using the `kubectl create secret generic` command:
+
[source,sh]
--
kubectl create secret generic logstash-beats-tls --from-file=ca.crt --from-file=client.crt --from-file=client.key --from-file=server.crt --from-file=server.pkcs8.key --dry-run=client -o yaml | kubectl label -f- --dry-run=client -o yaml --local app=logstash-demo > ../001-secret.yaml
--
+
The command generates a secrets file that looks resembles this.
+
[source,yaml]
--
apiVersion: v1
data:
ca.crt: <BASE64 representation of ca cert, used in beats client and logstash beats input>
client.crt: <BASE64 representation of beats client cert>
client.key: <BASE64 representation of beats client key>
server.crt: <BASE64 representation of server certificate, used in beats input>
server.pkcs8.key: <BASE64 representation of PKCS8 server key, used in beats input>
kind: Secret
metadata:
creationTimestamp: null
labels:
app: logstash-demo
name: logstash-beats-tls
--
[[qs-deployment]]
`002-deployment.yaml`::
Contains the configuration definition for {ls}.
+
[source,yaml]
--
spec:
replicas: 1
selector:
matchLabels:
app: logstash-demo
template:
metadata:
labels:
app: logstash-demo
spec:
containers:
- name: logstash
securityContext:
runAsNonRoot: true
runAsUser: 1000
image: {docker-image} <1>
env:
- name: LS_JAVA_OPTS <2>
value: "-Xmx1g -Xms1g"
- name: ELASTICSEARCH_PASSWORD <11>
valueFrom:
secretKeyRef:
name: demo-es-elastic-user
key: elastic
resources:
limits: <3>
cpu: 2000m
memory: 2Gi
requests:
cpu: 1000m
memory: 2Gi
ports: <4>
- containerPort: 9600
name: stats
- containerPort: 5044
name: beats
livenessProbe: <5>
httpGet:
path: /
port: 9600
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe: <6>
httpGet:
path: /
port: 9600
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts: <7>
- name: logstash-pipeline
mountPath: /usr/share/logstash/pipeline
- name: logstash-config <8>
mountPath: /usr/share/logstash/config/logstash.yml
subPath: logstash.yml
- name: es-certs <9>
mountPath: /usr/share/logstash/config/es_ca.crt
subPath: ca.crt
- name: logstash-beats-tls
mountPath: /usr/share/logstash/config/ca.crt
subPath: ca.crt
- name: logstash-beats-tls
mountPath: /usr/share/logstash/config/server.pkcs8.key
subPath: server.pkcs8.key
- name: logstash-beats-tls
mountPath: /usr/share/logstash/config/server.crt
subPath: server.crt
volumes:
- name: logstash-pipeline <7>
configMap:
name: logstash-pipeline
- name: logstash-config <8>
configMap:
name: logstash-config
- name: es-certs <9>
secret:
secretName: demo-es-http-certs-public
- name: logstash-beats-tls <10>
secret:
secretName: logstash-beats-tls
- name: es-user <11>
secret:
secretName: demo-es-elastic-user
--
<1> {ls} {logstash-ref}/docker.html[docker image]
<2> Set non-default JVM settings, such as memory allocation, here in the `LS_JAVA_OPTS` env variable to avoid the need to add a whole `jvm.options` file in a `ConfigMap`
<3> Resource/memory limits for the pod. Refer to Kubernetes documentation to set resources appropriately for each pod. Ensure that each pod has sufficient memory to handle the
heap specified in <2>, allowing enough memory to deal with direct memory. Check out {logstash-ref}/jvm-settings.html#heap-size[Logstash JVM settings] for details.
<4> Expose the necessary ports on the container. Here we are exposing port `5044` for the beats input, and `9600` for the metricbeat instance to query the logstash metrics API for stack monitoring purposes.
<5> Liveness probe to determine whether Logstash is running. Here we point to the Logstash Metrics API, an HTTP based API that will be ready shortly after logstash starts. Note that the endpoint shows no indication that Logstash is active, only that the API is available.
<6> Readiness probe to determine whether Logstash is running. Here we point to the {ls} Metrics API, an HTTP based API that will be ready shortly after {ls} starts. Note that the endpoint shows no indication that {ls} is active, only that the API is available.
<7> The pipeline configuration that we created in <<qs-configmap,the ConfigMap declaration>> needs a `volume` and a `volumeMount`. The `volume` refers to the created <<qs-configmap,config map>> and the `volumeMount` refers to the created `volume` and mounts in a location that logstash will read. Unless a separate `pipeline.yml` file is created by a further `ConfigMap` definition, the expected location of pipeline configurations is `/usr/share/logstash/pipelines` and the `mountPath` should be set accordingly.
<8> Name of the <<qs-configmap,Logstash configuration>> we created earlier. This file should contain key/value pairs intended to override the default values in {logstash-ref}/logstash-settings-file.html[logstash.yml], using the `flat key syntax` described in that document. To setup, this needs a `volume` and a `volumeMount`. The `volume` refers to the created <<qs-configmap,config map>> and the `volumeMount` refers to the created `volume` and mounts in a location that {ls} will read. The `mountPath` should be set to ` `/usr/share/logstash/logstash.yml`.
<9> `Volume` and `VolumeMount` definitions for certificates to use with Elasticsearch. This contains the CA certificate to output data to {es}. Refer to link:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html[TLS certificates] in the {eck} Guide for details.
<10> `Volume` and `VolumeMount` definitions for certificates to use with Beats.
<11> The {es} password is taken from `demo-es-elastic-user` and passed to the Logstash pipeline as an `ELASTICSEARCH_PASSWORD` environment variable. Refer to link:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-request-elasticsearch-endpoint.html[Access the {es} endpoint] in the {eck} Guide for details.
[[qs-service]]
`003-service.yaml`::
+
This file contains the Service definition, opening up ports on the logstash pods to the internal metricbeat (for stack monitoring) and filebeat in this instance.
[source,yaml]
--
spec:
type: ClusterIP
ports:
- port: 9600 <1>
name: "stats"
protocol: TCP
targetPort: 9600 <1>
- port: 5044 <2>
name: "beats"
protocol: TCP
targetPort: 5044 <2>
selector:
app: logstash-demo
--
<1> Opens port `9600` for {metricbeat} to connect to the {ls} metrics API.
<2> Opens port `5044` for {filebeat} to connect to the {beats} input defined in the <<qs-configmap,ConfigMap>>.
[[qs-additional-logstash-configuration]]
[[qs-autoscaler]]
`004-hpa.yml`::
+
This file sets up a horizontal pod autoscaler to scale {ls} instances up and down, depending on the load on the {ls} instance(s). See link:https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[kubernetes autoscaler docs] for more details.
[source,yaml]
--
apiVersion: autoscaling/v2 <1>
kind: HorizontalPodAutoscaler
metadata:
name: logstash
labels:
app: logstash-demo
spec:
minReplicas: 1 <2>
maxReplicas: 2
behavior:
scaleUp:
stabilizationWindowSeconds: 60 <3>
scaleDown:
stabilizationWindowSeconds: 180
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: logstash <4>
metrics:
- type: Resource <5>
resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
--
<1> Requires {k8s} `1.23` and higher.
<2> Specifies the maximum and minimum number of Logstashes desired for the cluster.
<3> Specifies stabilization windows to avoid rapidly scaling nodes up and down unnecessarily.
<4> `Deployment` created <<qs-deployment, earlier>>
[[qs-stack-monitoring-files]]
`006-metricbeat.yaml`::
Enables the {metricbeat} {ls} module and sets it to collect metrics data from `logstash:9600`:
+
[source,yaml]
--
- module: logstash <1>
metricsets:
- node
- node_stats
period: 10s
hosts:
- logstash:9600
xpack.enabled: true
--
<1> Definition for logstash module, defined under `spec.config.metricbeat.modules`
[[qs-filebeat-configuration]]
`005-filebeat.yaml`::
This file includes the configuration required for a beat to communicate with {ls}.
It includes the {ls} output definition, and makes the generated certs and key files from <<qs-secrets, the secrets file>> available to the beat to enable secure communication with {ls}.
+
[source,yaml]
--
volumes: <1>
- name: logstash-beats-tls
secret:
secretName: logstash-beats-tls
--
<1> Volume definition for certs/keys defined under `deployment.podTemplate.spec`.
+
[source,yaml]
--
volumeMounts: <1>
- name: logstash-beats-tls
mountPath: /usr/share/filebeat/ca.crt
subPath: ca.crt
- name: logstash-beats-tls
mountPath: /usr/share/filebeat/client.key
subPath: client.key
- name: logstash-beats-tls
mountPath: /usr/share/filebeat/client.crt
subPath: client.crt
--
<1> Volume mount definition for certs/keys defined under `deployment.podTemplate.spec.containers`.
+
[source,yaml]
--
output.logstash: <1>
hosts:
- "logstash:5044"
ssl.certificate_authorities: ["/usr/share/filebeat/ca.crt"]
ssl.certificate: "/usr/share/filebeat/client.crt"
ssl.key: "/usr/share/filebeat/client.key"
--
<1> Logstash output definition defined under `spec.config`.
[[qs-stack-configuration-files]]
`000-elasticsearch.yaml`::
Configures a single {es} instance to receive output data from {ls}.
`007-kibana.yaml`::
Configures a single {kib} instance to visualize the logs and metrics data.
Reference in a new issue View git blame Copy permalink