mirror of
https://github.com/elastic/kibana.git
synced 2025-06-27 18:51:07 -04:00
833c075362
## 📓 Summary
When retrieving the CPU stats for containerized (or non-container)
clusters, we were not considering a scenario where the user could run in
a cgroup but without limits set.
These changes re-write the conditions to determine whether we allow
treating limitless containers as non-containerized, covering the case
where a user run in a cgroup and for some reason hasn't set the limit.
## Testing
> Taken from https://github.com/elastic/kibana/pull/159351 since it
reproduced the same behaviours
There are 3 main states to test:
No limit set but Kibana configured to use container stats.
Limit changed during lookback period (to/from real value, to/from no
limit).
Limit set and CPU usage crossing threshold and then falling down to
recovery
**Note: Please also test the non-container use case for this rule to
ensure that didn't get broken during this refactor**
**1. Start Elasticsearch in a container without setting the CPU
limits:**
```
docker network create elastic
docker run --name es01 --net elastic -p 9201:9200 -e xpack.license.self_generated.type=trial -it docker.elastic.co/elasticsearch/elasticsearch:master-SNAPSHOT
```
(We're using `master-SNAPSHOT` to include a recent fix to reporting for
cgroup v2)
Make note of the generated password for the `elastic` user.
**2. Start another Elasticsearch instance to act as the monitoring
cluster**
**3. Configure Kibana to connect to the monitoring cluster and start
it**
**4. Configure Metricbeat to collect metrics from the Docker cluster and
ship them to the monitoring cluster, then start it**
Execute the below command next to the Metricbeat binary to grab the CA
certificate from the Elasticsearch cluster.
```
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .
```
Use the `elastic` password and the CA certificate to configure the
`elasticsearch` module:
```
- module: elasticsearch
xpack.enabled: true
period: 10s
hosts:
- "https://localhost:9201"
username: "elastic"
password: "PASSWORD"
ssl.certificate_authorities: "PATH_TO_CERT/http_ca.crt"
```
**5. Configure an alert in Kibana with a chosen threshold**
OBSERVE: Alert gets fired to inform you that there looks to be a
misconfiguration, together with reporting the current value for the
fallback metric (warning if the fallback metric is below threshold,
danger is if is above).
**6. Set limit**
First stop ES using `docker stop es01`, then set the limit using `docker
update --cpus=1 es01` and start it again using `docker start es01`.
After a brief delay you should now see the alert change to a warning
about the limits having changed during the alert lookback period and
stating that the CPU usage could not be confidently calculated.
Wait for change event to pass out of lookback window.
**7. Generate load on the monitored cluster**
[Slingshot](https://github.com/elastic/slingshot) is an option. After
you clone it, you need to update the `package.json` to match [this
change](
|
||
|---|---|---|
| .. | ||
| common | ||
| dev_docs | ||
| public | ||
| server | ||
| jest.config.js | ||
| kibana.jsonc | ||
| readme.md | ||
| tsconfig.json | ||
Documentation for Stack Monitoring developers
This plugin provides the Stack Monitoring kibana application.