github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-23 22:27:21 -04:00
Code Issues Projects Releases Packages Wiki Activity
logstash/.buildkite/scripts/benchmark
History
Exact
kaisecheng 4fd13730ee
[Backport 8.x] benchmark script (#17283) 
This commit cherry-picked the missing becnhmark script PRs
The deprecated artifacts-api is removed

[CI] benchmark uses the new artifacts-api (#17224)
[CI] benchmark readme (#16783)
Introduce a new flag to explicitly permit legacy monitoring (#16586) (Only take the benchmark script)
[ci] fix wrong queue type in benchmark marathon (#16465)
[CI] fix benchmark marathon (#16447)
[CI] benchmark dashboard and pipeline for testing against multiple versions (#16421)
2025-03-07 00:16:29 +00:00
..
config [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
save-objects [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
setup [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
core.sh [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
main.sh [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
marathon.sh [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
README.md [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00
util.sh [Backport 8.x] benchmark script (#17283) 2025-03-07 00:16:29 +00:00

README.md

Steps to set up GCP instance to run benchmark script

  • Create an instance "n2-standard-16" with Ubuntu image
  • Install docker
    • sudo snap install docker
    • sudo usermod -a -G docker $USER
  • Install jq
  • Install vault
    • sudo snap install vault
    • vault login --method github
    • vault kv get -format json secret/ci/elastic-logstash/benchmark
  • Setup Elasticsearch index mapping and alias with setup/*
  • Import Kibana dashboard with save-objects/*
  • Run the benchmark script
    • Send data to your own Elasticsearch. Customise VAULT_PATH="secret/ci/elastic-logstash/your/path"
    • Run the script main.sh
      • or run in background nohup bash -x main.sh > log.log 2>&1 &

Notes

  • Benchmarks should only be compared using the same hardware setup.
  • Please do not send the test metrics to the benchmark cluster. You can set VAULT_PATH to send data and metrics to your own server.
  • Run all.sh as calibration which gives you a baseline of performance in different versions.
  • #16586 allows legacy monitoring using the configuration xpack.monitoring.allow_legacy_collection: true, which is not recognized in version 8. To run benchmarks in version 8, use the script of the corresponding branch (e.g. 8.16) instead of main in buildkite.