[8.14] [BK] Migrate batch 7 (Performance) (#181133) (#182085)

# Backport

This will backport the following commits from `main` to `8.14`:
- [[BK] Migrate batch 7 (Performance)
(#181133)](https://github.com/elastic/kibana/pull/181133)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Alex
Szabo","email":"alex.szabo@elastic.co"},"sourceCommit":{"committedDate":"2024-04-30T09:20:45Z","message":"[BK]
Migrate batch 7 (Performance) (#181133)\n\n##
Summary\r\n\r\nValidation:\r\n - [x] RREs checked locally\r\n - [x]
Pipelines staged\r\n- [x] kibana /
single-user-performance\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/62\r\n
)\r\n- [x] kibana /
performance-data-set-extraction\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/60#018f0b96-7703-4e36-9924-f405073d0747\r\n
)\r\n- [x] kibana /
scalability-benchmarking\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/68\r\n
)\r\n- [x] kibana /
apis-capacity-testing\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/60#018f0b96-7703-4e36-9924-f405073d0747\r\n
)\r\n - [x] 7.17 / 8.14 validation (  not needed, no branch builds set
up)\r\n\r\n\r\nPart of:
https://github.com/elastic/kibana-operations/issues/79\r\n\r\nMigrates:
\r\n- kibana /
single-user-performance\r\n[kibana-performance-daily.yml](https://buildkite.com/elastic/kibana-single-user-performance)\r\n-
kibana /
performance-data-set-extraction\r\n[kibana-performance-data-set-extraction-daily.yml](https://buildkite.com/elastic/kibana-performance-data-set-extraction)\r\n-
kibana /
scalability-benchmarking\r\n[scalability_testing-daily.yml](https://buildkite.com/elastic/kibana-scalability-benchmarking-1)\r\n-
kibana /
apis-capacity-testing\r\n[kibana-apis-capacity-testing-daily.yml](https://buildkite.com/elastic/kibana-apis-capacity-testing)\r\n\r\nchore(BK):
Migrate batch 7 - performance and testing\r\n\r\nDepends on:
https://elasticco.atlassian.net/browse/ENGPRD-524","sha":"77961a68e05398731224f79c4189c783bb2ea142","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","backport:all-open","v8.15.0"],"title":"[BK]
Migrate batch 7
(Performance)","number":181133,"url":"https://github.com/elastic/kibana/pull/181133","mergeCommit":{"message":"[BK]
Migrate batch 7 (Performance) (#181133)\n\n##
Summary\r\n\r\nValidation:\r\n - [x] RREs checked locally\r\n - [x]
Pipelines staged\r\n- [x] kibana /
single-user-performance\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/62\r\n
)\r\n- [x] kibana /
performance-data-set-extraction\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/60#018f0b96-7703-4e36-9924-f405073d0747\r\n
)\r\n- [x] kibana /
scalability-benchmarking\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/68\r\n
)\r\n- [x] kibana /
apis-capacity-testing\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/60#018f0b96-7703-4e36-9924-f405073d0747\r\n
)\r\n - [x] 7.17 / 8.14 validation (  not needed, no branch builds set
up)\r\n\r\n\r\nPart of:
https://github.com/elastic/kibana-operations/issues/79\r\n\r\nMigrates:
\r\n- kibana /
single-user-performance\r\n[kibana-performance-daily.yml](https://buildkite.com/elastic/kibana-single-user-performance)\r\n-
kibana /
performance-data-set-extraction\r\n[kibana-performance-data-set-extraction-daily.yml](https://buildkite.com/elastic/kibana-performance-data-set-extraction)\r\n-
kibana /
scalability-benchmarking\r\n[scalability_testing-daily.yml](https://buildkite.com/elastic/kibana-scalability-benchmarking-1)\r\n-
kibana /
apis-capacity-testing\r\n[kibana-apis-capacity-testing-daily.yml](https://buildkite.com/elastic/kibana-apis-capacity-testing)\r\n\r\nchore(BK):
Migrate batch 7 - performance and testing\r\n\r\nDepends on:
https://elasticco.atlassian.net/browse/ENGPRD-524","sha":"77961a68e05398731224f79c4189c783bb2ea142"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181133","number":181133,"mergeCommit":{"message":"[BK]
Migrate batch 7 (Performance) (#181133)\n\n##
Summary\r\n\r\nValidation:\r\n - [x] RREs checked locally\r\n - [x]
Pipelines staged\r\n- [x] kibana /
single-user-performance\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/62\r\n
)\r\n- [x] kibana /
performance-data-set-extraction\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/60#018f0b96-7703-4e36-9924-f405073d0747\r\n
)\r\n- [x] kibana /
scalability-benchmarking\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/68\r\n
)\r\n- [x] kibana /
apis-capacity-testing\r\n(https://buildkite.com/elastic/kibana-migration-pipeline-staging/builds/60#018f0b96-7703-4e36-9924-f405073d0747\r\n
)\r\n - [x] 7.17 / 8.14 validation (  not needed, no branch builds set
up)\r\n\r\n\r\nPart of:
https://github.com/elastic/kibana-operations/issues/79\r\n\r\nMigrates:
\r\n- kibana /
single-user-performance\r\n[kibana-performance-daily.yml](https://buildkite.com/elastic/kibana-single-user-performance)\r\n-
kibana /
performance-data-set-extraction\r\n[kibana-performance-data-set-extraction-daily.yml](https://buildkite.com/elastic/kibana-performance-data-set-extraction)\r\n-
kibana /
scalability-benchmarking\r\n[scalability_testing-daily.yml](https://buildkite.com/elastic/kibana-scalability-benchmarking-1)\r\n-
kibana /
apis-capacity-testing\r\n[kibana-apis-capacity-testing-daily.yml](https://buildkite.com/elastic/kibana-apis-capacity-testing)\r\n\r\nchore(BK):
Migrate batch 7 - performance and testing\r\n\r\nDepends on:
https://elasticco.atlassian.net/browse/ENGPRD-524","sha":"77961a68e05398731224f79c4189c783bb2ea142"}}]}]
BACKPORT-->

Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
This commit is contained in:
Kibana Machine 2024-04-30 06:45:38 -04:00 committed by GitHub
parent e84fcf1aad
commit cd395daf25
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 258 additions and 14 deletions

View file

@ -0,0 +1,48 @@
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: bk-kibana-apis-capacity-testing
description: Runs capacity tests for Kibana apis
links:
- url: 'https://buildkite.com/elastic/kibana-apis-capacity-testing'
title: Pipeline link
spec:
type: buildkite-pipeline
owner: 'group:kibana-operations'
system: buildkite
implementation:
apiVersion: buildkite.elastic.dev/v1
kind: Pipeline
metadata:
name: kibana / apis-capacity-testing
description: Runs capacity tests for Kibana apis
spec:
env:
SLACK_NOTIFICATIONS_CHANNEL: '#kibana-performance-alerts'
BAZEL_CACHE_MODE: none
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
allow_rebuilds: true
branch_configuration: main
default_branch: main
repository: elastic/kibana
pipeline_file: .buildkite/pipelines/scalability/api_capacity_testing_daily.yml
skip_intermediate_builds: false
provider_settings:
build_branches: true
prefix_pull_request_fork_branch_names: false
skip_pull_request_builds_for_existing_commits: false
teams:
everyone:
access_level: BUILD_AND_READ
kibana-operations:
access_level: MANAGE_BUILD_AND_READ
appex-qa:
access_level: MANAGE_BUILD_AND_READ
kibana-tech-leads:
access_level: MANAGE_BUILD_AND_READ
schedules:
Capacity every 3h testing:
cronline: 0 1/3 * * * Europe/Berlin
message: Capacity every 3h testing
branch: main

View file

@ -0,0 +1,48 @@
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: bk-kibana-single-user-performance
description: Runs single user performance tests for kibana
links:
- url: 'https://buildkite.com/elastic/kibana-single-user-performance'
title: Pipeline link
spec:
type: buildkite-pipeline
owner: 'group:kibana-operations'
system: buildkite
implementation:
apiVersion: buildkite.elastic.dev/v1
kind: Pipeline
metadata:
name: kibana / single-user-performance
description: Runs single user performance tests for kibana
spec:
env:
SLACK_NOTIFICATIONS_CHANNEL: '#kibana-performance-alerts'
BAZEL_CACHE_MODE: none
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
allow_rebuilds: true
branch_configuration: main
default_branch: main
repository: elastic/kibana
pipeline_file: .buildkite/pipelines/performance/daily.yml
skip_intermediate_builds: false
provider_settings:
build_branches: true
prefix_pull_request_fork_branch_names: false
skip_pull_request_builds_for_existing_commits: false
teams:
everyone:
access_level: BUILD_AND_READ
kibana-operations:
access_level: MANAGE_BUILD_AND_READ
appex-qa:
access_level: MANAGE_BUILD_AND_READ
kibana-tech-leads:
access_level: MANAGE_BUILD_AND_READ
schedules:
Single user daily test:
cronline: 0 */3 * * * Europe/Berlin
message: Single user daily test
branch: main

View file

@ -0,0 +1,48 @@
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: bk-kibana-performance-data-set-extraction
description: Runs single user performance tests and extract APM traces
links:
- url: 'https://buildkite.com/elastic/kibana-performance-data-set-extraction'
title: Pipeline link
spec:
type: buildkite-pipeline
owner: 'group:kibana-operations'
system: buildkite
implementation:
apiVersion: buildkite.elastic.dev/v1
kind: Pipeline
metadata:
name: kibana / performance-data-set-extraction
description: Runs single user performance tests and extract APM traces
spec:
env:
SLACK_NOTIFICATIONS_CHANNEL: '#kibana-performance-alerts'
BAZEL_CACHE_MODE: none
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
allow_rebuilds: true
branch_configuration: main
default_branch: main
repository: elastic/kibana
pipeline_file: .buildkite/pipelines/performance/data_set_extraction_daily.yml
skip_intermediate_builds: false
provider_settings:
build_branches: true
prefix_pull_request_fork_branch_names: false
skip_pull_request_builds_for_existing_commits: false
teams:
everyone:
access_level: BUILD_AND_READ
kibana-operations:
access_level: MANAGE_BUILD_AND_READ
appex-qa:
access_level: MANAGE_BUILD_AND_READ
kibana-tech-leads:
access_level: MANAGE_BUILD_AND_READ
schedules:
Extract APM traces:
cronline: 0 3/8 * * * Europe/Berlin
message: Extract APM traces
branch: main

View file

@ -8,6 +8,7 @@ spec:
type: url
targets:
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-api-docs.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-apis-capacity-testing-daily.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-coverage-daily.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-es-serverless-snapshots.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-es-snapshots.yml
@ -16,9 +17,12 @@ spec:
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-fleet-packages-daily.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-migration-staging.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-on-merge-unsupported-ftrs.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-performance-daily.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-performance-data-set-extraction-daily.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-purge-cloud-deployments.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-serverless-release.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/kibana-serverless-security-solution-quality-gate-api-integration.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/scalability_testing-daily.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/security-solution-ess/security-solution-ess.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/security-solution-quality-gate/kibana-serverless-security-solution-quality-gate-defend-workflows.yml
- https://github.com/elastic/kibana/blob/main/.buildkite/pipeline-resource-definitions/security-solution-quality-gate/kibana-serverless-security-solution-quality-gate-detection-engine.yml

View file

@ -0,0 +1,48 @@
# yaml-language-server: $schema=https://gist.githubusercontent.com/elasticmachine/988b80dae436cafea07d9a4a460a011d/raw/rre.schema.json
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
name: bk-kibana-scalability-benchmarking
description: Runs scalability tests for Kibana server
links:
- url: 'https://buildkite.com/elastic/kibana-scalability-benchmarking'
title: Pipeline link
spec:
type: buildkite-pipeline
owner: 'group:kibana-operations'
system: buildkite
implementation:
apiVersion: buildkite.elastic.dev/v1
kind: Pipeline
metadata:
name: kibana / scalability-benchmarking
description: Runs scalability tests for Kibana server
spec:
env:
SLACK_NOTIFICATIONS_CHANNEL: '#kibana-performance-alerts'
BAZEL_CACHE_MODE: none
ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
allow_rebuilds: true
branch_configuration: main
default_branch: main
repository: elastic/kibana
pipeline_file: .buildkite/pipelines/scalability/daily.yml
skip_intermediate_builds: false
provider_settings:
build_branches: true
prefix_pull_request_fork_branch_names: false
skip_pull_request_builds_for_existing_commits: false
teams:
everyone:
access_level: BUILD_AND_READ
kibana-operations:
access_level: MANAGE_BUILD_AND_READ
appex-qa:
access_level: MANAGE_BUILD_AND_READ
kibana-tech-leads:
access_level: MANAGE_BUILD_AND_READ
schedules:
Scalability daily benchmarking:
cronline: 0 6 * * * Europe/Berlin
message: Scalability daily benchmarking
branch: main

View file

@ -2,14 +2,20 @@ steps:
- label: '👨‍🔧 Pre-Build'
command: .buildkite/scripts/lifecycle/pre_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2
- wait
- label: '🧑‍🏭 Build Kibana Distribution and Plugins'
command: .buildkite/scripts/steps/build_kibana.sh
agents:
queue: c2-16
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: c2-standard-16
key: build
if: "build.env('KIBANA_BUILD_ID') == null || build.env('KIBANA_BUILD_ID') == ''"
@ -26,7 +32,12 @@ steps:
- label: '📈 Report performance metrics to ci-stats'
command: .buildkite/scripts/steps/functional/report_performance_metrics.sh
agents:
queue: n2-2
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
localSsds: 1
localSsdInterface: nvme
machineType: n2-standard-2
depends_on: tests
- wait: ~
@ -35,4 +46,7 @@ steps:
- label: '🦸 Post-Build'
command: .buildkite/scripts/lifecycle/post_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2

View file

@ -2,7 +2,10 @@ steps:
- label: ':male-mechanic::skin-tone-2: Pre-Build'
command: .buildkite/scripts/lifecycle/pre_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2
timeout_in_minutes: 10
- wait
@ -10,14 +13,21 @@ steps:
- label: ':building_construction: Build Kibana Distribution and Plugins'
command: .buildkite/scripts/steps/build_kibana.sh
agents:
queue: c2-16
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: c2-standard-16
key: build
if: "build.env('KIBANA_BUILD_ID') == null || build.env('KIBANA_BUILD_ID') == ''"
- label: ':kibana: Performance Tests with Playwright config'
command: .buildkite/scripts/steps/functional/performance_playwright.sh
agents:
queue: n2-2-spot
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2
preemptible: true
depends_on: build
key: tests
timeout_in_minutes: 90
@ -31,7 +41,12 @@ steps:
- label: ':ship: Single user journeys dataset extraction for scalability benchmarking'
command: .buildkite/scripts/steps/functional/scalability_dataset_extraction.sh
agents:
queue: n2-2
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
localSsds: 1
localSsdInterface: nvme
machineType: n2-standard-2
depends_on: tests
- wait: ~
@ -40,4 +55,7 @@ steps:
- label: ':male_superhero::skin-tone-2: Post-Build'
command: .buildkite/scripts/lifecycle/post_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2

View file

@ -2,14 +2,21 @@ steps:
- label: 'Pre-Build'
command: .buildkite/scripts/lifecycle/pre_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2
- wait
- label: 'Build Kibana Distribution and Plugins'
command: .buildkite/scripts/steps/build_kibana.sh
agents:
queue: n2-16-spot
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-16
preemptible: true
key: build
if: "build.env('KIBANA_BUILD_ID') == null || build.env('KIBANA_BUILD_ID') == ''"
timeout_in_minutes: 60
@ -35,4 +42,7 @@ steps:
- label: 'Post-Build'
command: .buildkite/scripts/lifecycle/post_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2

View file

@ -2,7 +2,10 @@ steps:
- label: ':male-mechanic::skin-tone-2: Pre-Build'
command: .buildkite/scripts/lifecycle/pre_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2
timeout_in_minutes: 10
- wait
@ -23,5 +26,8 @@ steps:
- label: ':male_superhero::skin-tone-2: Post-Build'
command: .buildkite/scripts/lifecycle/post_build.sh
agents:
queue: kibana-default
image: family/kibana-ubuntu-2004
imageProject: elastic-images-qa
provider: gcp
machineType: n2-standard-2
timeout_in_minutes: 10