[8.x] [ci] Extract OAS check + add retry (#196534) (#196799)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[ci] Extract OAS check + add retry
(#196534)](https://github.com/elastic/kibana/pull/196534)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Alex
Szabo","email":"alex.szabo@elastic.co"},"sourceCommit":{"committedDate":"2024-10-17T13:03:28Z","message":"[ci]
Extract OAS check + add retry (#196534)\n\n## Summary\r\nIn the past
months, the capture OAS snapshot step has been quite flaky,\r\nbreaking
the `Checks` step, and thus breaking the `on-merge` jobs.\r\n\r\nThis PR
extracts the check to its own step and adds retries. The\r\nseparate
step is ideal because it is quite heavy compared to the other\r\nchecks
(we fire up ES + Kibana for the OAS snapshot).\r\n\r\nAlso, this PR
removes the `Checks` step altogether from
the\r\nkibana-chrome-forward-testing pipeline, as the Chrome versions
used do\r\nnot affect that aspect.\r\n\r\nThis test run includes a retry
within the Capture OAS Snapshot
step:\r\nhttps://buildkite.com/elastic/kibana-pull-request/builds/243187#01929612-dac7-4584-b440-120ea3fea7ea","sha":"2fbc843e0d6925a63e9bafdcab38ba7dffac1402","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Operations","release_note:skip","v9.0.0","backport:prev-major"],"number":196534,"url":"https://github.com/elastic/kibana/pull/196534","mergeCommit":{"message":"[ci]
Extract OAS check + add retry (#196534)\n\n## Summary\r\nIn the past
months, the capture OAS snapshot step has been quite flaky,\r\nbreaking
the `Checks` step, and thus breaking the `on-merge` jobs.\r\n\r\nThis PR
extracts the check to its own step and adds retries. The\r\nseparate
step is ideal because it is quite heavy compared to the other\r\nchecks
(we fire up ES + Kibana for the OAS snapshot).\r\n\r\nAlso, this PR
removes the `Checks` step altogether from
the\r\nkibana-chrome-forward-testing pipeline, as the Chrome versions
used do\r\nnot affect that aspect.\r\n\r\nThis test run includes a retry
within the Capture OAS Snapshot
step:\r\nhttps://buildkite.com/elastic/kibana-pull-request/builds/243187#01929612-dac7-4584-b440-120ea3fea7ea","sha":"2fbc843e0d6925a63e9bafdcab38ba7dffac1402"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196534","number":196534,"mergeCommit":{"message":"[ci]
Extract OAS check + add retry (#196534)\n\n## Summary\r\nIn the past
months, the capture OAS snapshot step has been quite flaky,\r\nbreaking
the `Checks` step, and thus breaking the `on-merge` jobs.\r\n\r\nThis PR
extracts the check to its own step and adds retries. The\r\nseparate
step is ideal because it is quite heavy compared to the other\r\nchecks
(we fire up ES + Kibana for the OAS snapshot).\r\n\r\nAlso, this PR
removes the `Checks` step altogether from
the\r\nkibana-chrome-forward-testing pipeline, as the Chrome versions
used do\r\nnot affect that aspect.\r\n\r\nThis test run includes a retry
within the Capture OAS Snapshot
step:\r\nhttps://buildkite.com/elastic/kibana-pull-request/builds/243187#01929612-dac7-4584-b440-120ea3fea7ea","sha":"2fbc843e0d6925a63e9bafdcab38ba7dffac1402"}}]}]
BACKPORT-->

.buildkite/pipelines/on_merge.yml

@@ -368,6 +368,20 @@ steps:
         - exit_status: '-1'
           limit: 3
 
+  - command: .buildkite/scripts/steps/checks/capture_oas_snapshot.sh
+    label: 'Check OAS Snapshot'
+    agents:
+      image: family/kibana-ubuntu-2004
+      imageProject: elastic-images-prod
+      provider: gcp
+      machineType: n2-standard-2
+      preemptible: true
+    timeout_in_minutes: 60
+    retry:
+      automatic:
+        - exit_status: '-1'
+          limit: 3
+
   - command: .buildkite/scripts/steps/storybooks/build_and_upload.sh
     label: 'Build Storybooks'
     agents:

.buildkite/pipelines/pull_request/base.yml

@@ -109,6 +109,17 @@ steps:
         - exit_status: '-1'
           limit: 3
 
+  - command: .buildkite/scripts/steps/checks/capture_oas_snapshot.sh
+    label: 'Check OAS Snapshot'
+    agents:
+      machineType: n2-standard-2
+      preemptible: true
+    timeout_in_minutes: 60
+    retry:
+      automatic:
+        - exit_status: '-1'
+          limit: 3
+
   - command: .buildkite/scripts/steps/api_docs/build_api_docs.sh
     label: 'Build API Docs'
     agents:

.buildkite/scripts/steps/checks.sh

@@ -10,7 +10,6 @@ if [[ "${FTR_ENABLE_FIPS_AGENT:-}" == "true" ]]; then
 fi
 .buildkite/scripts/steps/checks/saved_objects_compat_changes.sh
 .buildkite/scripts/steps/checks/saved_objects_definition_change.sh
-.buildkite/scripts/steps/capture_oas_snapshot.sh
 .buildkite/scripts/steps/code_generation/elastic_assistant_codegen.sh
 .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
 .buildkite/scripts/steps/openapi_bundling/security_solution_openapi_bundling.sh

.buildkite/scripts/steps/checks/capture_oas_snapshot.sh

@@ -2,6 +2,8 @@
 
 set -euo pipefail
 
+.buildkite/scripts/bootstrap.sh
+
 source .buildkite/scripts/common/util.sh
 
 echo --- Capture OAS snapshot
@@ -14,5 +16,10 @@ if [[ $BUILDKITE_PULL_REQUEST != "false" && "$BUILDKITE_PULL_REQUEST_BASE_BRANCH
   cmd="$cmd --no-serverless"
 fi
 
-eval "$cmd"
-check_for_changed_files "$cmd" true
+run_check() {
+  eval "$cmd"
+}
+
+retry 5 15 run_check
+
+check_for_changed_files "$cmd" true