github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 01:38:56 -04:00
Code Issues Projects Releases Packages Wiki Activity

[ci] retry scalability testing step on failure (#149399)

Browse source 
## Summary

Quite often bootstrap
[fails](https://buildkite.com/elastic/kibana-scalability-benchmarking/builds/957#0185e31d-dcc8-42f4-ac3a-e0399a175519)
on `kb-static-scalability-01` bare metal machine:

<details>
<summary>Bootstrap failure: [bazel] exited with code [1]</summary>

```
$ node scripts/kbn bootstrap
  | 2023-01-24 11:00:50 CEST | HINT
  | 2023-01-24 11:03:31 CEST | HINT If experiencing problems with node_modules try `yarn kbn bootstrap --force-install` or as last resort `yarn kbn reset && yarn kbn bootstrap`
  | 2023-01-24 11:03:31 CEST | HINT
  | 2023-01-24 11:03:31 CEST | ERROR  [bootstrap] failed: [bazel] exited with code [1]
  | 2023-01-24 11:03:31 CEST | output:
  | 2023-01-24 11:03:31 CEST | INFO: Invocation ID: 207c041d-23b8-4f07-bf56-8cb961819546
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | <--- Last few GCs --->
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | [1304562:0x74b83a0]   154462 ms: Mark-sweep 4023.8 (4130.2) -> 4011.3 (4131.0) MB, 1677.3 / 0.0 ms  (average mu = 0.149, current mu = 0.074) task; scavenge might not succeed
  | 2023-01-24 11:03:31 CEST | [1304562:0x74b83a0]   155522 ms: Mark-sweep 4024.7 (4131.0) -> 4012.0 (4131.7) MB, 924.0 / 0.0 ms  (average mu = 0.142, current mu = 0.128) task; scavenge might not succeed
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | <--- JS stacktrace --->
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
  | 2023-01-24 11:03:31 CEST | 1: 0xb6b850 node::Abort() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 2: 0xa806a6  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 3: 0xd52140 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 4: 0xd524e7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 5: 0xf2fbe5  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 6: 0xf30ae8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 7: 0xf40ff3  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 8: 0xf41e68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 9: 0xfb1124 v8::internal::ScavengeJob::Task::RunInternal() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 10: 0xe2187b non-virtual thunk to v8::internal::CancelableTask::Run() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 11: 0xbd6684  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 12: 0xbd9aee node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 13: 0x1652906  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 14: 0x1664e44  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 15: 0x165326e uv_run [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 16: 0xaafa2d node::SpinEventLoop(node::Environment*) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 17: 0xbb11f4 node::NodeMainInstance::Run() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 18: 0xb26c44 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 19: 0xb2a83f node::Start(int, char**) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 20: 0x7fbc908be083 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
  | 2023-01-24 11:03:31 CEST | 21: 0xaad7ee _start [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/yarn: line 19: 1304562 Aborted                 "$SCRIPT_DIR/node" "$SCRIPT_DIR/yarnpkg/bin/yarn.js" "$@"
  | 2023-01-24 11:03:31 CEST | INFO: Repository npm instantiated at:
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/builds/kb-static-scalability-01/elastic/kibana-scalability-benchmarking/kibana/WORKSPACE.bazel:47:13: in <toplevel>
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/index.bzl:83:18: in yarn_install
  | 2023-01-24 11:03:31 CEST | Repository rule yarn_install defined at:
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:956:31: in <toplevel>
  | 2023-01-24 11:03:31 CEST | ERROR: An error occurred during the fetch of repository 'npm':
  | 2023-01-24 11:03:31 CEST | Traceback (most recent call last):
  | 2023-01-24 11:03:31 CEST | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 949, column 13, in _yarn_install_impl
  | 2023-01-24 11:03:31 CEST | fail("yarn_install failed: %s (%s)" % (result.stdout, result.stderr))
  | 2023-01-24 11:03:31 CEST | Error in fail: yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | ERROR: /var/lib/buildkite-agent/builds/kb-static-scalability-01/elastic/kibana-scalability-benchmarking/kibana/WORKSPACE.bazel:47:13: fetching yarn_install rule //external:npm: Traceback (most recent call last):
  | 2023-01-24 11:03:31 CEST | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 949, column 13, in _yarn_install_impl
  | 2023-01-24 11:03:31 CEST | fail("yarn_install failed: %s (%s)" % (result.stdout, result.stderr))
  | 2023-01-24 11:03:31 CEST | Error in fail: yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | ERROR: Skipping '//packages/kbn-ui-shared-deps-src:shared_built_assets': no such package '@npm//webpack-cli': yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | ERROR: no such package '@npm//webpack-cli': yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | INFO: Elapsed time: 160.951s
  | 2023-01-24 11:03:31 CEST | INFO: 0 processes.}
  | 2023-01-24 11:03:31 CEST | error Command failed with exit code 1.
```
</details>

Re-try usually helps and should minimise the failures.

I think the worker is missing some configuration since
`.buildkite/scripts/bootstrap.sh` fails on it, but works totally fine on
`kb-static-ubuntu-01`.
This commit is contained in:
Dzmitry Lemechko 2023-01-26 16:02:23 +01:00 • committed by GitHub
parent 9e3cf8f0ec
commit 50393be0a8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.buildkite/pipelines/scalability/daily.yml
View file

@ -12,6 +12,10 @@ steps:
agents:
queue: kb-static-scalability
timeout_in_minutes: 90
retry:
automatic:
- exit_status: '*'
limit: 1
- wait: ~
continue_on_failure: true