github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-22 08:49:27 -04:00
Code Issues Projects Releases Packages Wiki Activity
kibana/.buildkite/pipelines
History
Exact
Dzmitry Lemechko 50393be0a8
[ci] retry scalability testing step on failure (#149399) 
## Summary

Quite often bootstrap
[fails](https://buildkite.com/elastic/kibana-scalability-benchmarking/builds/957#0185e31d-dcc8-42f4-ac3a-e0399a175519)
on `kb-static-scalability-01` bare metal machine:

<details>
<summary>Bootstrap failure: [bazel] exited with code [1]</summary>

```
$ node scripts/kbn bootstrap
  | 2023-01-24 11:00:50 CEST | HINT
  | 2023-01-24 11:03:31 CEST | HINT If experiencing problems with node_modules try `yarn kbn bootstrap --force-install` or as last resort `yarn kbn reset && yarn kbn bootstrap`
  | 2023-01-24 11:03:31 CEST | HINT
  | 2023-01-24 11:03:31 CEST | ERROR  [bootstrap] failed: [bazel] exited with code [1]
  | 2023-01-24 11:03:31 CEST | output:
  | 2023-01-24 11:03:31 CEST | INFO: Invocation ID: 207c041d-23b8-4f07-bf56-8cb961819546
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | <--- Last few GCs --->
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | [1304562:0x74b83a0]   154462 ms: Mark-sweep 4023.8 (4130.2) -> 4011.3 (4131.0) MB, 1677.3 / 0.0 ms  (average mu = 0.149, current mu = 0.074) task; scavenge might not succeed
  | 2023-01-24 11:03:31 CEST | [1304562:0x74b83a0]   155522 ms: Mark-sweep 4024.7 (4131.0) -> 4012.0 (4131.7) MB, 924.0 / 0.0 ms  (average mu = 0.142, current mu = 0.128) task; scavenge might not succeed
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | <--- JS stacktrace --->
  | 2023-01-24 11:03:31 CEST |  
  | 2023-01-24 11:03:31 CEST | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
  | 2023-01-24 11:03:31 CEST | 1: 0xb6b850 node::Abort() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 2: 0xa806a6  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 3: 0xd52140 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 4: 0xd524e7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 5: 0xf2fbe5  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 6: 0xf30ae8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 7: 0xf40ff3  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 8: 0xf41e68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 9: 0xfb1124 v8::internal::ScavengeJob::Task::RunInternal() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 10: 0xe2187b non-virtual thunk to v8::internal::CancelableTask::Run() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 11: 0xbd6684  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 12: 0xbd9aee node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 13: 0x1652906  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 14: 0x1664e44  [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 15: 0x165326e uv_run [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 16: 0xaafa2d node::SpinEventLoop(node::Environment*) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 17: 0xbb11f4 node::NodeMainInstance::Run() [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 18: 0xb26c44 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 19: 0xb2a83f node::Start(int, char**) [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | 20: 0x7fbc908be083 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
  | 2023-01-24 11:03:31 CEST | 21: 0xaad7ee _start [/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/nodejs/bin/node]
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/nodejs_linux_amd64/bin/yarn: line 19: 1304562 Aborted                 "$SCRIPT_DIR/node" "$SCRIPT_DIR/yarnpkg/bin/yarn.js" "$@"
  | 2023-01-24 11:03:31 CEST | INFO: Repository npm instantiated at:
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/builds/kb-static-scalability-01/elastic/kibana-scalability-benchmarking/kibana/WORKSPACE.bazel:47:13: in <toplevel>
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/index.bzl:83:18: in yarn_install
  | 2023-01-24 11:03:31 CEST | Repository rule yarn_install defined at:
  | 2023-01-24 11:03:31 CEST | /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:956:31: in <toplevel>
  | 2023-01-24 11:03:31 CEST | ERROR: An error occurred during the fetch of repository 'npm':
  | 2023-01-24 11:03:31 CEST | Traceback (most recent call last):
  | 2023-01-24 11:03:31 CEST | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 949, column 13, in _yarn_install_impl
  | 2023-01-24 11:03:31 CEST | fail("yarn_install failed: %s (%s)" % (result.stdout, result.stderr))
  | 2023-01-24 11:03:31 CEST | Error in fail: yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | ERROR: /var/lib/buildkite-agent/builds/kb-static-scalability-01/elastic/kibana-scalability-benchmarking/kibana/WORKSPACE.bazel:47:13: fetching yarn_install rule //external:npm: Traceback (most recent call last):
  | 2023-01-24 11:03:31 CEST | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/fbd7d220b25178fcebe6b4ba126fca45/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 949, column 13, in _yarn_install_impl
  | 2023-01-24 11:03:31 CEST | fail("yarn_install failed: %s (%s)" % (result.stdout, result.stderr))
  | 2023-01-24 11:03:31 CEST | Error in fail: yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | ERROR: Skipping '//packages/kbn-ui-shared-deps-src:shared_built_assets': no such package '@npm//webpack-cli': yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | ERROR: no such package '@npm//webpack-cli': yarn_install failed:  ()
  | 2023-01-24 11:03:31 CEST | INFO: Elapsed time: 160.951s
  | 2023-01-24 11:03:31 CEST | INFO: 0 processes.}
  | 2023-01-24 11:03:31 CEST | error Command failed with exit code 1.
```
</details>

Re-try usually helps and should minimise the failures.

I think the worker is missing some configuration since
`.buildkite/scripts/bootstrap.sh` fails on it, but works totally fine on
`kb-static-ubuntu-01`.
2023-01-26 16:02:23 +01:00
..
code_coverage [code coverage] removing instrumentation & plugin for functional tests (#148748) 2023-01-12 16:44:48 +01:00
es_snapshots [ci] Set elasticsearch snapshot build timeout (#144831) 2022-11-08 12:07:37 -06:00
flaky_tests [ci/cypress] Use spot instances (#147196) 2022-12-12 14:28:03 -06:00
fleet support disabling kibana build download in functional/common.sh (#148461) 2023-01-05 10:41:28 -07:00
performance [performance] Disable telemetry for journey by default (#148915) 2023-01-13 23:50:14 +01:00
pull_request [Defend Workflows] Init Cypress (#147822) 2023-01-04 17:05:13 +01:00
scalability [ci] retry scalability testing step on failure (#149399) 2023-01-26 16:02:23 +01:00
artifacts.yml [artifacts/cft] Soft fail on timeout (#141180) 2022-09-21 09:32:36 -05:00
artifacts_container_image.yml [ci] Add build scripts for latest docker image (#148606) 2023-01-13 13:46:40 -06:00
bazel_cache.yml [CI] Improve reliability of macos bazel cache jobs (#138554) 2022-08-10 13:11:59 -07:00
build_api_docs.yml [api-docs] 2022-06-14 Daily api_docs build (#134389) 2022-06-14 12:58:55 -07:00
on_merge.yml [ci/on-merge] Trigger container image build (#149350) 2023-01-23 13:16:56 -07:00
package_testing.yml [buildkite] Migrate package-testing pipeline from Jenkins (#122450) 2022-01-10 16:45:40 -05:00
purge_cloud_deployment.yml [cft] Add pipeline to shutdown deployment (#136659) 2022-07-20 10:22:25 -05:00
purge_cloud_deployments.yml [cft] Add pipeline to shutdown deployment (#136659) 2022-07-20 10:22:25 -05:00
update_demo_env.yml [ci] Ensure every kibana pipeline has a non-default queue defined (#126641) 2022-03-02 10:56:03 -05:00