We're seeing some timeouts recently in the 15 minute range.
Mac timeouts are already at 60 min.
(cherry picked from commit a2f3e3bd7c)
Co-authored-by: Jonathan Budzenski <jon@elastic.co>
* [ci] Conditional storybook builds
This moves storybook builds to run conditionally, either if there's a
storybook change or the `ci:build-storybooks` label is set.
These builds don't seem to be providing test coverage, and may not be utilized
in scenarios such as automated backports or unrelated changes.
* test regex
* formatting
* Revert "test regex"
This reverts commit dcef465414.
* fix revert
* move to spot instance
* fix merge
(cherry picked from commit 22b21645bb)
Co-authored-by: Jonathan Budzenski <jon@elastic.co>
* add scalability setup for login and promotion_tracking_dashboard journeys, re-enable step on ci
* update query to get single ftr transaction
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [QA][Code Coverage] Add a link to discover pinned to the current build.
When personnel other than myself wish to visually verify code coverage,
this link will be available in the ci output.
* Drop single quotes.
* Add annotation per J Budz.
* Fixup text.
* Drop debug.
## Summary
Fixes issue with security solution cypress flaky test runner
After introducing [dynamic split for cypress tests in Security Solution](https://github.com/elastic/kibana/pull/125986), there was discovered an issue with flaky test runner, which is using parallelism in slightly different manner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/697#_: tests were evenly split between N jobs, instead of repeating tests N times(as N jobs)
So, for flaky runner, I introducing a new ENV variable that would disable split for tests between parallel jobs and instead would run all test per each job.
Here is link to flaky runner build from this PR, which shows that all test runs per job
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/736
This test was added as a verification step for daily releasable
artifacts. We had originally planned on the release manager consuming
this context. The expected workflow has changed now to us building and
publishing the docker image instead, which already has coverage in the
Build and Deploy to Cloud step.
* [QA][Code Coverage] fixup for auto config handling pr
## Summary
Embed buildkite pipeline definition.
Follow up pr to change cc per auto config handling.
Also, resolves https://github.com/elastic/kibana/issues/132706
Increase worker count for `node scripts/build_kibana_platform_plugins` to 4 workers.
Normalize file names within coverage files such that nyc correctly builds the combined summaries.
_Ci runs this on myriad servers, so the paths are different, which "breaks" nyc's output_
Split the final merge of functional coverage into 2 passes due to [nyc issue](https://github.com/istanbuljs/nyc/issues/1263)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- adds `parallelism: 4` for security_solution cypress buildkite pipeline
- added parsing /integrations folder with cypress tests, to retrieve paths to individual test files using `glob` utility
- list of test files split equally between agents(there are approx 70+ tests files, split ~20 per job with **parallelism=4**)
- small refactoring of existing cypress runners for `security_solution`
Old metrics(before @MadameSheema https://github.com/elastic/kibana/pull/127558 performance improvements):
before split: average time of completion ~ 1h 40m for tests, 1h 55m for Kibana build
after split in 4 chunks: chunk completion between 20m - 30m, Kibana build 1h 20m
**Current metrics:**
before split: average time of completion ~ 1h for tests, 1h 10m for Kibana build
after split in 4 chunks: each chunk completion between 10m - 20m, 1h Kibana build 1h
* [artifacts] Test cloud context
This artifact currently has no consumers, but in the near future the
release-manager will produce Cloud images using the context. This
ensures builds are working before we publish artifacts.
* update permissions
* fix if statement
* DOCKER_CONTEXT -> KIBANA_DOCKER_CONTEXT
* [ci] extract apm traces after in single user benchmarking pipeline
* add performance-testing-dataset-extractor dependency
* use BUILD_ID for queries
* bump version for extractor, update script
* bump extractor version
* fix path
* upload artifacts to the public bucket
* [ftr] automatically determine config run order
* split lens config into two groups
* support ftr configs always running against CI
* Split detection_engine_api_integration rule exception list tests
* Add configs from previous commit
* [ftr] remove testMetadata and maintain a unique lifecycle instance per run
* Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run"
This reverts commit d2b4fdb824.
* Split alerting_api_integration/security_and_spaces tests
* Add groups to yaml
* Revert "Revert "[ftr] remove testMetadata and maintain a unique lifecycle instance per run""
This reverts commit 56232eea68.
* stop ES more forcefully and fix timeout
* only cleanup lifecycle phases when the cleanup is totally complete
* only use kill when cleaning up an esTestInstance
* fix broken import
* fix runOptions.alwaysUseSource implementation
* fix config access
* fix x-pack/ccs config
* fix ml import file paths
* update kibana build id
* revert array.concat() change
* fix baseConfig usage
* fix pie chart data
* split up maps tests
* pull in all of group5 so that es archives are loaded correctly
* add to ftr configs.yml
* fix pie chart data without breaking legacy version
* fix more pie_chart stuff in new vis lib
* restore normal PR tasks
* bump kibana-buildkite-library
* remove ciGroup validation
* remove the script which is no longer called from checks.sh
* [CI] Auto-commit changed files from 'yarn kbn run build -i @kbn/pm'
* adapt flaky test runner scripts to handle ftrConfig paths
* fix types in alerting_api_integration
* improve flaky config parsing and use non-local var name for passing explicit configs to ftr_configs.sh
* Split xpack dashboard tests
* Add configs
* [flaky] remove key from ftr-config steps
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* restore cypress builds
* remove ciGroups from FTR config files
* fixup some docs
* add temporary script to hunt for FTR config files
* use config.base.js naming for clarity
* use script to power ftr_configs.yml
* remove usage of removed x-pack/scripts/functional_tests
* fix test names in dashboard snapshots
* bump kibana-buildkite-library
* Try retrying only failed configs
* be a little quieter about trying to get testStats from configs with testRunners defined
* Remove test code
* bump kibana-buildkite-library
* update es_snapshot and on_merge jobs too
* track duration and exit code for each config and print it at the end of the script
* store results in order, rather than by key, in case there are duplicates in $config
* bash is hard
* fix env source and use +e rather than disabling e for whole file
* bash sucks
* print config summary in jest jobs too
* define results in jest_parallel.sh
* simplify config summary print, format times a little better
* fix reference to unbound time variable, use better variable name
* skip the newline between each result
* finish with the nitpicking
* sync changes with ftr_configs.sh
* refuse to execute config files which aren't listed in the .buildkite/ftr_configs.yml
* fix config.edge.js base config import paths
* fix some readmes
* resolve paths from ftr_configs manifest
* fix readConfigFile tests
* just allow __fixtures__ configs
* list a few more cypress config files
* install the main branch of kibana-buildkite-library
* split up lens group1
* move ml data_visualizer tests to their own config
* fix import paths
* fix more imports
* install specific commit of buildkite-pipeline-library
* sort configs in ftr_configs.yml
* bump kibana-buildkite-library
* remove temporary script
* fix env var for limiting config types
* Update docs/developer/contributing/development-functional-tests.asciidoc
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* produce a JUnit report for saved objects field count
* apply standard concurrency limits from flaky test runner
* support customizing FTR concurrency via the env
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
