mirror of
https://github.com/elastic/kibana.git
synced 2025-06-28 19:13:14 -04:00
14c3235182
## Summary This PR adds `spaceTest` interface to `kbn-scout` to run space aware tests, that can be executed in parallel. Most of Discover tests were converted to parallel run because we see runtime improvement with 2 parallel workers. Experiment 1: **ES data pre-ingested**, running 9 Discover **stateful** tests in **5 files** locally | Run setup | Took time | | ------------- | ------------- | | 1 worker | `1.3` min | | 2 workers | `58.7` sec | | 3 workers | `48.3` sec | | 4 workers | **tests fail** | Conclusion: using **2** workers is the optimal solution to continue Experiment 2: Running Discover tests for stateful/serverless in **Kibana CI** (starting servers, ingesting ES data, running tests) | Run setup | 1 worker | 2 workers | diff | ------------- | ------------- |------------- |------------- | | stateful, 9 tests / 5 files | `1.7` min | `1.2` min | `-29.4%`| | svl ES, 8 tests / 4 files | `1.7` min | `1.3` min | `-23.5%`| | svl Oblt, 8 tests / 4 files | `1.8` min | `1.4` min | `-22.2%`| | svl Search, 5 tests / 2 files | `59.9` sec | `51.6` sec | `-13.8%`| Conclusion: parallel run effectiveness benefits from tests being split in **more test files**. Experiment 3: Clone existing tests to have **3 times more test files** and re-run tests for stateful/serverless in **Kibana CI** (starting servers, ingesting ES data, running tests) | Run setup | 1 worker | 2 workers | diff | ------------- | ------------- |------------- |------------- | | stateful, 27 tests / 15 files | `4.3` min | `2.7` min | `-37.2%`| | svl ES, 24 tests / 12 files | `4.3` min | `2.7` min | `-37.2%`| Conclusion: parallel run effectiveness is **increasing** with more test files in place, **not linear** but with good test design we can expect **up to 40%** or maybe a bit more. How parallel run works: - `scoutSpace` fixture is loaded on Playwright worker setup (using `auto: true` config), creates a new Kibana Space, expose its id to other fixtures and deletes the space on teardown. - `browserAuth` fixture for parallel run caches Cookie per worker/space like `role:spaceId`. It is needed because Playwright doesn't spin up new browser for worker, but only new context. - kbnClient was updated to allow passing `createNewCopies: true` in query, it is needed to load the same Saved Objects in parallel workers/spaces and generate new ids to work with them. `scoutSpace` caches ids and allows to reach saved object by its name. This logic is different from single thread run, where we can use default ids from kbnArchives. How to run parallel tests locally, e.g. for stateful: ``` node scripts/scout run-tests --stateful --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts ```
31 lines
967 B
TypeScript
31 lines
967 B
TypeScript
/*
|
|
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
|
* or more contributor license agreements. Licensed under the "Elastic License
|
|
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
|
|
* Public License v 1"; you may not use this file except in compliance with, at
|
|
* your election, the "Elastic License 2.0", the "GNU Affero General Public
|
|
* License v3.0 only", or the "Server Side Public License, v 1".
|
|
*/
|
|
|
|
export * as cli from './src/cli';
|
|
export {
|
|
expect,
|
|
test,
|
|
spaceTest,
|
|
tags,
|
|
createPlaywrightConfig,
|
|
createLazyPageObject,
|
|
ingestTestDataHook,
|
|
} from './src/playwright';
|
|
export type {
|
|
ScoutPlaywrightOptions,
|
|
ScoutTestOptions,
|
|
ScoutPage,
|
|
PageObjects,
|
|
ScoutTestFixtures,
|
|
ScoutWorkerFixtures,
|
|
ScoutParallelTestFixtures,
|
|
ScoutParallelWorkerFixtures,
|
|
} from './src/playwright';
|
|
|
|
export type { Client, KbnClient, KibanaUrl, SamlSessionManager, ToolingLog } from './src/types';
|