[9.0] [scout] use project deps as global hooks for parallel tests (#211409) (#214645)

# Backport

This will backport the following commits from `main` to `9.0`:
- [[scout] use project deps as global hooks for parallel tests
(#211409)](https://github.com/elastic/kibana/pull/211409)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Dzmitry
Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2025-03-13T09:06:26Z","message":"[scout]
use project deps as global hooks for parallel tests (#211409)\n\n##
Summary\n\nCurrently we are using `globalSetup` [script in
configuration\nfile](https://playwright.dev/docs/test-global-setup-teardown#option-2-configure-globalsetup-and-globalteardown)\nto
ingest Elasticsearch data before running the tests in parallel\nagainst
the same ES/Kibana instances.\n\nThis approach doesn't work well when
you need to adjust `globalSetup`\nlogic based on specific condition,
e.g. configuration file defining\nwhere servers are hosted, its
credentials, etc.\n\nNot only global hook, but `ScoutConfig` fixture
expects an argument to\ndefine where servers configuration is
defined:\n\n\ncd502acea1/packages/kbn-scout/src/playwright/fixtures/worker/core_fixtures.ts (L65-L75)\n\n`testInfo`
is how Playwright exposes currently running configuration in\na form of
`project`
interface:\n[projects](https://playwright.dev/docs/test-projects) can be
used to\ngroup tests, e.g. for specific envs or
browsers.\n\nUnfortunately `testInfo` is not exposed in global scripts,
because in\nPlaywright project design `globalSetup` scripts are run
before multiple\nprojects and projects can have its own `setup` hooks
via\n[dependencies](https://playwright.dev/docs/test-global-setup-teardown#option-1-project-dependencies):\n\n```\n
{\n name: 'setup',\n testMatch: /global.setup\\.ts/,\n },\n {\n name:
'local',\n use: { ...devices['Desktop Chrome'], configName: 'local' },\n
dependencies: 'setup',\n },\n``` \n\nWe already use project API to get
`serversConfigDir` path, where we plan\nto store local and cloud server
configurations. This PR proposes to\ndefine projects as `local` and
`cloud` (maybe even separate `cloud-mki`,\n`cloud-ech`) as a way to
provide playwright information about
servers\nconfiguration.\n\nAdvantages:\n1. we can re-use existing
fixtures as-is, without adding custom exported\nhelper functions for ES
data ingestion\n2. project dependency is displayed as `setup` in
Playwright report\n3. way better and simpler design for
consumers:\n```\nimport { globalSetupHook } from
'@kbn/scout';\n\nglobalSetupHook('Ingest data to Elasticsearch', async
({ esArchiver, log }) => {\n // add archives to load, if needed\n const
archives = [\n testData.ES_ARCHIVES.LOGSTASH,\n ];\n\n
log.debug('[setup] loading test data (only if indexes do not
exist)...');\n for (const archive of archives) {\n await
esArchiver.loadIfNeeded(archive);\n }\n});\n```\n4. it is supported by
VSCode Playwright plugin\n<img width=\"1271\" alt=\"Screenshot
2025-02-17 at 11 26
12\"\nsrc=\"https://github.com/user-attachments/assets/ba7eeb38-d39d-4785-9c11-18647599ec4a\"\n/>\n\nI
find it extremely useful because you don't need to change env var
when\nyou want to switch b/w local or cloud run, all the configurations
are\nloaded automatically and you just tick the
checkbox!\n\nDisadvantages:\n1. it is important to run `playwright test`
with `--project` flag to use\nthe proper configuration\n2. we have to
define how `projects` are used for local and cloud\nconfiguration, and
make sure it meets requirements of multiple teams. We\ncan expose a way
to pass custom project definitions in\n`createPlaywrightConfig`
function, but it might complicate the support\neffort when every Team
has too many custom projects.\n3. `project` term is something we can't
change and might be confusing\n4. Since it is a Playwright feature, we
might not have consistency with\nAPI tests runner under Scout\n\nFor
reviewers: \n\nPlaying with it locally might give a better understanding
about the\npros/cons, especially with IDE playwright plugin
installed.\n\nRunning servers with tests:\n```\nnode scripts/scout.js
run-tests --serverless=oblt --testTarget=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n\nnode
scripts/scout.js run-tests --serverless=oblt --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n```\n\nRunning
test only requires passing `project` argument:\n\n```\nnpx playwright
test --project=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n\nnpx
playwright test --project=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts\n```\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
jennypavlova
<jennypavlova94@gmail.com>","sha":"1b30686181df5410e92983ede4002ac5f693a164","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:obs-ux-infra_services","backport:version","test:scout","v9.1.0","v8.19.0"],"title":"[scout]
use project deps as global hooks for parallel
tests","number":211409,"url":"https://github.com/elastic/kibana/pull/211409","mergeCommit":{"message":"[scout]
use project deps as global hooks for parallel tests (#211409)\n\n##
Summary\n\nCurrently we are using `globalSetup` [script in
configuration\nfile](https://playwright.dev/docs/test-global-setup-teardown#option-2-configure-globalsetup-and-globalteardown)\nto
ingest Elasticsearch data before running the tests in parallel\nagainst
the same ES/Kibana instances.\n\nThis approach doesn't work well when
you need to adjust `globalSetup`\nlogic based on specific condition,
e.g. configuration file defining\nwhere servers are hosted, its
credentials, etc.\n\nNot only global hook, but `ScoutConfig` fixture
expects an argument to\ndefine where servers configuration is
defined:\n\n\ncd502acea1/packages/kbn-scout/src/playwright/fixtures/worker/core_fixtures.ts (L65-L75)\n\n`testInfo`
is how Playwright exposes currently running configuration in\na form of
`project`
interface:\n[projects](https://playwright.dev/docs/test-projects) can be
used to\ngroup tests, e.g. for specific envs or
browsers.\n\nUnfortunately `testInfo` is not exposed in global scripts,
because in\nPlaywright project design `globalSetup` scripts are run
before multiple\nprojects and projects can have its own `setup` hooks
via\n[dependencies](https://playwright.dev/docs/test-global-setup-teardown#option-1-project-dependencies):\n\n```\n
{\n name: 'setup',\n testMatch: /global.setup\\.ts/,\n },\n {\n name:
'local',\n use: { ...devices['Desktop Chrome'], configName: 'local' },\n
dependencies: 'setup',\n },\n``` \n\nWe already use project API to get
`serversConfigDir` path, where we plan\nto store local and cloud server
configurations. This PR proposes to\ndefine projects as `local` and
`cloud` (maybe even separate `cloud-mki`,\n`cloud-ech`) as a way to
provide playwright information about
servers\nconfiguration.\n\nAdvantages:\n1. we can re-use existing
fixtures as-is, without adding custom exported\nhelper functions for ES
data ingestion\n2. project dependency is displayed as `setup` in
Playwright report\n3. way better and simpler design for
consumers:\n```\nimport { globalSetupHook } from
'@kbn/scout';\n\nglobalSetupHook('Ingest data to Elasticsearch', async
({ esArchiver, log }) => {\n // add archives to load, if needed\n const
archives = [\n testData.ES_ARCHIVES.LOGSTASH,\n ];\n\n
log.debug('[setup] loading test data (only if indexes do not
exist)...');\n for (const archive of archives) {\n await
esArchiver.loadIfNeeded(archive);\n }\n});\n```\n4. it is supported by
VSCode Playwright plugin\n<img width=\"1271\" alt=\"Screenshot
2025-02-17 at 11 26
12\"\nsrc=\"https://github.com/user-attachments/assets/ba7eeb38-d39d-4785-9c11-18647599ec4a\"\n/>\n\nI
find it extremely useful because you don't need to change env var
when\nyou want to switch b/w local or cloud run, all the configurations
are\nloaded automatically and you just tick the
checkbox!\n\nDisadvantages:\n1. it is important to run `playwright test`
with `--project` flag to use\nthe proper configuration\n2. we have to
define how `projects` are used for local and cloud\nconfiguration, and
make sure it meets requirements of multiple teams. We\ncan expose a way
to pass custom project definitions in\n`createPlaywrightConfig`
function, but it might complicate the support\neffort when every Team
has too many custom projects.\n3. `project` term is something we can't
change and might be confusing\n4. Since it is a Playwright feature, we
might not have consistency with\nAPI tests runner under Scout\n\nFor
reviewers: \n\nPlaying with it locally might give a better understanding
about the\npros/cons, especially with IDE playwright plugin
installed.\n\nRunning servers with tests:\n```\nnode scripts/scout.js
run-tests --serverless=oblt --testTarget=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n\nnode
scripts/scout.js run-tests --serverless=oblt --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n```\n\nRunning
test only requires passing `project` argument:\n\n```\nnpx playwright
test --project=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n\nnpx
playwright test --project=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts\n```\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
jennypavlova
<jennypavlova94@gmail.com>","sha":"1b30686181df5410e92983ede4002ac5f693a164"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/211409","number":211409,"mergeCommit":{"message":"[scout]
use project deps as global hooks for parallel tests (#211409)\n\n##
Summary\n\nCurrently we are using `globalSetup` [script in
configuration\nfile](https://playwright.dev/docs/test-global-setup-teardown#option-2-configure-globalsetup-and-globalteardown)\nto
ingest Elasticsearch data before running the tests in parallel\nagainst
the same ES/Kibana instances.\n\nThis approach doesn't work well when
you need to adjust `globalSetup`\nlogic based on specific condition,
e.g. configuration file defining\nwhere servers are hosted, its
credentials, etc.\n\nNot only global hook, but `ScoutConfig` fixture
expects an argument to\ndefine where servers configuration is
defined:\n\n\ncd502acea1/packages/kbn-scout/src/playwright/fixtures/worker/core_fixtures.ts (L65-L75)\n\n`testInfo`
is how Playwright exposes currently running configuration in\na form of
`project`
interface:\n[projects](https://playwright.dev/docs/test-projects) can be
used to\ngroup tests, e.g. for specific envs or
browsers.\n\nUnfortunately `testInfo` is not exposed in global scripts,
because in\nPlaywright project design `globalSetup` scripts are run
before multiple\nprojects and projects can have its own `setup` hooks
via\n[dependencies](https://playwright.dev/docs/test-global-setup-teardown#option-1-project-dependencies):\n\n```\n
{\n name: 'setup',\n testMatch: /global.setup\\.ts/,\n },\n {\n name:
'local',\n use: { ...devices['Desktop Chrome'], configName: 'local' },\n
dependencies: 'setup',\n },\n``` \n\nWe already use project API to get
`serversConfigDir` path, where we plan\nto store local and cloud server
configurations. This PR proposes to\ndefine projects as `local` and
`cloud` (maybe even separate `cloud-mki`,\n`cloud-ech`) as a way to
provide playwright information about
servers\nconfiguration.\n\nAdvantages:\n1. we can re-use existing
fixtures as-is, without adding custom exported\nhelper functions for ES
data ingestion\n2. project dependency is displayed as `setup` in
Playwright report\n3. way better and simpler design for
consumers:\n```\nimport { globalSetupHook } from
'@kbn/scout';\n\nglobalSetupHook('Ingest data to Elasticsearch', async
({ esArchiver, log }) => {\n // add archives to load, if needed\n const
archives = [\n testData.ES_ARCHIVES.LOGSTASH,\n ];\n\n
log.debug('[setup] loading test data (only if indexes do not
exist)...');\n for (const archive of archives) {\n await
esArchiver.loadIfNeeded(archive);\n }\n});\n```\n4. it is supported by
VSCode Playwright plugin\n<img width=\"1271\" alt=\"Screenshot
2025-02-17 at 11 26
12\"\nsrc=\"https://github.com/user-attachments/assets/ba7eeb38-d39d-4785-9c11-18647599ec4a\"\n/>\n\nI
find it extremely useful because you don't need to change env var
when\nyou want to switch b/w local or cloud run, all the configurations
are\nloaded automatically and you just tick the
checkbox!\n\nDisadvantages:\n1. it is important to run `playwright test`
with `--project` flag to use\nthe proper configuration\n2. we have to
define how `projects` are used for local and cloud\nconfiguration, and
make sure it meets requirements of multiple teams. We\ncan expose a way
to pass custom project definitions in\n`createPlaywrightConfig`
function, but it might complicate the support\neffort when every Team
has too many custom projects.\n3. `project` term is something we can't
change and might be confusing\n4. Since it is a Playwright feature, we
might not have consistency with\nAPI tests runner under Scout\n\nFor
reviewers: \n\nPlaying with it locally might give a better understanding
about the\npros/cons, especially with IDE playwright plugin
installed.\n\nRunning servers with tests:\n```\nnode scripts/scout.js
run-tests --serverless=oblt --testTarget=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n\nnode
scripts/scout.js run-tests --serverless=oblt --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n```\n\nRunning
test only requires passing `project` argument:\n\n```\nnpx playwright
test --project=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts\n\nnpx
playwright test --project=local --config
x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts\n```\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by:
jennypavlova
<jennypavlova94@gmail.com>","sha":"1b30686181df5410e92983ede4002ac5f693a164"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
This commit is contained in:
Dzmitry Lemechko 2025-03-18 15:26:38 +01:00 committed by GitHub
parent c5c18bbe5e
commit 1101afbf99
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
24 changed files with 165 additions and 68 deletions

View file

@ -1474,7 +1474,8 @@ module.exports = {
'playwright/no-slowed-test': 'error',
'playwright/no-standalone-expect': 'error',
'playwright/no-unsafe-references': 'error',
'playwright/no-wait-for-selector': 'warn',
'playwright/no-useless-await': 'error',
'playwright/no-wait-for-selector': 'error',
'playwright/max-nested-describe': ['error', { max: 1 }],
'playwright/missing-playwright-await': 'error',
'playwright/prefer-comparison-matcher': 'error',

View file

@ -12,10 +12,19 @@ import { getRunTarget, stripRunCommand } from './cli_processing';
describe('cli_processing', () => {
describe('stripRunCommand', () => {
it(`should return the correct run command when started with 'npx'`, () => {
const argv = ['npx', 'playwright', 'test', '--config', 'path/to/config', '--grep=@svlSearch'];
const argv = [
'npx',
'playwright',
'test',
'--config',
'path/to/config',
'--project',
'local',
'--grep=@svlSearch',
];
expect(stripRunCommand(argv)).toBe(
'npx playwright test --config path/to/config --grep=@svlSearch'
'npx playwright test --config path/to/config --project local --grep=@svlSearch'
);
});
@ -26,11 +35,13 @@ describe('cli_processing', () => {
'test',
'--config',
'path/to/config',
'--project',
'local',
'--grep=@svlSearch',
];
expect(stripRunCommand(argv)).toBe(
'npx playwright test --config path/to/config --grep=@svlSearch'
'npx playwright test --config path/to/config --project local --grep=@svlSearch'
);
});

View file

@ -186,9 +186,11 @@ If the servers are already running, you can execute tests independently using ei
- Command Line: Use the following command to run tests:
```bash
npx playwright test --config <plugin-path>/ui_tests/playwright.config.ts
npx playwright test --config <plugin-path>/ui_tests/playwright.config.ts --project local
```
We use `project` flag to define test target, where tests to be run: local servers or Elastic Cloud. Currently we only support local servers.
### Contributing
We welcome contributions to improve and extend `kbn-scout`. This guide will help you get started, add new features, and align with existing project standards.

View file

@ -12,11 +12,10 @@ export {
expect,
test,
spaceTest,
globalSetupHook,
tags,
createPlaywrightConfig,
createLazyPageObject,
ingestTestDataHook,
ingestSynthtraceDataHook,
} from './src/playwright';
export type {
ScoutPlaywrightOptions,

View file

@ -64,7 +64,8 @@ describe('createPlaywrightConfig', () => {
expect(config.timeout).toBe(60000);
expect(config.expect?.timeout).toBe(10000);
expect(config.outputDir).toBe('./output/test-artifacts');
expect(config.projects![0].name).toEqual('chromium');
expect(config.projects).toHaveLength(1);
expect(config.projects![0].name).toEqual('local');
});
it('should return a Playwright configuration with Scout reporters', () => {
@ -96,12 +97,17 @@ describe('createPlaywrightConfig', () => {
]);
});
it(`should override 'workers' count in Playwright configuration`, () => {
it(`should override 'workers' count and add 'setup' project dependency`, () => {
const testDir = './my_tests';
const workers = 2;
const config = createPlaywrightConfig({ testDir, workers });
expect(config.workers).toBe(workers);
expect(config.projects).toHaveLength(2);
expect(config.projects![0].name).toEqual('setup');
expect(config.projects![1].name).toEqual('local');
expect(config.projects![1]).toHaveProperty('dependencies', ['setup']);
});
it('should generate and cache runId in process.env.TEST_RUN_ID', () => {

View file

@ -27,9 +27,34 @@ export function createPlaywrightConfig(options: ScoutPlaywrightOptions): Playwri
process.env.TEST_RUN_ID = runId;
}
const scoutProjects: PlaywrightTestConfig<ScoutTestOptions>['projects'] = [
{
name: 'local',
use: { ...devices['Desktop Chrome'], configName: 'local' },
},
];
/**
* For parallel tests, we need to add a setup project that runs before the tests project.
*/
if (options.workers && options.workers > 1) {
const parentProject = scoutProjects.find((p) => p.use?.configName);
scoutProjects.unshift({
name: 'setup',
use: parentProject?.use ? { ...parentProject.use } : {},
testMatch: /global.setup\.ts/,
});
scoutProjects.forEach((project) => {
if (project.name !== 'setup') {
project.dependencies = ['setup'];
}
});
}
return defineConfig<ScoutTestOptions>({
testDir: options.testDir,
globalSetup: options.globalSetup,
/* Run tests in files in parallel */
fullyParallel: false,
/* Fail the build on CI if you accidentally left test.only in the source code. */
@ -47,6 +72,7 @@ export function createPlaywrightConfig(options: ScoutPlaywrightOptions): Playwri
],
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
use: {
// 'configName' is not defined by default to enforce using '--project' flag when running the tests
testIdAttribute: 'data-test-subj',
serversConfigDir: SCOUT_SERVERS_ROOT,
[VALID_CONFIG_MARKER]: true,
@ -70,24 +96,6 @@ export function createPlaywrightConfig(options: ScoutPlaywrightOptions): Playwri
outputDir: './output/test-artifacts', // For other test artifacts (screenshots, videos, traces)
/* Configure projects for major browsers */
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
// {
// name: 'firefox',
// use: { ...devices['Desktop Firefox'] },
// },
],
/* Run your local dev server before starting the tests */
// webServer: {
// command: 'npm run start',
// url: 'http://127.0.0.1:3000',
// reuseExistingServer: !process.env.CI,
// },
projects: scoutProjects,
});
}

View file

@ -8,7 +8,13 @@
*/
import { mergeTests } from 'playwright/test';
import { apiFixtures, coreWorkerFixtures, scoutSpaceParallelFixture } from './worker';
import {
apiFixtures,
coreWorkerFixtures,
esArchiverFixture,
scoutSpaceParallelFixture,
synthtraceFixture,
} from './worker';
import type {
ApiParallelWorkerFixtures,
EsClient,
@ -52,3 +58,10 @@ export interface ScoutParallelWorkerFixtures extends ApiParallelWorkerFixtures {
esClient: EsClient;
scoutSpace: ScoutSpaceParallelFixture;
}
export const globalSetup = mergeTests(
coreWorkerFixtures,
esArchiverFixture,
synthtraceFixture,
apiFixtures
);

View file

@ -66,10 +66,15 @@ export const coreWorkerFixtures = base.extend<
*/
config: [
({ log }, use, workerInfo) => {
const configName = 'local';
const projectUse = workerInfo.project.use as ScoutTestOptions;
if (!projectUse.configName) {
throw new Error(
`Failed to read the 'configName' property. Make sure to run tests with '--project' flag and target enviroment (local or cloud),
e.g. 'npx playwright test --project local --config <path_to_Playwright.config.ts>'`
);
}
const serversConfigDir = projectUse.serversConfigDir;
const configInstance = createScoutConfig(serversConfigDir, configName, log);
const configInstance = createScoutConfig(serversConfigDir, projectUse.configName, log);
use(configInstance);
},

View file

@ -8,7 +8,13 @@
*/
import { Readable } from 'stream';
import type { ApmFields, Fields, InfraDocument, OtelDocument } from '@kbn/apm-synthtrace-client';
import type {
ApmFields,
Fields,
InfraDocument,
OtelDocument,
SynthtraceGenerator,
} from '@kbn/apm-synthtrace-client';
import Url from 'url';
import type { SynthtraceEsClient } from '@kbn/apm-synthtrace/src/lib/shared/base_client';
import {
@ -17,10 +23,9 @@ import {
getOtelSynthtraceEsClient,
} from '../../../common/services/synthtrace';
import { coreWorkerFixtures } from './core_fixtures';
import type { SynthtraceEvents } from '../../global_hooks/synthtrace_ingestion';
interface SynthtraceFixtureEsClient<TFields extends Fields> {
index: (events: SynthtraceEvents<TFields>) => Promise<void>;
index: (events: SynthtraceGenerator<TFields>) => Promise<void>;
clean: SynthtraceEsClient<TFields>['clean'];
}
@ -34,15 +39,12 @@ const useSynthtraceClient = async <TFields extends Fields>(
client: SynthtraceEsClient<TFields>,
use: (client: SynthtraceFixtureEsClient<TFields>) => Promise<void>
) => {
const index = async (events: SynthtraceEvents<TFields>) =>
const index = async (events: SynthtraceGenerator<TFields>) =>
await client.index(Readable.from(Array.from(events).flatMap((event) => event.serialize())));
const clean = async () => await client.clean();
await use({ index, clean });
// cleanup function after all tests have ran
await client.clean();
};
export const synthtraceFixture = coreWorkerFixtures.extend<{}, SynthtraceFixture>({

View file

@ -65,6 +65,9 @@ const getSynthtraceClient = (
}
};
/**
* @deprecated Use `globalSetupHook` and synthtrace fixtures instead
*/
export async function ingestSynthtraceDataHook(config: FullConfig, data: SynthtraceIngestionData) {
const log = getLogger();

View file

@ -7,7 +7,7 @@
* License v3.0 only", or the "Server Side Public License, v 1".
*/
import { scoutFixtures, scoutParallelFixtures } from './fixtures';
import { scoutFixtures, scoutParallelFixtures, globalSetup } from './fixtures';
// Scout core fixtures: worker & test scope
export const test = scoutFixtures;
@ -15,6 +15,8 @@ export const test = scoutFixtures;
// Scout core 'space aware' fixtures: worker & test scope
export const spaceTest = scoutParallelFixtures;
export const globalSetupHook = globalSetup;
export { createPlaywrightConfig } from './config';
export { createLazyPageObject } from './page_objects/utils';
export { expect } from './expect';

View file

@ -72,6 +72,7 @@ describe('parseTestFlags', () => {
expect(result).toEqual({
mode: 'serverless=oblt',
configPath: '/path/to/config',
testTarget: 'local',
headed: false,
esFrom: undefined,
installDir: undefined,
@ -82,6 +83,7 @@ describe('parseTestFlags', () => {
it(`should parse with correct config and stateful flags`, async () => {
const flags = new FlagsReader({
config: '/path/to/config',
testTarget: 'local',
stateful: true,
logToFile: false,
headed: true,
@ -93,10 +95,41 @@ describe('parseTestFlags', () => {
expect(result).toEqual({
mode: 'stateful',
configPath: '/path/to/config',
testTarget: 'local',
headed: true,
esFrom: 'snapshot',
installDir: undefined,
logsDir: undefined,
});
});
it(`should throw an error with incorrect '--testTarget' flag`, async () => {
const flags = new FlagsReader({
config: '/path/to/config',
testTarget: 'a',
stateful: true,
logToFile: false,
headed: true,
esFrom: 'snapshot',
});
await expect(parseTestFlags(flags)).rejects.toThrow(
'invalid --testTarget, expected one of "local", "cloud"'
);
});
it(`should throw an error with incorrect '--testTarget' flag set to 'cloud'`, async () => {
const flags = new FlagsReader({
config: '/path/to/config',
testTarget: 'cloud',
stateful: true,
logToFile: false,
headed: true,
esFrom: 'snapshot',
});
validatePlaywrightConfigMock.mockResolvedValueOnce();
await expect(parseTestFlags(flags)).rejects.toThrow(
'Running tests against Cloud / MKI is not supported yet'
);
});
});

View file

@ -19,6 +19,7 @@ export interface RunTestsOptions {
configPath: string;
headed: boolean;
mode: CliSupportedServerModes;
testTarget: 'local' | 'cloud';
esFrom: 'serverless' | 'source' | 'snapshot' | undefined;
installDir: string | undefined;
logsDir: string | undefined;
@ -27,11 +28,12 @@ export interface RunTestsOptions {
export const TEST_FLAG_OPTIONS: FlagOptions = {
...SERVER_FLAG_OPTIONS,
boolean: [...(SERVER_FLAG_OPTIONS.boolean || []), 'headed'],
string: [...(SERVER_FLAG_OPTIONS.string || []), 'config'],
default: { headed: false },
string: [...(SERVER_FLAG_OPTIONS.string || []), 'config', 'testTarget'],
default: { headed: false, testTarget: 'local' },
help: `${SERVER_FLAG_OPTIONS.help}
--config Playwright config file path
--headed Run Playwright with browser head
--testTarget Run tests agaist locally started servers or Cloud deployment / MKI project
`,
};
@ -39,6 +41,11 @@ export async function parseTestFlags(flags: FlagsReader) {
const options = parseServerFlags(flags);
const configPath = flags.string('config');
const headed = flags.boolean('headed');
const testTarget = flags.enum('testTarget', ['local', 'cloud']) || 'local';
if (testTarget === 'cloud') {
throw createFlagError(`Running tests against Cloud / MKI is not supported yet`);
}
if (!configPath) {
throw createFlagError(`Path to playwright config is required: --config <file path>`);
@ -51,5 +58,6 @@ export async function parseTestFlags(flags: FlagsReader) {
...options,
configPath,
headed,
testTarget,
};
}

View file

@ -32,7 +32,12 @@ export async function runTests(log: ToolingLog, options: RunTestsOptions) {
const playwrightConfigPath = options.configPath;
const cmd = resolve(REPO_ROOT, './node_modules/.bin/playwright');
const cmdArgs = ['test', `--config=${playwrightConfigPath}`, `--grep=${playwrightGrepTag}`];
const cmdArgs = [
'test',
`--config=${playwrightConfigPath}`,
`--grep=${playwrightGrepTag}`,
`--project=${options.testTarget}`,
];
await withProcRunner(log, async (procs) => {
log.info(`scout: Validate Playwright config has tests`);
@ -78,7 +83,7 @@ export async function runTests(log: ToolingLog, options: RunTestsOptions) {
// wait for 5 seconds
await silence(log, 5000);
// Running 'npx playwright test --config=${playwrightConfigPath}'
// Running 'npx playwright test --config=${playwrightConfigPath} --project local'
await procs.run(`playwright`, {
cmd,
args: [...cmdArgs, ...(options.headed ? ['--headed'] : [])],

View file

@ -13,14 +13,15 @@ export type Protocol = 'http' | 'https';
export const VALID_CONFIG_MARKER = Symbol('validConfig');
export type ScoutConfigName = 'local';
export interface ScoutTestOptions extends PlaywrightTestOptions {
serversConfigDir: string;
configName: ScoutConfigName;
[VALID_CONFIG_MARKER]: boolean;
}
export interface ScoutPlaywrightOptions
extends Pick<PlaywrightTestConfig, 'testDir' | 'workers' | 'globalSetup'> {
export interface ScoutPlaywrightOptions extends Pick<PlaywrightTestConfig, 'testDir' | 'workers'> {
testDir: string;
workers?: 1 | 2 | 3; // to keep performance consistent within test suites
globalSetup?: string;
}

View file

@ -52,7 +52,7 @@ export async function startServers(log: ToolingLog, options: StartServerOptions)
'\n\n' +
dedent`
Elasticsearch and Kibana are ready for functional testing.
Use 'npx playwright test --config <path_to_Playwright.config.ts>' to run tests'
Use 'npx playwright test --config <path_to_Playwright.config.ts> --project local' to run tests'
` +
'\n\n'
);

View file

@ -13,10 +13,10 @@ Then you can run the tests multiple times in another terminal with:
```bash
// ESS
npx playwright test --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts --grep @ess
npx playwright test --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts --project local --grep @ess
// Serverless
npx playwright test --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts --grep @svlSearch
npx playwright test --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts --project local --grep @svlSearch
// @svlOblt, @svlSecurity
```

View file

@ -9,7 +9,6 @@ import { createPlaywrightConfig } from '@kbn/scout';
// eslint-disable-next-line import/no-default-export
export default createPlaywrightConfig({
globalSetup: require.resolve('./parallel_tests/global_setup'),
testDir: './parallel_tests',
workers: 2,
});

View file

@ -5,11 +5,10 @@
* 2.0.
*/
import { ingestTestDataHook } from '@kbn/scout';
import { type FullConfig } from '@playwright/test';
import { globalSetupHook } from '@kbn/scout';
import { testData } from '../fixtures';
async function globalSetup(config: FullConfig) {
globalSetupHook('Ingest data to Elasticsearch', async ({ esArchiver, log }) => {
// add archives to load, if needed
const archives = [
testData.ES_ARCHIVES.LOGSTASH,
@ -17,8 +16,8 @@ async function globalSetup(config: FullConfig) {
testData.ES_ARCHIVES.ECOMMERCE,
];
return ingestTestDataHook(config, archives);
}
// eslint-disable-next-line import/no-default-export
export default globalSetup;
log.debug('[setup] loading test data (only if indexes do not exist)...');
for (const archive of archives) {
await esArchiver.loadIfNeeded(archive);
}
});

View file

@ -17,10 +17,10 @@ export type {
// re-export from @kbn/scout
export {
expect,
globalSetupHook,
tags,
createPlaywrightConfig,
createLazyPageObject,
ingestTestDataHook,
} from '@kbn/scout';
export type {
@ -29,7 +29,6 @@ export type {
KibanaUrl,
ScoutLogger,
ScoutPage,
PageObjects,
ScoutServerConfig,
ScoutTestConfig,
ScoutPlaywrightOptions,

View file

@ -16,20 +16,22 @@ Some tests are designed to run sequentially:
```bash
// ESS
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/playwright.config.ts --grep @ess
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/playwright.config.ts --project=local --grep @ess
// Serverless
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/playwright.config.ts --grep @svlOblt
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/playwright.config.ts --project=local --grep @svlOblt
```
Some tests are designed to run concurrently (preferred option):
```bash
// ESS
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/parallel_playwright.config.ts --grep @ess
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/parallel_playwright.config.ts
--project=local --grep @ess
// Serverless
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/parallel_playwright.config.ts --grep @svlOblt
npx playwright test --config x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/parallel_playwright.config.ts
--project=local --grep @svlOblt
```
Test results are available in `x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/output`

View file

@ -5,7 +5,7 @@
* 2.0.
*/
import { createPlaywrightConfig } from '@kbn/scout';
import { createPlaywrightConfig } from '@kbn/scout-oblt';
// eslint-disable-next-line import/no-default-export
export default createPlaywrightConfig({

View file

@ -5,7 +5,7 @@
* 2.0.
*/
import { createPlaywrightConfig } from '@kbn/scout';
import { createPlaywrightConfig } from '@kbn/scout-oblt';
// eslint-disable-next-line import/no-default-export
export default createPlaywrightConfig({

View file

@ -7,7 +7,6 @@
"**/*"
],
"kbn_references": [
"@kbn/scout",
"@kbn/scout-oblt"
],
"exclude": [