[8.x] [scout] support tests run against Cloud / MKI (#216705) (#217488)

# Backport This will backport the following commits from `main` to `8.x`: - [[scout] support tests run against Cloud / MKI (#216705)](https://github.com/elastic/kibana/pull/216705)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
2025-04-23 09:19:04 -04:00 · 2025-04-08 14:57:19 +02:00 · 2025-04-08 14:57:19 +02:00 · ccaf01ed71
commit ccaf01ed71
parent 1fb1f1945b
20 changed files with 466 additions and 209 deletions
--- a/.buildkite/pipelines/pull_request/scout_tests.yml
+++ b/.buildkite/pipelines/pull_request/scout_tests.yml
@ -8,7 +8,7 @@ steps:
      SCOUT_CONFIGS_SCRIPT: '.buildkite/scripts/steps/test/scout_configs.sh'
    retry:
      automatic:
-        # No retry when Scout CI config validation fails
+        # No retry when Scout configs fail
        - exit_status: 10
          limit: 0
        - exit_status: '*'
--- a/src/platform/packages/shared/kbn-scout/README.md
+++ b/src/platform/packages/shared/kbn-scout/README.md
@ -159,38 +159,129 @@ The instance of the `Config` class is passed to start servers for the specific d

 ### How to Use

+Scout uses Playwright's [projects concept](https://playwright.dev/docs/test-projects) to define the environment where tests are executed. The following projects are supported:
+
+- **`local`**: Tests are executed against servers started locally. Configuration is auto-generated by Scout and saved to `KIBANA_REPO_ROOT/.scout/servers/local.json`.
+- **`ech`**: Tests are executed against a Stateful deployment created in Elastic Cloud. Configuration is **manually** defined in `KIBANA_REPO_ROOT/.scout/servers/cloud_ech.json`.
+
+```json
+{
+  "serverless": false,
+  "isCloud": true,
+  "cloudHostName": "elastic_cloud_hostname_qa_staging_prod",
+  "cloudUsersFilePath": "/path_to_your_cloud_users/role_users.json",
+  "hosts": {
+    "kibana": "https://my.cloud.deployment.kb.co",
+    "elasticsearch": "https://my.cloud.deployment.es.co"
+  },
+  "auth": {
+    "username": "deployment_username",
+    "password": "deployment_password"
+  }
+}
+```
+
+- **`mki`**: Tests are executed against a Serverless project created in Elastic Cloud (MKI). Configuration is **manually** defined in `KIBANA_REPO_ROOT/.scout/servers/cloud_mki.json`.
+
+```json
+{
+  "serverless": true
+  "projectType": "es",
+  "isCloud": true,
+  "cloudHostName": "elastic_cloud_hostname_qa_staging_prod",
+  "cloudUsersFilePath": "/path_to_your_cloud_users/role_users.json",
+  "hosts": {
+    "kibana": "https://my.es.project.kb.co",
+    "elasticsearch": "https://my.es.project.es.co"
+  },
+  "auth": {
+    "username": "operator_username",
+    "password": "operator_password"
+  }
+}
+```
+
 #### Starting Servers Only

-To start the servers without running tests, use the following command:
+To start the servers locally without running tests, use the following command:

 ```bash
 node scripts/scout.js start-server [--stateful|--serverless=[es|oblt|security]]
 ```

-This is useful for manual testing or running tests via an IDE.
+- **`--stateful`**: Starts servers in a stateful mode.
+- **`--serverless`**: Starts servers in a serverless mode. You can specify additional options like `es` (Elasticsearch), `oblt` (Observability), or `security`.

-#### Running Servers and Tests
+This command is useful for manual testing or running tests via an IDE.

-To start the servers and run tests, use:
+#### Running Servers and Tests Locally
+
+To start the servers locally and run tests in one step, use:

 ```bash
 node scripts/scout.js run-tests [--stateful|--serverless=[es|oblt|security]] --config <plugin-path>/ui_tests/playwright.config.ts
 ```

-This command starts the required servers and then automatically executes the tests using Playwright.
+- **`--stateful`** or **`--serverless`**: Specifies the deployment type.
+- **`--config`**: Path to the Playwright configuration file for the plugin.
+
+This command starts the required servers and automatically executes the tests using Playwright.

 #### Running Tests Separately

-If the servers are already running, you can execute tests independently using either:
+If the servers are already running, you can execute tests independently using one of the following methods:

- Playwright Plugin in IDE: Run tests directly within your IDE using Playwright's integration.
- Command Line: Use the following command to run tests:
+1. **Playwright Plugin in IDE**: Run tests directly within your IDE using Playwright's integration.
+2. **Command Line**: Use the following command:

 ```bash
 npx playwright test --config <plugin-path>/ui_tests/playwright.config.ts --project local
 ```

-We use `project` flag to define test target, where tests to be run: local servers or Elastic Cloud. Currently we only support local servers.
+- **`--project`**: Specifies the test target as `local` ( `ech` or `mki` for Cloud targets, see below).
+
+#### Running Tests Against Cloud
+
+To run tests against a Cloud deployment, you can use either the Scout CLI or the Playwright CLI.
+
+**Using Scout CLI:**
+
+```bash
+node scripts/scout.js run-tests \
+  --stateful \
+  --testTarget=cloud \
+  --config <plugin-path>/ui_tests/playwright.config.ts
+```
+
+```bash
+node scripts/scout.js run-tests \
+  --serverless=oblt \
+  --testTarget=cloud \
+  --config <plugin-path>/ui_tests/playwright.config.ts
+```
+
+- **`--testTarget=cloud`**: Specifies that tests should run against a Cloud deployment.
+
+**Using Playwright CLI:**
+
+```bash
+npx playwright test \
+  --project=ech \
+  --grep=@ess \
+  --config <plugin-path>/ui_tests/playwright.config.ts
+```
+
+```bash
+npx playwright test \
+  --project=mki \
+  --grep=@svlOblt \
+  --config <plugin-path>/ui_tests/playwright.config.ts
+```
+
+- **`--project`**: Specifies the test target (`ech` for Stateful or `mki` for Serverless).
+- **`--grep`**: Filters tests by tags (e.g., `@svlSearch` for Elasticsearch or `@svlOblt` for Observability).
+
+By following these steps, you can efficiently run tests in various environments using Scout.

 ### Contributing

@ -310,6 +401,8 @@ export const scoutTestFixtures = mergeTests(


 ### Running tests on CI
+
+#### Enabling tests for execution
 Scout is still in active development, which means frequent code changes may sometimes cause test failures. To maintain stability, we currently do not run Scout tests for every PR and encourage teams to limit the number of tests they add for now.

 If a test is difficult to stabilize within a reasonable timeframe, we reserve the right to disable it or even all tests for particular plugin.
@ -324,3 +417,12 @@ On CI we run Scout tests only for `enabled` plugins:

 For PRs, Scout tests run only if there are changes to registered plugins or Scout-related packages.
 On merge commits, Scout tests run in a non-blocking mode.
+
+#### Scout exit codes
+
+| Exit code | Description |
+|--------|--------|
+| 0 | All tests passed |
+| 1 | Missing configuration (e.g. SCOUT_CONFIG_GROUP_KEY and SCOUT_CONFIG_GROUP_TYPE environment variables not set) |
+| 2 | No tests in Playwright config | 
+| 10| Tests failed | 
--- a/src/platform/packages/shared/kbn-scout/src/cli/run_tests.ts
+++ b/src/platform/packages/shared/kbn-scout/src/cli/run_tests.ts
@ -36,8 +36,13 @@ export const runTestsCmd: Command<void> = {
    This also handles server starts. Make sure a Scout test server is not already running before invoking this command.

  Common usage:
+    Running tests against local servers:
    node scripts/scout run-tests --stateful --config <playwright_config_path>
    node scripts/scout run-tests --serverless=es --headed --config <playwright_config_path>
+
+    Running tests against Cloud deployment / MKI project:
+    node scripts/scout run-tests --stateful --testTarget=cloud --config <playwright_config_path>
+    node scripts/scout run-tests --serverless=es --testTarget=cloud --config <playwright_config_path>
  `,
  flags: TEST_FLAG_OPTIONS,
  run: async ({ flagsReader, log }) => {
--- a/src/platform/packages/shared/kbn-scout/src/common/services/clients.ts
+++ b/src/platform/packages/shared/kbn-scout/src/common/services/clients.ts
@ -45,6 +45,7 @@ export function getEsClient(config: ScoutTestConfig, log: ScoutLogger) {

    esClientInstance = createEsClientForTesting({
      esUrl: elasticsearchUrl,
+      isCloud: config.isCloud,
      authOverride: { username, password },
    });
  }
--- a/src/platform/packages/shared/kbn-scout/src/common/services/saml_auth.ts
+++ b/src/platform/packages/shared/kbn-scout/src/common/services/saml_auth.ts
@ -59,6 +59,7 @@ export const createSamlSessionManager = (
    hostOptions: createKibanaHostOptions(config),
    log,
    isCloud: config.isCloud,
+    cloudHostName: config.cloudHostName,
    supportedRoles: {
      roles: supportedRoles,
      sourcePath: rolesDefinitionPath,
--- a/src/platform/packages/shared/kbn-scout/src/playwright/config/create_config.test.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/config/create_config.test.ts
@ -66,7 +66,7 @@ describe('createPlaywrightConfig', () => {
    expect(config.timeout).toBe(60000);
    expect(config.expect?.timeout).toBe(10000);
    expect(config.outputDir).toBe('./output/test-artifacts');
-    expect(config.projects).toHaveLength(1);
+    expect(config.projects).toHaveLength(3);
    expect(config.projects![0].name).toEqual('local');
  });

@ -106,10 +106,16 @@ describe('createPlaywrightConfig', () => {
    const config = createPlaywrightConfig({ testDir, workers });
    expect(config.workers).toBe(workers);

-    expect(config.projects).toHaveLength(2);
-    expect(config.projects![0].name).toEqual('setup');
+    expect(config.projects).toHaveLength(6);
+    expect(config.projects![0].name).toEqual('setup-local');
    expect(config.projects![1].name).toEqual('local');
-    expect(config.projects![1]).toHaveProperty('dependencies', ['setup']);
+    expect(config.projects![1]).toHaveProperty('dependencies', ['setup-local']);
+    expect(config.projects![2].name).toEqual('setup-ech');
+    expect(config.projects![3].name).toEqual('ech');
+    expect(config.projects![3]).toHaveProperty('dependencies', ['setup-ech']);
+    expect(config.projects![4].name).toEqual('setup-mki');
+    expect(config.projects![5].name).toEqual('mki');
+    expect(config.projects![5]).toHaveProperty('dependencies', ['setup-mki']);
  });

  it('should generate and cache runId in process.env.TEST_RUN_ID', () => {
--- a/src/platform/packages/shared/kbn-scout/src/playwright/config/create_config.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/config/create_config.ts
@ -27,31 +27,39 @@ export function createPlaywrightConfig(options: ScoutPlaywrightOptions): Playwri
    process.env.TEST_RUN_ID = runId;
  }

-  const scoutProjects: PlaywrightTestConfig<ScoutTestOptions>['projects'] = [
+  const scoutDefaultProjects: PlaywrightTestConfig<ScoutTestOptions>['projects'] = [
    {
      name: 'local',
      use: { ...devices['Desktop Chrome'], configName: 'local' },
    },
+    {
+      name: 'ech',
+      use: { ...devices['Desktop Chrome'], configName: 'cloud_ech' },
+    },
+    {
+      name: 'mki',
+      use: { ...devices['Desktop Chrome'], configName: 'cloud_mki' },
+    },
  ];

+  let scoutProjects: PlaywrightTestConfig<ScoutTestOptions>['projects'] = [];
+
  /**
-   * For parallel tests, we need to add a setup project that runs before the tests project.
+   * For parallel tests, we need to add a setup as a project dependency. While Playwright doesn't allow to read 'use'
+   * from the parent project, we have to create a setup project with the explicit 'use' object for each parent project.
+   * This is a workaround for https://github.com/microsoft/playwright/issues/32547
   */
-  if (options.workers && options.workers > 1) {
-    const parentProject = scoutProjects.find((p) => p.use?.configName);
-
-    scoutProjects.unshift({
-      name: 'setup',
-      use: parentProject?.use ? { ...parentProject.use } : {},
-      testMatch: /global.setup\.ts/,
-    });
-
-    scoutProjects.forEach((project) => {
-      if (project.name !== 'setup') {
-        project.dependencies = ['setup'];
-      }
-    });
-  }
+  scoutProjects =
+    options.workers && options.workers > 1
+      ? scoutDefaultProjects.flatMap((project) => [
+          {
+            name: `setup-${project?.name}`,
+            use: project?.use ? { ...project.use } : {},
+            testMatch: /global.setup\.ts/,
+          },
+          { ...project, dependencies: [`setup-${project?.name}`] },
+        ])
+      : scoutDefaultProjects;

  return defineConfig<ScoutTestOptions>({
    testDir: options.testDir,
--- a/src/platform/packages/shared/kbn-scout/src/playwright/runner/flags.test.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/runner/flags.test.ts
@ -58,7 +58,7 @@ describe('parseTestFlags', () => {
    );
  });

-  it(`should parse with correct config and serverless flags`, async () => {
+  it(`should parse with serverless flag for local target`, async () => {
    const flags = new FlagsReader({
      config: '/path/to/config',
      stateful: false,
@ -80,7 +80,7 @@ describe('parseTestFlags', () => {
    });
  });

-  it(`should parse with correct config and stateful flags`, async () => {
+  it(`should parse with stateful flag for local target`, async () => {
    const flags = new FlagsReader({
      config: '/path/to/config',
      testTarget: 'local',
@ -118,7 +118,30 @@ describe('parseTestFlags', () => {
    );
  });

-  it(`should throw an error with incorrect '--testTarget' flag set to 'cloud'`, async () => {
+  it(`should parse with serverless flag for cloud target`, async () => {
+    const flags = new FlagsReader({
+      config: '/path/to/config',
+      testTarget: 'cloud',
+      stateful: false,
+      serverless: 'oblt',
+      logToFile: false,
+      headed: false,
+    });
+    validatePlaywrightConfigMock.mockResolvedValueOnce();
+    const result = await parseTestFlags(flags);
+
+    expect(result).toEqual({
+      mode: 'serverless=oblt',
+      configPath: '/path/to/config',
+      testTarget: 'cloud',
+      headed: false,
+      esFrom: undefined,
+      installDir: undefined,
+      logsDir: undefined,
+    });
+  });
+
+  it(`should parse with stateful flag for cloud target`, async () => {
    const flags = new FlagsReader({
      config: '/path/to/config',
      testTarget: 'cloud',
@ -128,8 +151,16 @@ describe('parseTestFlags', () => {
      esFrom: 'snapshot',
    });
    validatePlaywrightConfigMock.mockResolvedValueOnce();
-    await expect(parseTestFlags(flags)).rejects.toThrow(
-      'Running tests against Cloud / MKI is not supported yet'
-    );
+    const result = await parseTestFlags(flags);
+
+    expect(result).toEqual({
+      mode: 'stateful',
+      configPath: '/path/to/config',
+      testTarget: 'cloud',
+      headed: true,
+      esFrom: 'snapshot',
+      installDir: undefined,
+      logsDir: undefined,
+    });
  });
 });
--- a/src/platform/packages/shared/kbn-scout/src/playwright/runner/flags.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/runner/flags.ts
@ -43,10 +43,6 @@ export async function parseTestFlags(flags: FlagsReader) {
  const headed = flags.boolean('headed');
  const testTarget = flags.enum('testTarget', ['local', 'cloud']) || 'local';

-  if (testTarget === 'cloud') {
-    throw createFlagError(`Running tests against Cloud / MKI is not supported yet`);
-  }
-
  if (!configPath) {
    throw createFlagError(`Path to playwright config is required: --config <file path>`);
  }
--- a/src/platform/packages/shared/kbn-scout/src/playwright/runner/run_tests.test.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/runner/run_tests.test.ts
@ -0,0 +1,75 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+import { ToolingLog } from '@kbn/tooling-log';
+import { hasTestsInPlaywrightConfig } from './run_tests';
+import { execPromise } from '../utils';
+
+jest.mock('../utils', () => ({
+  execPromise: jest.fn(),
+}));
+
+describe('hasTestsInPlaywrightConfig', () => {
+  let mockLog: ToolingLog;
+  const execPromiseMock = execPromise as jest.Mock;
+
+  beforeEach(() => {
+    mockLog = {
+      debug: jest.fn(),
+      info: jest.fn(),
+      error: jest.fn(),
+    } as unknown as ToolingLog;
+  });
+
+  afterEach(() => {
+    jest.clearAllMocks();
+    jest.resetAllMocks();
+  });
+
+  it('should log the last line of stdout when tests are found', async () => {
+    execPromiseMock.mockImplementationOnce(() =>
+      Promise.resolve({
+        stdout: 'Listing tests:\n[local] > spec.ts > Suite > Test\nTotal: 1 test in 1 file\n',
+        stderr: '',
+      })
+    );
+
+    const result = await hasTestsInPlaywrightConfig(
+      mockLog,
+      'playwright',
+      ['test pwArgs'],
+      'configPath/playwright.config.ts'
+    );
+
+    expect(mockLog.debug).toHaveBeenCalledWith(
+      `scout: running 'SCOUT_REPORTER_ENABLED=false playwright test pwArgs --list'`
+    );
+    expect(mockLog.info).toHaveBeenCalledTimes(2);
+    expect(mockLog.info).toHaveBeenNthCalledWith(1, 'scout: Validate Playwright config has tests');
+    expect(mockLog.info).toHaveBeenNthCalledWith(2, 'scout: Total: 1 test in 1 file');
+    expect(result).toEqual(true);
+  });
+
+  it('should log an error and return false when no tests are found', async () => {
+    execPromiseMock.mockRejectedValueOnce(new Error('Command failed'));
+
+    const result = await hasTestsInPlaywrightConfig(
+      mockLog,
+      'playwright',
+      ['test pwArgs'],
+      'configPath/playwright.config.ts'
+    );
+
+    expect(mockLog.info).toHaveBeenCalledWith('scout: Validate Playwright config has tests');
+    expect(mockLog.error).toHaveBeenCalledWith(
+      'scout: No tests found in [configPath/playwright.config.ts]'
+    );
+    expect(result).toEqual(false);
+  });
+});
--- a/src/platform/packages/shared/kbn-scout/src/playwright/runner/run_tests.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/runner/run_tests.ts
@ -8,12 +8,8 @@
 */

 import { resolve } from 'path';
-import { exec } from 'child_process';
-import { promisify } from 'util';
-const execPromise = promisify(exec);
-
 import { ToolingLog } from '@kbn/tooling-log';
-import { withProcRunner } from '@kbn/dev-proc-runner';
+import { ProcRunner, withProcRunner } from '@kbn/dev-proc-runner';
 import { getTimeReporter } from '@kbn/ci-stats-reporter';
 import { REPO_ROOT } from '@kbn/repo-info';
 import { runElasticsearch, runKibanaServer } from '../../servers';
@ -21,86 +17,131 @@ import { loadServersConfig } from '../../config';
 import { silence } from '../../common';
 import { RunTestsOptions } from './flags';
 import { getExtraKbnOpts } from '../../servers/run_kibana_server';
-import { getPlaywrightGrepTag } from '../utils';
+import { getPlaywrightGrepTag, execPromise } from '../utils';
+import { ScoutPlaywrightProjects } from '../types';
+
+export const getPlaywrightProject = (
+  testTarget: RunTestsOptions['testTarget'],
+  mode: RunTestsOptions['mode']
+): ScoutPlaywrightProjects => {
+  if (testTarget === 'cloud') {
+    return mode === 'stateful' ? 'ech' : 'mki';
+  }
+
+  return 'local';
+};
+
+async function runPlaywrightTest(procs: ProcRunner, cmd: string, args: string[]) {
+  return procs.run(`playwright`, {
+    cmd,
+    args,
+    cwd: resolve(REPO_ROOT),
+    env: {
+      ...process.env,
+    },
+    wait: true,
+  });
+}
+
+export async function hasTestsInPlaywrightConfig(
+  log: ToolingLog,
+  cmd: string,
+  cmdArgs: string[],
+  configPath: string
+): Promise<boolean> {
+  log.info(`scout: Validate Playwright config has tests`);
+  try {
+    const validationCmd = `SCOUT_REPORTER_ENABLED=false ${cmd} ${cmdArgs.join(' ')} --list`;
+    log.debug(`scout: running '${validationCmd}'`);
+
+    const result = await execPromise(validationCmd);
+    const lastLine = result.stdout.trim().split('\n').pop() || '';
+
+    log.info(`scout: ${lastLine}`);
+    return true; // success
+  } catch (err) {
+    log.error(`scout: No tests found in [${configPath}]`);
+    return false; // failure
+  }
+}
+
+async function runLocalServersAndTests(
+  procs: ProcRunner,
+  log: ToolingLog,
+  options: RunTestsOptions,
+  cmd: string,
+  cmdArgs: string[]
+) {
+  const config = await loadServersConfig(options.mode, log);
+  const abortCtrl = new AbortController();
+
+  const onEarlyExit = (msg: string) => {
+    log.error(msg);
+    abortCtrl.abort();
+  };
+
+  let shutdownEs;
+
+  try {
+    shutdownEs = await runElasticsearch({
+      onEarlyExit,
+      config,
+      log,
+      esFrom: options.esFrom,
+      logsDir: options.logsDir,
+    });
+
+    await runKibanaServer({
+      procs,
+      onEarlyExit,
+      config,
+      installDir: options.installDir,
+      extraKbnOpts: getExtraKbnOpts(options.installDir, config.get('serverless')),
+    });
+
+    // wait for 5 seconds
+    await silence(log, 5000);
+
+    await runPlaywrightTest(procs, cmd, cmdArgs);
+  } finally {
+    try {
+      await procs.stop('kibana');
+    } finally {
+      if (shutdownEs) {
+        await shutdownEs();
+      }
+    }
+  }
+}

 export async function runTests(log: ToolingLog, options: RunTestsOptions) {
  const runStartTime = Date.now();
  const reportTime = getTimeReporter(log, 'scripts/scout run-tests');

-  const config = await loadServersConfig(options.mode, log);
-  const playwrightGrepTag = getPlaywrightGrepTag(config);
-  const playwrightConfigPath = options.configPath;
+  const pwGrepTag = getPlaywrightGrepTag(options.mode);
+  const pwConfigPath = options.configPath;
+  const pwProject = getPlaywrightProject(options.testTarget, options.mode);

-  const cmd = resolve(REPO_ROOT, './node_modules/.bin/playwright');
-  const cmdArgs = [
+  const pwBinPath = resolve(REPO_ROOT, './node_modules/.bin/playwright');
+  const pwCmdArgs = [
    'test',
-    `--config=${playwrightConfigPath}`,
-    `--grep=${playwrightGrepTag}`,
-    `--project=${options.testTarget}`,
+    `--config=${pwConfigPath}`,
+    `--grep=${pwGrepTag}`,
+    `--project=${pwProject}`,
+    ...(options.headed ? ['--headed'] : []),
  ];

  await withProcRunner(log, async (procs) => {
-    log.info(`scout: Validate Playwright config has tests`);
-    try {
-      // '--list' flag tells Playwright to collect all the tests, but do not run it
-      // We disable scout reporter explicitly to avoid creating directories and collecting stats
-      const result = await execPromise(
-        `SCOUT_REPORTER_ENABLED=false ${cmd} ${cmdArgs.join(' ')} --list`
-      );
-      const lastLine = result.stdout.trim().split('\n').pop();
-      log.info(`scout: ${lastLine}`);
-    } catch (err) {
-      log.error(`scout: No tests found in [${playwrightConfigPath}]`);
+    const hasTests = await hasTestsInPlaywrightConfig(log, pwBinPath, pwCmdArgs, pwConfigPath);
+
+    if (!hasTests) {
      process.exit(2); // code "2" means no tests found
    }

-    const abortCtrl = new AbortController();
-
-    const onEarlyExit = (msg: string) => {
-      log.error(msg);
-      abortCtrl.abort();
-    };
-
-    let shutdownEs;
-
-    try {
-      shutdownEs = await runElasticsearch({
-        onEarlyExit,
-        config,
-        log,
-        esFrom: options.esFrom,
-        logsDir: options.logsDir,
-      });
-
-      await runKibanaServer({
-        procs,
-        onEarlyExit,
-        config,
-        installDir: options.installDir,
-        extraKbnOpts: getExtraKbnOpts(options.installDir, config.get('serverless')),
-      });
-
-      // wait for 5 seconds
-      await silence(log, 5000);
-
-      // Running 'npx playwright test --config=${playwrightConfigPath} --project local'
-      await procs.run(`playwright`, {
-        cmd,
-        args: [...cmdArgs, ...(options.headed ? ['--headed'] : [])],
-        cwd: resolve(REPO_ROOT),
-        env: {
-          ...process.env,
-        },
-        wait: true,
-      });
-    } finally {
-      try {
-        await procs.stop('kibana');
-      } finally {
-        if (shutdownEs) {
-          await shutdownEs();
-        }
-      }
+    if (pwProject === 'local') {
+      await runLocalServersAndTests(procs, log, options, pwBinPath, pwCmdArgs);
+    } else {
+      await runPlaywrightTest(procs, pwBinPath, pwCmdArgs);
    }

    reportTime(runStartTime, 'ready', {
--- a/src/platform/packages/shared/kbn-scout/src/playwright/types/index.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/types/index.ts
@ -13,7 +13,9 @@ export type Protocol = 'http' | 'https';

 export const VALID_CONFIG_MARKER = Symbol('validConfig');

-export type ScoutConfigName = 'local';
+export type ScoutPlaywrightProjects = 'local' | 'ech' | 'mki';
+
+export type ScoutConfigName = 'local' | 'cloud_ech' | 'cloud_mki';

 export interface ScoutTestOptions extends PlaywrightTestOptions {
  serversConfigDir: string;
--- a/src/platform/packages/shared/kbn-scout/src/playwright/utils/index.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/utils/index.ts
@ -7,4 +7,4 @@
 * License v3.0 only", or the "Server Side Public License, v 1".
 */

-export { isValidUTCDate, formatTime, getPlaywrightGrepTag } from './runner_utils';
+export { isValidUTCDate, formatTime, getPlaywrightGrepTag, execPromise } from './runner_utils';
--- a/src/platform/packages/shared/kbn-scout/src/playwright/utils/runner_utils.test.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/utils/runner_utils.test.ts
@ -57,52 +57,13 @@ describe('formatTime', () => {
 });

 describe('getPlaywrightGrepTag', () => {
-  const mockConfig = {
-    getScoutTestConfig: jest.fn(),
-  };
-
  it('should return the correct tag for serverless mode', () => {
-    mockConfig.getScoutTestConfig.mockReturnValue({
-      serverless: true,
-      projectType: 'oblt',
-    });
-
-    const result = getPlaywrightGrepTag(mockConfig as any);
-
-    expect(mockConfig.getScoutTestConfig).toHaveBeenCalled();
+    const result = getPlaywrightGrepTag('serverless=oblt');
    expect(result).toBe('@svlOblt');
  });

  it('should return the correct tag for stateful mode', () => {
-    mockConfig.getScoutTestConfig.mockReturnValue({
-      serverless: false,
-    });
-
-    const result = getPlaywrightGrepTag(mockConfig as any);
-
-    expect(mockConfig.getScoutTestConfig).toHaveBeenCalled();
+    const result = getPlaywrightGrepTag('stateful');
    expect(result).toBe('@ess');
  });
-
-  it('should throw an error if projectType is missing in serverless mode', () => {
-    mockConfig.getScoutTestConfig.mockReturnValue({
-      serverless: true,
-      projectType: undefined,
-    });
-
-    expect(() => getPlaywrightGrepTag(mockConfig as any)).toThrow(
-      `'projectType' is required to determine tags for 'serverless' mode.`
-    );
-  });
-
-  it('should throw an error if unknown projectType is set in serverless mode', () => {
-    mockConfig.getScoutTestConfig.mockReturnValue({
-      serverless: true,
-      projectType: 'a',
-    });
-
-    expect(() => getPlaywrightGrepTag(mockConfig as any)).toThrow(
-      `No tags found for projectType: 'a'.`
-    );
-  });
 });
--- a/src/platform/packages/shared/kbn-scout/src/playwright/utils/runner_utils.ts
+++ b/src/platform/packages/shared/kbn-scout/src/playwright/utils/runner_utils.ts
@ -8,8 +8,12 @@
 */

 import moment from 'moment';
-import { Config } from '../../config';
+import { exec } from 'child_process';
+import { promisify } from 'util';
 import { tagsByMode } from '../tags';
+import { CliSupportedServerModes } from '../../types';
+
+export const execPromise = promisify(exec);

 export const isValidUTCDate = (date: string): boolean => {
  return !isNaN(Date.parse(date)) && new Date(date).toISOString() === date;
@ -19,22 +23,22 @@ export function formatTime(date: string, fmt: string = 'MMM D, YYYY @ HH:mm:ss.S
  return moment.utc(date, fmt).format();
 }

-export const getPlaywrightGrepTag = (config: Config): string => {
-  const serversConfig = config.getScoutTestConfig();
+const getServerlessTag = (projectType: string): string => {
+  if (!projectType) {
+    throw new Error(`'projectType' is required to determine tags for 'serverless' mode.`);
+  }
+  const tag = tagsByMode.serverless[projectType as 'security' | 'es' | 'oblt'];
+  if (!tag) {
+    throw new Error(`No tags found for projectType: '${projectType}'.`);
+  }
+  return tag;
+};

-  if (serversConfig.serverless) {
-    const { projectType } = serversConfig;
+export const getPlaywrightGrepTag = (mode: CliSupportedServerModes): string => {
+  const [distro, projectType] = mode.split('=');

-    if (!projectType) {
-      throw new Error(`'projectType' is required to determine tags for 'serverless' mode.`);
-    }
-
-    const tag = tagsByMode.serverless[projectType];
-    if (!tag) {
-      throw new Error(`No tags found for projectType: '${projectType}'.`);
-    }
-
-    return tag;
+  if (distro === 'serverless') {
+    return getServerlessTag(projectType);
  }

  return tagsByMode.stateful;
--- a/src/platform/packages/shared/kbn-scout/src/types/test_config.d.ts
+++ b/src/platform/packages/shared/kbn-scout/src/types/test_config.d.ts
@ -13,8 +13,9 @@ export interface ScoutTestConfig {
  serverless: boolean;
  projectType?: ServerlessProjectType;
  isCloud: boolean;
-  license: string;
+  cloudHostName?: string;
  cloudUsersFilePath: string;
+  license: string;
  hosts: {
    kibana: string;
    elasticsearch: string;
--- a/src/platform/packages/shared/kbn-test/src/auth/saml_auth.ts
+++ b/src/platform/packages/shared/kbn-test/src/auth/saml_auth.ts
@ -16,7 +16,7 @@ import * as cheerio from 'cheerio';
 import { Cookie, parse as parseCookie } from 'tough-cookie';
 import Url from 'url';
 import { randomInt } from 'crypto';
-import { isValidHostname, isValidUrl } from './helper';
+import { isValidUrl } from './helper';
 import {
  CloudSamlSessionParams,
  CreateSamlSessionParams,
@ -74,15 +74,6 @@ const getCookieFromResponseHeaders = (response: AxiosResponse, errorMessage: str
  return cookie;
 };

-const getCloudHostName = () => {
-  const hostname = process.env.TEST_CLOUD_HOST_NAME;
-  if (!hostname || !isValidHostname(hostname)) {
-    throw new Error('SAML Authentication requires TEST_CLOUD_HOST_NAME env variable to be set');
-  }
-
-  return hostname;
-};
-
 const getCloudUrl = (hostname: string, pathname: string) => {
  return Url.format({
    protocol: 'https',
@ -166,7 +157,7 @@ export const createCloudSession = async (
        await delay(retryParams.attemptDelay);
      } else {
        log.error(
-          `Failed to create the new cloud session with ${retryParams.attemptsCount} attempts`
+          `Failed to create the new cloud session with ${retryParams.attemptsCount} attempts for ${email} user`
        );
        // throw original error with stacktrace
        throw ex;
@ -282,7 +273,6 @@ export const finishSAMLHandshake = async ({
  while (attemptsLeft > 0) {
    try {
      authResponse = await axios.request(request);
-
      // SAML callback should return 302
      if (authResponse.status === 302) {
        return getCookieFromResponseHeaders(
@ -351,8 +341,7 @@ export const getSecurityProfile = async ({
 };

 export const createCloudSAMLSession = async (params: CloudSamlSessionParams) => {
-  const { email, password, kbnHost, kbnVersion, log } = params;
-  const hostname = getCloudHostName();
+  const { hostname, email, password, kbnHost, kbnVersion, log } = params;
  const ecSession = await createCloudSession({ hostname, email, password, log });
  const { location, sid } = await createSAMLRequest(kbnHost, kbnVersion, log);
  const samlResponse = await createSAMLResponse({ location, ecSession, email, kbnHost, log });
--- a/src/platform/packages/shared/kbn-test/src/auth/session_manager.test.ts
+++ b/src/platform/packages/shared/kbn-test/src/auth/session_manager.test.ts
@ -32,7 +32,6 @@ const cloudUsersFilePath = resolve(REPO_ROOT, SERVERLESS_ROLES_ROOT_PATH, 'role_
 const createLocalSAMLSessionMock = jest.spyOn(samlAuth, 'createLocalSAMLSession');
 const getSecurityProfileMock = jest.spyOn(samlAuth, 'getSecurityProfile');
 const readCloudUsersFromFileMock = jest.spyOn(helper, 'readCloudUsersFromFile');
-const isValidHostnameMock = jest.spyOn(helper, 'isValidHostname');

 const getTestToken = () => 'kbn_cookie_' + crypto.randomBytes(16).toString('hex');

@ -214,14 +213,20 @@ describe('SamlSessionManager', () => {
      log,
      cloudUsersFilePath,
    };
+
    const cloudCookieInstance = Cookie.parse(
      'sid=cloud_cookie_value; Path=/; Expires=Wed, 01 Oct 2023 07:00:00 GMT'
    )!;
+    const cloudHostName = 'cloud.env.co';
    const cloudEmail = 'viewer@elastic.co';
    const cloudFullname = 'Test Viewer';
    const cloudUsers = new Array<[Role, User]>();
    cloudUsers.push(['viewer', { email: 'viewer@elastic.co', password: 'p1234' }]);
    cloudUsers.push(['editor', { email: 'editor@elastic.co', password: 'p1234' }]);
+    const samlSMOptionsWithCloudHostName = {
+      ...samlSessionManagerOptions,
+      cloudHostName,
+    };

    describe('handles errors', () => {
      beforeEach(() => {
@ -232,18 +237,22 @@ describe('SamlSessionManager', () => {
        get.mockImplementationOnce(() => Promise.resolve('8.12.0'));

        readCloudUsersFromFileMock.mockReturnValue(cloudUsers);
+        delete process.env.TEST_CLOUD_HOST_NAME; // Ensure variable is unset
      });

      test('should throw error if TEST_CLOUD_HOST_NAME is not set', async () => {
-        createCloudSAMLSessionMock.mockRestore();
-        isValidHostnameMock.mockReturnValueOnce(false);
-        const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
-        await expect(
-          samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(roleViewer)
-        ).rejects.toThrow(
-          'SAML Authentication requires TEST_CLOUD_HOST_NAME env variable to be set'
+        expect(() => new SamlSessionManager(samlSessionManagerOptions)).toThrow(
+          `'cloudHostName' is required for Cloud authentication. Provide it in the constructor or via the TEST_CLOUD_HOST_NAME environment variable.`
        );
      });
+
+      test(`should throw error if 'cloudHostName' is not a valid host`, async () => {
+        const invalidHostName = 'invalid_host';
+        expect(
+          () =>
+            new SamlSessionManager({ ...samlSessionManagerOptions, cloudHostName: invalidHostName })
+        ).toThrow(`TEST_CLOUD_HOST_NAME is not a valid hostname: ${invalidHostName}`);
+      });
    });

    beforeEach(() => {
@ -255,15 +264,22 @@ describe('SamlSessionManager', () => {

      createCloudSAMLSessionMock.mockResolvedValue(new Session(cloudCookieInstance, cloudEmail));
      readCloudUsersFromFileMock.mockReturnValue(cloudUsers);
+      delete process.env.TEST_CLOUD_HOST_NAME;
    });

-    test('should create an instance of SamlSessionManager', () => {
+    test(`should create an instance of SamlSessionManager with 'cloudHostName' passed`, () => {
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
+      expect(samlSessionManager).toBeInstanceOf(SamlSessionManager);
+    });
+
+    test(`should create an instance of SamlSessionManager with 'TEST_CLOUD_HOST_NAME' variable set`, () => {
+      process.env.TEST_CLOUD_HOST_NAME = 'cloud.env.co'; // Mock the environment variable
      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
      expect(samlSessionManager).toBeInstanceOf(SamlSessionManager);
    });

    test(`'getSessionCookieForRole' should return the actual cookie value`, async () => {
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      createCloudSAMLSessionMock.mockResolvedValue(new Session(cloudCookieInstance, cloudEmail));
      const cookie = await samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(
        roleViewer
@ -272,13 +288,13 @@ describe('SamlSessionManager', () => {
    });

    test(`'getApiCredentialsForRole' should return {Cookie: <cookieString>}`, async () => {
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      const credentials = await samlSessionManager.getApiCredentialsForRole(roleViewer);
      expect(credentials).toEqual({ Cookie: `${cloudCookieInstance.cookieString()}` });
    });

    test(`'getSessionCookieForRole' should call 'createCloudSAMLSession' only once for the same role`, async () => {
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      await samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(roleViewer);
      await samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(roleEditor);
      await samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(roleViewer);
@ -287,7 +303,7 @@ describe('SamlSessionManager', () => {
    });

    test(`'getSessionCookieForRole' should call 'createCloudSAMLSession' again if 'forceNewSession = true'`, async () => {
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      createCloudSAMLSessionMock.mockResolvedValueOnce(
        new Session(
          Cookie.parse(`sid=${getTestToken()}; Path=/; Expires=Wed, 01 Oct 2023 07:00:00 GMT`)!,
@ -315,22 +331,20 @@ describe('SamlSessionManager', () => {
    });

    test(`'getEmail' return the correct email`, async () => {
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      const email = await samlSessionManager.getEmail(roleViewer);
      expect(email).toBe(cloudEmail);
    });

    test(`'getSupportedRoles' return empty array when roles by default`, async () => {
-      const samlSessionManager = new SamlSessionManager({
-        ...samlSessionManagerOptions,
-      });
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      const roles = samlSessionManager.getSupportedRoles();
      expect(roles).toEqual([]);
    });

    test(`'getSupportedRoles' return the correct roles when roles were defined`, async () => {
      const samlSessionManager = new SamlSessionManager({
-        ...samlSessionManagerOptions,
+        ...samlSMOptionsWithCloudHostName,
        supportedRoles,
      });
      const roles = samlSessionManager.getSupportedRoles();
@ -347,7 +361,7 @@ describe('SamlSessionManager', () => {
        elastic_cloud_user: true,
      };
      getSecurityProfileMock.mockResolvedValueOnce(testData);
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      const userData = await samlSessionManager.getUserData(roleViewer);

      expect(userData).toEqual(testData);
@ -359,7 +373,7 @@ describe('SamlSessionManager', () => {
        ', '
      )}. Add role descriptor in ${supportedRoles.sourcePath} to enable it for testing`;
      const samlSessionManager = new SamlSessionManager({
-        ...samlSessionManagerOptions,
+        ...samlSMOptionsWithCloudHostName,
        supportedRoles,
      });
      await expect(
@ -376,7 +390,7 @@ describe('SamlSessionManager', () => {

    test(`throws error for non-existing role when 'supportedRoles' is not defined`, async () => {
      const nonExistingRole = 'tester';
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      await expect(
        samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(nonExistingRole)
      ).rejects.toThrow(`User with '${nonExistingRole}' role is not defined`);
@ -391,7 +405,7 @@ describe('SamlSessionManager', () => {

    test(`throws error when credentials are not specified for the role`, async () => {
      const noCredentialsRole = 'admin';
-      const samlSessionManager = new SamlSessionManager(samlSessionManagerOptions);
+      const samlSessionManager = new SamlSessionManager(samlSMOptionsWithCloudHostName);
      await expect(
        samlSessionManager.getInteractiveUserSessionCookieWithRoleScope(noCredentialsRole)
      ).rejects.toThrow(`User with '${noCredentialsRole}' role is not defined`);
--- a/src/platform/packages/shared/kbn-test/src/auth/session_manager.ts
+++ b/src/platform/packages/shared/kbn-test/src/auth/session_manager.ts
@ -10,7 +10,7 @@
 import { ToolingLog } from '@kbn/tooling-log';
 import Url from 'url';
 import { KbnClient } from '../kbn_client';
-import { readCloudUsersFromFile } from './helper';
+import { isValidHostname, readCloudUsersFromFile } from './helper';
 import {
  createCloudSAMLSession,
  createLocalSAMLSession,
@ -31,6 +31,7 @@ export interface SamlSessionManagerOptions {
  hostOptions: HostOptions;
  isCloud: boolean;
  supportedRoles?: SupportedRoles;
+  cloudHostName?: string;
  cloudUsersFilePath: string;
  log: ToolingLog;
 }
@ -56,6 +57,7 @@ export class SamlSessionManager {
  private readonly roleToUserMap: Map<Role, User>;
  private readonly sessionCache: Map<Role, Session>;
  private readonly supportedRoles?: SupportedRoles;
+  private readonly cloudHostName?: string;
  private readonly cloudUsersFilePath: string;

  constructor(options: SamlSessionManagerOptions) {
@ -67,6 +69,10 @@ export class SamlSessionManager {
    };
    this.kbnHost = Url.format(hostOptionsWithoutAuth);
    this.isCloud = options.isCloud;
+    this.cloudHostName = options.cloudHostName || process.env.TEST_CLOUD_HOST_NAME;
+    if (this.isCloud) {
+      this.validateCloudHostName();
+    }
    this.kbnClient = new KbnClient({
      log: this.log,
      url: Url.format({
@ -81,6 +87,18 @@ export class SamlSessionManager {
    this.validateCloudSetting();
  }

+  private validateCloudHostName() {
+    if (!this.cloudHostName) {
+      throw new Error(
+        `'cloudHostName' is required for Cloud authentication. Provide it in the constructor or via the TEST_CLOUD_HOST_NAME environment variable.`
+      );
+    }
+
+    if (!isValidHostname(this.cloudHostName)) {
+      throw new Error(`TEST_CLOUD_HOST_NAME is not a valid hostname: ${this.cloudHostName}`);
+    }
+  }
+
  /**
   * Validates if the 'kbnHost' points to Cloud, even if 'isCloud' was set to false
   */
@ -151,6 +169,7 @@ Set env variable 'TEST_CLOUD=1' to run FTR against your Cloud deployment`
      const kbnVersion = await this.kbnClient.version.get();
      const { email, password } = this.getCloudUserByRole(role);
      session = await createCloudSAMLSession({
+        hostname: this.cloudHostName!,
        email,
        password,
        kbnHost: this.kbnHost,
--- a/src/platform/packages/shared/kbn-test/src/auth/types.ts
+++ b/src/platform/packages/shared/kbn-test/src/auth/types.ts
@ -10,6 +10,7 @@
 import { ToolingLog } from '@kbn/tooling-log';

 export interface CloudSamlSessionParams {
+  hostname: string;
  kbnHost: string;
  kbnVersion: string;
  email: string;