[Scout] add initial docs (#204050)

## Summary

Adding initial documentation for the new Kibana Test Framework. 

To reviewers: please read it carefully and provide your feedback/changes
so that we have a solid documentation in place.

packages/kbn-scout/README.md

@@ -1,9 +1,258 @@
 # @kbn/scout
 
-The package is designed to streamline the setup and execution of Playwright tests for Kibana. It consolidates server management and testing capabilities by wrapping both the Kibana/Elasticsearch server launcher and the Playwright test runner. It includes:
+`kbn-scout` is a modern test framework for Kibana. It uses Playwright for UI integration tests. Its primary goal is to enhance the developer experience by offering a lightweight and flexible testing solution to create UI tests next to the plugin source code. This README explains the structure of the `kbn-scout` package and provides an overview of its key components.
 
-    - core test and worker-scoped fixtures for reliable setup across test suites
-    - page objects combined into the fixture for for core Kibana apps UI interactions
-    - configurations for seamless test execution in both local and CI environments
+### Table of Contents
 
-This package aims to simplify test setup and enhance modularity, making it easier to create, run, and maintain deployment-agnostic tests, that are located in the plugin they actually test.
+1. Overview
+2. Folder Structure
+3. Key Components
+4. How to Use
+5. Contributing
+
+### Overview
+
+The `kbn-scout` framework provides:
+
+- **Ease of integration:** a simplified mechanism to write and run tests closer to plugins.
+- **Deployment-agnostic tests:** enables the testing of Kibana features across different environments (e.g., Stateful, Serverless).
+- **Fixture-based design:** built on Playwright's fixture model to modularize and standardize test setup.
+- **Focus on Developer Productivity:** faster test execution and minimal boilerplate for writing tests.
+
+### Folder Structure
+
+The `kbn-scout` structure includes the following key directories and files:
+
+```
+packages/kbn-scout/
+├── src/
+│   ├── cli/
+│   ├── common/
+│   │   ├── services/
+│   │   ├── utils/
+│   ├── config/
+│   │   ├── loader/
+│   │   ├── schema/
+│   │   └── serverless/
+│   │   └── stateful/
+│   │   └── config.ts
+│   ├── playwright/
+│   │   ├── config/
+│   │   └── fixtures
+│   │   │   └── test/
+│   │   │   └── worker/
+│   │   └── page_objects/
+│   │   └── runner
+│   │   │   └── config_validator.ts
+│   │   │   └── run_tests.ts
+│   ├── servers/
+│   │   ├── run_elasticsearch.ts
+│   │   └── run_kibana_server.ts
+│   │   └── start_servers.ts
+│   ├── types/
+│   └── index.ts
+├── package.json
+├── tsconfig.json
+```
+
+### Key Components
+
+1. **src/cli/**
+
+Contains the logic to start servers, with or without running tests. It is accessed through the `scripts/scout` script.
+
+2. **src/common/**
+
+`services` directory includes test helpers used across UI and API integration tests, such as Kibana and Elasticsearch `clients`, `esArchiver`, and `samlSessionManager`. These services are used to initialize instances and expose them to tests via Playwright worker fixtures.
+
+3. **src/config/**
+
+`config` directory holds configurations for running servers locally. `serverless` and `stateful` directories contain deployment-specific configurations. Configuration attributes are defined in `schema` directory.
+The `Config` class in config.ts serves as the main entry point. It is instantiated using the config loader in
+the `loader` directory. This instance is compatible with the `kbn-test` input format and is passed to functions
+for starting servers.
+
+4. **src/playwright/**
+
+#### Config
+
+`playwright` directory manages the default Playwright configuration. It exports the `createPlaywrightConfig` function, which is used by Kibana plugins to define Scout playwright configurations and serves as the entry point to run tests.
+
+```ts
+import { createPlaywrightConfig } from '@kbn/scout';
+
+// eslint-disable-next-line import/no-default-export
+export default createPlaywrightConfig({
+    testDir: './tests',
+    workers: 2,
+});
+```
+
+Scout relies on configuration to determine the test files and opt-in [parallel test execution](https://playwright.dev/docs/test-parallel) against the single Elastic cluster.
+
+The Playwright configuration should only be created this way to ensure compatibility with Scout functionality. For configuration
+verification, we use a marker `VALID_CONFIG_MARKER`, and Scout will throw an error if the configuration is invalid.
+
+#### Fixtures
+
+The `fixtures` directory contains core Scout capabilities required for testing the majority of Kibana plugins. [Fixtures](https://playwright.dev/docs/test-fixtures) can be
+scoped to either `test` or `worker`. Scope decides when to init a new fixture instance: once per worker or for every test function. It is important to choose the correct scope to keep test execution optimally fast: if **a new instance is not needed for every test**, the fixture should be scoped to **worker**. Otherwise, it should be scoped to **test**.
+
+**Core `worker` scoped fixtures:**
+- `log`
+- `config`
+- `esClient`
+- `kbnClient`
+- `esArchiver`
+- `samlAuth`
+
+```ts
+    test.beforeAll(async ({ kbnClient }) => {
+      await kbnClient.importExport.load(testData.KBN_ARCHIVES.ECOMMERCE);
+    });
+```
+
+**Core `test` scoped fixtures:**
+- `browserAuth`
+- `pageObjects`
+- `page`
+
+```ts
+    test.beforeEach(async ({ browserAuth }) => {
+      await browserAuth.loginAsViewer();
+    });
+```
+
+If a new fixture depends on a fixture with a `test` scope, it must also be `test` scoped.
+
+#### Page Objects
+
+The `page_objects` directory contains all the Page Objects that represent Platform core functionality such as Discover, Dashboard, Index Management, etc.
+
+If a Page Object is likely to be used in more than one plugin, it should be added here. This allows other teams to reuse it, improving collaboration across teams, reducing code duplication, and simplifying support and adoption.
+
+Page Objects must be registered with the `createLazyPageObject` function, which guarantees its instance is lazy-initialized. This way, we can have all the page objects available in the test context, but only the ones that are called will be actually initialized:
+
+```ts
+export function createCorePageObjects(page: ScoutPage): PageObjects {
+  return {
+    dashboard: createLazyPageObject(DashboardApp, page),
+    discover: createLazyPageObject(DiscoverApp, page),
+    // Add new page objects here
+  };
+}
+```
+
+All registered Page Objects are available via the `pageObjects` fixture:
+
+```ts
+test.beforeEach(async ({ pageObjects }) => {
+    await pageObjects.discover.goto();
+});
+```
+
+5. **src/servers/**
+
+Here we have logic to start Kibana and Elasticsearch servers using `kbn-test` functionality in Scout flavor.
+The instance of the `Config` class is passed to start servers for the specific deployment type. The `loadServersConfig` function not only returns a `kbn-test` compatible config instance, but also converts it to `ScoutServiceConfig` format and saves it on disk to `./scout/servers/local.json` in the Kibana root directory. Scout `config` fixture reads it and expose to UI tests.
+
+### How to Use
+
+#### Starting Servers Only
+
+To start the servers without running tests, use the following command:
+
+```bash
+node scripts/scout.js start-server [--stateful|--serverless=[es|oblt|security]]
+```
+
+This is useful for manual testing or running tests via an IDE.
+
+#### Running Servers and Tests
+
+To start the servers and run tests, use:
+
+```bash
+node scripts/scout.js run-tests [--stateful|--serverless=[es|oblt|security]] --config <plugin-path>/ui_tests/playwright.config.ts
+```
+
+This command starts the required servers and then automatically executes the tests using Playwright.
+
+#### Running Tests Separately
+
+If the servers are already running, you can execute tests independently using either:
+
+- Playwright Plugin in IDE: Run tests directly within your IDE using Playwright's integration.
+- Command Line: Use the following command to run tests:
+
+```bash
+npx playwright test --config <plugin-path>/ui_tests/playwright.config.ts
+```
+
+### Contributing
+
+We welcome contributions to improve and extend `kbn-scout`. This guide will help you get started, add new features, and align with existing project standards.
+
+#### Setting Up the Environment
+
+Ensure you have the latest local copy of the Kibana repository.
+
+Install dependencies by running the following commands:
+- `yarn kbn bootstrap` to install dependencies.
+- `node scripts/build_kibana_platform_plugins.js` to build plugins.
+
+Move to the `packages/kbn-scout` directory to begin development.
+
+#### Adding or Modifying Features
+
+Contributions to sharable fixtures and page objects are highly encouraged to promote reusability, stability, and ease of adoption. Follow these steps:
+
+Create a New Page Object: Add your Page Object to the `src/playwright/page_objects` directory. For instance:
+
+#### Adding Page Objects
+
+1. **Create a New Page Object:** Add your Page Object to the src/playwright/page_objects directory. For instance:
+
+```ts
+export class NewPage {
+  constructor(private readonly page: ScoutPage) {}
+
+  // implementation
+}
+```
+2. **Register the Page Object:** Update the index file to include the new Page Object:
+```ts
+export function createCorePageObjects(page: ScoutPage): PageObjects {
+  return {
+    ...
+    newPage: createLazyPageObject(NewPage, page),
+  };
+}
+```
+#### Adding Fixtures
+1. **Determine Fixture Scope:** Decide if your fixture should apply to the `test` (per-test) or `worker` (per-worker) scope.
+
+2. **Implement the Fixture:** Add the implementation to `src/playwright/fixtures/test` or `src/playwright/fixtures/worker`.
+
+```ts
+export const newTestFixture = base.extend<ScoutTestFixtures, ScoutWorkerFixtures>({
+  newFixture: async ({}, use) => {
+    const myFn = // implementation
+    await use(myFn);
+    // optionally, cleanup on test completion
+  },
+});
+```
+3. **Register the Fixture:** Add the fixture to the appropriate scope:
+```ts
+export const scoutTestFixtures = mergeTests(
+  ...
+  newTestFixture,
+);
+```
+
+#### Best Practices
+- **Reusable Code:** When creating Page Objects or Fixtures that apply to more than one plugin, ensure they are added to the kbn-scout package.
+- **Adhere to Existing Structure:** Maintain consistency with the project's architecture.
+- **Add Unit Tests:** Include tests for new logic where applicable, ensuring it works as expected.
+- **Playwright documentation:** [Official best practices](https://playwright.dev/docs/best-practices)

packages/kbn-scout/src/config/loader/config_load.ts

@@ -8,10 +8,16 @@
  */
 
 import path from 'path';
-import { ToolingLog } from '@kbn/tooling-log';
 import { Config } from '../config';
 
-export const loadConfig = async (configPath: string, log: ToolingLog): Promise<Config> => {
+/**
+ * Dynamically loads server configuration file in the "kbn-scout" framework. It reads
+ * and validates the configuration file, ensuring the presence of essential servers
+ * information required to initialize the testing environment.
+ * @param configPath Path to the configuration file to be loaded.
+ * @returns Config instance that is used to start local servers
+ */
+export const loadConfig = async (configPath: string): Promise<Config> => {
   try {
     const absolutePath = path.resolve(configPath);
     const configModule = await import(absolutePath);

packages/kbn-scout/src/config/utils.ts

@@ -16,6 +16,7 @@ import { SCOUT_SERVERS_ROOT } from '@kbn/scout-info';
 import { CliSupportedServerModes, ScoutServerConfig } from '../types';
 import { getConfigFilePath } from './get_config_file';
 import { loadConfig } from './loader/config_load';
+import type { Config } from './config';
 
 export const formatCurrentDate = () => {
   const now = new Date();
@@ -29,6 +30,11 @@ export const formatCurrentDate = () => {
   );
 };
 
+/**
+ * Saves Scout server configuration to the disk.
+ * @param testServersConfig configuration to be saved
+ * @param log Logger instance to report errors or debug information.
+ */
 const saveTestServersConfigOnDisk = (testServersConfig: ScoutServerConfig, log: ToolingLog) => {
   const configFilePath = path.join(SCOUT_SERVERS_ROOT, `local.json`);
 
@@ -48,16 +54,26 @@ const saveTestServersConfigOnDisk = (testServersConfig: ScoutServerConfig, log:
   }
 };
 
-export async function loadServersConfig(mode: CliSupportedServerModes, log: ToolingLog) {
+/**
+ * Loads server configuration based on the mode, creates "kbn-test" compatible Config
+ * instance, that can be used to start local servers and saves its "Scout"-format copy
+ * to the disk.
+ * @param mode server local run mode
+ * @param log Logger instance to report errors or debug information.
+ * @returns "kbn-test" compatible Config instance
+ */
+export async function loadServersConfig(
+  mode: CliSupportedServerModes,
+  log: ToolingLog
+): Promise<Config> {
   // get path to one of the predefined config files
   const configPath = getConfigFilePath(mode);
   // load config that is compatible with kbn-test input format
-  const config = await loadConfig(configPath, log);
+  const config = await loadConfig(configPath);
   // construct config for Playwright Test
   const scoutServerConfig = config.getTestServersConfig();
   // save test config to the file
   saveTestServersConfigOnDisk(scoutServerConfig, log);
-
   return config;
 }
 

packages/kbn-scout/src/playwright/fixtures/test/browser_auth.ts

@@ -14,6 +14,11 @@ import { serviceLoadedMsg } from '../../utils';
 
 type LoginFunction = (role: string) => Promise<void>;
 
+/**
+ * The "browserAuth" fixture simplifies the process of logging into Kibana with
+ * different roles during tests. It uses the "samlAuth" fixture to create an authentication session
+ * for the specified role and the "context" fixture to update the cookie with the role-scoped session.
+ */
 export const browserAuthFixture = base.extend<{ browserAuth: LoginFixture }, ScoutWorkerFixtures>({
   browserAuth: async ({ log, context, samlAuth, config }, use) => {
     const setSessionCookie = async (cookieValue: string) => {

packages/kbn-scout/src/playwright/fixtures/test/page_objects.ts

@@ -11,6 +11,14 @@ import { test as base } from '@playwright/test';
 import { ScoutTestFixtures, ScoutWorkerFixtures } from '../types';
 import { createCorePageObjects } from '../../page_objects';
 
+/**
+ * The "pageObjects" fixture provides a centralized and consistent way to access and
+ * interact with reusable Page Objects in tests. This fixture automatically
+ * initializes core Page Objects and makes them available to tests, promoting
+ * modularity and reducing redundant setup.
+ *
+ * Note: Page Objects are lazily instantiated on first access.
+ */
 export const pageObjectsFixture = base.extend<ScoutTestFixtures, ScoutWorkerFixtures>({
   pageObjects: async ({ page }, use) => {
     const corePageObjects = createCorePageObjects(page);

packages/kbn-scout/src/playwright/fixtures/types/test_scope.ts

@@ -17,8 +17,20 @@ export interface ScoutTestFixtures {
 }
 
 export interface LoginFixture {
+  /**
+   * Logs in as a user with viewer-only permissions.
+   * @returns A Promise that resolves once the cookie in browser is set.
+   */
   loginAsViewer: () => Promise<void>;
+  /**
+   * Logs in as a user with administrative privileges
+   * @returns A Promise that resolves once the cookie in browser is set.
+   */
   loginAsAdmin: () => Promise<void>;
+  /**
+   * Logs in as a user with elevated, but not admin, permissions.
+   * @returns A Promise that resolves once the cookie in browser is set.
+   */
   loginAsPrivilegedUser: () => Promise<void>;
 }
 

packages/kbn-scout/src/playwright/fixtures/types/worker_scope.ts

@@ -18,6 +18,13 @@ import { ScoutServerConfig } from '../../../types';
 import { KibanaUrl } from '../../../common/services/kibana_url';
 
 export interface EsArchiverFixture {
+  /**
+   * Loads an Elasticsearch archive if the specified data index is not present.
+   * @param name The name of the archive to load.
+   * @param performance An object of type LoadActionPerfOptions to measure and
+   * report performance metrics during the load operation.
+   * @returns A Promise that resolves to an object containing index statistics.
+   */
   loadIfNeeded: (
     name: string,
     performance?: LoadActionPerfOptions | undefined
@@ -25,11 +32,30 @@ export interface EsArchiverFixture {
 }
 
 export interface UiSettingsFixture {
+  /**
+   * Applies one or more UI settings
+   * @param values (UiSettingValues): An object containing key-value pairs of UI settings to apply.
+   * @returns A Promise that resolves once the settings are applied.
+   */
   set: (values: UiSettingValues) => Promise<void>;
+  /**
+   * Resets specific UI settings to their default values.
+   * @param values A list of UI setting keys to unset.
+   * @returns A Promise that resolves after the settings are unset.
+   */
   unset: (...values: string[]) => Promise<any>;
+  /**
+   * Sets the default time range for Kibana.
+   * @from The start time of the default time range.
+   * @to The end time of the default time range.
+   * @returns A Promise that resolves once the default time is set.
+   */
   setDefaultTime: ({ from, to }: { from: string; to: string }) => Promise<void>;
 }
 
+/**
+ * The `ScoutWorkerFixtures` type defines the set of fixtures that are available
+ */
 export interface ScoutWorkerFixtures {
   log: ToolingLog;
   config: ScoutServerConfig;

packages/kbn-scout/src/playwright/fixtures/worker/core.ts

@@ -22,7 +22,16 @@ import {
 import { ScoutWorkerFixtures } from '../types/worker_scope';
 import { ScoutTestOptions } from '../../types';
 
+/**
+ * The coreWorkerFixtures setup defines foundational fixtures that are essential
+ * for running tests in the "kbn-scout" framework. These fixtures provide reusable,
+ * scoped resources for each Playwright worker, ensuring that tests have consistent
+ * and isolated access to critical services such as logging, configuration, and
+ * clients for interacting with Kibana and Elasticsearch.
+ */
 export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
+  // Provides a scoped logger instance for each worker. This logger is shared across
+  // all other fixtures within the worker scope.
   log: [
     ({}, use) => {
       use(createLogger());
@@ -30,6 +39,11 @@ export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
     { scope: 'worker' },
   ],
 
+  /**
+   * Loads the test server configuration from the source file based on local or cloud
+   * target, located by default in '.scout/servers' directory. It supplies Playwright
+   * with all server-related information including hosts, credentials, type of deployment, etc.
+   */
   config: [
     ({ log }, use, testInfo) => {
       const configName = 'local';
@@ -42,6 +56,10 @@ export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
     { scope: 'worker' },
   ],
 
+  /**
+   * Generates and exposes a Kibana URL object based on the configuration, allowing tests
+   * and fixtures to programmatically construct and validate URLs.
+   */
   kbnUrl: [
     ({ config, log }, use) => {
       use(createKbnUrl(config, log));
@@ -49,6 +67,9 @@ export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
     { scope: 'worker' },
   ],
 
+  /**
+   * Instantiates an Elasticsearch client, enabling API-level interactions with ES.
+   */
   esClient: [
     ({ config, log }, use) => {
       use(createEsClient(config, log));
@@ -56,6 +77,9 @@ export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
     { scope: 'worker' },
   ],
 
+  /**
+   * Creates a Kibana client, enabling API-level interactions with Kibana.
+   */
   kbnClient: [
     ({ log, config }, use) => {
       use(createKbnClient(config, log));
@@ -63,10 +87,17 @@ export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
     { scope: 'worker' },
   ],
 
+  /**
+   * Provides utilities for managing test data in Elasticsearch. The "loadIfNeeded" method
+   * optimizes test execution by loading data archives only if required, avoiding redundant
+   * data ingestion.
+   *
+   * Note: In order to speedup test execution and avoid the overhead of deleting the data
+   * we only expose capability to ingest the data indexes.
+   */
   esArchiver: [
     ({ log, esClient, kbnClient }, use) => {
       const esArchiverInstance = createEsArchiver(esClient, kbnClient, log);
-      // to speedup test execution we only allow to ingest the data indexes and only if index doesn't exist
       const loadIfNeeded = async (name: string, performance?: LoadActionPerfOptions | undefined) =>
         esArchiverInstance!.loadIfNeeded(name, performance);
 
@@ -75,6 +106,13 @@ export const coreWorkerFixtures = base.extend<{}, ScoutWorkerFixtures>({
     { scope: 'worker' },
   ],
 
+  /**
+   * Creates a SAML session manager, that handles authentication tasks for tests involving
+   * SAML-based authentication.
+   *
+   * Note: In order to speedup execution of tests, we cache the session cookies for each role
+   * after first call.
+   */
   samlAuth: [
     ({ log, config }, use) => {
       use(createSamlSessionManager(config, log));

packages/kbn-scout/src/playwright/fixtures/worker/ui_settings.ts

@@ -12,6 +12,9 @@ import { UiSettingValues } from '@kbn/test/src/kbn_client/kbn_client_ui_settings
 import { ScoutWorkerFixtures } from '../types';
 import { isValidUTCDate, formatTime } from '../../utils';
 
+/**
+ * This fixture provides a way to interact with Kibana UI settings.
+ */
 export const uiSettingsFixture = base.extend<{}, ScoutWorkerFixtures>({
   uiSettings: [
     ({ kbnClient }, use) => {

x-pack/plugins/discover_enhanced/ui_tests/README.md

@@ -3,9 +3,10 @@ First start the servers with
 
 ```bash
 // ESS
-node scripts/scout_start_servers.js --stateful
+node scripts/scout.js start-server --stateful
+
 // Serverless
-node scripts/scout_start_servers.js --serverless=es
+node scripts/scout.js start-server --serverless=[es|oblt|security]
 ```
 
 Then you can run the tests multiple times in another terminal with:
@@ -13,8 +14,11 @@ Then you can run the tests multiple times in another terminal with:
 ```bash
 // ESS
 npx playwright test --config x-pack/plugins/discover_enhanced/ui_tests/playwright.config.ts --grep @ess
+
 // Serverless
-npx playwright test --config x-pack/plugins/discover_enhanced/ui_tests/playwright.config.ts --grep @svlSearch // @svlOblt, @svlSecurity
+npx playwright test --config x-pack/plugins/discover_enhanced/ui_tests/playwright.config.ts --grep @svlSearch
+
+// @svlOblt, @svlSecurity
 ```
 
 Test results are available in `x-pack/plugins/discover_enhanced/ui_tests/output`