kibana/x-pack/test/load/runner.ts
Gloria Hornero 4041d274b3
[ON-WEEK][POC] Playwright (#190803)
## UPDATE
It has been removed the execution of the playwright tests on buildkite,
the execution will be re-enabled as soon as we are ready and as
described below in the PR, there are still steps pending to be done.

## Motivation

**Cypress is not performing well lately.**
* We have been facing significant performance issues with Cypress. For
instance, it takes a long time to open the visual interface and start
executing tests.

**Teams are finding it increasingly challenging to write new tests and
debug existing ones.**
* The time and effort required to create new tests or troubleshoot
existing ones have become burdensome.

**Concern about the impact this could have on our testing practices.**
* Lose motivation to write tests or, worse, skip writing crucial tests.

## Why Playwright?

* Compared to Cypress, Playwright seems to be known for its faster
execution times and lower resource consumption. What could have a
positive impact by having faster feedback during development and
execution of new tests as well as more efficient use of CI resources.

* Provides powerful debugging tools which can make easier to write,
debug and execute tests.

* Seems to provide the same capabilities we currently use in our Cypress
tests.

* Given Playwright's active development and backing by Microsoft, it is
likely to continue evolving rapidly, making it a safe long-term choice.

Considering all the above, Playwright seems to be a strong candidate to
replace Cypress and address all the issues we are facing lately
regarding UI test automation.

## Objective of this POC

To write in Playwright a couple of tests we currently have on Cypress to
check the performance of the tool as well as the development experience.

The tests selected have been:
-
[enable_risk_score_redirect.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/dashboards/enable_risk_score_redirect.cy.ts)
- Owned by Entity Analytics team and selected by its simplicity since it
does not need any special setup to be executed and is short.

-
[manual_rule_run.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_gaps/manual_rule_run.cy.ts)
- Owned by Detection Engine team and selected because is short and adds
a bit more of complexity due to it needs of clean-up and setting up
initial data through the API.
  
## How to execute the tests

### Visual mode
- Navigate to: `x-pack/test/security_solution_playwright`
- Execute: `yarn open:ess` for ESS environment or `yarn open:serverless`
for serverless environment.

### Headless mode
- Navigate to: `x-pack/test/security_solution_playwright`
- Execute: `yarn run:ess` for ESS environment or `yarn run:serverless`
for serverless environment.

### From VScode
- Install `Playwright Test for VScode` extension by Microsoft
- Navigate to: `x-pack/test/security_solution_playwright`
- Execute: `yarn open:ess` for ESS environment or `yarn open:serverless`
for serverless environment.
- Open your IDE
- Click on the `Testing` icon
- On the `Test Explorer` click on the three dots to select the profile
you are going to execute `ess` or `serverless`
- Click on the test you want to execute or navigate to the spec file of
the test and execute it from the same spec.

## My experience
- Tests are way easier to implement than with Cypress.
- Playwright does not rely on chainable commands. Chainable commands on
Cypress can lead to confusing code.
- Without chainable commands, the flow of the tests is more explicit and
easier to understand.
- You can notice that the tool has been designed with Typescript in
mind.
- Is super easy to implement the Page Object Model pattern (POM).
- With POM the test code is clean and focused on "what" rather than
"how".
- Love the fact that you can execute the tests from the same IDE without
having to switch windows during test development.
- The visual mode execution gives you lots of information out of the
box.

## The scope of this PR
- Sets the initial infrastructure to write and execute tests with
Playwright.
- Has examples and set a basis about how to write tests using the POM.
- Allows the execution of the tests in ESS and serverless (just
stateless environment).
- Integrates the execution of the tests with buildkite.

## Pending to be done/investigate
- Proper readme
- How to split tests and PO between the different teams
- Good reports on CI
- Upload screenshots on CI
- Flaky test suite runner 
- Complete the labeling
- Execution of the tests on MKI environments

## FAQ
**Can I start adding tests to playwright?**
Currently, you can explore and experiment with Playwright, but there is
still work pending to be done to make the tool officially usable.

**Why security engineering productivity is the owner of the playwright
folder?**
This is something temporary to make sure that good practices are
followed.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: dkirchan <diamantis.kirchantzoglou@elastic.co>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Jon <jon@budzenski.me>
2024-09-06 13:09:18 +02:00

93 lines
3.1 KiB
TypeScript

/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { withProcRunner } from '@kbn/dev-proc-runner';
import { resolve } from 'path';
import { REPO_ROOT } from '@kbn/repo-info';
import Fs from 'fs';
import { createFlagError } from '@kbn/dev-cli-errors';
import { setTimeout as setTimeoutAsync } from 'timers/promises';
import { FtrProviderContext } from '../functional/ftr_provider_context';
const baseSimulationPath = 'src/test/scala/org/kibanaLoadTest/simulation';
const simulationPackage = 'org.kibanaLoadTest.simulation';
const simulationFIleExtension = '.scala';
const gatlingProjectRootPath: string =
process.env.GATLING_PROJECT_PATH || resolve(REPO_ROOT, '../kibana-load-testing');
const puppeteerProjectRootPath: string = resolve(gatlingProjectRootPath, 'puppeteer');
const simulationEntry: string = process.env.GATLING_SIMULATIONS || 'branch.DemoJourney';
if (!Fs.existsSync(gatlingProjectRootPath)) {
throw createFlagError(
`Incorrect path to load testing project: '${gatlingProjectRootPath}'\n
Clone 'elastic/kibana-load-testing' and set path using 'GATLING_PROJECT_PATH' env var`
);
}
const dropEmptyLines = (s: string) =>
s
.split(',')
.filter((i) => i.length > 0)
.map((i) => (i.includes('.') ? i : `branch.${i}`));
const simulationClasses = dropEmptyLines(simulationEntry);
const simulationsRootPath = resolve(gatlingProjectRootPath, baseSimulationPath);
simulationClasses.map((className) => {
const simulationClassPath = resolve(
simulationsRootPath,
className.replace('.', '/') + simulationFIleExtension
);
if (!Fs.existsSync(simulationClassPath)) {
throw createFlagError(`Simulation class is not found: '${simulationClassPath}'`);
}
});
/**
*
* GatlingTestRunner is used to run load simulation against local Kibana instance
*
* Use GATLING_SIMULATIONS to pass comma-separated class names
* Use GATLING_PROJECT_PATH to override path to 'kibana-load-testing' project
*/
export async function GatlingTestRunner({ getService }: FtrProviderContext) {
const log = getService('log');
await withProcRunner(log, async (procs) => {
for (let i = 0; i < simulationClasses.length; i++) {
await procs.run('node build/index.js', {
cmd: 'node',
args: [
'build/index.js',
`--simulation='${simulationClasses[i]}'`,
`--config='./config.json'`,
],
cwd: puppeteerProjectRootPath,
env: {
...process.env,
},
wait: true,
});
await procs.run('gatling: test', {
cmd: 'mvn',
args: [
'gatling:test',
'-q',
`-Dgatling.simulationClass=${simulationPackage}.${simulationClasses[i]}`,
],
cwd: gatlingProjectRootPath,
env: {
...process.env,
},
wait: true,
});
// wait a minute between simulations, skip for the last one
if (i < simulationClasses.length - 1) {
await setTimeoutAsync(60 * 1000);
}
}
});
}