mirror of
https://github.com/elastic/kibana.git
synced 2025-04-20 16:03:20 -04:00
## UPDATE It has been removed the execution of the playwright tests on buildkite, the execution will be re-enabled as soon as we are ready and as described below in the PR, there are still steps pending to be done. ## Motivation **Cypress is not performing well lately.** * We have been facing significant performance issues with Cypress. For instance, it takes a long time to open the visual interface and start executing tests. **Teams are finding it increasingly challenging to write new tests and debug existing ones.** * The time and effort required to create new tests or troubleshoot existing ones have become burdensome. **Concern about the impact this could have on our testing practices.** * Lose motivation to write tests or, worse, skip writing crucial tests. ## Why Playwright? * Compared to Cypress, Playwright seems to be known for its faster execution times and lower resource consumption. What could have a positive impact by having faster feedback during development and execution of new tests as well as more efficient use of CI resources. * Provides powerful debugging tools which can make easier to write, debug and execute tests. * Seems to provide the same capabilities we currently use in our Cypress tests. * Given Playwright's active development and backing by Microsoft, it is likely to continue evolving rapidly, making it a safe long-term choice. Considering all the above, Playwright seems to be a strong candidate to replace Cypress and address all the issues we are facing lately regarding UI test automation. ## Objective of this POC To write in Playwright a couple of tests we currently have on Cypress to check the performance of the tool as well as the development experience. The tests selected have been: - [enable_risk_score_redirect.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/dashboards/enable_risk_score_redirect.cy.ts) - Owned by Entity Analytics team and selected by its simplicity since it does not need any special setup to be executed and is short. - [manual_rule_run.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_gaps/manual_rule_run.cy.ts) - Owned by Detection Engine team and selected because is short and adds a bit more of complexity due to it needs of clean-up and setting up initial data through the API. ## How to execute the tests ### Visual mode - Navigate to: `x-pack/test/security_solution_playwright` - Execute: `yarn open:ess` for ESS environment or `yarn open:serverless` for serverless environment. ### Headless mode - Navigate to: `x-pack/test/security_solution_playwright` - Execute: `yarn run:ess` for ESS environment or `yarn run:serverless` for serverless environment. ### From VScode - Install `Playwright Test for VScode` extension by Microsoft - Navigate to: `x-pack/test/security_solution_playwright` - Execute: `yarn open:ess` for ESS environment or `yarn open:serverless` for serverless environment. - Open your IDE - Click on the `Testing` icon - On the `Test Explorer` click on the three dots to select the profile you are going to execute `ess` or `serverless` - Click on the test you want to execute or navigate to the spec file of the test and execute it from the same spec. ## My experience - Tests are way easier to implement than with Cypress. - Playwright does not rely on chainable commands. Chainable commands on Cypress can lead to confusing code. - Without chainable commands, the flow of the tests is more explicit and easier to understand. - You can notice that the tool has been designed with Typescript in mind. - Is super easy to implement the Page Object Model pattern (POM). - With POM the test code is clean and focused on "what" rather than "how". - Love the fact that you can execute the tests from the same IDE without having to switch windows during test development. - The visual mode execution gives you lots of information out of the box. ## The scope of this PR - Sets the initial infrastructure to write and execute tests with Playwright. - Has examples and set a basis about how to write tests using the POM. - Allows the execution of the tests in ESS and serverless (just stateless environment). - Integrates the execution of the tests with buildkite. ## Pending to be done/investigate - Proper readme - How to split tests and PO between the different teams - Good reports on CI - Upload screenshots on CI - Flaky test suite runner - Complete the labeling - Execution of the tests on MKI environments ## FAQ **Can I start adding tests to playwright?** Currently, you can explore and experiment with Playwright, but there is still work pending to be done to make the tool officially usable. **Why security engineering productivity is the owner of the playwright folder?** This is something temporary to make sure that good practices are followed. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: dkirchan <diamantis.kirchantzoglou@elastic.co> Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com> Co-authored-by: Jon <jon@budzenski.me>
93 lines
3.1 KiB
TypeScript
93 lines
3.1 KiB
TypeScript
/*
|
|
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
|
* or more contributor license agreements. Licensed under the Elastic License
|
|
* 2.0; you may not use this file except in compliance with the Elastic License
|
|
* 2.0.
|
|
*/
|
|
|
|
import { withProcRunner } from '@kbn/dev-proc-runner';
|
|
import { resolve } from 'path';
|
|
import { REPO_ROOT } from '@kbn/repo-info';
|
|
import Fs from 'fs';
|
|
import { createFlagError } from '@kbn/dev-cli-errors';
|
|
import { setTimeout as setTimeoutAsync } from 'timers/promises';
|
|
import { FtrProviderContext } from '../functional/ftr_provider_context';
|
|
|
|
const baseSimulationPath = 'src/test/scala/org/kibanaLoadTest/simulation';
|
|
const simulationPackage = 'org.kibanaLoadTest.simulation';
|
|
const simulationFIleExtension = '.scala';
|
|
const gatlingProjectRootPath: string =
|
|
process.env.GATLING_PROJECT_PATH || resolve(REPO_ROOT, '../kibana-load-testing');
|
|
const puppeteerProjectRootPath: string = resolve(gatlingProjectRootPath, 'puppeteer');
|
|
const simulationEntry: string = process.env.GATLING_SIMULATIONS || 'branch.DemoJourney';
|
|
|
|
if (!Fs.existsSync(gatlingProjectRootPath)) {
|
|
throw createFlagError(
|
|
`Incorrect path to load testing project: '${gatlingProjectRootPath}'\n
|
|
Clone 'elastic/kibana-load-testing' and set path using 'GATLING_PROJECT_PATH' env var`
|
|
);
|
|
}
|
|
|
|
const dropEmptyLines = (s: string) =>
|
|
s
|
|
.split(',')
|
|
.filter((i) => i.length > 0)
|
|
.map((i) => (i.includes('.') ? i : `branch.${i}`));
|
|
const simulationClasses = dropEmptyLines(simulationEntry);
|
|
const simulationsRootPath = resolve(gatlingProjectRootPath, baseSimulationPath);
|
|
|
|
simulationClasses.map((className) => {
|
|
const simulationClassPath = resolve(
|
|
simulationsRootPath,
|
|
className.replace('.', '/') + simulationFIleExtension
|
|
);
|
|
if (!Fs.existsSync(simulationClassPath)) {
|
|
throw createFlagError(`Simulation class is not found: '${simulationClassPath}'`);
|
|
}
|
|
});
|
|
|
|
/**
|
|
*
|
|
* GatlingTestRunner is used to run load simulation against local Kibana instance
|
|
*
|
|
* Use GATLING_SIMULATIONS to pass comma-separated class names
|
|
* Use GATLING_PROJECT_PATH to override path to 'kibana-load-testing' project
|
|
*/
|
|
export async function GatlingTestRunner({ getService }: FtrProviderContext) {
|
|
const log = getService('log');
|
|
|
|
await withProcRunner(log, async (procs) => {
|
|
for (let i = 0; i < simulationClasses.length; i++) {
|
|
await procs.run('node build/index.js', {
|
|
cmd: 'node',
|
|
args: [
|
|
'build/index.js',
|
|
`--simulation='${simulationClasses[i]}'`,
|
|
`--config='./config.json'`,
|
|
],
|
|
cwd: puppeteerProjectRootPath,
|
|
env: {
|
|
...process.env,
|
|
},
|
|
wait: true,
|
|
});
|
|
await procs.run('gatling: test', {
|
|
cmd: 'mvn',
|
|
args: [
|
|
'gatling:test',
|
|
'-q',
|
|
`-Dgatling.simulationClass=${simulationPackage}.${simulationClasses[i]}`,
|
|
],
|
|
cwd: gatlingProjectRootPath,
|
|
env: {
|
|
...process.env,
|
|
},
|
|
wait: true,
|
|
});
|
|
// wait a minute between simulations, skip for the last one
|
|
if (i < simulationClasses.length - 1) {
|
|
await setTimeoutAsync(60 * 1000);
|
|
}
|
|
}
|
|
});
|
|
}
|