mirror of
https://github.com/elastic/kibana.git
synced 2025-04-19 23:39:42 -04:00
4041d274b3
## UPDATE It has been removed the execution of the playwright tests on buildkite, the execution will be re-enabled as soon as we are ready and as described below in the PR, there are still steps pending to be done. ## Motivation **Cypress is not performing well lately.** * We have been facing significant performance issues with Cypress. For instance, it takes a long time to open the visual interface and start executing tests. **Teams are finding it increasingly challenging to write new tests and debug existing ones.** * The time and effort required to create new tests or troubleshoot existing ones have become burdensome. **Concern about the impact this could have on our testing practices.** * Lose motivation to write tests or, worse, skip writing crucial tests. ## Why Playwright? * Compared to Cypress, Playwright seems to be known for its faster execution times and lower resource consumption. What could have a positive impact by having faster feedback during development and execution of new tests as well as more efficient use of CI resources. * Provides powerful debugging tools which can make easier to write, debug and execute tests. * Seems to provide the same capabilities we currently use in our Cypress tests. * Given Playwright's active development and backing by Microsoft, it is likely to continue evolving rapidly, making it a safe long-term choice. Considering all the above, Playwright seems to be a strong candidate to replace Cypress and address all the issues we are facing lately regarding UI test automation. ## Objective of this POC To write in Playwright a couple of tests we currently have on Cypress to check the performance of the tool as well as the development experience. The tests selected have been: - [enable_risk_score_redirect.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/dashboards/enable_risk_score_redirect.cy.ts) - Owned by Entity Analytics team and selected by its simplicity since it does not need any special setup to be executed and is short. - [manual_rule_run.cy.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_gaps/manual_rule_run.cy.ts) - Owned by Detection Engine team and selected because is short and adds a bit more of complexity due to it needs of clean-up and setting up initial data through the API. ## How to execute the tests ### Visual mode - Navigate to: `x-pack/test/security_solution_playwright` - Execute: `yarn open:ess` for ESS environment or `yarn open:serverless` for serverless environment. ### Headless mode - Navigate to: `x-pack/test/security_solution_playwright` - Execute: `yarn run:ess` for ESS environment or `yarn run:serverless` for serverless environment. ### From VScode - Install `Playwright Test for VScode` extension by Microsoft - Navigate to: `x-pack/test/security_solution_playwright` - Execute: `yarn open:ess` for ESS environment or `yarn open:serverless` for serverless environment. - Open your IDE - Click on the `Testing` icon - On the `Test Explorer` click on the three dots to select the profile you are going to execute `ess` or `serverless` - Click on the test you want to execute or navigate to the spec file of the test and execute it from the same spec. ## My experience - Tests are way easier to implement than with Cypress. - Playwright does not rely on chainable commands. Chainable commands on Cypress can lead to confusing code. - Without chainable commands, the flow of the tests is more explicit and easier to understand. - You can notice that the tool has been designed with Typescript in mind. - Is super easy to implement the Page Object Model pattern (POM). - With POM the test code is clean and focused on "what" rather than "how". - Love the fact that you can execute the tests from the same IDE without having to switch windows during test development. - The visual mode execution gives you lots of information out of the box. ## The scope of this PR - Sets the initial infrastructure to write and execute tests with Playwright. - Has examples and set a basis about how to write tests using the POM. - Allows the execution of the tests in ESS and serverless (just stateless environment). - Integrates the execution of the tests with buildkite. ## Pending to be done/investigate - Proper readme - How to split tests and PO between the different teams - Good reports on CI - Upload screenshots on CI - Flaky test suite runner - Complete the labeling - Execution of the tests on MKI environments ## FAQ **Can I start adding tests to playwright?** Currently, you can explore and experiment with Playwright, but there is still work pending to be done to make the tool officially usable. **Why security engineering productivity is the owner of the playwright folder?** This is something temporary to make sure that good practices are followed. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: dkirchan <diamantis.kirchantzoglou@elastic.co> Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com> Co-authored-by: Jon <jon@budzenski.me>
160 lines
No EOL
3.9 KiB
Text
160 lines
No EOL
3.9 KiB
Text
.aws-config.json
|
|
.signing-config.json
|
|
.ackrc
|
|
/.es
|
|
/.chromium
|
|
.DS_Store
|
|
.node_binaries
|
|
/.beats
|
|
.native_modules
|
|
node_modules
|
|
!/src/dev/npm/integration_tests/__fixtures__/fixture1/node_modules
|
|
!/src/dev/notice/__fixtures__/node_modules
|
|
!/packages/kbn-import-resolver/src/__fixtures__/node_modules
|
|
!/packages/kbn-import-resolver/src/__fixtures__/packages/box/node_modules
|
|
trash
|
|
/optimize
|
|
/built_assets
|
|
target
|
|
/build
|
|
.jruby
|
|
.idea
|
|
*.iml
|
|
*.log
|
|
types.eslint.config.js
|
|
__tmp__
|
|
|
|
# Ignore example plugin builds
|
|
/examples/*/build
|
|
/x-pack/examples/*/build
|
|
|
|
# Ignore certain functional test runner artifacts
|
|
/test/*/failure_debug
|
|
/test/*/screenshots/diff
|
|
/test/*/screenshots/failure
|
|
/test/*/screenshots/session
|
|
/test/*/screenshots/visual_regression_gallery.html
|
|
|
|
# Ignore the same artifacts in x-pack
|
|
/x-pack/test/*/failure_debug
|
|
/x-pack/test/*/screenshots/diff
|
|
/x-pack/test/*/screenshots/failure
|
|
/x-pack/test/*/screenshots/session
|
|
/x-pack/test/*/screenshots/visual_regression_gallery.html
|
|
/x-pack/test/functional/apps/*/*/reporting/reports/failure
|
|
|
|
# Ignore the same artifacts in x-pack/test_serverless
|
|
/x-pack/test_serverless/*/failure_debug
|
|
/x-pack/test_serverless/*/screenshots/diff
|
|
/x-pack/test_serverless/*/screenshots/failure
|
|
/x-pack/test_serverless/*/screenshots/session
|
|
/x-pack/test_serverless/*/screenshots/visual_regression_gallery.html
|
|
/x-pack/test_serverless/functional/apps/*/*/reporting/reports/failure
|
|
|
|
/html_docs
|
|
.eslintcache
|
|
/plugins/
|
|
/data
|
|
disabledPlugins
|
|
webpackstats.json
|
|
/config/*
|
|
!/config/kibana.yml
|
|
!/config/README.md
|
|
!/config/serverless.yml
|
|
!/config/serverless.es.yml
|
|
!/config/serverless.oblt.yml
|
|
!/config/serverless.security.yml
|
|
!/config/node.options
|
|
coverage
|
|
!/test/common/fixtures/plugins/coverage
|
|
selenium
|
|
.babel_register_cache.json
|
|
.webpack.babelcache
|
|
*.swp
|
|
*.swo
|
|
*.out
|
|
package-lock.json
|
|
!/.buildkite/package-lock.json
|
|
.yo-rc.json
|
|
.vscode
|
|
*.sublime-*
|
|
npm-debug.log*
|
|
.tern-project
|
|
.nyc_output
|
|
.gradle
|
|
.vagrant
|
|
.envrc
|
|
|
|
## Snyk
|
|
.dccache
|
|
|
|
## @cypress/snapshot from apm plugin
|
|
/snapshots.js
|
|
/apm-diagnostics*.json
|
|
/x-pack/plugins/observability_solution/apm/ftr_e2e/cypress/e2e/service_map/snapshots/*.actual.png
|
|
/x-pack/plugins/observability_solution/apm/ftr_e2e/cypress/e2e/service_map/snapshots/*.diff.png
|
|
|
|
# transpiled cypress config
|
|
x-pack/plugins/fleet/cypress.config.d.ts
|
|
x-pack/plugins/fleet/cypress.config.js
|
|
x-pack/plugins/osquery/cypress.config.d.ts
|
|
x-pack/plugins/osquery/cypress.config.js
|
|
x-pack/plugins/enterprise_search/cypress.config.d.ts
|
|
x-pack/plugins/enterprise_search/cypress.config.js
|
|
x-pack/plugins/security_solution/public/management/cypress.config.d.ts
|
|
x-pack/plugins/security_solution/public/management/cypress.config.js
|
|
x-pack/plugins/security_solution/public/management/cypress_endpoint.config.d.ts
|
|
x-pack/plugins/security_solution/public/management/cypress_endpoint.config.js
|
|
|
|
# release notes script output
|
|
report.csv
|
|
report.asciidoc
|
|
|
|
# TS incremental build cache
|
|
*.tsbuildinfo
|
|
|
|
# Automatically generated and user-modifiable
|
|
/tsconfig.refs.json
|
|
*.type_check.json
|
|
|
|
# Yarn local mirror content
|
|
.yarn-local-mirror
|
|
|
|
# Bazel
|
|
.ijwb
|
|
/bazel
|
|
/bazel-*
|
|
.bazelrc.user
|
|
.bazelrc.cache
|
|
|
|
elastic-agent-*
|
|
fleet-server-*
|
|
elastic-agent.yml
|
|
fleet-server.yml
|
|
/packages/*/package-map.json
|
|
/packages/*/config-paths.json
|
|
/packages/kbn-synthetic-package-map/
|
|
**/.synthetics/
|
|
**/.journeys/
|
|
x-pack/test/security_api_integration/plugins/audit_log/audit.log
|
|
|
|
# ignore FTR temp directory
|
|
.ftr
|
|
role_users.json
|
|
|
|
|
|
.devcontainer/.env
|
|
|
|
# Ignore temporary files in oas_docs
|
|
oas_docs/output/kibana.serverless.tmp*.yaml
|
|
oas_docs/output/kibana.tmp*.yaml
|
|
oas_docs/output/kibana.new.yaml
|
|
oas_docs/output/kibana.serverless.new.yaml
|
|
|
|
# Security Solution Playwright
|
|
x-pack/test/security_solution_playwright/test-results/
|
|
x-pack/test/security_solution_playwright/playwright-report/
|
|
x-pack/test/security_solution_playwright/blob-report/
|
|
x-pack/test/security_solution_playwright/playwright/.cache/
|
|
x-pack/test/security_solution_playwright/.auth/
|
|
x-pack/test/security_solution_playwright/.env |