github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 09:19:04 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Security Solution] Cypress documentation updates (#86835)

Browse source 
* Update/refactor some cypress documentation

* Fixes some whitespace/grammar/typos
* Condenses the explanation/instructions for the different modes of
  execution

* Condense Artifacts section

This is a big sprawling file; trying to cut down on the noise.

* Move test-running section to top of README

This is going to be what 90% of readers are looking for, methinks.

* Adds Security Solution's cypress suite to x-pack testing README

* Fix broken link

This file was moved as part of #64368.

* Remove broken link

This file was deleted in #67138.

* Apply suggestions from code review

Co-authored-by: Devin W. Hurley <snowmiser111@gmail.com>

* Fix typo

Co-authored-by: Devin W. Hurley <snowmiser111@gmail.com>
This commit is contained in:
Ryland Herrick 2021-01-04 18:47:08 -06:00 • committed by GitHub
parent 42ba5a8e9f
commit 1363a2aa73
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 143 additions and 184 deletions
Download patch file Download diff file Expand all files Collapse all files

7
x-pack/README.md
View file

@ -84,8 +84,9 @@ Jest integration tests can be used to test behavior with Elasticsearch and the K
yarn test:jest_integration
```
An example test exists at [test_utils/jest/integration_tests/example_integration.test.ts](test_utils/jest/integration_tests/example_integration.test.ts)
#### Running Reporting functional tests
See [here](test/reporting/README.md) for more information on running reporting tests.
See [here](./test/functional/apps/dashboard/reporting/README.md) for more information on running reporting tests.
#### Running Security Solution Cypress E2E/integration tests
See [here](./plugins/security_solution/cypress/README.md) for information on running this test suite.

320
x-pack/plugins/security_solution/cypress/README.md
View file

@ -2,190 +2,177 @@
The `security_solution/cypress` directory contains functional UI tests that execute using [Cypress](https://www.cypress.io/).
## Running the tests
There are currently four ways to run the tests, comprised of two execution modes and two target environments, which will be detailed below.
### Execution modes
#### Interactive mode
When you run Cypress in interactive mode, an interactive runner is displayed that allows you to see commands as they execute while also viewing the application under test. For more information, please see [cypress documentation](https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview).
#### Headless mode
A headless browser is a browser simulation program that does not have a user interface. These programs operate like any other browser, but do not display any UI. This is why meanwhile you are executing the tests on this mode you are not going to see the application under test. Just the output of the test is displayed on the terminal once the execution is finished.
### Target environments
#### FTR (CI)
This is the configuration used by CI. It uses the FTR to spawn both a Kibana instance (http://localhost:5620) and an Elasticsearch instance (http://localhost:9220) with a preloaded minimum set of data (see preceding "Test data" section), and then executes cypress against this stack. You can find this configuration in `x-pack/test/security_solution_cypress`
#### Custom Targets
This configuration runs cypress tests against an arbitrary host.
**WARNING**: When using your own instances you need to take into account that if you already have data on it, the tests may fail, as well as, they can put your instances in an undesired state, since our tests uses es_archive to populate data.
### Test Execution: Examples
#### FTR + Headless
Since this is how tests are run on CI, this will likely be the configuration you want to reproduce failures locally, etc.
```shell
# bootstrap kibana from the project root
yarn kbn bootstrap
# build the plugins/assets that cypress will execute against
node scripts/build_kibana_platform_plugins
# launch the cypress test runner
cd x-pack/plugins/security_solution
yarn cypress:run-as-ci
```
#### FTR + Interactive
This is the preferred mode for developing new tests.
```shell
# bootstrap kibana from the project root
yarn kbn bootstrap
# build the plugins/assets that cypress will execute against
node scripts/build_kibana_platform_plugins
# launch the cypress test runner
cd x-pack/plugins/security_solution
yarn cypress:open-as-ci
```
#### Custom Target + Headless
This mode may be useful for testing a release, e.g. spin up a build candidate
and point cypress at it to catch regressions.
```shell
# bootstrap kibana from the project root
yarn kbn bootstrap
# load auditbeat data needed for test execution (which FTR normally does for us)
cd x-pack/plugins/security_solution
node ../../../scripts/es_archiver load auditbeat --dir ../../test/security_solution_cypress/es_archives --config ../../../test/functional/config.js --es-url http(s)://<username>:<password>@<elsUrl> --kibana-url http(s)://<userName>:<password>@<kbnUrl>
# launch the cypress test runner with overridden environment variables
cd x-pack/plugins/security_solution
CYPRESS_BASE_URL=http(s)://<username>:<password>@<kbnUrl> CYPRESS_ELASTICSEARCH_URL=http(s)://<username>:<password>@<elsUrl> CYPRESS_ELASTICSEARCH_USERNAME=<username> CYPRESS_ELASTICSEARCH_PASSWORD=password yarn cypress:run
```
## Folder Structure
### Fixtures (Cypress native folder)
### integration/
Fixtures are used as external pieces of static data when we stub responses.
Cypress convention. Contains the specs that are going to be executed.
### Integration (Cypress native folder)
### fixtures/
Contains the specs that are going to be executed.
Cypress convention. Fixtures are used as external pieces of static data when we stub responses.
### Objects
### plugins/
Objects are a representation of data used accross different tests.
Cypress convention. As a convenience, by default Cypress will automatically include the plugins file cypress/plugins/index.js before every single spec file it runs.
### Pluggins (Cypress native folder)
### objects/
By default Cypress will automatically include the plugins file cypress/plugins/index.js before every single spec file it runs. They do this purely as a convenience mechanism so you don’t have to import this that in every single one of your spec files.
Contains representations of data used across different tests; our domain objects.
### Screens
### screens/
In _screens_ folder we are going to find all the elements we want to interact in our tests.
Contains the elements we want to interact with in our tests.
Each file inside the tasks folder represents a screen of our application. When the screens are complex i.e. Hosts contains multiple tabs, the page is represented by a folder and the different important parts are represented by files.
Each file inside the screens folder represents a screen in our application. When the screens are complex, e.g. Hosts with its multiple tabs, the page is represented by a folder and the different important parts are represented by files.
i.e.
- tasks
Example:
- screens
- hosts
- all_hosts.ts
- authentications.ts
- events.ts
- main.ts
- uncommon_processes.ts
### Tasks
_Tasks_ are functions that my be re-used across tests.
Each file inside the tasks folder represents a screen of our application. When the screens are complex i.e. Hosts contains multiple tabs, the page is represented by a folder and the different important parts are represented by files.
i.e.
- tasks
- hosts
- all_hosts.ts
- authentications.ts
- authentications.ts
- events.ts
- main.ts
- uncommon_processes.ts
### URLs
### tasks/
_Tasks_ are functions that may be reused across tests.
Each file inside the tasks folder represents a screen of our application. When the screens are complex, e.g. Hosts with its multiple tabs, the page is represented by a folder and the different important parts are represented by files.
Example:
- tasks
- hosts
- all_hosts.ts
- authentications.ts
- events.ts
- main.ts
- uncommon_processes.ts
### urls/
Represents all the URLs used during the tests execution.
## Test data
The data the tests need:
- Is generated on the fly using our application APIs (preferred way)
- Is ingested on the ELS instance using es_archive
By default when running the tests on Jenkins mode a base set of data is ingested on the ELS instance: an empty kibana index and a set of auditbeat data (the `empty_kibana` and `auditbeat` archives, respectively). This is usually enough to cover most of the scenarios that we are testing.
- Is generated on the fly using our application APIs (preferred way)
- Is ingested on the ELS instance using the `es_archive` utility
By default, when running the tests in Jenkins mode, a base set of data is ingested on the ELS instance: an empty kibana index and a set of auditbeat data (the `empty_kibana` and `auditbeat` archives, respectively). This is usually enough to cover most of the scenarios that we are testing.
### How to generate a new archive
We are using es_archiver in order to manage the data that our Cypress tests needs.
**Note:** As mentioned above, archives are only meant to contain external data, e.g. beats data. Due to the tendency for archived domain objects (rules, signals) to quickly become out of date, it is strongly suggested that you generate this data within the test, through interaction with either the UI or the API.
1. Setup if possible a clean instance of kibana and elasticsearch (if not, possible please try to clean the data that you are going to generate).
We use es_archiver to manage the data that our Cypress tests need.
1. Set up a clean instance of kibana and elasticsearch (if this is not possible, try to clean/minimize the data that you are going to archive).
2. With the kibana and elasticsearch instance up and running, create the data that you need for your test.
3. When you are sure that you have all the data you need run the following command from: `x-pack/plugins/security_solution`
```sh
```sh
node ../../../scripts/es_archiver save <nameOfTheFolderWhereDataIsSaved> <indexPatternsToBeSaved> --dir ../../test/security_solution_cypress/es_archives --config ../../../test/functional/config.js --es-url http://<elasticsearchUsername>:<elasticsearchPassword>@<elasticsearchHost>:<elasticsearchPort>
```
Example:
Example:
```sh
node ../../../scripts/es_archiver save custom_rules ".kibana",".siem-signal*" --dir ../../test/security_solution_cypress/es_archives --config ../../../test/functional/config.js --es-url http://elastic:changeme@localhost:9220
```
Note that the command is going to create the folder if does not exist in the directory with the imported data.
Note that the command will create the folder if it does not exist.
## Running the tests
You can run the tests in interactive or headless mode, emulating the Jenkins pipeline or using your own instances.
### Interactive vs Headless mode
#### Interactive
When you run the Cypress on interactive mode, an interactive runner is displayed that allows you to see commands as they execute while also viewing the application under test.
For more information, please visit: https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview
#### Headless mode
A headless browser is a browser simulation program that does not have a user interface. These programs operate like any other browser, but do not display any UI. This is why meanwhile you are executing the tests on this mode you are not going to see the application under test. Just the output of the test is displayed on the terminal once the execution is finished.
### Emulating Jenkins vs your own instances
#### Emulating Jenkins
With this mode we use the FTR to run the Cypress tests and automatically, a Kibana instance (http://localhost:5620) and Elastic Search instance (http://localhost:9220) with a preloaded minimum set of data.
You can find the configuration of this mode in `x-pack/test/security_solution_cypress`
#### Your own instances
When using your own instances you need to take into account that if you already have data on it, the tests may fail, as well as, they can put your instances in an undesired state, since our tests uses es_archive to populate data.
### Running Cypress in Headless mode as a Jenkins execution (The preferred way when running regressions on your machine)
1. First bootstrap kibana changes from the Kibana root directory:
```sh
yarn kbn bootstrap
```
2. Build the plugins
```sh
node scripts/build_kibana_platform_plugins
```
3. Launch Cypress command line test runner:
```sh
cd x-pack/plugins/security_solution
yarn cypress:run-as-ci
```
As explained previously, this type of execution you don't need to have running a kibana and elasticsearch instance. This is because the command, as it would happen in the CI, will launch the instances. The elasticsearch instance will be fed data found in: `x-pack/test/security_solution_cypress/es_archives`
### Running Cypress in Interactive mode as a Jenkins execution (The preferred way when developing new cypress tests)
1. First bootstrap kibana changes from the Kibana root directory:
```sh
yarn kbn bootstrap
```
2. Build the plugins
```sh
node scripts/build_kibana_platform_plugins
```
3. Launch Cypress command line test runner:
```sh
cd x-pack/plugins/security_solution
yarn cypress:open-as-ci
```
As explained previously, this type of execution you don't need to have running a kibana and elasticsearch instance. This is because the command, as it would happen in the CI, will launch the instances. The elasticsearch instance will be fed data found in: `x-pack/test/security_solution_cypress/es_archives`
### Running Cypress in your own instances (Recommended just for releases regressions)
1. First bootstrap kibana changes from the Kibana root directory:
```sh
yarn kbn bootstrap
```
2. Load the initial auditbeat set of data needed for the test execution:
```sh
cd x-pack/plugins/security_solution
node ../../../scripts/es_archiver load auditbeat --dir ../../test/security_solution_cypress/es_archives --config ../../../test/functional/config.js --es-url http(s)://<username>:<password>@<elsUrl> --kibana-url http(s)://<userName>:<password>@<kbnUrl>
```
3. Launch Cypress overriden some of the environment variables:
```sh
CYPRESS_BASE_URL=http(s)://<username>:<password>@<kbnUrl> CYPRESS_ELASTICSEARCH_URL=http(s)://<username>:<password>@<elsUrl> CYPRESS_ELASTICSEARCH_USERNAME=<username> CYPRESS_ELASTICSEARCH_PASSWORD=password yarn cypress:run
```
## Best Practices
## Development Best Practices
### Clean up the state
Remember to use the `cleanKibana` method before starting the execution of the test
Remember to clean up the state of the test after its execution, typically with the `cleanKibana` function. Be mindful of failure scenarios, as well: if your test fails, will it leave the environment in a recoverable state?
### Minimize the use of es_archive
When possible, create all the data that you need for executing the tests using the application APIS.
When possible, create all the data that you need for executing the tests using the application APIS or the UI.
### Speed up test execution time
@ -194,54 +181,25 @@ taken into consideration until another solution is implemented:
- Group the tests that are similar in different contexts.
- For every context login only once, clean the state between tests if needed without re-loading the page.
- All tests in a spec file must be order-independent.
- All tests in a spec file must be order-independent.
Remember that minimizing the number of times the web page is loaded, we minimize as well the execution time.
## Reporting
## Test Artifacts
When Cypress tests are run on the command line via non visual mode
reporting artifacts are generated under the `target` directory in the root
of the Kibana, as detailed for each artifact type in the sections below.
When Cypress tests are run headless on the command line, artifacts
are generated under the `target` directory in the root of Kibana as follows:
### HTML Reports
- HTML Reports
- location: `target/kibana-security-solution/cypress/results/output.html`
- `junit` Reports
- location: `target/kibana-security-solution/cypress/results`
- Screenshots (of failed tests)
- location: `target/kibana-security-solution/cypress/screenshots`
- Videos
- disabled by default, can be enabled by setting env var `CYPRESS_video=true`
- location: `target/kibana-security-solution/cypress/videos`
An HTML report (e.g. for email notifications) is output to:
```
target/kibana-security-solution/cypress/results/output.html
```
### Screenshots
Screenshots of failed tests are output to:
```
target/kibana-security-solution/cypress/screenshots
```
### `junit` Reports
The Kibana CI process reports `junit` test results from the `target/junit` directory.
Cypress `junit` reports are generated in `target/kibana-security-solution/cypress/results`
and copied to the `target/junit` directory.
### Videos (optional)
Videos are disabled by default, but can optionally be enabled by setting the
`CYPRESS_video=true` environment variable:
```
CYPRESS_video=true yarn cypress:run
```
Videos are (optionally) output to:
```
target/kibana-security-solution/cypress/videos
```
## Linting
## Linting
Optional linting rules for Cypress and linting setup can be found [here](https://github.com/cypress-io/eslint-plugin-cypress#usage)