mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 01:38:56 -04:00
[Security Solution] Create test plan for prebuilt rule installation and upgrade workflows (#160685)
## Summary Combines the two following test plans into one: - [Test plan for the legacy workflows of installing/upgrading prebuilt rules](https://docs.google.com/document/d/1d_1DYnHlnCaPznWTjeCxhoaRUwxc2O_V0LToAPG0xLE/edit) - this google doc is deprecated and will be replaced by the consolidated test plan created in this PR - [Test plan for the new workflows of installing/upgrading prebuilt rules](https://docs.google.com/document/d/1cYvwtpzk0uLn5R88BlCX-fRIwR2n_NoqkSjIXkY9k34/edit) - this google doc is deprecated and will be replaced by the consolidated test plan created in this PR Adds a template for writing new test plans. --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Juan Pablo Djeredjian
GitHub
parent
8ef1287017
commit
e379f0a97d
4 changed files with 712 additions and 3 deletions
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
This folder contains test plans for the features of Security Solution.
|
||||
|
||||
## Folder Structure
|
||||
## Folder structure
|
||||
|
||||
The folder is first split into major Security Solution domains:
|
||||
|
||||
|
|
@ -28,6 +28,15 @@ Within each subdomain, you can organize test plans as you like, for example:
|
|||
- you might want to have a folder per feature, if your features are large and you have multiple test plans per feature
|
||||
- or you might want to have a plain list of test plans if features are relatively small
|
||||
|
||||
## Ownership
|
||||
## Folder ownership
|
||||
|
||||
Each subdomain folder should be owned by a single GitHub team in the `.github/CODEOWNERS` file.
|
||||
|
||||
## Test plan structure
|
||||
|
||||
Some examples for reference:
|
||||
|
||||
- [Test plan template](./test_plan_template.md).
|
||||
- [Installation and Upgrade of Prebuilt Rules](./detection_response/prebuilt_rules/installation_and_upgrade.md).
|
||||
|
||||
Feel free to tune the structure whenever it makes sense and improves readability or maintainability of your plan: add more sections to `Useful info`, add more top-level sections in addition to `Useful info` and `Scenarios`, etc.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,600 @@
|
|||
# Installation and Upgrade of Prebuilt Rules
|
||||
|
||||
This is a test plan for the workflows of installing and upgrading prebuilt rules.
|
||||
|
||||
Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic. It does not cover any past functionality that was removed or functionality to be implemented in the future. The plan is about to change in the future Milestones.
|
||||
|
||||
## Useful information
|
||||
|
||||
### Tickets
|
||||
|
||||
- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic
|
||||
- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176)
|
||||
- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192)
|
||||
- [Document the new UI for installing and upgrading prebuilt detection rules](https://github.com/elastic/security-docs/issues/3496)
|
||||
|
||||
### Terminology
|
||||
|
||||
- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**.
|
||||
|
||||
- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects).
|
||||
|
||||
- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet.
|
||||
|
||||
- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_.
|
||||
|
||||
- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment.
|
||||
|
||||
- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action.
|
||||
- CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page.
|
||||
- CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page.
|
||||
|
||||
### Assumptions
|
||||
|
||||
- Below scenarios only apply to prebuilt detection rules.
|
||||
- Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher.
|
||||
- EPR is available for fetching the package unless explicitly indicated otherwise.
|
||||
- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored.
|
||||
|
||||
### Non-functional requirements
|
||||
|
||||
- Notifications, rule installation and rule upgrade workflows should work:
|
||||
- regardless of the package type: with historical rule versions or without;
|
||||
- regardless of the package registry availability: i.e., they should also work in air-gapped environments.
|
||||
- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet.
|
||||
- Kibana should not crash with Out Of Memory exception during package installation.
|
||||
- For test purposes, it should be possible to use detection rules package versions lower than the latest.
|
||||
|
||||
## Scenarios
|
||||
|
||||
### Package installation
|
||||
|
||||
#### **Scenario: Package is installed via Fleet**
|
||||
|
||||
**Automation**: 2 e2e tests that install the real package.
|
||||
|
||||
```Gherkin
|
||||
Given the package is not installed
|
||||
When user opens the Rule Management page
|
||||
Then the package gets installed in the background from EPR
|
||||
```
|
||||
|
||||
#### **Scenario: Package is installed via bundled Fleet package in Kibana**
|
||||
|
||||
**Automation**: 1 integration test.
|
||||
|
||||
```Gherkin
|
||||
Given the package is not installed
|
||||
And user is in an air-gapped environment
|
||||
When user opens the Rule Management page
|
||||
Then the package gets installed in the background from packages bundled into Kibana
|
||||
```
|
||||
|
||||
#### **Scenario: Large package can be installed on a small Kibana instance**
|
||||
|
||||
**Automation**: 1 integration test.
|
||||
|
||||
```Gherkin
|
||||
Given the package is not installed
|
||||
And the package contains the largest amount of historical rule versions (15000)
|
||||
And the Kibana instance has a memory heap size of 700 Mb (see note below)
|
||||
When user opens the Rule Management page
|
||||
Then the package is installed without Kibana crashing with an Out Of Memory error
|
||||
```
|
||||
|
||||
**Note**: 600 Mb seems to always crash Kibana with an OOM error. 700 Mb runs with no issues in the Flaky test runner with 100 iterations: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2215.
|
||||
|
||||
### Rule installation and upgrade via the Prebuilt rules API
|
||||
|
||||
There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions.
|
||||
|
||||
#### **Scenario: API can install all prebuilt rules**
|
||||
|
||||
**Automation**: 8 integration tests with mock rules: 4 examples below * 2 (we split checking API response and installed rules into two different tests).
|
||||
|
||||
```Gherkin
|
||||
Given the package <package_type> is installed
|
||||
And the package contains N rules
|
||||
When user installs all rules via install <api>
|
||||
Then the endpoint should return 200 with <install_response>
|
||||
And N rule objects should be created
|
||||
And each rule object should have correct id and version
|
||||
|
||||
Examples:
|
||||
| package_type | api | install_response |
|
||||
| with historical versions | legacy | installed: N, updated: 0 |
|
||||
| w/o historical versions | legacy | installed: N, updated: 0 |
|
||||
| with historical versions | new | total: N, succeeded: N |
|
||||
| w/o historical versions | new | total: N, succeeded: N |
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Legacy API:
|
||||
- install: `PUT /api/detection_engine/rules/prepackaged`
|
||||
- New API:
|
||||
- install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform`
|
||||
|
||||
#### **Scenario: API can install prebuilt rules that are not yet installed**
|
||||
|
||||
**Automation**: 4 integration tests with mock rules.
|
||||
|
||||
```Gherkin
|
||||
Given the package <package_type> is installed
|
||||
And the package contains N rules
|
||||
When user installs all rules via install <api>
|
||||
And deletes one of the installed rules
|
||||
And gets prebuilt rules status via status <api>
|
||||
Then the endpoint should return 200 with <status_response>
|
||||
When user installs all rules via install <api> again
|
||||
Then the endpoint should return 200 with <install_response>
|
||||
|
||||
Examples:
|
||||
| package_type | api | status_response | install_response |
|
||||
| with historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 |
|
||||
| w/o historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 |
|
||||
| with historical versions | new | to_install: 1 | total: 1, succeeded: 1 |
|
||||
| w/o historical versions | new | to_install: 1 | total: 1, succeeded: 1 |
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Legacy API:
|
||||
- install: `PUT /api/detection_engine/rules/prepackaged`
|
||||
- status: `GET /api/detection_engine/rules/prepackaged/_status`
|
||||
- New API:
|
||||
- install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform`
|
||||
- status: `GET /internal/detection_engine/prebuilt_rules/status`
|
||||
|
||||
#### **Scenario: API can upgrade prebuilt rules that are outdated**
|
||||
|
||||
**Automation**: 4 integration tests with mock rules.
|
||||
|
||||
```Gherkin
|
||||
Given the package <package_type> is installed
|
||||
And the package contains N rules
|
||||
When user installs all rules via install <api>
|
||||
And new X+1 version of a rule asset <assets_update>
|
||||
And user gets prebuilt rules status via status <api>
|
||||
Then the endpoint should return 200 with <status_response>
|
||||
When user upgrades all rules via upgrade <api>
|
||||
Then the endpoint should return 200 with <upgrade_response>
|
||||
|
||||
Examples:
|
||||
| package_type | api | assets_update | status_response | upgrade_response |
|
||||
| with historical versions | legacy | gets added | not_updated: 1 | installed: 0, updated: 1 |
|
||||
| w/o historical versions | legacy | replaces X | not_updated: 1 | installed: 0, updated: 1 |
|
||||
| with historical versions | new | gets added | to_upgrade: 1 | total: 1, succeeded: 1 |
|
||||
| w/o historical versions | new | replaces X | to_upgrade: 1 | total: 1, succeeded: 1 |
|
||||
```
|
||||
|
||||
TODO: Check why for the legacy API Dmitrii has added 2 integration tests for `rule package with historical versions` instead of 1:
|
||||
|
||||
- `should update outdated prebuilt rules when previous historical versions available`
|
||||
- `should update outdated prebuilt rules when previous historical versions unavailable`
|
||||
|
||||
Notes:
|
||||
|
||||
- Legacy API:
|
||||
- install: `PUT /api/detection_engine/rules/prepackaged`
|
||||
- upgrade: `PUT /api/detection_engine/rules/prepackaged`
|
||||
- status: `GET /api/detection_engine/rules/prepackaged/_status`
|
||||
- New API:
|
||||
- install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform`
|
||||
- upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform`
|
||||
- status: `GET /internal/detection_engine/prebuilt_rules/status`
|
||||
|
||||
#### **Scenario: API does not install or upgrade prebuilt rules if they are up to date**
|
||||
|
||||
**Automation**: 4 integration tests with mock rules.
|
||||
|
||||
```Gherkin
|
||||
Given the package <package_type> is installed
|
||||
And the package contains N rules
|
||||
When user installs all rules via install <api>
|
||||
And user gets prebuilt rules status via status <api>
|
||||
Then the endpoint should return 200 with <status_response>
|
||||
When user calls install <api>
|
||||
Then the endpoint should return 200 with <install_response>
|
||||
When user calls upgrade <api>
|
||||
Then the endpoint should return 200 with <upgrade_response>
|
||||
|
||||
Examples:
|
||||
| package_type | api | status_response | install_response | upgrade_response |
|
||||
| with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 |
|
||||
| w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 |
|
||||
| with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 |
|
||||
| w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 |
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Legacy API:
|
||||
- install: `PUT /api/detection_engine/rules/prepackaged`
|
||||
- upgrade: `PUT /api/detection_engine/rules/prepackaged`
|
||||
- status: `GET /api/detection_engine/rules/prepackaged/_status`
|
||||
- New API:
|
||||
- install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform`
|
||||
- upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform`
|
||||
- status: `GET /internal/detection_engine/prebuilt_rules/status`
|
||||
|
||||
### Scenarios for the real package
|
||||
|
||||
#### **Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new pckage**
|
||||
|
||||
**Automation**: 1 integration test with real packages.
|
||||
|
||||
```Gherkin
|
||||
Given there are two package versions: N-1 and N
|
||||
And the package of N-1 version is installed
|
||||
When user calls the status endpoint
|
||||
Then it should return a 200 response with some number of rules to install and 0 rules to upgrade
|
||||
When user calls the installation/_review endpoint
|
||||
Then it should return a 200 response matching the response of the status endpoint
|
||||
When user calls the installation/_perform_ endpoint
|
||||
Then it should return a 200 response matching the response of the status endpoint
|
||||
And rules returned in this response should exist as alert saved objects
|
||||
When user installs the package of N version
|
||||
Then it should be installed successfully
|
||||
When user calls the status endpoint
|
||||
Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade
|
||||
When user calls the installation/_review endpoint
|
||||
Then it should return a 200 response matching the response of the status endpoint
|
||||
When user calls the installation/_perform_ endpoint
|
||||
Then rules returned in this response should exist as alert saved objects
|
||||
When user calls the upgrade/_review endpoint
|
||||
Then it should return a 200 response matching the response of the status endpoint
|
||||
When user calls the upgrade/_perform_ endpoint
|
||||
Then rules returned in this response should exist as alert saved objects
|
||||
```
|
||||
|
||||
### Rule installation and upgrade notifications on the Rule Management page
|
||||
|
||||
#### **Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given no prebuilt rules are installed in Kibana
|
||||
And there are X prebuilt rules available to install
|
||||
When user opens the Rule Management page
|
||||
Then user should see a CTA to install prebuilt rules
|
||||
And user should see a number of rules available to install (X)
|
||||
And user should NOT see a CTA to upgrade prebuilt rules
|
||||
And user should NOT see a number of rules available to upgrade
|
||||
And user should NOT see the Rule Updates table
|
||||
```
|
||||
|
||||
#### **Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given no prebuilt rules are installed in Kibana
|
||||
And no prebuilt rule assets exist
|
||||
When user opens the Rule Management page
|
||||
Then user should NOT see a CTA to install prebuilt rules
|
||||
And user should NOT see a number of rules available to install
|
||||
And user should NOT see a CTA to upgrade prebuilt rules
|
||||
And user should NOT see a number of rules available to upgrade
|
||||
And user should NOT see the Rule Updates table
|
||||
```
|
||||
|
||||
#### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given all the latest prebuilt rules are installed in Kibana
|
||||
When user opens the Rule Management page
|
||||
Then user should NOT see a CTA to install prebuilt rules
|
||||
And user should NOT see a number of rules available to install
|
||||
And user should NOT see a CTA to upgrade prebuilt rules
|
||||
And user should NOT see a number of rules available to upgrade
|
||||
And user should NOT see the Rule Updates table
|
||||
```
|
||||
|
||||
#### **Scenario: User is notified when some prebuilt rules can be installed**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And there are Y more prebuilt rules available to install
|
||||
And for all X installed rules there are no new versions available
|
||||
When user opens the Rule Management page
|
||||
Then user should see a CTA to install prebuilt rules
|
||||
And user should see the number of rules available to install (Y)
|
||||
And user should NOT see a CTA to upgrade prebuilt rules
|
||||
And user should NOT see a number of rules available to upgrade
|
||||
And user should NOT see the Rule Updates table
|
||||
```
|
||||
|
||||
#### **Scenario: User is notified when some prebuilt rules can be upgraded**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And there are no more prebuilt rules available to install
|
||||
And for Z of the installed rules there are new versions available
|
||||
When user opens the Rule Management page
|
||||
Then user should NOT see a CTA to install prebuilt rules
|
||||
And user should NOT see a number of rules available to install
|
||||
And user should see a CTA to upgrade prebuilt rules
|
||||
And user should see the number of rules available to upgrade (Z)
|
||||
And user should see the Rule Updates table
|
||||
```
|
||||
|
||||
#### **Scenario: User is notified when both rules to install and upgrade are available**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And there are Y more prebuilt rules available to install
|
||||
And for Z of the installed rules there are new versions available
|
||||
When user opens the Rule Management page
|
||||
Then user should see a CTA to install prebuilt rules
|
||||
And user should see the number of rules available to install (Y)
|
||||
And user should see a CTA to upgrade prebuilt rules
|
||||
And user should see the number of rules available to upgrade (Z)
|
||||
And user should see the Rule Updates table
|
||||
```
|
||||
|
||||
#### **Scenario: User is notified after a prebuilt rule gets deleted**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And there are no more prebuilt rules available to install
|
||||
When user opens the Rule Management page
|
||||
And user deletes Y prebuilt rules
|
||||
Then user should see a CTA to install prebuilt rules
|
||||
And user should see the number of rules available to install (Y)
|
||||
```
|
||||
|
||||
### Rule installation workflow: base cases
|
||||
|
||||
#### **Scenario: User can install prebuilt rules one by one**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/* endpoints in integration.
|
||||
|
||||
```Gherkin
|
||||
Given no prebuilt rules are installed in Kibana
|
||||
And there are X prebuilt rules available to install
|
||||
When user opens the Add Rules page
|
||||
Then prebuilt rules available for installation should be displayed in the table
|
||||
When user installs one individual rule
|
||||
Then success message should be displayed after installation
|
||||
And the installed rule should be removed from the table
|
||||
When user navigates back to the Rule Management page
|
||||
Then user should see a CTA to install prebuilt rules
|
||||
And user should see the number of rules available to install decreased by 1
|
||||
```
|
||||
|
||||
#### **Scenario: User can install multiple prebuilt rules selected on the page**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/* endpoints in integration.
|
||||
|
||||
```Gherkin
|
||||
Given no prebuilt rules are installed in Kibana
|
||||
And there are X prebuilt rules available to install
|
||||
When user opens the Add Rules page
|
||||
Then prebuilt rules available for installation should be displayed in the table
|
||||
When user selects <Y> rules
|
||||
Then user should see a CTA to install <Y> number of rules
|
||||
When user clicks the CTA
|
||||
Then success message should be displayed after installation
|
||||
And all the installed rules should be removed from the table
|
||||
When user navigates back to the Rule Management page
|
||||
Then user should see a CTA to install prebuilt rules
|
||||
And user should see the number of rules available to install decreased by <Y> number of installed rules
|
||||
|
||||
Examples:
|
||||
| Y |
|
||||
| a few rules on the page, e.g. 2 |
|
||||
| all rules on the page, e.g. 12 |
|
||||
```
|
||||
|
||||
#### **Scenario: User can install all available prebuilt rules at once**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/* endpoints in integration.
|
||||
|
||||
```Gherkin
|
||||
Given no prebuilt rules are installed in Kibana
|
||||
And there are X prebuilt rules available to install
|
||||
When user opens the Add Rules page
|
||||
Then prebuilt rules available for installation should be displayed in the table
|
||||
When user installs all rules
|
||||
Then success message should be displayed after installation
|
||||
And all the rules should be removed from the table
|
||||
And user should see a message indicating that all available rules have been installed
|
||||
And user should see a CTA that leads to the Rule Management page
|
||||
When user clicks on the CTA
|
||||
Then user should be navigated back to Rule Management page
|
||||
And user should NOT see a CTA to install prebuilt rules
|
||||
And user should NOT see a number of rules available to install
|
||||
```
|
||||
|
||||
#### **Scenario: Empty screen is shown when all prebuilt rules are installed**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 1 integration test.
|
||||
|
||||
```Gherkin
|
||||
Given all the available prebuilt rules are installed in Kibana
|
||||
When user opens the Add Rules page
|
||||
Then user should see a message indicating that all available rules have been installed
|
||||
And user should see a CTA that leads to the Rule Management page
|
||||
```
|
||||
|
||||
### Rule installation workflow: filtering, sorting, pagination
|
||||
|
||||
TODO: add scenarios
|
||||
|
||||
### Rule installation workflow: misc cases
|
||||
|
||||
#### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**
|
||||
|
||||
**Automation**: unit tests.
|
||||
|
||||
```Gherkin
|
||||
Given prebuilt rules package is not installed
|
||||
When user opens the Add Rules page
|
||||
Then user should see a loading skeleton until the package installation is completed
|
||||
```
|
||||
|
||||
#### **Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**
|
||||
|
||||
**Automation**: 1 e2e test.
|
||||
|
||||
```Gherkin
|
||||
Given user is on the Add Rules page
|
||||
When user navigates to the Rule Management page via breadcrumbs
|
||||
Then the Rule Management page should be displayed
|
||||
```
|
||||
|
||||
### Rule upgrade workflow: base cases
|
||||
|
||||
#### **Scenario: User can upgrade prebuilt rules one by one**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/* endpoints in integration.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And for Y of the installed rules there are new versions available
|
||||
And user is on the Rule Management page
|
||||
When user opens the Rule Updates table
|
||||
Then Y rules available for upgrade should be displayed in the table
|
||||
When user upgrades one individual rule
|
||||
Then success message should be displayed after upgrade
|
||||
And the upgraded rule should be removed from the table
|
||||
And user should see the number of rules available to upgrade decreased by 1
|
||||
```
|
||||
|
||||
#### **Scenario: User can upgrade multiple prebuilt rules selected on the page**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/* endpoints in integration.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And for Y of the installed rules there are new versions available
|
||||
And user is on the Rule Management page
|
||||
When user opens the Rule Updates table
|
||||
Then Y rules available for upgrade should be displayed in the table
|
||||
When user selects <Z> rules
|
||||
Then user should see a CTA to upgrade <Z> number of rules
|
||||
When user clicks the CTA
|
||||
Then success message should be displayed after upgrade
|
||||
And all the <Z> upgraded rules should be removed from the table
|
||||
And user should see the number of rules available to upgrade decreased by <Z> number of upgraded rules
|
||||
|
||||
Examples:
|
||||
| Z |
|
||||
| a few rules on the page, e.g. 2 |
|
||||
| all rules on the page, e.g. 12 |
|
||||
```
|
||||
|
||||
#### **Scenario: User can upgrade all available prebuilt rules at once**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/* endpoints in integration.
|
||||
|
||||
```Gherkin
|
||||
Given X prebuilt rules are installed in Kibana
|
||||
And for Y of the installed rules there are new versions available
|
||||
And user is on the Rule Management page
|
||||
When user opens the Rule Updates table
|
||||
Then Y rules available for upgrade should be displayed in the table
|
||||
When user upgrades all rules
|
||||
Then success message should be displayed after upgrade
|
||||
And user should NOT see a CTA to upgrade prebuilt rules
|
||||
And user should NOT see a number of rules available to upgrade
|
||||
And user should NOT see the Rule Updates table
|
||||
```
|
||||
|
||||
### Rule upgrade workflow: filtering, sorting, pagination
|
||||
|
||||
TODO: add scenarios
|
||||
|
||||
### Rule upgrade workflow: misc cases
|
||||
|
||||
#### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed**
|
||||
|
||||
**Automation**: unit tests.
|
||||
|
||||
```Gherkin
|
||||
Given prebuilt rules package is not installed
|
||||
When user opens the Rule Management page
|
||||
Then user should NOT see the Rule Updates tab until the package installation is completed and there are rules available for upgrade
|
||||
```
|
||||
|
||||
### Error handling
|
||||
|
||||
#### **Scenario: Error is handled when any operation on prebuilt rules fails**
|
||||
|
||||
**Automation**: unit tests.
|
||||
|
||||
```Gherkin
|
||||
When user is <operation> prebuilt rules
|
||||
And this operation fails
|
||||
Then user should see an error message
|
||||
|
||||
Examples:
|
||||
| operation |
|
||||
| installing all |
|
||||
| installing selected |
|
||||
| installing individual |
|
||||
| upgrading all |
|
||||
| upgrading selected |
|
||||
| upgrading individual |
|
||||
```
|
||||
|
||||
### Authorization / RBAC
|
||||
|
||||
#### **Scenario: User with read privileges on Security Solution cannot install prebuilt rules**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and installation endpoints.
|
||||
|
||||
```Gherkin
|
||||
Given user with "Security: read" privileges on Security Solution
|
||||
And no prebuilt rules are installed in Kibana
|
||||
And there are prebuilt rules available to install
|
||||
When user opens the Add Rules page
|
||||
Then user should see prebuilt rules available to install
|
||||
But user should not be able to install them
|
||||
```
|
||||
|
||||
#### **Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**
|
||||
|
||||
**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints.
|
||||
|
||||
```Gherkin
|
||||
Given user with "Security: read" privileges on Security Solution
|
||||
And X prebuilt rules are installed in Kibana
|
||||
And for Y of the installed rules there are new versions available
|
||||
When user opens the Rule Management page
|
||||
And user opens the Rule Updates table
|
||||
Then user should see prebuilt rules available to upgrade
|
||||
But user should not be able to upgrade them
|
||||
```
|
||||
|
||||
### Kibana upgrade
|
||||
|
||||
#### **Scenario: User can use prebuilt rules after upgrading Kibana from version A to B**
|
||||
|
||||
**Automation**: not automated, manual testing required.
|
||||
|
||||
```Gherkin
|
||||
Given user is upgrading Kibana from version <A> to version <B>
|
||||
And the <A> instance contains already installed prebuilt rules
|
||||
When the upgrade is complete
|
||||
Then user should be able to install new prebuilt rules
|
||||
And delete installed prebuilt rules
|
||||
And upgrade installed prebuilt rules that have newer versions in <B>
|
||||
|
||||
Examples:
|
||||
| A | B |
|
||||
| 8.7 | 8.9.0 |
|
||||
| 7.17.x | 8.9.0 |
|
||||
```
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add test plans.
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
# Awesome Feature
|
||||
|
||||
<!-- Elaborate on what are you testing here, explain what the Awesome Feature is about. -->
|
||||
This is a test plan for ...
|
||||
|
||||
<!-- Add any additional info, e.g. are you expecting to finalize it in your PR, or later. -->
|
||||
Status: `in progress`. <!-- `in progress` | `done` -->
|
||||
|
||||
## Useful information
|
||||
|
||||
### Tickets
|
||||
|
||||
<!-- Add links to any related tickets. -->
|
||||
|
||||
- [Awesome Feature](https://github.com/elastic/security-team/issues/9999) epic
|
||||
- [Add tests for the new awesome feature](https://github.com/elastic/kibana/issues/999999)
|
||||
- [Document the new awesome feature](https://github.com/elastic/security-docs/issues/9999)
|
||||
|
||||
### Terminology
|
||||
|
||||
<!--
|
||||
Explain special terminology around the feature.
|
||||
This would allow you to write more concise scenarios, which would improve readability.
|
||||
-->
|
||||
|
||||
- **Term 1**: explanation.
|
||||
|
||||
- **Term 2**: explanation.
|
||||
|
||||
### Assumptions
|
||||
|
||||
<!--
|
||||
Mention any assumptions for the scenarios that are not explicitly stated in their steps.
|
||||
For example, you could describe:
|
||||
- license requirements: the feature is available under the Basic license
|
||||
- RBAC requirements: user should have certain privileges to normally access the feature
|
||||
- data setup: user should have certain saved objects, source events, alerts, etc in the system
|
||||
-->
|
||||
|
||||
- Assumption 1.
|
||||
- Assumption 2.
|
||||
|
||||
### Non-functional requirements
|
||||
|
||||
<!--
|
||||
Describe any non-function requirements for the feature, if you have any. These could be about:
|
||||
- existence or lack of any data in Elasticsearch
|
||||
- scale: size of data (number or size of objects), number of Elasticsearch or Kibana nodes, etc
|
||||
- performance
|
||||
- resilience and error handling
|
||||
- observability: APM instrumentation, console logging, event log, correlation ids
|
||||
- testing
|
||||
-->
|
||||
|
||||
- Requirement 1.
|
||||
- Requirement 2.
|
||||
|
||||
## Scenarios
|
||||
|
||||
<!--
|
||||
Add scenarios for the feature. Split them into meaningful sections (groups) of related scenarios.
|
||||
The goal of having sections is to make it easier to navigate the test plan:
|
||||
- there shouldn't be too many sections with few scenarios in each -- it would be hard to see
|
||||
the whole picture of how the feature works
|
||||
- there shouldn't bee too few sections with a lot of scenarios in each
|
||||
|
||||
For example, here's some typical sections you might want to add:
|
||||
- "Core functionality". Happy paths, base use cases, etc. Split it into several sections if
|
||||
there's too many scenarios for it.
|
||||
- "Error handling"
|
||||
- "Authorization / RBAC"
|
||||
- "Kibana upgrade"
|
||||
-->
|
||||
|
||||
### Section 1
|
||||
|
||||
#### **Scenario: Awesome feature works**
|
||||
|
||||
<!-- Describe how are you planning to automate this scenario -->
|
||||
**Automation**: X e2e tests + Y integration tests + unit tests.
|
||||
|
||||
<!-- Use Gherkin syntax to describe the scenario https://cucumber.io/docs/gherkin/ -->
|
||||
```Gherkin
|
||||
Given ...
|
||||
When ...
|
||||
Then ...
|
||||
```
|
||||
|
||||
<!-- Consider adding any other useful notes and clarifications -->
|
||||
|
||||
### Section 2
|
||||
|
||||
#### **Scenario: ?**
|
||||
|
||||
**Automation**: X e2e tests + Y integration tests + unit tests.
|
||||
|
||||
```Gherkin
|
||||
Given ...
|
||||
When ...
|
||||
Then ...
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue