github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-06-27 18:51:07 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Security Solution] Actualize prebuilt rule upgrade test plans (#222606)

Browse source 
**Addresses:** https://github.com/elastic/kibana/issues/202078
**Resolves:** https://github.com/elastic/kibana/issues/166215

## Summary

This PR actualizes prebuilt rule upgrade test plans to correspond to the current feature state.

## Details

The changes are summarized in the following items

- Missing scenarios were added
- https://github.com/elastic/kibana/issues/166215 was addressed
- The wording was fixed to make the test scenarios shorter and focused
- Prebuilt Rules Customization Milestone 2 test scenarios were migrated to a separate test plan `prebuilt_rule_json_diff.md`. The functionality is still relevant and used for rule type changes and under low-tier licenses.
This commit is contained in:
Maxim Palenov 2025-06-24 12:09:52 +02:00 • committed by GitHub
parent 6907186433
commit 1a59438b12
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 996 additions and 1035 deletions
Download patch file Download diff file Expand all files Collapse all files

154
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_installation.md
View file

@ -46,6 +46,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing)
- [**Scenario: Optional tabs and sections without content should be hidden in preview before installing**](#scenario-optional-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing)
- [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination)
- [**Scenario: User can search prebuilt rules by rule name, index pattern or MITRE ATT\&CK™ tactic or technique on the Prebuilt Rules installation page**](#scenario-user-can-search-prebuilt-rules-by-rule-name-index-pattern-or-mitre-attck-tactic-or-technique-on-the-prebuilt-rules-installation-page)
- [**Scenario: User can filter prebuilt rules by tags on the Prebuilt Rules installation page**](#scenario-user-can-filter-prebuilt-rules-by-tags-on-the-prebuilt-rules-installation-page)
- [**Scenario: User can sort prebuilt rules on Prebuilt Rules installation page**](#scenario-user-can-sort-prebuilt-rules-on-prebuilt-rules-installation-page)
- [**Scenario: User can paginate over prebuilt rules on Prebuilt Rules installation page**](#scenario-user-can-paginate-over-prebuilt-rules-on-prebuilt-rules-installation-page)
- [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases)
- [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed)
- [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs)
@ -108,79 +112,6 @@ Previewing properties of a prebuilt rule before installing it:
- If user chooses to preview a prebuilt rule to be installed, we currently show this preview in a flyout.
- In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed.
Examples of rule properties we show in the prebuilt rule preview flyout:
```Gherkin
Examples:
| rule_type | property | tab | section |
│ All rule types │ Author │ Overview │ About │
│ All rule types │ Building block │ Overview │ About │
│ All rule types │ Severity │ Overview │ About │
│ All rule types │ Severity override │ Overview │ About │
│ All rule types │ Risk score │ Overview │ About │
│ All rule types │ Risk score override │ Overview │ About │
│ All rule types │ Reference URLs │ Overview │ About │
│ All rule types │ False positive examples │ Overview │ About │
│ All rule types │ Custom highlighted fields │ Overview │ About │
│ All rule types │ License │ Overview │ About │
│ All rule types │ Rule name override │ Overview │ About │
│ All rule types │ MITRE ATT&CK™ │ Overview │ About │
│ All rule types │ Timestamp override │ Overview │ About │
│ All rule types │ Tags │ Overview │ About │
│ All rule types │ Type │ Overview │ Definition │
│ All rule types │ Related integrations │ Overview │ Definition │
│ All rule types │ Required fields │ Overview │ Definition │
│ All rule types │ Timeline template │ Overview │ Definition │
│ All rule types │ Runs every │ Overview │ Schedule │
│ All rule types │ Additional look-back time │ Overview │ Schedule │
│ All rule types │ Setup guide │ Overview │ Setup guide │
│ All rule types │ Investigation guide │ Investigation guide │ Investigation guide │
│ Custom Query │ Index patterns │ Overview │ Definition │
│ Custom Query │ Data view ID │ Overview │ Definition │
│ Custom Query │ Data view index pattern │ Overview │ Definition │
│ Custom Query │ Custom query │ Overview │ Definition │
│ Custom Query │ Filters │ Overview │ Definition │
│ Custom Query │ Saved query name │ Overview │ Definition │
│ Custom Query │ Saved query filters │ Overview │ Definition │
│ Custom Query │ Saved query │ Overview │ Definition │
│ Custom Query │ Suppress alerts by │ Overview │ Definition │
│ Custom Query │ Suppress alerts for │ Overview │ Definition │
│ Custom Query │ If a suppression field is missing │ Overview │ Definition │
│ Machine Learning │ Anomaly score threshold │ Overview │ Definition │
│ Machine Learning │ Machine Learning job │ Overview │ Definition │
│ Threshold │ Threshold │ Overview │ Definition │
│ Threshold │ Index patterns │ Overview │ Definition │
│ Threshold │ Data view ID │ Overview │ Definition │
│ Threshold │ Data view index pattern │ Overview │ Definition │
│ Threshold │ Custom query │ Overview │ Definition │
│ Threshold │ Filters │ Overview │ Definition │
│ Event Correlation │ EQL query │ Overview │ Definition │
│ Event Correlation │ Filters │ Overview │ Definition │
│ Event Correlation │ Index patterns │ Overview │ Definition │
│ Event Correlation │ Data view ID │ Overview │ Definition │
│ Event Correlation │ Data view index pattern │ Overview │ Definition │
│ Indicator Match │ Indicator index patterns │ Overview │ Definition │
│ Indicator Match │ Indicator mapping │ Overview │ Definition │
│ Indicator Match │ Indicator filters │ Overview │ Definition │
│ Indicator Match │ Indicator index query │ Overview │ Definition │
│ Indicator Match │ Index patterns │ Overview │ Definition │
│ Indicator Match │ Data view ID │ Overview │ Definition │
│ Indicator Match │ Data view index pattern │ Overview │ Definition │
│ Indicator Match │ Custom query │ Overview │ Definition │
│ Indicator Match │ Filters │ Overview │ Definition │
│ New Terms │ Fields │ Overview │ Definition │
│ New Terms │ History Window Size │ Overview │ Definition │
│ New Terms │ Index patterns │ Overview │ Definition │
│ New Terms │ Data view ID │ Overview │ Definition │
│ New Terms │ Data view index pattern │ Overview │ Definition │
│ New Terms │ Custom query │ Overview │ Definition │
│ New Terms │ Filters │ Overview │ Definition │
│ ESQL │ ESQL query │ Overview │ Definition │
│ ESQL │ Suppress alerts by │ Overview │ Definition │
│ ESQL │ Suppress alerts for │ Overview │ Definition │
│ ESQL │ If a suppression field is missing │ Overview │ Definition │
```
## Scenarios
### Rule installation notifications on the Rule Management page
@ -197,7 +128,7 @@ Then user should NOT see a CTA to install prebuilt rules
And user should NOT see a number of rules available to install
And user should NOT see a CTA to upgrade prebuilt rules
And user should NOT see a number of rules available to upgrade
And user should NOT see the Rule Updates table
And user should NOT see the Prebuilt Rules Upgrades page
```
#### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date**
@ -212,7 +143,7 @@ Then user should NOT see a CTA to install prebuilt rules
And user should NOT see a number of rules available to install
And user should NOT see a CTA to upgrade prebuilt rules
And user should NOT see a number of rules available to upgrade
And user should NOT see the Rule Updates table
And user should NOT see the Prebuilt Rules Upgrades page
```
#### **Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**
@ -228,7 +159,7 @@ Then user should see a CTA to install prebuilt rules
And user should see a number of rules available to install (X)
And user should NOT see a CTA to upgrade prebuilt rules
And user should NOT see a number of rules available to upgrade
And user should NOT see the Rule Updates table
And user should NOT see the Prebuilt Rules Upgrades page
```
#### **Scenario: User is notified when some prebuilt rules can be installed**
@ -245,7 +176,7 @@ Then user should see a CTA to install prebuilt rules
And user should see the number of rules available to install (Y)
And user should NOT see a CTA to upgrade prebuilt rules
And user should NOT see a number of rules available to upgrade
And user should NOT see the Rule Updates table
And user should NOT see the Prebuilt Rules Upgrades page
```
#### **Scenario: User is notified when both rules to install and upgrade are available**
@ -418,7 +349,73 @@ And the Investigation Guide tab should NOT be displayed
### Rule installation workflow: filtering, sorting, pagination
TODO: add scenarios https://github.com/elastic/kibana/issues/166215
#### **Scenario: User can search prebuilt rules by rule name, index pattern or MITRE ATT&CK™ tactic or technique on the Prebuilt Rules installation page**
**Automation**: 1 e2e test with mock rules
```Gherkin
Given multiple prebuilt rules available for installation
When user opens the Prebuilt Rules installation page
Then the available prebuilt rules should be shown
When user enters <text> in the search field
Then only the available prebuilt rules matching the <text> should be shown
```
**Examples:**
- `<text>`
- rule name or its part
- index pattern
- MITRE ATT&CK™ tactic or technique
#### **Scenario: User can filter prebuilt rules by tags on the Prebuilt Rules installation page**
**Automation**: 1 e2e test with mock rules
```Gherkin
Given multiple prebuilt rules available for installation
When user opens the Prebuilt Rules installation page
Then the available prebuilt rules should be shown
When user filters the available prebuilt rules by one or more tags
Then only the available prebuilt rules having these tags should be shown
```
#### **Scenario: User can sort prebuilt rules on Prebuilt Rules installation page**
**Automation**: 1 e2e test with mock rules
```Gherkin
Given multiple prebuilt rules available for installation
When user opens the Prebuilt Rules installation page
Then the available prebuilt rules should be shown
When user clicks on <field> header by picking the sorting direction
Then the available prebuilt rules should be sorted by <field> in the expected order
```
**Examples:**
- `<field>`
- rule name
- risk score
- severity
#### **Scenario: User can paginate over prebuilt rules on Prebuilt Rules installation page**
**Automation**: 1 e2e test with mock rules
```Gherkin
Given multiple prebuilt rules available for installation
When user opens the Prebuilt Rules installation page
Then the available prebuilt rules should be shown
When user picks the desired number of <rows_per_page>
Then the <rows_per_page> of the available prebuilt rules should be shown on the page
When user navigates to the next pages
Then the next page of the available prebuilt rules should be shown
```
**Examples:**
`<rows_per_page>` = 5 | 10 | 20 | 50 | 100
### Rule installation workflow: misc cases
@ -504,7 +501,6 @@ Notes:
- install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform`
- status: `GET /internal/detection_engine/prebuilt_rules/status`
#### **Scenario: API does not install prebuilt rules if they are up to date**
**Automation**: 4 integration tests with mock rules.

282
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_json_diff.md Normal file
View file

@ -0,0 +1,282 @@
# Test plan: upgrading prebuilt rules one-by-one with preview <!-- omit from toc -->
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for license agnostic all/per-field rule upgrade JSON diff view.
## Table of contents <!-- omit from toc -->
<!--
Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
-->
[Useful information](#useful-information)
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view)
- [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view)
- [**Scenario: User can see changes when updated rule is a different rule type**](#scenario-user-can-see-changes-when-updated-rule-is-a-different-rule-type)
- [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel)
- [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs)
- [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview)
- [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view)
- [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade)
- [**Scenario: Rule actions and exception lists SHOULDN'T be shown as modified**](#scenario-rule-actions-and-exception-lists-shouldnt-be-shown-as-modified)
- [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview)
- [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview)
- [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified)
- [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default)
- [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically)
## Useful information
### Tickets
- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal)
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
- [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078)
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Technical requirements
Non-functional requirements for the functionality outlined in this test plan.
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
## Scenarios
> These scenarios are applicable to rule type change upgrade and low-license tier.
### Rule upgrade workflow: viewing rule changes in per-field diff view
#### **Scenario: User can see changes in a side-by-side per-field diff view**
**Automation**: 1 e2e test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then the per-field upgrade preview should open
And rule changes should be displayed in a two-column diff view with each field in its own accordion component
And all field diff accordions should be open by default
And correct rule version numbers should be displayed in their respective columns
When the user selects another rule without closing the preview
Then the preview should display the changes for the newly selected rule
```
#### **Scenario: User can see changes when updated rule is a different rule type**
**Automation**: 1 e2e test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then the rule type changes should be displayed in grouped field diffs with corresponding query fields
# When tooltip enhancement is added, this step needs to be added to the corresponding test scenario
And a tooltip is displayed with information about changing rule types
```
#### **Scenario: Field groupings should be rendered together in the same accordion panel**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
The <field> diff accordion panel should display its grouped rule properties
And each property should have its name displayed inside the panel above its value
```
**Examples:**
- `<field>`
- `data_source`
- `kql_query`
- `eql_query`
- `esql_query`
- `threat_query`
- `rule_schedule`
- `rule_name_override`
- `timestamp_override`
- `timeline_template`
- `building_block`
- `threshold`
#### **Scenario: Undefined values are displayed with empty diffs**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then the preview should open
And the old/new field should render an empty panel
```
#### **Scenario: Field diff components have the same grouping and order as in rule details overview**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then the multiple field diff accordions should be sorted in the same order as on the rule details overview tab
And the field diff accordions should be grouped inside its corresponding <section> accordion
And any <section> accordion that doesn't have fields inside it shouldn't be displayed
```
**Examples:**
- `<section>`
- About
- Definition
- Schedule
- Setup Guide
### Rule upgrade workflow: viewing rule changes in JSON diff view
> This section is license agnostic. JSON view is displayed at a separate tab.
#### **Scenario: User can see precisely how property values would change after upgrade**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
And the upgrade preview for this prebuilt rule is shown
Then each line of <column> that was <change_type> should have <bg_color> background
And marked with <line_badge> badge
And each changed word in <column> should be highlighted with <accent_color>
```
**Examples:**
| change_type | column | bg_color | accent_color | line_badge |
| ----------- | -------------- | ---------------- | -------------------- | ---------- |
| updated | Current rule | removed_bg_color | removed_accent_color | - |
| updated | Elastic update | added_bg_color | added_accent_color | + |
| removed | Current rule | removed_bg_color | none | - |
| removed | Elastic update | none | none | none |
| added | Current rule | none | none | none |
| added | Elastic update | added_bg_color | none | + |
#### **Scenario: Rule actions and exception lists SHOULDN'T be shown as modified**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade not including any actions nor exception lists
And user adds actions and an exception list for this rule
When user opens Prebuilt Rule Upgrade Flyout for this rule
Then the JSON diff shouldn't show any modifications to rule's actions or exception list
```
#### **Scenario: Dynamic properties should not be included in preview**
**Automation**: 1 e2e test
```Gherkin
Given a prebuilt rule with an upgrade
And this prebuilt rule has executed at least once
When user opens the upgrade preview
Then the JSON diff shouldn't show any <property> properties on both sides
```
**Examples:**
`<property>` = `property` | `execution_summary` | `enabled`
#### **Scenario: Technical properties should not be included in preview**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then the JSON diff shouldn't show any <technical_property> properties on both sides
```
**Examples:**
`<technical_property>` = `revision` | `updated_at` | `updated_by` | `created_at` | `created_by`
#### **Scenario: Properties with semantically equal values should not be shown as modified**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
And the upgrade has properties with different, but semantically equal values
When user opens the upgrade preview
Then the JSON diff shouldn't show any changes to properties with semantically equal values
```
**Examples:**
- Duration:
- 1h
- 60m
- 3600s
- Empty value:
- no value
- ''
- []
- undefined
- null
#### **Scenario: Unchanged sections of a rule should be hidden by default**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then only the sections of the diff that have changes should be visible
And unchanged sections should be hidden behind a button with a number of unchanged lines
When user clicks on the hidden section button
Then the section should expand and show the unchanged properties
```
#### **Scenario: Properties should be sorted alphabetically**
**Automation**: 1 UI integration test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the upgrade preview
Then visible properties should be sorted alphabetically
When user expands all hidden sections
Then all properties of the rule should be sorted alphabetically
```

591
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md
View file

@ -28,40 +28,45 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews)
- [**Scenario: User can preview prebuilt rules having upgrades**](#scenario-user-can-preview-prebuilt-rules-having-upgrades)
- [**Scenario: User can upgrade a prebuilt rule using the rule preview**](#scenario-user-can-upgrade-a-prebuilt-rule-using-the-rule-preview)
- [**Scenario: User can see correct rule information in the preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-the-preview-before-upgrading)
- [**Scenario: Tabs and sections without content should be hidden in the preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-the-preview-before-upgrading)
- [Rule upgrade field preview](#rule-upgrade-field-preview)
- [Preview non-customized field that has an upgrade (AAB)](#preview-non-customized-field-that-has-an-upgrade-aab)
- [Preview customized field that doesn't have an upgrade (ABA)](#preview-customized-field-that-doesnt-have-an-upgrade-aba)
- [Preview customized field that has a matching upgrade (ABB)](#preview-customized-field-that-has-a-matching-upgrade-abb)
- [Preview customized field that has an upgrade resulting in a solvable conflict (ABC, conflict solvable by diff algo)](#preview-customized-field-that-has-an-upgrade-resulting-in-a-solvable-conflict-abc-conflict-solvable-by-diff-algo)
- [Preview customized field that has an upgrade resulting in a non-solvable conflict (ABC, conflict non-solvable by diff algo)](#preview-customized-field-that-has-an-upgrade-resulting-in-a-non-solvable-conflict-abc-conflict-non-solvable-by-diff-algo)
- [**Scenario: Preview non-customized field that has an upgrade (AAB)**](#scenario-preview-non-customized-field-that-has-an-upgrade-aab)
- [**Scenario: Preview customized field that doesn't have an upgrade (ABA)**](#scenario-preview-customized-field-that-doesnt-have-an-upgrade-aba)
- [**Scenario: Preview customized field that has a matching upgrade (ABB)**](#scenario-preview-customized-field-that-has-a-matching-upgrade-abb)
- [**Scenario: Preview customized field that has an upgrade resulting in a solvable conflict (ABC, conflict solvable by diff algo)**](#scenario-preview-customized-field-that-has-an-upgrade-resulting-in-a-solvable-conflict-abc-conflict-solvable-by-diff-algo)
- [**Scenario: Preview customized field that has an upgrade resulting in a non-solvable conflict (ABC, conflict non-solvable by diff algo)**](#scenario-preview-customized-field-that-has-an-upgrade-resulting-in-a-non-solvable-conflict-abc-conflict-non-solvable-by-diff-algo)
- [Rule upgrade field preview Diff View options](#rule-upgrade-field-preview-diff-view-options)
- [Preview customized field that doesn't have an upgrade (AAB diff case)](#preview-customized-field-that-doesnt-have-an-upgrade-aab-diff-case)
- [Preview non-customized field that has an upgrade (ABA diff case)](#preview-non-customized-field-that-has-an-upgrade-aba-diff-case)
- [Preview customized field diff that has a matching upgrade (ABB diff case)](#preview-customized-field-diff-that-has-a-matching-upgrade-abb-diff-case)
- [Preview customized field diff that has an upgrade with a solvable conflict (ABC diff case, conflict solvable by diff algo)](#preview-customized-field-diff-that-has-an-upgrade-with-a-solvable-conflict-abc-diff-case-conflict-solvable-by-diff-algo)
- [Preview customized field diff that has an upgrade with a non-solvable conflict (ABC diff case, conflict non-solvable by diff algo)](#preview-customized-field-diff-that-has-an-upgrade-with-a-non-solvable-conflict-abc-diff-case-conflict-non-solvable-by-diff-algo)
- [**Scenario: Preview customized field that doesn't have an upgrade (AAB diff case)**](#scenario-preview-customized-field-that-doesnt-have-an-upgrade-aab-diff-case)
- [**Scenario: Preview non-customized field that has an upgrade (ABA diff case)**](#scenario-preview-non-customized-field-that-has-an-upgrade-aba-diff-case)
- [**Scenario: Preview customized field diff that has a matching upgrade (ABB diff case)**](#scenario-preview-customized-field-diff-that-has-a-matching-upgrade-abb-diff-case)
- [**Scenario: Preview customized field diff that has an upgrade with a solvable conflict (ABC diff case, conflict solvable by diff algo)**](#scenario-preview-customized-field-diff-that-has-an-upgrade-with-a-solvable-conflict-abc-diff-case-conflict-solvable-by-diff-algo)
- [**Scenario: Preview customized field diff that has an upgrade with a non-solvable conflict (ABC diff case, conflict non-solvable by diff algo)**](#scenario-preview-customized-field-diff-that-has-an-upgrade-with-a-non-solvable-conflict-abc-diff-case-conflict-non-solvable-by-diff-algo)
- [Field editing](#field-editing)
- [Validation blocks saving field form when value is invalid](#validation-blocks-saving-field-form-when-value-is-invalid)
- [Saving unchanged field form value doesn't add up or remove anything to the field diff in Diff View](#saving-unchanged-field-form-value-doesnt-add-up-or-remove-anything-to-the-field-diff-in-diff-view)
- [**Scenario: Validation blocks saving field form when value is invalid**](#scenario-validation-blocks-saving-field-form-when-value-is-invalid)
- [**Scenario: Saving unchanged field form value doesn't add up or remove anything to the field diff in Diff View**](#scenario-saving-unchanged-field-form-value-doesnt-add-up-or-remove-anything-to-the-field-diff-in-diff-view)
- [Rule upgrade button](#rule-upgrade-button)
- [Rule upgrade button is disabled when num of conflicts \>= 1](#rule-upgrade-button-is-disabled-when-num-of-conflicts--1)
- [Rule upgrade button is disabled when num fields in edit mode \>= 1](#rule-upgrade-button-is-disabled-when-num-fields-in-edit-mode--1)
- [Rule upgrade button is disabled when num of conflicts \>= 1 or num fields in edit mode \>= 1](#rule-upgrade-button-is-disabled-when-num-of-conflicts--1-or-num-fields-in-edit-mode--1)
- [**Scenario: Rule upgrade button is disabled when num of conflicts \>= 1**](#scenario-rule-upgrade-button-is-disabled-when-num-of-conflicts--1)
- [**Scenario: Rule upgrade button is disabled when num fields in edit mode \>= 1**](#scenario-rule-upgrade-button-is-disabled-when-num-fields-in-edit-mode--1)
- [**Scenario: Rule upgrade button is disabled when num of conflicts \>= 1 or num fields in edit mode \>= 1**](#scenario-rule-upgrade-button-is-disabled-when-num-of-conflicts--1-or-num-fields-in-edit-mode--1)
- [Rule upgrade after field preview](#rule-upgrade-after-field-preview)
- [Non-customized rule upgrade after preview (AAB diff case)](#non-customized-rule-upgrade-after-preview-aab-diff-case)
- [Non-customized rule upgrade after preview and customizing field values (AAB diff case)](#non-customized-rule-upgrade-after-preview-and-customizing-field-values-aab-diff-case)
- [Customized rule upgrade after preview customized fields that don't have upgrades (ABA diff case)](#customized-rule-upgrade-after-preview-customized-fields-that-dont-have-upgrades-aba-diff-case)
- [Customized rule upgrade after preview customized fields that don't have upgrades and changing that field values (ABA diff case)](#customized-rule-upgrade-after-preview-customized-fields-that-dont-have-upgrades-and-changing-that-field-values-aba-diff-case)
- [Customized rule upgrade after preview and accepting solvable conflicts (ABC diff case, conflict solvable by diff algo)](#customized-rule-upgrade-after-preview-and-accepting-solvable-conflicts-abc-diff-case-conflict-solvable-by-diff-algo)
- [Customized rule upgrade after preview and accepting edited solvable conflicts (ABC diff case, conflict solvable by diff algo)](#customized-rule-upgrade-after-preview-and-accepting-edited-solvable-conflicts-abc-diff-case-conflict-solvable-by-diff-algo)
- [Customized rule upgrade after preview non-solvable conflicts and accepting suggested field value (ABC diff case, non-solvable by diff algo)](#customized-rule-upgrade-after-preview-non-solvable-conflicts-and-accepting-suggested-field-value-abc-diff-case-non-solvable-by-diff-algo)
- [Customized rule upgrade after preview non-solvable conflicts and accepting edited field value (ABC diff case, non-solvable by diff algo)](#customized-rule-upgrade-after-preview-non-solvable-conflicts-and-accepting-edited-field-value-abc-diff-case-non-solvable-by-diff-algo)
- [**Scenario: Non-customized rule upgrade after preview (AAB diff case)**](#scenario-non-customized-rule-upgrade-after-preview-aab-diff-case)
- [**Scenario: Non-customized rule upgrade after preview and customizing field values (AAB diff case)**](#scenario-non-customized-rule-upgrade-after-preview-and-customizing-field-values-aab-diff-case)
- [**Scenario: Customized rule upgrade after preview customized fields that don't have upgrades (ABA diff case)**](#scenario-customized-rule-upgrade-after-preview-customized-fields-that-dont-have-upgrades-aba-diff-case)
- [**Scenario: Customized rule upgrade after preview customized fields that don't have upgrades and changing that field values (ABA diff case)**](#scenario-customized-rule-upgrade-after-preview-customized-fields-that-dont-have-upgrades-and-changing-that-field-values-aba-diff-case)
- [**Scenario: Customized rule upgrade after preview and accepting solvable conflicts (ABC diff case, conflict solvable by diff algo)**](#scenario-customized-rule-upgrade-after-preview-and-accepting-solvable-conflicts-abc-diff-case-conflict-solvable-by-diff-algo)
- [**Scenario: Customized rule upgrade after preview and accepting edited solvable conflicts (ABC diff case, conflict solvable by diff algo)**](#scenario-customized-rule-upgrade-after-preview-and-accepting-edited-solvable-conflicts-abc-diff-case-conflict-solvable-by-diff-algo)
- [**Scenario: Customized rule upgrade after preview non-solvable conflicts and accepting suggested field value (ABC diff case, non-solvable by diff algo)**](#scenario-customized-rule-upgrade-after-preview-non-solvable-conflicts-and-accepting-suggested-field-value-abc-diff-case-non-solvable-by-diff-algo)
- [**Scenario: Customized rule upgrade after preview non-solvable conflicts and accepting edited field value (ABC diff case, non-solvable by diff algo)**](#scenario-customized-rule-upgrade-after-preview-non-solvable-conflicts-and-accepting-edited-field-value-abc-diff-case-non-solvable-by-diff-algo)
- [Rule type upgrade](#rule-type-upgrade)
- [Non-customized rule upgrade to a different rule type after preview](#non-customized-rule-upgrade-to-a-different-rule-type-after-preview)
- [Customized rule upgrade to a different rule type after preview](#customized-rule-upgrade-to-a-different-rule-type-after-preview)
- [**Scenario: Non-customized rule upgrade to a different rule type after preview**](#scenario-non-customized-rule-upgrade-to-a-different-rule-type-after-preview)
- [**Scenario: Customized rule upgrade to a different rule type after preview**](#scenario-customized-rule-upgrade-to-a-different-rule-type-after-preview)
- [Concurrency control](#concurrency-control)
- [User gets notified after someone edited a rule being previewed](#user-gets-notified-after-someone-edited-a-rule-being-previewed)
- [User gets notified after a new rule versions is released](#user-gets-notified-after-a-new-rule-versions-is-released)
- [**Scenario: User gets notified after someone edited a rule being previewed**](#scenario-user-gets-notified-after-someone-edited-a-rule-being-previewed)
- [**Scenario: User gets notified after a new rule versions is released**](#scenario-user-gets-notified-after-a-new-rule-versions-is-released)
- [Licensing](#licensing)
- [**Scenario: User can NOT modify field values in upgrade preview when license is insufficient**](#scenario-user-can-not-modify-field-values-in-upgrade-preview-when-license-is-insufficient)
- [**Scenario: User is warned about losing their customizations in upgrade preview when license is insufficient**](#scenario-user-is-warned-about-losing-their-customizations-in-upgrade-preview-when-license-is-insufficient)
@ -73,6 +78,7 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal)
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
- [Relax the rules of handling missing base versions of prebuilt rules](https://github.com/elastic/kibana/issues/210358)
- [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078)
### Terminology
@ -89,6 +95,7 @@ Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- A prebuilt rule is shown in the Rule Upgrade table when there's a newer version of this rule in the currently installed package with prebuilt rules.
- It's expected the Prebuilt Rules upgrade workflow works seamlessly even if some or all prebuilt rules may have their **base versions** missing.
### Technical requirements
@ -133,16 +140,66 @@ What should be inside the Rule Upgrade flyout:
## Scenarios
### Rule upgrade workflow: rule previews
#### **Scenario: User can preview prebuilt rules having upgrades**
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the rule preview for the prebuilt rule
Then the preview should open
When user closes the preview
Then it should disappear
```
#### **Scenario: User can upgrade a prebuilt rule using the rule preview**
**Automation**: 1 e2e test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens the rule preview for the prebuilt rule
And upgrades the rule using a CTA in the rule preview
Then the rule should be upgraded to the latest version
And a success message should be displayed after upgrade
And the rule should be removed from the Prebuilt Rules Upgrades page
And user should see the number of rules available to upgrade as initial number minus 1
```
#### **Scenario: User can see correct rule information in the preview before upgrading**
**Automation**: 1 e2e test
```Gherkin
Given a prebuilt rule with an upgrade
When user opens a rule preview for the prebuilt rule
Then the "Updates" tab should be active
When user selects the "Overview" tab
Then all properties of the new version of a rule should be displayed in the correct tab and section of the preview
```
#### **Scenario: Tabs and sections without content should be hidden in the preview before upgrading**
**Automation**: 1 e2e test
```Gherkin
Given a prebuilt rule is installed
And this rule upgrade doesn't have Setup and Investigation guides
When user opens a rule preview for the prebuilt rule
Then the Setup Guide section should NOT be displayed in the Overview tab
And the Investigation Guide tab should NOT be displayed
```
### Rule upgrade field preview
#### Preview non-customized field that has an upgrade (AAB)
#### **Scenario: Preview non-customized field that has an upgrade (AAB)**
**Automation**: Jest functional test for each \<field\>.
```Gherkin
Given an installed prebuilt rule
And that rule has no customizations
And <field> has an upgrade
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is non-customized
And the <field> has an upgrade
When user opens the Rule Update Flyout
Then user should see <field> has no conflicts
And <field> is shown collapsed
@ -154,14 +211,14 @@ Examples:
<field> = all customizable fields
```
#### Preview customized field that doesn't have an upgrade (ABA)
#### **Scenario: Preview customized field that doesn't have an upgrade (ABA)**
**Automation**: Jest functional test for each \<field\>.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has no upgrades
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has no upgrades
When user opens the Rule Update Flyout
Then user should see <field> has a customized value
And <field> has no conflicts
@ -169,19 +226,20 @@ And <field> is shown collapsed
When user expands <field>
Then user should see <field> Diff View
And user should see <field> Readonly View
Examples:
<field> = all customizable fields
```
#### Preview customized field that has a matching upgrade (ABB)
**Examples:**
`<field>` = all customizable fields
#### **Scenario: Preview customized field that has a matching upgrade (ABB)**
**Automation**: Jest functional test for each \<field\>.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has an upgrade matching customization
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has an upgrade matching customization
When user opens the Rule Update Flyout
Then user should see <field> has matching upgrade
And <field> has no conflicts
@ -189,19 +247,20 @@ And <field> is shown collapsed
When user expands <field>
Then user should see <field> Diff View
And user should see <field> Readonly View
Examples:
<field> = all customizable fields
```
#### Preview customized field that has an upgrade resulting in a solvable conflict (ABC, conflict solvable by diff algo)
**Examples:**
`<field>` = all customizable fields
#### **Scenario: Preview customized field that has an upgrade resulting in a solvable conflict (ABC, conflict solvable by diff algo)**
**Automation**: Jest functional test for each \<field\>.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has an upgrade resulting in a solvable conflict
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has an upgrade resulting in a solvable conflict
When user opens the Rule Update Flyout
Then user should see <field> has a customized value
And <field> has a conflict
@ -212,26 +271,29 @@ And <field> Readonly View displays a merged value
When user switches to edit form
Then user should see <field> edit form
And <field> edit form has merged value
Examples: <field> whose diff algo supports values merging
| data_source |
| tags |
| description |
| references |
| note |
| setup |
| threat_index |
| new_terms_fields |
```
#### Preview customized field that has an upgrade resulting in a non-solvable conflict (ABC, conflict non-solvable by diff algo)
**Examples:**
`<field>` whose diff algo supports values merging:
- `data_source`
- `tags`
- `description`
- `references`
- `note`
- `setup`
- `threat_index`
- `new_terms_fields`
#### **Scenario: Preview customized field that has an upgrade resulting in a non-solvable conflict (ABC, conflict non-solvable by diff algo)**
**Automation**: Jest functional test for each \<field\>.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has an upgrade resulting in a non-solvable conflict
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has an upgrade resulting in a non-solvable conflict
When user opens the Rule Update Flyout
Then user should see <field> has a customized value
And <field> has a conflict
@ -242,61 +304,65 @@ And <field> edit form displays current rule version field value
When user saves and accepts the form
Then user should see <field> Readonly mode
And <field> Readonly mode displays the current rule version field value
Examples:
<field> = all customizable fields, but always mergeable fields "tags", "references", "threat_index", "new_terms_fields"
```
**Examples:**
`<field>` = all customizable fields besides always mergeable fields (`tags`, `references`, `threat_index`, `new_terms_fields`)
### Rule upgrade field preview Diff View options
#### Preview customized field that doesn't have an upgrade (AAB diff case)
#### **Scenario: Preview customized field that doesn't have an upgrade (AAB diff case)**
**Automation**: 1 Jest integration test.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> doesn't have an upgrade
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> doesn't have an upgrade
When user opens the Rule Update Flyout
Then user should see <field> Diff View
And it shows a diff between original and current <field> values
When user edits and saves the <field> form
Then user should see a diff between original and edited <field> values
Examples:
<field> = all customizable fields
```
#### Preview non-customized field that has an upgrade (ABA diff case)
**Examples:**
`<field>` = all customizable fields
#### **Scenario: Preview non-customized field that has an upgrade (ABA diff case)**
**Automation**: 1 Jest integration test.
```Gherkin
Given an installed prebuilt rule
And <field> isn't customized
And <field> has an upgrade
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has no upgrades
When user opens the Rule Update Flyout
Then user should see <field> Diff View
And it shows a diff between original and upgraded <field> values
When user edits and saves the <field> form
Then user should see a diff between original and edited <field> values
And user should have an ability to see <diff option>
Examples:
<field> = all customizable fields
<diff option>
- Elastic update, a diff between original and upgrade field values
```
#### Preview customized field diff that has a matching upgrade (ABB diff case)
**Examples:**
`<field>` = all customizable fields
`<diff option>`
- Elastic update, a diff between original and upgrade field values
#### **Scenario: Preview customized field diff that has a matching upgrade (ABB diff case)**
**Automation**: 1 Jest integration test.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has a matching upgrade
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has a matching upgrade
When user opens the Rule Update Flyout
Then user should see <field> Diff View
And it shows a diff between original and customized <field> values
@ -304,22 +370,24 @@ And user should have an ability to see a <diff option>
When user edits and saves the <field> form
Then user should see a diff between original and edited <field> values
And user should have an ability to see a <diff option>
Examples:
<field> = all customizable fields
<diff option>
- Elastic update, a diff between original and upgrade field values
```
#### Preview customized field diff that has an upgrade with a solvable conflict (ABC diff case, conflict solvable by diff algo)
**Examples:**
`<field>` = all customizable fields
`<diff option>`
- Elastic update, a diff between original and upgrade field values
#### **Scenario: Preview customized field diff that has an upgrade with a solvable conflict (ABC diff case, conflict solvable by diff algo)**
**Automation**: 1 Jest integration test.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has an upgrade resulting in a solvable conflict
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has an upgrade resulting in a solvable conflict
When user opens the Rule Update Flyout
Then user should see <field> Diff View
And it shows a diff between original and merged (customization + upgrade) <field> values
@ -327,23 +395,25 @@ And user should have an ability to see a <diff option>
When user edits and saves the <field> form
Then user should see a diff between original and edited <field> values
And user should have an ability to see a <diff option>
Examples:
<field> = all customizable fields
<diff option>
- Elastic update, a diff between original and upgrade field values
- Original field customization, a diff between original and customized field values
```
#### Preview customized field diff that has an upgrade with a non-solvable conflict (ABC diff case, conflict non-solvable by diff algo)
**Examples:**
`<field>` = all customizable fields
`<diff option>`
- Elastic update, a diff between original and upgrade field values
- Original field customization, a diff between original and customized field values
#### **Scenario: Preview customized field diff that has an upgrade with a non-solvable conflict (ABC diff case, conflict non-solvable by diff algo)**
**Automation**: 1 Jest integration test.
```Gherkin
Given an installed prebuilt rule
And <field> is customized
And <field> has an upgrade resulting in a non-solvable conflict
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> is customized
And the <field> has an upgrade resulting in a non-solvable conflict
When user opens the Rule Update Flyout
Then user should see <field> Diff View
And it shows a diff between original and customized <field> values
@ -351,78 +421,81 @@ And user should have an ability to see a <diff option A>
When user edits and saves the <field> form
Then user should see a diff between original and edited <field> values
And user should have an ability to see a <diff option B>
Examples:
<field> = all customizable fields
<diff option A>
- Elastic upgrade, a diff between original and upgrade field values
<diff option B>
- Elastic upgrade, a diff between original and upgrade field values
- Original customization, a diff between original and customized field values
```
**Examples:**
`<field>` = all customizable fields
`<diff option A>`
- Elastic upgrade, a diff between original and upgrade field values
`<diff option B\>`
- Elastic upgrade, a diff between original and upgrade field values
- Original customization, a diff between original and customized field values
### Field editing
#### Validation blocks saving field form when value is invalid
#### **Scenario: Validation blocks saving field form when value is invalid**
**Automation**: 1 Jest integration test per \<field\> + \<diff case\> variation.
```Gherkin
Given an installed prebuilt rule
And <field> corresponds to a <diff case>
And <field> appears in the Rule Update Flyout
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> corresponds to a <diff case>
And the <field> appears in the Rule Update Flyout
When user edits <field>'s in a <field> form
And enters an invalid value
Then Save button should be disabled
And user should not be able to save the <field> form until a valid value is entered
Examples:
<field> = all customizable fields
<diff case>
- AAB = a customized field that has doesn't have an upgrade
- ABA = a non-customized field that has an upgrade
- ABB = a customized field diff that has a matching upgrade
- ABC solvable = customized field diff that has an upgrade with a solvable conflict
- ABC non-solvable = customized field diff that has an upgrade with a non-solvable conflict
```
#### Saving unchanged field form value doesn't add up or remove anything to the field diff in Diff View
**Examples:**
- `<field>` = all customizable fields
- `<diff case>`
- `AAB` = a customized field that has doesn't have an upgrade
- `ABA` = a non-customized field that has an upgrade
- `ABB` = a customized field diff that has a matching upgrade
- `ABC solvable` = customized field diff that has an upgrade with a solvable conflict
- `ABC non-solvable` = customized field diff that has an upgrade with a non-solvable conflict
#### **Scenario: Saving unchanged field form value doesn't add up or remove anything to the field diff in Diff View**
**Automation**: 1 Jest integration test per \<field\> + \<diff case\> variation.
```Gherkin
Given an installed prebuilt rule
And <field> corresponds to a <diff case>
Given a prebuilt rule with an upgrade
And this prebuilt rule's <field> corresponds to a <diff case>
And <field> appears in the Rule Update Flyout
When user opens a <field> form
And saves the form without changes
Then <field> Diff View should not have any new lines added up or removed
Examples:
<field> = all customizable fields
<diff case>
- AAB = a customized field that has doesn't have an upgrade
- ABA = a non-customized field that has an upgrade
- ABB = a customized field diff that has a matching upgrade
- ABC solvable = customized field diff that has an upgrade with a solvable conflict
- ABC non-solvable = customized field diff that has an upgrade with a non-solvable conflict
```
**Examples:**
- `<field>` = all customizable fields
- `<diff case>`
- `AAB` = a customized field that has doesn't have an upgrade
- `ABA` = a non-customized field that has an upgrade
- `ABB` = a customized field diff that has a matching upgrade
- `ABC solvable` = customized field diff that has an upgrade with a solvable conflict
- `ABC non-solvable` = customized field diff that has an upgrade with a non-solvable conflict
### Rule upgrade button
#### Rule upgrade button is disabled when num of conflicts >= 1
#### **Scenario: Rule upgrade button is disabled when num of conflicts >= 1**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has customizations
And it has an upgrade resulting to conflicts
When user opens the Rule Update Flyout
Given a prebuilt rule with an upgrade resulting to conflicts
When user opens the upgrade preview
Then user should see INACTIVE CTA to upgrade the prebuilt rule
When user hover on the INACTIVE CTA
Then explanation tooltip appears
@ -430,14 +503,13 @@ When user resolves all conflicts
Then the INACTIVE CTA becomes ACTIVE
```
#### Rule upgrade button is disabled when num fields in edit mode >= 1
#### **Scenario: Rule upgrade button is disabled when num fields in edit mode >= 1**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And it has an upgrade without conflicts
When user opens the Rule Update Flyout
Given a prebuilt rule with an upgrade without conflicts
When user opens the upgrade preview
Then user should see ACTIVE CTA to upgrade the prebuilt rule
When user switch one or more fields to edit mode
Then user should see INACTIVE CTA
@ -447,15 +519,14 @@ When user every field in readonly mode
Then the INACTIVE CTA becomes ACTIVE
```
#### Rule upgrade button is disabled when num of conflicts >= 1 or num fields in edit mode >= 1
#### **Scenario: Rule upgrade button is disabled when num of conflicts >= 1 or num fields in edit mode >= 1**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has customizations
And it has an upgrade resulting to conflicts
When user opens the Rule Update Flyout
Given a prebuilt rule with an upgrade resulting to conflicts
And this rule is customized
When user opens the upgrade preview
Then user should see INACTIVE CTA to upgrade the prebuilt rule
When user resolves all conflicts
Then the INACTIVE CTA becomes ACTIVE
@ -465,14 +536,13 @@ Then user should see INACTIVE CTA
### Rule upgrade after field preview
#### Non-customized rule upgrade after preview (AAB diff case)
#### **Scenario: Non-customized rule upgrade after preview (AAB diff case)**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule does not have any customizations
And <field> has an upgrade
Given a non-customized prebuilt rule
And this rule's <field> has an upgrade
When user opens the Rule Update Flyout
Then user should see a CTA to upgrade the prebuilt rule
And user should see <field> has no conflicts
@ -481,19 +551,19 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <field> has an upgraded value
Examples:
<field> = all customizable fields
```
#### Non-customized rule upgrade after preview and customizing field values (AAB diff case)
**Examples:**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
`<field>` = all customizable fields
#### **Scenario: Non-customized rule upgrade after preview and customizing field values (AAB diff case)**
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule does not have any customizations
And <field> has an upgrade
Given a non-customized prebuilt rule
And this rule's <field> has an upgrade
When user opens the Rule Update Flyout
Then user should see a CTA to upgrade the prebuilt rule
And user should see <field> has no conflicts
@ -504,19 +574,20 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <field> has a new value
Examples:
<field> = all customizable fields
```
#### Customized rule upgrade after preview customized fields that don't have upgrades (ABA diff case)
**Examples:**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
`<field>` = all customizable fields
#### **Scenario: Customized rule upgrade after preview customized fields that don't have upgrades (ABA diff case)**
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has <field> customized
And it has an upgrade for <fieldB> (<fieldB> != <fieldA>)
Given a prebuilt rule is installed
And this rule's <fieldA> is customized
And this rule's <fieldB> has an upgrade (<fieldB> != <fieldA>)
When user opens the Rule Update Flyout
Then user should see a CTA to upgrade the prebuilt rule
And user should see <fieldA> has a customized value and there are no conflicts
@ -525,19 +596,20 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <fieldA> has preserved the customized value
Examples:
<field> = all customizable fields
```
#### Customized rule upgrade after preview customized fields that don't have upgrades and changing that field values (ABA diff case)
**Examples:**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
`<field>` = all customizable fields
#### **Scenario: Customized rule upgrade after preview customized fields that don't have upgrades and changing that field values (ABA diff case)**
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has <fieldA> customized
And it has an upgrade for <fieldB> (<fieldB> != <fieldA>)
Given a prebuilt rule is installed
And this rule's <fieldA> is customized
And this rule's <fieldB> has an upgrade (<fieldB> != <fieldA>)
When user opens the Rule Update Flyout
Then user should see a CTA to upgrade the prebuilt rule
And user should see <fieldA> has a customized value and there are no conflicts
@ -548,19 +620,20 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <fieldA> has a new value
Examples:
<field> = all customizable fields
```
#### Customized rule upgrade after preview and accepting solvable conflicts (ABC diff case, conflict solvable by diff algo)
**Examples:**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
`<field>` = all customizable fields
#### **Scenario: Customized rule upgrade after preview and accepting solvable conflicts (ABC diff case, conflict solvable by diff algo)**
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has <field> customized
And it has an upgrade resulting in a solvable conflict
Given a prebuilt rule is installed
And this rule's <field> is customized
And the <field> has an upgrade resulting in a solvable conflict
When user opens the Rule Update Flyout
Then user should see INACTIVE CTA to upgrade the prebuilt rule
And user should see <field> has a conflict
@ -571,26 +644,29 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <field> has an upgraded value user accepted
Examples: <field> is one of
| data_source |
| tags |
| description |
| references |
| note |
| setup |
| threat_index |
| new_terms_fields |
```
#### Customized rule upgrade after preview and accepting edited solvable conflicts (ABC diff case, conflict solvable by diff algo)
**Examples:**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
`<field>` is one of
- `data_source`
- `tags`
- `description`
- `references`
- `note`
- `setup`
- `threat_index`
- `new_terms_fields`
#### **Scenario: Customized rule upgrade after preview and accepting edited solvable conflicts (ABC diff case, conflict solvable by diff algo)**
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has <field> customized
And it has an upgrade resulting in a solvable conflict
Given a prebuilt rule is installed
And this rule's <field> is customized
And the <field> has an upgrade resulting in a solvable conflict
When user opens the Rule Update Flyout
Then user should see INACTIVE CTA to upgrade the prebuilt rule
And user should see <field> has a conflict
@ -605,26 +681,29 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <field> has an upgraded value user entered and saved in the form
Examples: <field> is one of
| data_source |
| tags |
| description |
| references |
| note |
| setup |
| threat_index |
| new_terms_fields |
```
#### Customized rule upgrade after preview non-solvable conflicts and accepting suggested field value (ABC diff case, non-solvable by diff algo)
**Examples:**
`<field>` is one of
- `data_source`
- `tags`
- `description`
- `references`
- `note`
- `setup`
- `threat_index`
- `new_terms_fields`
#### **Scenario: Customized rule upgrade after preview non-solvable conflicts and accepting suggested field value (ABC diff case, non-solvable by diff algo)**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has <field> customized
And it has an upgrade resulting to a non-solvable conflict
Given a prebuilt rule is installed
And this rule's <field> is customized
And the <field> has an upgrade resulting to a non-solvable conflict
When user opens the Rule Update Flyout
Then user should see INACTIVE CTA to upgrade the prebuilt rule
And <field> has a conflict
@ -637,18 +716,19 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <field> has an upgraded value accepted by user
Examples:
<field> = all customizable fields
```
#### Customized rule upgrade after preview non-solvable conflicts and accepting edited field value (ABC diff case, non-solvable by diff algo)
**Examples:**
**Automation**: Jest integration test per \<field\> and 1 bulk Cypress test.
`<field>` = all customizable fields
#### **Scenario: Customized rule upgrade after preview non-solvable conflicts and accepting edited field value (ABC diff case, non-solvable by diff algo)**
**Automation**: Jest integration test per `<field>` and 1 bulk Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has <field> customized
Given a prebuilt rule is installed
And this prebuilt rule's <field> is customized
And it has an upgrade resulting to a non-solvable conflict
When user opens the Rule Update Flyout
Then user should see INACTIVE CTA to upgrade the prebuilt rule
@ -662,21 +742,20 @@ Then success message should be displayed after upgrade
And upgraded prebuilt rule should be removed from the table
When user opens rule details page for that prebuilt rule
Then user should see <field> has an upgraded value user entered and saved in the form
Examples:
<field> = all customizable fields, but always mergeable fields "tags", "references", "threat_index", "new_terms_fields"
```
**Examples:**
`<field>` = all customizable fields besides always mergeable fields (`tags`, `references`, `threat_index`, `new_terms_fields`)
### Rule type upgrade
#### Non-customized rule upgrade to a different rule type after preview
#### **Scenario: Non-customized rule upgrade to a different rule type after preview**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has no customizations
And it has an upgrade
Given a non-customized prebuilt rule with an upgrade
When user opens the Rule Update Flyout
Then user should see a CTA to upgrade the prebuilt rule
And a warning message saying rule upgrade changes its type
@ -688,14 +767,12 @@ Then user should see <field> has changed its type
And has upgraded field values
```
#### Customized rule upgrade to a different rule type after preview
#### **Scenario: Customized rule upgrade to a different rule type after preview**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has customizations
And it has an upgrade
Given a customized prebuilt rule with an upgrade
When user opens the Rule Update Flyout
Then user should see a CTA to upgrade the prebuilt rule
And a warning message saying rule upgrade changes its type
@ -710,13 +787,12 @@ And has upgraded field values
### Concurrency control
#### User gets notified after someone edited a rule being previewed
#### **Scenario: User gets notified after someone edited a rule being previewed**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has an upgrade
Given a prebuilt rule with an upgrade
And <userA> opened Rule Update Preview
And saved custom field values via field forms
When <userB> edits the same rule providing changed field values
@ -724,13 +800,12 @@ Then <userA> should see a notification that rule has been edited
And saved custom field values got discarded
```
#### User gets notified after a new rule versions is released
#### **Scenario: User gets notified after a new rule versions is released**
**Automation**: 1 Cypress test.
```Gherkin
Given an installed prebuilt rule
And that rule has an upgrade
Given a prebuilt rule with an upgrade
And user opened Rule Update Preview
And saved custom field values via field forms
When a new version of the same rule gets available
@ -746,8 +821,7 @@ And saved custom field values got discarded
```Gherkin
Given a Kibana instance running under an insufficient license
And a prebuilt rule is installed
And this rule is outdated (a new version is available for this rule)
And a prebuilt rule with an upgrade
When user opens an upgrade preview for this rule
Then user should see a preview of rule field updates
And there should NOT be a possibility to edit any field values
@ -759,10 +833,8 @@ And there should NOT be a possibility to edit any field values
```Gherkin
Given a Kibana instance running under an insufficient license
And a prebuilt rule is installed
And a base version exists for this rule
And this rule is customized
And this rule is outdated (a new version is available for this rule)
And a customized prebuilt rule with an upgrade
And the base version exists for this rule
When user opens an upgrade preview for this rule
Then user should see a warning that their customizations will be lost on upgrade
```
@ -773,14 +845,15 @@ Then user should see a warning that their customizations will be lost on upgrade
```Gherkin
Given a Kibana instance running under an insufficient license
And a prebuilt rule is installed
And a prebuilt rule with an upgrade
And a base version exists for this rule
And this rule is outdated (a new version is available for this rule)
And this rule is <customization_state>
When user opens an upgrade preview for this rule and clicks on CTA
Then success message should be displayed after upgrade
And the upgraded prebuilt rule should be removed from the table
And all customizable rule fields should be equal to the target version
<customization_state> = customized | not customized
```
**Examples:**
`<customization_state>` = `customized` | `not customized`

999
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md
View file

File diff suppressed because it is too large Load diff

5
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_common_info.md
View file

@ -107,6 +107,10 @@ Terminology related to prebuilt rule customization:
- **insufficient license**: a license or a product tier that doesn't allow rule customization. In Serverless environments customization is only allowed on Security Essentials product tier. In non-Serverless environments customization is only allowed on Trial and Enterprise licenses.
- **upgrade to target version**: a process of upgrading a prebuilt rule to its latest version from Elastic. After the upgrade, all customizable field values in the rule will match those of the latest version from Elastic.
Terminology related to prebuilt rule upgrade workflow:
- **upgrade conflict**, **conflicting upgrade**: mostly it means the incoming rule upgrade has changes to the customized fields. Depending on the field type it may be possible to **solve** the conflict (a.k.a. **solvable conflict**, **auto-solving conflict**) otherwise the conflict is **non-solvable** (a.k.a. **unresolved conflict**). In any case the conflict means the prebuilt rule upgrade is unsafe and should be reviewed.
Terminology related to the "rule source" object:
- **Rule source**, also known as `ruleSource` and `rule_source`: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source.
@ -116,7 +120,6 @@ Terminology related to UI and UX:
- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action.
## Common assumptions
**Assumptions about test environments and scenarios** outlined in all of the test plans related to prebuilt rules.