github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 09:48:58 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)

Browse source 
**Epic:** https://github.com/elastic/kibana/issues/174168
**Partially addresses:**
https://github.com/elastic/kibana/issues/202068,
https://github.com/elastic/kibana/issues/202078,
https://github.com/elastic/kibana/issues/202079
**Follow-up to:** https://github.com/elastic/kibana/pull/211472

## Summary

We're cleaning up and refactoring our existing test plans for prebuilt
rule customization, upgrade, and export/import workflows.

Specifically, this PR:

- Updates the common structure of test plans:
- Adds a tip to each test plan's header to help with navigating all our
test plans for prebuilt rules.
- Introduces a new `Requirements` section to each one and consolidates
assumptions, technical requirements and product requirements under it.
- Each `Product requirements` section now contains relevant user stories
and other product details. In the document flow this is kept close to
the `Scenarios` section.
  - Updates the test plan template according to the updated structure.
- Moves the documentation about customizable and non-customizable rule
fields to the common info document.
- Restores the lost changes from commit
bde4612e52.

No "functional" changes have been made to any test plans, such as
adding, removing, or updating any scenarios. The only functional-like
changes were made to the user stories when creating/updating the product
requirements sections.

This PR wraps up the refactoring efforts aimed to prepare the test plans
for being "functionally" changed and improved in follow-up PRs. In the
following PRs, we're going to cover the logic described in
https://github.com/elastic/kibana/issues/210358, address any gaps in the
coverage, and improve the scenarios structure and wording.

## Review tip

It might be easier to review this PR commit-by-commit as each of them
contains logically cohesive changes.
This commit is contained in:
Georgii Gorbachev 2025-02-26 09:39:32 +01:00 • committed by GitHub
parent d685e54225
commit c9616593c1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 458 additions and 299 deletions
Download patch file Download diff file Expand all files Collapse all files

2
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/README.md
View file

@ -37,6 +37,6 @@ Each subdomain folder should be owned by a single GitHub team in the `.github/CO
Some examples for reference:
- [Test plan template](./test_plan_template.md).
- [Installation and Upgrade of Prebuilt Rules](./detection_response/prebuilt_rules/installation_and_upgrade.md).
- [Test plans for prebuilt rules](./detection_response/prebuilt_rules/prebuilt_rules.md).
Feel free to tune the structure whenever it makes sense and improves readability or maintainability of your plan: add more sections to `Useful info`, add more top-level sections in addition to `Useful info` and `Scenarios`, etc.

54
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflows of customizing prebuilt rules via:
@ -24,10 +27,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Editing prebuilt rules](#editing-prebuilt-rules)
- [**Scenario: User can edit a non-customized prebuilt rule from the rule edit page**](#scenario-user-can-edit-a-non-customized-prebuilt-rule-from-the-rule-edit-page)
@ -60,9 +64,35 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
- [Tests for prebuilt rule customization workflow](https://github.com/elastic/kibana/issues/202068)
### User stories
### Terminology
**Prebuilt rule customization workflow:**
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **Rule source**, or **`ruleSource`**: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source.
- **`is_customized`**: a field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version.
- **non-semantic change**: a change to a rule field that is functionally different. We normalize certain fields so for a time-related field such as `from`, `1m` vs `60s` are treated as the same value. We also trim leading and trailing whitespace for query fields.
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- Rule package used will have all previous rule versions present (no missing base versions).
### Technical requirements
Non-functional requirements for the functionality outlined in this test plan.
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
User stories:
- User can edit a single prebuilt rule from the Rule Details page.
- User can edit single prebuilt rules one-by-one from the Rule Management page.
@ -76,22 +106,6 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- User can see which rules are customized on the Rule Management page in the Upgrade table.
- User can un-customize a prebuilt rule by editing it and reverting its parameters back to their original values.
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **Rule source**, or **`ruleSource`**: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source.
- **`is_customized`**: a field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version.
- **non-semantic change**: a change to a rule field that is functionally different. We normalize certain fields so for a time-related field such as `from`, `1m` vs `60s` are treated as the same value. We also trim leading and trailing whitespace for query fields.
### Assumptions
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- Rule package used will have all previous rule versions present (no missing base versions)
### Non-functional requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
## Scenarios
### Editing prebuilt rules

46
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_export.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflows of:
@ -25,10 +28,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Core Functionality](#core-functionality)
- [Scenario: Exporting prebuilt rule individually from rule details page](#scenario-exporting-prebuilt-rule-individually-from-rule-details-page)
@ -52,9 +56,31 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Support exporting prebuilt rules from the Rule Details page](https://github.com/elastic/kibana/issues/180176)
- [Tests for prebuilt rule import/export workflow](https://github.com/elastic/kibana/issues/202079)
### User stories
### Terminology
**Prebuilt rule export workflow:**
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Technical requirements
Non-functional requirements for the functionality outlined in this test plan.
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
User stories:
- User can export a single prebuilt rule from the Rule Details page.
- User can export multiple prebuilt rules one-by-one from the Rule Management page.
@ -63,18 +89,6 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- User can export prebuilt customized rules.
- User can export any combination of prebuilt non-customized, prebuilt customized, and custom rules.
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
### Assumptions
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Non-functional requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
## Scenarios
### Core Functionality

141
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_import.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflows of:
@ -21,26 +24,28 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Core Functionality](#core-functionality)
- [Scenario: Importing an unmodified prebuilt rule with a matching rule_id and version](#scenario-importing-an-unmodified-prebuilt-rule-with-a-matching-rule_id-and-version)
- [Scenario: Importing a customized prebuilt rule with a matching rule_id and version](#scenario-importing-a-customized-prebuilt-rule-with-a-matching-rule_id-and-version)
- [Scenario: Importing a custom rule with a matching rule_id and version](#scenario-importing-a-custom-rule-with-a-matching-rule_id-and-version)
- [Scenario: Importing a prebuilt rule with a matching rule_id but no matching version](#scenario-importing-a-prebuilt-rule-with-a-matching-rule_id-but-no-matching-version)
- [Scenario: Importing a prebuilt rule with a non-existent rule_id](#scenario-importing-a-prebuilt-rule-with-a-non-existent-rule_id)
- [Scenario: Importing a prebuilt rule without a rule_id field](#scenario-importing-a-prebuilt-rule-without-a-rule_id-field)
- [Scenario: Importing a prebuilt rule with a matching rule_id but missing a version field](#scenario-importing-a-prebuilt-rule-with-a-matching-rule_id-but-missing-a-version-field)
- [Scenario: Importing an existing custom rule missing a version field](#scenario-importing-an-existing-custom-rule-missing-a-version-field)
- [Scenario: Importing a new custom rule missing a version field](#scenario-importing-a-new-custom-rule-missing-a-version-field)
- [Scenario: Importing a rule with overwrite flag set to true](#scenario-importing-a-rule-with-overwrite-flag-set-to-true)
- [Scenario: Importing a rule with overwrite flag set to false](#scenario-importing-a-rule-with-overwrite-flag-set-to-false)
- [Scenario: Importing a non-customized prebuilt rule with a matching `rule_id` and `version`](#scenario-importing-a-non-customized-prebuilt-rule-with-a-matching-rule_id-and-version)
- [Scenario: Importing a customized prebuilt rule with a matching `rule_id` and `version`](#scenario-importing-a-customized-prebuilt-rule-with-a-matching-rule_id-and-version)
- [Scenario: Importing a custom rule with a matching prebuilt `rule_id` and `version`](#scenario-importing-a-custom-rule-with-a-matching-prebuilt-rule_id-and-version)
- [Scenario: Importing a custom rule with a matching custom `rule_id` and `version`](#scenario-importing-a-custom-rule-with-a-matching-custom-rule_id-and-version)
- [Scenario: Importing a prebuilt rule with a matching `rule_id` but no matching `version`](#scenario-importing-a-prebuilt-rule-with-a-matching-rule_id-but-no-matching-version)
- [Scenario: Importing a prebuilt rule with a non-existent `rule_id`](#scenario-importing-a-prebuilt-rule-with-a-non-existent-rule_id)
- [Scenario: Importing a prebuilt rule without a `rule_id` field](#scenario-importing-a-prebuilt-rule-without-a-rule_id-field)
- [Scenario: Importing a prebuilt rule with a matching `rule_id` but missing a `version` field](#scenario-importing-a-prebuilt-rule-with-a-matching-rule_id-but-missing-a-version-field)
- [Scenario: Importing an existing custom rule missing a `version` field](#scenario-importing-an-existing-custom-rule-missing-a-version-field)
- [Scenario: Importing a new custom rule missing a `version` field](#scenario-importing-a-new-custom-rule-missing-a-version-field)
- [Scenario: Importing a rule with `overwrite` flag set to true](#scenario-importing-a-rule-with-overwrite-flag-set-to-true)
- [Scenario: Importing a rule with `overwrite` flag set to false](#scenario-importing-a-rule-with-overwrite-flag-set-to-false)
- [Scenario: Importing both custom and prebuilt rules](#scenario-importing-both-custom-and-prebuilt-rules)
- [Scenario: Importing prebuilt rules when the rules package is not installed](#scenario-importing-prebuilt-rules-when-the-rules-package-is-not-installed)
- [Scenario: User imports a custom rule before a prebuilt rule asset is created with the same rule_id](#scenario-user-imports-a-custom-rule-before-a-prebuilt-rule-asset-is-created-with-the-same-rule_id)
- [Scenario: Importing a prebuilt rule when the rules package is not installed](#scenario-importing-a-prebuilt-rule-when-the-rules-package-is-not-installed)
- [Scenario: Importing a custom rule before a prebuilt rule asset is created with the same `rule_id`](#scenario-importing-a-custom-rule-before-a-prebuilt-rule-asset-is-created-with-the-same-rule_id)
## Useful information
@ -52,9 +57,31 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Benchmark performance of importing a large number of prebuilt rules](https://github.com/elastic/kibana/issues/195632)
- [Tests for prebuilt rule import/export workflow](https://github.com/elastic/kibana/issues/202079)
### User stories
### Terminology
**Prebuilt rule import workflow:**
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Technical requirements
Non-functional requirements for the functionality outlined in this test plan.
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
User stories:
- User can import a single prebuilt rule on the Rule Management page.
- User can import multiple prebuilt rules on the Rule Management page.
@ -62,40 +89,29 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- User can import prebuilt customized rules.
- User can import any combination of prebuilt non-customized, prebuilt customized, and custom rules.
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
### Assumptions
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Non-functional requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
## Scenarios
### Core Functionality
#### Scenario: Importing an unmodified prebuilt rule with a matching rule_id and version
#### Scenario: Importing a non-customized prebuilt rule with a matching `rule_id` and `version`
**Automation**: 1 cypress test and 1 integration test.
```Gherkin
Given the import payload contains an unmodified prebuilt rule
Given the import payload contains a non-customized prebuilt rule
And its rule_id and version match a rule asset from the installed package
When the user imports the rule
Then the rule should be created or updated
And the ruleSource type should be "external"
And isCustomized should be false
```
#### Scenario: Importing a customized prebuilt rule with a matching rule_id and version
#### Scenario: Importing a customized prebuilt rule with a matching `rule_id` and `version`
**Automation**: 1 cypress test and 1 integration test.
```Gherkin
Given the import payload contains a modified prebuilt rule
Given the import payload contains a customized prebuilt rule
And its rule_id and version match a rule asset from the installed package
When the user imports the rule
Then the rule should be created or updated
@ -103,56 +119,59 @@ And the ruleSource type should be "external"
And isCustomized should be true
```
#### Scenario: Importing a custom rule with a matching prebuilt rule_id and version
#### Scenario: Importing a custom rule with a matching prebuilt `rule_id` and `version`
**Automation**: 1 cypress test and 1 integration test.
```Gherkin
Given the import payload contains a custom rule with a matching rule_id and version
Given the import payload contains a custom rule
And its rule_id and version match a rule asset from the installed package
When the user imports the rule
Then the rule should be created or updated
And the ruleSource type should be "external"
And isCustomized should be true
```
#### Scenario: Importing a custom rule with a matching custom rule_id and version
#### Scenario: Importing a custom rule with a matching custom `rule_id` and `version`
**Automation**: 1 cypress test and 1 integration test.
```Gherkin
Given the import payload contains a custom rule with a matching rule_id and version
Given the import payload contains a custom rule
And its rule_id and version match a rule asset from the installed package
And the overwrite flag is set to true
When the user imports the rule
Then the rule should be created or updated
And the ruleSource type should be "internal"
```
#### Scenario: Importing a prebuilt rule with a matching rule_id but no matching version
#### Scenario: Importing a prebuilt rule with a matching `rule_id` but no matching `version`
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains a prebuilt rule
And its rule_id matches a rule asset from the installed package
And the version does not match the rule asset's version
And its rule_id matches one or a few rule assets from the installed package
And its version does NOT match any of those rule assets
When the user imports the rule
Then the rule should be created or updated
And the ruleSource type should be "external"
And isCustomized should be true
```
#### Scenario: Importing a prebuilt rule with a non-existent rule_id
#### Scenario: Importing a prebuilt rule with a non-existent `rule_id`
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains a prebuilt rule
And its rule_id does NOT match a rule asset from the installed package
And its rule_id does NOT match any rule assets from the installed package
When the user imports the rule
Then the rule should be created
And the ruleSource type should be "internal"
```
#### Scenario: Importing a prebuilt rule without a rule_id field
#### Scenario: Importing a prebuilt rule without a `rule_id` field
**Automation**: 1 integration test.
@ -162,41 +181,46 @@ When the user imports the rule
Then the import should be rejected with a message "rule_id field is required"
```
#### Scenario: Importing a prebuilt rule with a matching rule_id but missing a version field
#### Scenario: Importing a prebuilt rule with a matching `rule_id` but missing a `version` field
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains a prebuilt rule without a version field
And its rule_id matches one or a few rule assets from the installed package
When the user imports the rule
Then the import should be rejected with a message "version field is required"
```
#### Scenario: Importing an existing custom rule missing a version field
#### Scenario: Importing an existing custom rule missing a `version` field
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains an existing custom rule without a version field
Given the import payload contains a custom rule without a version field
And its rule_id does NOT match any rule assets from the installed package
And this custom rule has already been created
When the user imports the rule
Then the rule should be updated
And the ruleSource type should be "internal"
And the "version" field should be set to the existing rule's "version"
```
#### Scenario: Importing a new custom rule missing a version field
#### Scenario: Importing a new custom rule missing a `version` field
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains a new custom rule without a version field
Given the import payload contains a custom rule without a version field
And its rule_id does NOT match any rule assets from the installed package
And this custom rule hasn't been created yet
When the user imports the rule
Then the rule should be created
And the ruleSource type should be "internal"
And the "version" field should be set to 1
```
#### Scenario: Importing a rule with overwrite flag set to true
#### Scenario: Importing a rule with `overwrite` flag set to true
**Automation**: 1 integration test.
@ -209,7 +233,7 @@ Then the rule should be overwritten
And the ruleSource should be based on rule_id and version
```
#### Scenario: Importing a rule with overwrite flag set to false
#### Scenario: Importing a rule with `overwrite` flag set to false
**Automation**: 1 integration test.
@ -228,25 +252,28 @@ CASE: should have the same outcome for all rule types
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains modified and unmodified, custom and prebuilt rules
When the user imports the rule
Given the import payload contains prebuilt non-customized, prebuilt customized, and custom rules
When the user imports these rules
Then custom rules should be created or updated, with versions defaulted to 1
And prebuilt rules should be created or updated,
And prebuilt rules missing versions should be rejected
```
#### Scenario: Importing prebuilt rules when the rules package is not installed
#### Scenario: Importing a prebuilt rule when the rules package is not installed
**Automation**: 1 integration test.
```Gherkin
Given the import payload contains prebuilt rules
And no rules package has been installed locally
Given the import payload contains a prebuilt rule
And its rule_id matches one or a few rule assets from the latest package
And the package hasn't been installed yet
When the user imports the rule
Then the latest prebuilt rules package should get installed automatically
Then the latest package should get installed automatically
And the rule should be created or updated
And the ruleSource type should be "external"
```
#### Scenario: User imports a custom rule before a prebuilt rule asset is created with the same rule_id
#### Scenario: Importing a custom rule before a prebuilt rule asset is created with the same `rule_id`
**Automation**: 1 integration test.

53
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_installation.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflows of:
@ -20,11 +23,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Functional requirements](#functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Rule installation notifications on the Rule Management page](#rule-installation-notifications-on-the-rule-management-page)
- [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets)
@ -63,34 +66,46 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Users can Customize Prebuilt Detection Rules: Milestone 2](https://github.com/elastic/kibana/issues/174167)
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
### User stories
**Prebuilt rule installation workflow:**
- User can install single prebuilt rules one-by-one from the Rule Installation page.
- User can install multiple prebuilt rules in bulk from the Rule Installation page.
- User can install all available prebuilt rules in bulk from the Rule Installation page.
- User can preview properties of a prebuilt rule before installing it.
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **CTA to install prebuilt rules**: a link button with a counter on the Rule Management page.
- **CTA to upgrade prebuilt rules**: a tab with a counter on the Rule Management page.
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Non-functional requirements
### Technical requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
Non-functional requirements for the functionality outlined in this test plan.
### Functional requirements
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
- Users should be able to install prebuilt rules on the `Basic` license and higher.
- User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties).
- If user chooses to preview a prebuilt rule to be installed/upgraded, we currently show this preview in a flyout.
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
User stories for the main rule installation features:
- User can install single prebuilt rules one-by-one from the Rule Installation page.
- User can install multiple prebuilt rules in bulk from the Rule Installation page.
- User can install all available prebuilt rules in bulk from the Rule Installation page.
- User can preview properties of a prebuilt rule before installing it.
User stories for licensing and RBAC:
- User can install prebuilt rules on the `Basic` license and higher.
Previewing properties of a prebuilt rule before installing it:
- If user chooses to preview a prebuilt rule to be installed, we currently show this preview in a flyout.
- In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed.
Examples of rule properties we show in the prebuilt rule preview flyout:
@ -555,4 +570,4 @@ And no prebuilt rules are installed
When user opens the Add Rules page
Then user should see prebuilt rules available to install
But user should not be able to install them
```
```

23
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_diff_algorithms.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the diff algorithms used in the workflows of upgrading prebuilt rules and specifically in the `upgrade/_review` endpoint.
@ -18,8 +21,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Rule field doesn't have an update and has no custom value - `AAA`](#rule-field-doesnt-have-an-update-and-has-no-custom-value---aaa)
- [**Scenario: `AAA` - Rule field is any type**](#scenario-aaa---rule-field-is-any-type)
@ -76,15 +81,27 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- `eql_query`: an object that contains a `query` string field, a `language` field that always has the value: `eql`, and a `filters` field that contains an array of kibana filters.
- `esql_query`: an object that contains a `query` string field and a `language` field that always has the value: `esql`.
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- All scenarios will contain at least 1 prebuilt rule installed in Kibana.
- A new version will be available for rule(s).
### Non-functional requirements
### Technical requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
Non-functional requirements for the functionality outlined in this test plan.
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
## Scenarios

72
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflow of:
@ -19,11 +22,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Functional requirements](#functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Rule upgrade field preview](#rule-upgrade-field-preview)
- [Preview non-customized field that has an upgrade (AAB)](#preview-non-customized-field-that-has-an-upgrade-aab)
@ -68,51 +71,60 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
- [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078)
### User stories
**Prebuilt rule upgrade workflow:**
- User can upgrade a single prebuilt rule to its latest version from the Rule Upgrade table with previewing incoming updates from Elastic and user customizations in the Rule Upgrade flyout.
- Specifically, when upgrading a rule, in the Rule Upgrade flyout:
- User can preview updates from Elastic, for each rule field that has an update from Elastic.
- User can preview their customizations, for each rule field that was customized.
- User can compare their customizations with updates from Elastic and see if there are any conflicts between them, per each rule field.
- User can manually resolve conflicts between their customizations and updates from Elastic, per each rule field.
- User can edit the final field values before submitting the update.
- User can upgrade a rule if its type has been changed by Elastic in the latest version, but can only accept the incoming changes; any user customizations will be lost.
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **CTA to upgrade the prebuilt rule**: the button to upgrade the prebuilt rule currently shown in the Rule Upgrade flyout.
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- A prebuilt rule is shown in the Rule Upgrade table when there's a newer version of this rule in the currently installed package with prebuilt rules.
### Non-functional requirements
### Technical requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
Non-functional requirements for the functionality outlined in this test plan.
### Functional requirements
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
User stories:
- User can upgrade a single prebuilt rule to its latest version from the Rule Upgrade table with previewing incoming updates from Elastic and user customizations in the Rule Upgrade flyout. See below user stories that describe this workflow in more detail.
- User can upgrade single prebuilt rules one-by-one via the Rule Upgrade flyout.
User stories for upgrading a rule via the Rule Upgrade flyout:
- User can preview what updates they would receive from Elastic in the latest rule version, per each rule field that has an update from Elastic.
- User can preview their customizations, and which of them will be retained in the rule or overwritten by updates from Elastic, per each rule field that was customized.
- User can compare their customizations with updates from Elastic and see if there are any conflicts between them, per each rule field.
- User can manually resolve conflicts between their customizations and updates from Elastic, per each rule field.
- User can edit the final field values before submitting the update.
User stories for edge cases:
- User can upgrade a rule if its type has been changed by Elastic in the latest version, but can only accept the incoming changes. User customizations, if the rule has any, will be lost on upgrade. We force the user to preview such changes, meaning that users can only do this via the Rule Upgrade flyout, quick upgrade right from the table is not supported.
What should be inside the Rule Upgrade flyout:
- User should be able to upgrade prebuilt rules one-by-one with the ability to preview:
- what updates they would receive from Elastic in the latest rule version;
- what user customizations they would retain in the rule;
- are there any conflicts between the updates from Elastic and the user customizations;
- what they should pay attention to in case there are any conflicts.
- A preview should be shown in the Rule Upgrade flyout.
- The Rule Upgrade flyout should contain a few tabs:
- The "Updates" tab.
- The "Overview" tab.
- The "Investigation guide" tab.
- On the "Updates" tab:
- We should show the updates from Elastic and the user-customized fields.
- We should show only those fields that are [customizable](./shared_assets/customizable_rule_fields.md).
- We shouldn't show technical fields and those that are [not customizable](./shared_assets/non_customizable_rule_fields.md).
- User should be able to upgrade a prebuilt rule that has some updates to [non-customizable fields](./shared_assets/non_customizable_rule_fields.md) in the latest version.
- Any other fields that are not involved in the rule upgrade workflow, such as `enabled` or `exceptions`, should stay unchanged after rule upgrade.
- We show the updates from Elastic and the user-customized fields.
- We show only those fields that are [customizable](./prebuilt_rules_common_info.md#customizable-rule-fields).
- We don't show technical fields and those that are [not customizable](./prebuilt_rules_common_info.md#non-customizable-rule-fields). They are handled by the upgrade workflow automatically under the hood.
- We don't show some non-technical fields (rule parameters) as they are also handled by the upgrade workflow automatically under the hood. For example, the `enabled` or `exception_list` fields stay unchanged after upgrade (we retain their values from the current rule version).
## Scenarios

103
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflows of:
@ -20,11 +23,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Functional requirements](#functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Rule upgrade notifications on the Rule Management page](#rule-upgrade-notifications-on-the-rule-management-page)
- [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date)
@ -86,84 +89,58 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
- [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078)
### User stories
**Prebuilt rule upgrade workflow:**
- User can upgrade a single prebuilt rule to its latest version without previewing the incoming updates.
- User can bulk upgrade multiple prebuilt rules to their latest versions without previewing the incoming updates.
### Terminology
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **CTA to install prebuilt rules**: a link button with a counter on the Rule Management page.
- **CTA to upgrade prebuilt rules**: a tab with a counter on the Rule Management page.
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Non-functional requirements
### Technical requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
Non-functional requirements for the functionality outlined in this test plan.
### Functional requirements
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
User should be able to upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions).
### Product requirements
User should be able to upgrade prebuilt rules with and without customizations. Where the following fields support customizations:
Functional requirements for the functionality outlined in this test plan.
| Rule type | Field name in UI | Diffable rule field |
| ---------------- | ------------------------- | ------------------------- |
| All rule types | Rule name | `name` |
| All rule types | Rule description | `description` |
| All rule types | Tags | `tags` |
| All rule types | Default severity | `severity` |
| All rule types | Severity Override | `severity_mapping` |
| All rule types | Default risk score | `risk_score` |
| All rule types | Risk score override | `risk_score_mapping` |
| All rule types | Reference URLs | `references` |
| All rule types | False positive examples | `false_positives` |
| All rule types | MITRE ATT&CK™ threats | `threat` |
| All rule types | Setup guide | `setup` |
| All rule types | Investigation guide | `note` |
| All rule types | Related integrations | `related_integrations` |
| All rule types | Required fields | `required_fields` |
| All rule types | Rule schedule | `rule_schedule` |
| All rule types | Max alerts per run | `max_signals` |
| All rule types | Rule name override | `rule_name_override` |
| All rule types | Timestamp override | `timestamp_override` |
| All rule types | Timeline template | `timeline_template` |
| All rule types | Building block `*` | `building_block` |
| All rule types | Investigation fields | `investigation_fields` |
| All rule types | Data source `**` | `data_source` |
| All rule types | Suppress alerts | `alert_suppression` |
| Custom Query | Custom query | `kql_query` |
| Saved Query | Custom query | `kql_query` |
| EQL | EQL query | `eql_query` |
| ESQL | ESQL query | `esql_query` |
| Threat Match | Custom query | `kql_query` |
| Threat Match | Indicator index patterns | `threat_index` |
| Threat Match | Indicator index query | `threat_query` |
| Threat Match | Indicator mapping | `threat_mapping` |
| Threat Match | Indicator prefix override | `threat_indicator_path` |
| Threshold | Custom query | `kql_query` |
| Threshold | Threshold config | `threshold` |
| Machine Learning | Machine Learning job | `machine_learning_job_id` |
| Machine Learning | Anomaly score threshold | `anomaly_threshold` |
| New Terms | Custom query | `kql_query` |
| New Terms | Fields | `new_terms_fields` |
| New Terms | History Window Size | `history_window_start` |
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
- `*` Building block field is used to mark alerts as building block alerts.
- `**` Data Source represents index patterns or a data view. Machine Learning rules don't have data_source field.
User stories for upgrading single prebuilt rules one-by-one:
User should be able to upgrade prebuilt rules with updates in the following non-customizable fields:
- User can upgrade a single prebuilt rule to its latest version without previewing the incoming updates:
- if the rule doesn't have any conflicts with its latest version.
- User can't upgrade a single prebuilt rule to its latest version without previewing the incoming updates:
- if the rule has any solvable conflicts with its latest version;
- if the rule has any non-solvable conflicts with its latest version;
- if the rule's type has been changed in the latest version by Elastic (this is considered a non-solvable conflict);
- in these situations user is required to upgrade the rule with preview.
| Field name | Diffable rule field |
| --------------------- | ------------------- |
| Rule type | `type` |
| Rule version | `version` |
User stories for upgrading multiple prebuilt rules in bulk:
- User can bulk upgrade multiple prebuilt rules to their latest versions without previewing the incoming updates, but only those:
- that don't have any conflicts with their latest versions;
- that have some solvable conflicts but don't have any non-solvable conflicts with their latest versions.
- User can't bulk upgrade multiple prebuilt rules to their latest versions without previewing the incoming updates:
- if these rules have any non-solvable conflicts with their latest versions;
- if these rules don't have any non-solvable conflicts, but have some solvable conflicts and the user doesn't confirm their intention to upgrade such rules;
- if these rules' types have been changed in their latest versions by Elastic (this is considered a non-solvable conflict);
- in these situations user is required to upgrade each rule one-by-one with preview.
- User can bulk upgrade prebuilt rules with solvable conflicts only if the user confirms their intention to upgrade such rules.
- User can "bulk upgrade" a single prebuilt rule via the bulk actions. In this case, the "user stories for upgrading multiple prebuilt rules in bulk" apply instead of the "user stories for upgrading single prebuilt rules one-by-one".
User stories, misc:
- In general, user can upgrade a prebuilt rule without preview regardless of the fact if the rule is customized or not. The ability to do so depends on the fact if this customization conflicts with the latest version or not, and if yes, is this conflict solvable or non-solvable.
## Scenarios

2
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules.md
View file

@ -11,7 +11,7 @@ See [common information](./prebuilt_rules_common_info.md) about prebuilt rules,
## Test plans
Understand how we distribute prebuilt rules via a Fleet package and how prebuilt rules become available for installation and upgrade in the system.
Understand how we distribute prebuilt rules via a Fleet package and how prebuilt rules become available for installation and upgrade in the system:
- [Test plan: prebuilt rules package](./prebuilt_rules_package.md)

85
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_common_info.md
View file

@ -1,5 +1,10 @@
# Common information about prebuilt rules <!-- omit from toc -->
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](#features).
## Table of contents <!-- omit from toc -->
<!--
@ -11,8 +16,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Features](#features)
- [Common terminology](#common-terminology)
- [Common assumptions](#common-assumptions)
- [Common non-functional requirements](#common-non-functional-requirements)
- [Common functional requirements](#common-functional-requirements)
- [Common technical requirements](#common-technical-requirements)
- [Common product requirements](#common-product-requirements)
- [Customizable rule fields](#customizable-rule-fields)
- [Non-customizable rule fields](#non-customizable-rule-fields)
## Tickets
@ -95,8 +102,8 @@ Terminology related to prebuilt rule customization:
- For a customized field, `current_version.field` != `base_version.field`.
- **Non-customized field**: a prebuilt rule's field that has the original value from the originally installed prebuilt rule.
- For a non-customized field, `current_version.field` == `base_version.field`.
- **Customizable rule field**: a rule field that is able to be customized on a prebuilt rule. A comprehenseive list can be found in `./shared_assets/customizable_rule_fields.md`.
- **Non-customizable rule field**: a rule field that is unable to be customized on a prebuilt rule. A comprehenseive list can be found in `./shared_assets/non_customizable_rule_fields.md`.
- **Customizable rule field**: a rule field that is able to be customized on a prebuilt rule. A comprehenseive list can be found [below](#customizable-rule-fields).
- **Non-customizable rule field**: a rule field that is unable to be customized on a prebuilt rule. A comprehenseive list can be found [below](#non-customizable-rule-fields).
Terminology related to the "rule source" object:
@ -109,6 +116,8 @@ Terminology related to UI and UX:
## Common assumptions
**Assumptions about test environments and scenarios** outlined in all of the test plans related to prebuilt rules.
Unless explicitly indicated otherwise:
- Scenarios in the test plans only apply to prebuilt detection rules. Some scenarios may apply to both prebuilt and custom detection rules, in which case it should be clearly stated.
@ -119,7 +128,9 @@ Unless explicitly indicated otherwise:
- on the `Essentials` tier in a Serverless Security environment.
- User has the required [privileges for managing detection rules](https://www.elastic.co/guide/en/security/current/detections-permissions-section.html).
## Common non-functional requirements
## Common technical requirements
**Non-functional requirements** for the functionality of prebuilt rules. These apply to all of / most of the prebuilt rules workflows and test plans.
- Package installation, rule installation and rule upgrade workflows should work:
- regardless of the package type: with historical rule versions or without;
@ -128,6 +139,66 @@ Unless explicitly indicated otherwise:
- Kibana should not crash with Out Of Memory exception during package installation.
- For test purposes, it should be possible to use detection rules package versions lower than the latest.
## Common functional requirements
## Common product requirements
TBD
**Functional requirements** for the functionality of prebuilt rules. These apply to all of / most of the prebuilt rules workflows and test plans.
### Customizable rule fields
These are the fields of prebuilt rules that users should be able to customize:
| Rule type | Field name in UI | Diffable rule field |
| ---------------- | ------------------------- | ------------------------- |
| All rule types | Rule name | `name` |
| All rule types | Rule description | `description` |
| All rule types | Tags | `tags` |
| All rule types | Default severity | `severity` |
| All rule types | Severity Override | `severity_mapping` |
| All rule types | Default risk score | `risk_score` |
| All rule types | Risk score override | `risk_score_mapping` |
| All rule types | Reference URLs | `references` |
| All rule types | False positive examples | `false_positives` |
| All rule types | MITRE ATT&CK™ threats | `threat` |
| All rule types | Setup guide | `setup` |
| All rule types | Investigation guide | `note` |
| All rule types | Related integrations | `related_integrations` |
| All rule types | Required fields | `required_fields` |
| All rule types | Rule schedule | `rule_schedule` |
| All rule types | Max alerts per run | `max_signals` |
| All rule types | Rule name override | `rule_name_override` |
| All rule types | Timestamp override | `timestamp_override` |
| All rule types | Timeline template | `timeline_template` |
| All rule types | Building block `*` | `building_block` |
| All rule types | Investigation fields | `investigation_fields` |
| All rule types | Data source `**` | `data_source` |
| All rule types | Suppress alerts | `alert_suppression` |
| Custom Query | Custom query | `kql_query` |
| Saved Query | Custom query | `kql_query` |
| EQL | EQL query | `eql_query` |
| ESQL | ESQL query | `esql_query` |
| Threat Match | Custom query | `kql_query` |
| Threat Match | Indicator index patterns | `threat_index` |
| Threat Match | Indicator index query | `threat_query` |
| Threat Match | Indicator mapping | `threat_mapping` |
| Threat Match | Indicator prefix override | `threat_indicator_path` |
| Threshold | Custom query | `kql_query` |
| Threshold | Threshold config | `threshold` |
| Machine Learning | Machine Learning job | `machine_learning_job_id` |
| Machine Learning | Anomaly score threshold | `anomaly_threshold` |
| New Terms | Custom query | `kql_query` |
| New Terms | Fields | `new_terms_fields` |
| New Terms | History Window Size | `history_window_start` |
- `*` Building block field is used to mark alerts as building block alerts.
- `**` Data Source represents index patterns or a data view. Machine Learning rules don't have data_source field.
### Non-customizable rule fields
These are the fields of prebuilt rules that users should not be able to customize:
| Field name | Diffable rule field |
| ------------ | ------------------- |
| Rule type | `type` |
| Rule version | `version` |
| Rule author | `author` |
| Rule license | `license` |

32
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_package.md
View file

@ -2,6 +2,9 @@
**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->
This is a test plan for the workflow of installing and updating the Fleet package with prebuilt rules.
@ -18,9 +21,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Functional requirements](#functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Package installation](#package-installation)
- [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet)
@ -45,22 +49,34 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Common terminology](./prebuilt_rules_common_info.md#common-terminology), see the terminology related to the package with prebuilt rules.
## Requirements
### Assumptions
Assumptions about test environments and scenarios outlined in this test plan.
- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
### Non-functional requirements
### Technical requirements
- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
Non-functional requirements for the functionality outlined in this test plan.
### Functional requirements
- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).
- Package should be installable and updatable:
### Product requirements
Functional requirements for the functionality outlined in this test plan.
- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).
User stories:
- Package can be installed and updated:
- on any license in self-hosted and ECH environments;
- on any tier in Serverless Security environments;
- regardless of user privileges, as long as the user has access to Security Solution.
- User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties).
- User should be able to upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions).
- User can install prebuilt rules with and without previewing what exactly they would install (rule properties).
- User can upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions).
## Scenarios

48
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/shared_assets/customizable_rule_fields.md
View file

@ -1,48 +0,0 @@
# Customizable rule fields
These are fields in the detection rule schema that are able to be customized on a prebuilt rule.
| Rule type | Field name in UI | Diffable rule field |
| ---------------- | ------------------------- | ------------------------- |
| All rule types | Rule name | `name` |
| All rule types | Rule description | `description` |
| All rule types | Tags | `tags` |
| All rule types | Default severity | `severity` |
| All rule types | Severity Override | `severity_mapping` |
| All rule types | Default risk score | `risk_score` |
| All rule types | Risk score override | `risk_score_mapping` |
| All rule types | Reference URLs | `references` |
| All rule types | False positive examples | `false_positives` |
| All rule types | MITRE ATT&CK™ threats | `threat` |
| All rule types | Setup guide | `setup` |
| All rule types | Investigation guide | `note` |
| All rule types | Related integrations | `related_integrations` |
| All rule types | Required fields | `required_fields` |
| All rule types | Rule schedule | `rule_schedule` |
| All rule types | Max alerts per run | `max_signals` |
| All rule types | Rule name override | `rule_name_override` |
| All rule types | Timestamp override | `timestamp_override` |
| All rule types | Timeline template | `timeline_template` |
| All rule types | Building block `*` | `building_block` |
| All rule types | Investigation fields | `investigation_fields` |
| All rule types | Data source `**` | `data_source` |
| All rule types | Suppress alerts | `alert_suppression` |
| Custom Query | Custom query | `kql_query` |
| Saved Query | Custom query | `kql_query` |
| EQL | EQL query | `eql_query` |
| ESQL | ESQL query | `esql_query` |
| Threat Match | Custom query | `kql_query` |
| Threat Match | Indicator index patterns | `threat_index` |
| Threat Match | Indicator index query | `threat_query` |
| Threat Match | Indicator mapping | `threat_mapping` |
| Threat Match | Indicator prefix override | `threat_indicator_path` |
| Threshold | Custom query | `kql_query` |
| Threshold | Threshold config | `threshold` |
| Machine Learning | Machine Learning job | `machine_learning_job_id` |
| Machine Learning | Anomaly score threshold | `anomaly_threshold` |
| New Terms | Custom query | `kql_query` |
| New Terms | Fields | `new_terms_fields` |
| New Terms | History Window Size | `history_window_start` |
- `*` Building block field is used to mark alerts as building block alerts.
- `**` Data Source represents index patterns or a data view. Machine Learning rules don't have data_source field.

10
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/shared_assets/non_customizable_rule_fields.md
View file

@ -1,10 +0,0 @@
# Non-customizable rule fields
These are fields in the detection rule schema that cannot be customized for a prebuilt rule.
| Field name | Diffable rule field |
| ------------ | ------------------- |
| Rule type | `type` |
| Rule version | `version` |
| Rule author | `author` |
| Rule license | `license` |

86
x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/test_plan_template.md
View file

@ -1,10 +1,33 @@
# Awesome Feature
# Test plan: my awesome feature <!-- omit from toc -->
<!-- Elaborate on what are you testing here, explain what the Awesome Feature is about. -->
<!-- Convey the plan's current status, e.g. are you expecting to finalize it in your PR, or later. -->
**Status**: `in progress`. <!-- `in progress` | `done` -->
## Summary <!-- omit from toc -->
<!-- Elaborate on what are we testing here, explain what the Awesome Feature is about. -->
This is a test plan for ...
<!-- Add any additional info, e.g. are you expecting to finalize it in your PR, or later. -->
Status: `in progress`. <!-- `in progress` | `done` -->
## Table of contents <!-- omit from toc -->
<!--
Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
-->
- [Useful information](#useful-information)
- [Tickets](#tickets)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Section 1](#section-1)
- [**Scenario: Title of scenario 1.1**](#scenario-title-of-scenario-11)
- [**Scenario: Title of scenario 1.2**](#scenario-title-of-scenario-12)
- [Section 2](#section-2)
- [**Scenario: Title of scenario 2.1**](#scenario-title-of-scenario-21)
## Useful information
@ -12,7 +35,7 @@ Status: `in progress`. <!-- `in progress` | `done` -->
<!-- Add links to any related tickets. -->
- [Awesome Feature](https://github.com/elastic/security-team/issues/9999) epic
- [Awesome Feature's epic](https://github.com/elastic/security-team/issues/9999)
- [Add tests for the new awesome feature](https://github.com/elastic/kibana/issues/999999)
- [Document the new awesome feature](https://github.com/elastic/security-docs/issues/9999)
@ -24,23 +47,26 @@ Status: `in progress`. <!-- `in progress` | `done` -->
-->
- **Term 1**: explanation.
- **Term 2**: explanation.
## Requirements
### Assumptions
<!--
<!--
Mention any assumptions for the scenarios that are not explicitly stated in their steps.
For example, you could describe:
- license requirements: the feature is available under the Basic license
- RBAC requirements: user should have certain privileges to normally access the feature
- data setup: user should have certain saved objects, source events, alerts, etc in the system
- license assumptions: all scenarios are executed under the Basic license, unless indicated otherwise
- RBAC assumptions: user has the required privileges to normally access the feature
- data setup: user has certain saved objects, source events, alerts, etc in the system
-->
Assumptions about test environments and scenarios outlined in this test plan.
- Assumption 1.
- Assumption 2.
### Non-functional requirements
### Technical requirements
<!--
Describe any non-function requirements for the feature, if you have any. These could be about:
@ -52,9 +78,27 @@ Status: `in progress`. <!-- `in progress` | `done` -->
- testing
-->
Non-functional requirements for the functionality outlined in this test plan.
- Requirement 1.
- Requirement 2.
### Product requirements
<!--
Describe any function requirements for the feature. This may include:
- user stories
- acceptance criteria
- any other relevant details and comments about the UX or UI
-->
Functional requirements for the functionality outlined in this test plan.
User stories:
- User can do X.
- User can do Y.
## Scenarios
<!--
@ -74,7 +118,7 @@ Status: `in progress`. <!-- `in progress` | `done` -->
### Section 1
#### **Scenario: Awesome feature works**
#### **Scenario: Title of scenario 1.1**
<!-- Describe how are you planning to automate this scenario -->
**Automation**: X e2e tests + Y integration tests + unit tests.
@ -86,11 +130,21 @@ When ...
Then ...
```
<!-- Consider adding any other useful notes and clarifications -->
<!-- Consider adding any other useful notes and clarifications for the scenario -->
### Section 2
#### **Scenario: ?**
#### **Scenario: Title of scenario 1.2**
**Automation**: X e2e tests + Y integration tests + unit tests.
```Gherkin
Given ...
When ...
Then ...
```
### Section 2
#### **Scenario: Title of scenario 2.1**
**Automation**: X e2e tests + Y integration tests + unit tests.