Enable Security's Cypress tests on all PRs (#167516)

## Summary

Security Solution writes e2e and other tests using Cypress. In the past,
these tests, if they failed on a tracked branch, couldn't be easily
skipped. They also weren't run in parallelized jobs. For primarily these
reasons, they didn't run on most Kibana PRs.

This PR moves these Cypress tests back to the main PR pipeline. Tests
that fail on tracked branches create (or update) Github issues which can
be used with the skip-test github workflow script to easily skip the
failing tests. The pipeline steps are parallelized and run in under 40
minutes.

### Open Questions

- [ ] Should this PR enable Serverless Security Defend Workflows Cypress
Tests @patrykkopycinski

### Some buildkite pipelines that used to run only on Security PRs now
run on all PRs:

These steps run on all PRs with these changes

- Security Solution Cypress Tests (general tests that haven't been
organized into an area team)
- Explore tests
- Investigations Tests
- Defend Workflows Tests
- Defend Workflows Serverless
- Threat Intel Tests
- OS Query Tests
- Security Solution Burning Changed Specs (these run only recently
changed specs a few extra times)
- Security Solution OpenAPI codegen
- OSQuery burning
- OSQuery Serverless

<details>
  <summary><b>And these already run on all PRs</b></summary>

  - Serverless Security Cypress Tests
  - Serverless Explore tests
  - Serverless Investigations Tests
</details>

### Security Cypress tests run in the main `on merge` pipeline instead
of the `on merge unsupported ftrs` pipeline:

These steps run in the `on merge` pipeline with these changes:

- Security Solution Cypress Tests
- Explore Cypress Tests
- Investigations Cypress Tests
- Defend Workflows Cypress Tests
- Defend Workflows Serverless Cypress Tests
- Threat Intelligence Cypress Tests
- Osquery Cypress Tests

<details>
<summary><b>and these already run on the `on merge`
pipeline</b></summary>

- Serverless Security Cypress Tests
- Serverless Explore - Security Solution Cypress Tests
- Serverless Investigations - Security Solution Cypress Tests

</details>

### Additional work to be done:

We need to consolidate build steps, enhance test skipping to support
Cypress-grep flags, avoid out-of-memory errors in cypress, enhance
parallelization, improve Cypress reporting, and probably other things.
These are tracked separately. Reach out to me if you need details.

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
This commit is contained in:
Robert Austin 2023-10-06 11:58:31 -04:00 committed by GitHub
parent 8e44bcada4
commit ea0a1a073e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 227 additions and 251 deletions

View file

@ -115,6 +115,90 @@ steps:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution.sh
label: 'Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 16
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_explore.sh
label: 'Explore - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 4
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
label: 'Investigations - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/defend_workflows.sh
label: 'Defend Workflows Cypress Tests'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh
label: 'Defend Workflows Cypress Tests on Serverless'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 120
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/threat_intelligence.sh
label: 'Threat Intelligence Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 120
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/osquery_cypress.sh
label: 'Osquery Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: '.buildkite/scripts/steps/functional/on_merge_unsupported_ftrs.sh'
label: Trigger unsupported ftr tests
timeout_in_minutes: 10

View file

@ -49,66 +49,6 @@ steps:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution.sh
label: 'Security Solution Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 120
parallelism: 14
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
label: 'Investigations - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 120
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_explore.sh
label: 'Explore - Security Solution Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/defend_workflows.sh
label: 'Defend Workflows Cypress Tests'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 120
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/osquery_cypress.sh
label: 'Osquery Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/synthetics_plugin.sh
label: 'Synthetics @elastic/synthetics Tests'
agents:

View file

@ -93,6 +93,136 @@ steps:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution.sh
label: 'Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 16
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_explore.sh
label: 'Explore - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 4
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
label: 'Investigations - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/defend_workflows.sh
label: 'Defend Workflows Cypress Tests'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh
label: 'Defend Workflows Cypress Tests on Serverless'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 120
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/threat_intelligence.sh
label: 'Threat Intelligence Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 120
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/osquery_cypress.sh
label: 'Osquery Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
artifact_paths:
- 'target/kibana-osquery/**/*'
- command: .buildkite/scripts/steps/functional/security_solution_burn.sh
label: 'Security Solution Cypress tests, burning changed specs'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 1
retry:
automatic: false
soft_fail: true
- command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
label: 'Security Solution OpenAPI codegen'
agents:
queue: n2-2-spot
timeout_in_minutes: 60
parallelism: 1
- command: .buildkite/scripts/steps/functional/osquery_cypress_burn.sh
label: 'Osquery Cypress Tests, burning changed specs'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
soft_fail: true
retry:
automatic: false
artifact_paths:
- 'target/kibana-osquery/**/*'
- command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh
label: 'Serverless Osquery Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
artifact_paths:
- 'target/kibana-osquery/**/*'
# status_exception: Native role management is not enabled in this Elasticsearch instance
# - command: .buildkite/scripts/steps/functional/security_serverless_defend_workflows.sh
# label: 'Serverless Security Defend Workflows Cypress Tests'

View file

@ -1,26 +0,0 @@
steps:
- command: .buildkite/scripts/steps/functional/defend_workflows.sh
label: 'Defend Workflows Cypress Tests'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 60
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh
label: 'Defend Workflows Cypress Tests on Serverless'
agents:
queue: n2-4-virt
depends_on: build
timeout_in_minutes: 120
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1

View file

@ -1,40 +0,0 @@
steps:
- command: .buildkite/scripts/steps/functional/osquery_cypress.sh
label: 'Osquery Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
artifact_paths:
- "target/kibana-osquery/**/*"
- command: .buildkite/scripts/steps/functional/osquery_cypress_burn.sh
label: 'Osquery Cypress Tests, burning changed specs'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
soft_fail: true
retry:
automatic: false
artifact_paths:
- "target/kibana-osquery/**/*"
- command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh
label: 'Serverless Osquery Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 50
parallelism: 6
retry:
automatic:
- exit_status: '*'
limit: 1
artifact_paths:
- "target/kibana-osquery/**/*"

View file

@ -1,54 +0,0 @@
steps:
- command: .buildkite/scripts/steps/functional/security_solution.sh
label: 'Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 16
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_explore.sh
label: 'Explore - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 4
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_investigations.sh
label: 'Investigations - Security Solution Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 8
retry:
automatic:
- exit_status: '*'
limit: 1
- command: .buildkite/scripts/steps/functional/security_solution_burn.sh
label: 'Security Solution Cypress tests, burning changed specs'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 60
parallelism: 1
retry:
automatic: false
soft_fail: true
- command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
label: 'Security Solution OpenAPI codegen'
agents:
queue: n2-2-spot
timeout_in_minutes: 60
parallelism: 1

View file

@ -1,12 +0,0 @@
steps:
- command: .buildkite/scripts/steps/functional/threat_intelligence.sh
label: 'Threat Intelligence Cypress Tests'
agents:
queue: n2-4-spot
depends_on: build
timeout_in_minutes: 120
parallelism: 2
retry:
automatic:
- exit_status: '*'
limit: 1

View file

@ -59,38 +59,6 @@ const uploadPipeline = (pipelineContent: string | object) => {
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/kbn_handlebars.yml'));
}
if (
(await doAnyChangesMatch([
/^src\/plugins\/controls/,
/^packages\/kbn-securitysolution-.*/,
/^x-pack\/plugins\/lists/,
/^x-pack\/plugins\/security_solution/,
/^x-pack\/plugins\/timelines/,
/^x-pack\/plugins\/triggers_actions_ui\/public\/application\/sections\/action_connector_form/,
/^x-pack\/plugins\/triggers_actions_ui\/public\/application\/sections\/alerts_table/,
/^x-pack\/plugins\/triggers_actions_ui\/public\/application\/context\/actions_connectors_context\.tsx/,
/^x-pack\/test\/defend_workflows_cypress/,
/^x-pack\/test\/security_solution_cypress/,
/^fleet_packages\.json/, // It contains reference to prebuilt detection rules, we want to run security solution tests if it changes
])) ||
GITHUB_PR_LABELS.includes('ci:all-cypress-suites')
) {
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/security_solution.yml'));
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/defend_workflows.yml'));
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml'));
}
if (
(await doAnyChangesMatch([
/^x-pack\/plugins\/threat_intelligence/,
/^x-pack\/test\/threat_intelligence_cypress/,
/^x-pack\/plugins\/security_solution\/public\/threat_intelligence/,
])) ||
GITHUB_PR_LABELS.includes('ci:all-cypress-suites')
) {
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/threat_intelligence.yml'));
}
if (
(await doAnyChangesMatch([
/^src\/plugins\/data/,
@ -140,16 +108,6 @@ const uploadPipeline = (pipelineContent: string | object) => {
GITHUB_PR_LABELS.includes('ci:all-cypress-suites')
) {
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/fleet_cypress.yml'));
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/defend_workflows.yml'));
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml'));
}
if (
((await doAnyChangesMatch([/^x-pack\/plugins\/osquery/, /^x-pack\/test\/osquery_cypress/])) ||
GITHUB_PR_LABELS.includes('ci:all-cypress-suites')) &&
!GITHUB_PR_LABELS.includes('ci:skip-cypress-osquery')
) {
pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml'));
}
if (await doAnyChangesMatch([/^x-pack\/plugins\/exploratory_view/])) {

View file

@ -94,6 +94,7 @@ export const InstallPrebuiltRulesResponse = z.object({
timelines_updated: z.number().int().min(0),
});
```
## Programmatic API
Alternatively, you can use the code generator programmatically. You can create a script file and run it with `node` command. This could be useful if you want to set up code generation in your CI pipeline. Here's an example of such script:
@ -135,26 +136,26 @@ check_for_changed_files "yarn openapi:generate" true
This scripts sets up the minimal environment required fro code generation and runs the code generation script. Then it checks if there are any changes and commits them if there are any using the `check_for_changed_files` function.
Then add the code generation script to your plugin build pipeline. Open your plugin build pipeline, for example `.buildkite/pipelines/pull_request/security_solution.yml`, and add the following command to the steps list adjusting the path to your code generation script:
Then add the code generation script to your plugin build pipeline. Open your plugin build pipeline, for example `.buildkite/pipelines/pull_request/base.yml`, and add the following command to the steps list adjusting the path to your code generation script:
```yaml
- command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
label: 'Security Solution OpenAPI codegen'
agents:
queue: n2-2-spot
timeout_in_minutes: 60
parallelism: 1
- command: .buildkite/scripts/steps/code_generation/security_solution_codegen.sh
label: 'Security Solution OpenAPI codegen'
agents:
queue: n2-2-spot
timeout_in_minutes: 60
parallelism: 1
```
Now on every pull request the code generation script will run and commit the changes if there are any.
## OpenAPI Schema
The code generator supports the OpenAPI definitions described in the request, response, and component sections of the document.
The code generator supports the OpenAPI definitions described in the request, response, and component sections of the document.
For every API operation (GET, POST, etc) it is required to specify the `operationId` field. This field is used to generate the name of the generated types. For example, if the `operationId` is `InstallPrebuiltRules` then the generated types will be named `InstallPrebuiltRulesResponse` and `InstallPrebuiltRulesRequest`. If the `operationId` is not specified then the code generation will throw an error.
The `x-codegen-enabled` field is used to enable or disable code generation for the operation. If it is not specified then code generation is disabled by default. This field could be also used to disable code generation of common components described in the `components` section of the OpenAPI specification.
The `x-codegen-enabled` field is used to enable or disable code generation for the operation. If it is not specified then code generation is disabled by default. This field could be also used to disable code generation of common components described in the `components` section of the OpenAPI specification.
Keep in mind that disabling code generation for common components that are referenced by external OpenAPI specifications could lead to errors during code generation.
@ -198,4 +199,4 @@ paths:
application/json:
schema:
$ref: '../../../model/rule_schema/rule_schemas.schema.yaml#/components/schemas/RuleResponse'
```
```

View file

@ -84,14 +84,9 @@ You can use this approach to load separate data dumps for every test case, to co
### CI Execution
The entry point for PR testing is `.buildkite/pipelines/pull_request/threat_intelligence.yml` file, see that for details on
The entry point for PR testing is `.buildkite/pipelines/pull_request/base.yml` file, see that for details on
how the test suite is executed & extra options regarding parallelism, retrying etc.
E2E tests for this plugin will only be executed if any of the files changed within the PR matches dependency list here:
`.buildkite/scripts/pipelines/pull_request/pipeline.ts`
It is also possible to run all tests by attaching a PR flag: `ci:all-cypress-suites`.
## FAQ
### How is the Threat Intelligence code loaded in Kibana?
@ -110,4 +105,4 @@ See [CONTRIBUTING.md](https://github.com/elastic/kibana/blob/main/x-pack/plugins
## Issues
Please report any issues in [this GitHub project](https://github.com/orgs/elastic/projects/758/).
Please report any issues in [this GitHub project](https://github.com/orgs/elastic/projects/758/).