**Epics:** https://github.com/elastic/security-team/issues/8058,
https://github.com/elastic/security-team/issues/6726 (internal)
**Partially addresses:**
https://github.com/elastic/security-team/issues/7991 (internal)
## Summary
The main benefit of this PR is shown in `rule_request_schema.test.ts`,
where the error messages are now more accurate and concise. With regular
unions, `zod` has to try validating the input against all schemas in the
union and reports the errors from every schema in the union. Switching
to discriminated unions, with `type` as the discriminator, allows `zod`
to pick the right rule type schema from the union and only validate
against that rule type. This means the error message reports that either
the discriminator is invalid, in any case where `type` is not valid, or
if `type` is valid but another field is wrong for that type of rule then
the error message is the validation result from only that rule type.
To make it possible to use discriminated unions, we need to switch from
using zod's `.and()` for intersections to `.merge()` because `.and()`
returns an intersection type that is incompatible with discriminated
unions in zod. Similarly, we need to remove the usage of `.transform()`
because it returns a ZodEffect that is incompatible with `.merge()`.
Instead of using `.transform()` to turn properties from optional to
possibly undefined, we can use `requiredOptional` explicitly in specific
places to convert the types. Similarly, the `RequiredOptional` type can
be used on the return type of conversion functions between API and
internal schemas to enforce that all properties are explicitly specified
in the conversion.
Future work:
- better alignment of codegen with OpenAPI definitions of anyOf/oneOf.
https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/#oneof
oneOf requires that the input match exactly one schema from the list,
which is different from z.union. anyOf should be z.union, oneOf should
be z.discriminatedUnion
- flatten the schema structure further to avoid `Type instantiation is
excessively deep and possibly infinite`. Seems to be a common issue with
zod (https://github.com/microsoft/TypeScript/issues/34933) Limiting the
number of `.merge` and other zod operations needed to build a particular
schema object seems to help resolve the error. Combining
`ResponseRequiredFields` and `ResponseOptionalFields` into a single
object rather than merging them solved the immediate problem. However,
we may still be near the depth limit. Changing `RuleResponse` as seen
below also solved the problem in testing, and may give us more headroom
for future changes if we apply techniques like this here and in other
places. The difference here is that `SharedResponseProps` is only
intersected with the type specific schemas after they're combined in a
discriminated union, whereas in `main` we merge `SharedResponseProps`
with each individual schema then merge them all together.
- combine other Required and Optional schemas, like
QueryRuleRequiredFields and QueryRuleOptionalFields
```ts
export type RuleResponse = z.infer<typeof RuleResponse>;
export const RuleResponse = SharedResponseProps.and(z.discriminatedUnion('type', [
EqlRuleResponseFields,
QueryRuleResponseFields,
SavedQueryRuleResponseFields,
ThresholdRuleResponseFields,
ThreatMatchRuleResponseFields,
MachineLearningRuleResponseFields,
NewTermsRuleResponseFields,
EsqlRuleResponseFields,
]));
```
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
❓ Are there tests for @kbn/openapi-generator? I couldn't find them ❓
Our team had a `@timestamp` field in an API response, this caused the
zod schema to break because the key was not being escaped in the object
e.g:
```js
export const AssetCriticalityRecord = CreateAssetCriticalityRecord.and(z.object({
// ❌ invalid JS
@timestamp: z.string().datetime(),
}));
```
With the fix, the `@timestamp` key is surrounded by quotes.
```js
export const AssetCriticalityRecord = CreateAssetCriticalityRecord.and(z.object({
// ✅
'@timestamp': z.string().datetime(),
}));
```
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
**Related to: https://github.com/elastic/security-team/issues/7491**
## Summary
Migrated remaining public Detection Engine endpoints to OpenAPI schema
and code generation:
- `POST /api/detection_engine/rules/_bulk_action`
- `GET /api/detection_engine/rules/_find`
Also completed the migration of internal APIs:
- `GET /internal/detection_engine/rules/{ruleId}/execution/events`
- `GET /internal/detection_engine/rules/{ruleId}/execution/results`
### Other notable changes
- Changed how we compose Zod error messages for unions, see
`packages/kbn-zod-helpers/src/stringify_zod_error.ts`. Now we are trying
to list the validation errors of all union members but limiting the
total number of validation errors displayed to users.
- Addressed some remaining `TODO
https://github.com/elastic/security-team/issues/7491`
- Removed dependencies of the risk engine and timelines on detection
engine schemas
- Removed outdated legacy rule schemas that are no longer in use
- Added new schema helpers that work with query params:
`BooleanFromString` and `ArrayFromString`
**Parent meta ticket:
https://github.com/elastic/security-team/issues/7491**
Resolves: https://github.com/elastic/security-team/issues/7582
Resolves: https://github.com/elastic/security-team/issues/7580
Resolves: https://github.com/elastic/security-team/issues/7581
## Summary
This PR migrates the rules schema to OpenAPI, Zod, and code generation.
The following APIs now have complete OpenAPI specifications and are
enabled for code generation:
| Method | Endpoint | OpenAPI spec | Fully migrated |
| ------ |
---------------------------------------------------------------- |
------------ | -------------- |
| POST | /api/detection_engine/rules | ✅ | ✅ |
| GET | /api/detection_engine/rules | ✅ | ✅ |
| PUT | /api/detection_engine/rules | ✅ | ✅ |
| PATCH | /api/detection_engine/rules | ✅ | ✅ |
| DELETE | /api/detection_engine/rules | ✅ | ✅ |
| POST | /api/detection_engine/rules/\_bulk_create | ✅ | ✅ |
| PUT | /api/detection_engine/rules/\_bulk_update | ✅ | ✅ |
| PATCH | /api/detection_engine/rules/\_bulk_update | ✅ | ✅ |
| DELETE | /api/detection_engine/rules/\_bulk_delete | ✅ | ✅ |
| POST | /api/detection_engine/rules/\_bulk_delete | ✅ | ✅ |
### Rule schemas are now forward-compatible
We now allow extra fields in schemas for forward compatibility, but we
remove them from the payload during parsing. So from now on, extra
fields are simply ignored and won't lead to validation errors.
## Summary
Security Solution writes e2e and other tests using Cypress. In the past,
these tests, if they failed on a tracked branch, couldn't be easily
skipped. They also weren't run in parallelized jobs. For primarily these
reasons, they didn't run on most Kibana PRs.
This PR moves these Cypress tests back to the main PR pipeline. Tests
that fail on tracked branches create (or update) Github issues which can
be used with the skip-test github workflow script to easily skip the
failing tests. The pipeline steps are parallelized and run in under 40
minutes.
### Open Questions
- [ ] Should this PR enable Serverless Security Defend Workflows Cypress
Tests @patrykkopycinski
### Some buildkite pipelines that used to run only on Security PRs now
run on all PRs:
These steps run on all PRs with these changes
- Security Solution Cypress Tests (general tests that haven't been
organized into an area team)
- Explore tests
- Investigations Tests
- Defend Workflows Tests
- Defend Workflows Serverless
- Threat Intel Tests
- OS Query Tests
- Security Solution Burning Changed Specs (these run only recently
changed specs a few extra times)
- Security Solution OpenAPI codegen
- OSQuery burning
- OSQuery Serverless
<details>
<summary><b>And these already run on all PRs</b></summary>
- Serverless Security Cypress Tests
- Serverless Explore tests
- Serverless Investigations Tests
</details>
### Security Cypress tests run in the main `on merge` pipeline instead
of the `on merge unsupported ftrs` pipeline:
These steps run in the `on merge` pipeline with these changes:
- Security Solution Cypress Tests
- Explore Cypress Tests
- Investigations Cypress Tests
- Defend Workflows Cypress Tests
- Defend Workflows Serverless Cypress Tests
- Threat Intelligence Cypress Tests
- Osquery Cypress Tests
<details>
<summary><b>and these already run on the `on merge`
pipeline</b></summary>
- Serverless Security Cypress Tests
- Serverless Explore - Security Solution Cypress Tests
- Serverless Investigations - Security Solution Cypress Tests
</details>
### Additional work to be done:
We need to consolidate build steps, enhance test skipping to support
Cypress-grep flags, avoid out-of-memory errors in cypress, enhance
parallelization, improve Cypress reporting, and probably other things.
These are tracked separately. Reach out to me if you need details.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
