# Backport
This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Support for kibana spaces in openapi generated
securitySolutionApi service for integration tests
(#194029)](https://github.com/elastic/kibana/pull/194029)
<!--- Backport version: 8.9.8 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Tiago Vila
Verde","email":"tiago.vilaverde@elastic.co"},"sourceCommit":{"committedDate":"2024-09-26T17:53:25Z","message":"[Security
Solution] Support for kibana spaces in openapi generated
securitySolutionApi service for integration tests (#194029)\n\n###
Summary\r\n\r\nThis PR adds support for Kibana Spaces in the
generated\r\n`securitySolutionApi` service for integration
tests.\r\n\r\nUsers can now pass an extra parameter `kibanaSpace` when
calling a route\r\nfrom the service. The provided space will be prefixed
to the API's path.\r\nIf no argument is provided, it is default'ed to
`'default'`\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"2f45c90d2f58e9f0e3b5ff90f7cbece482478c00","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","v8.16.0"],"number":194029,"url":"https://github.com/elastic/kibana/pull/194029","mergeCommit":{"message":"[Security
Solution] Support for kibana spaces in openapi generated
securitySolutionApi service for integration tests (#194029)\n\n###
Summary\r\n\r\nThis PR adds support for Kibana Spaces in the
generated\r\n`securitySolutionApi` service for integration
tests.\r\n\r\nUsers can now pass an extra parameter `kibanaSpace` when
calling a route\r\nfrom the service. The provided space will be prefixed
to the API's path.\r\nIf no argument is provided, it is default'ed to
`'default'`\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"2f45c90d2f58e9f0e3b5ff90f7cbece482478c00"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194029","number":194029,"mergeCommit":{"message":"[Security
Solution] Support for kibana spaces in openapi generated
securitySolutionApi service for integration tests (#194029)\n\n###
Summary\r\n\r\nThis PR adds support for Kibana Spaces in the
generated\r\n`securitySolutionApi` service for integration
tests.\r\n\r\nUsers can now pass an extra parameter `kibanaSpace` when
calling a route\r\nfrom the service. The provided space will be prefixed
to the API's path.\r\nIf no argument is provided, it is default'ed to
`'default'`\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"2f45c90d2f58e9f0e3b5ff90f7cbece482478c00"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Creates CLI script tooling for building data, rules, exceptions, and
lists in any (local, cloud, serverless) environment for manual testing.
The initial commits here add generated clients for accessing security
solution, exceptions, and lists APIs and a placeholder script where
those clients are set up for use. See README for more details.
Much of the code in this PR is auto-generated clients. The hand written
code is intended to be primarily in `quickstart/modules/`, where we can
add wrapper code to simplify the process for common test environment
setup. For example, `createValueListException` takes an array of items
and some metadata and automatically creates a new value list and an
exception that references that value list. `/modules/data/` contains
functions to generate documents of arbitrary size, and we can add more
functions to create various other types of documents.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#187638
## Summary
In this [PR](https://github.com/elastic/kibana/pull/186190), we
introduced @kbn/zod package and an OAS convertor to automatically
generate Open API Specifications for the routes that use zod for their
validation. In this PR, we add an eslint rule to enforce importing from
@kbn/zod instead of zod directly.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Relates to:** https://github.com/elastic/security-team/issues/9723 (internal)
## Summary
This PR adds a User Guide for OpenAPI Code Generator. It's based on hands on experience while adding and modifying OpenAPI specs plus questions incoming from various engineers.
The User Guides should help streamlining code generation with OpenAPI specs by answering frequent questions.
**Relates to:** https://github.com/elastic/security-team/issues/9401
## Summary
Disabling OpenAPI spec linting in https://github.com/elastic/kibana/pull/179074 lead to accumulating invalid OpenAPi specs.
This PR enables OpenAPI linting for Security Solution plugin and make appropriate fixes to make the linting pass.
## Details
OpenAPI linting is a part of code generation. It runs automatically but can be disabled via `skipLinting: true`. Code generation with disabled linting isn't able to catch all possible problems in processing specs.
The majority of problems came from Entity Analytics and Osquery OpenAPI specs. These specs were fixed and refactored to enable code generation and integrate generated artefacts into routes to make sure OpenAPI spec match API endpoints they describe. It helped to catch some subtle inconsistencies.
**Closes:** https://github.com/elastic/kibana/issues/182928
## Summary
This PR modifies `kbn-openapi-generator` to produce Zod schemas and related TS types for shared schemas in `components.schemas` in a proper order independent from OpenAPI spec schemas definition order.
## Details
Current `kbn-openapi-generator` implementation to generate Zod schemas and related TS types for OpenAPI's spec file `components.schemas` is straightforward. It repeats schemas definition order which can lead to cases when a dependent artefact is defined before its dependencies. Engineers have to manually order schemas in OpenAPI spec files to make sure produce artefacts are valid TS files while OpenAPI specification doesn't impose such requirements.
To illustrate the problem let's consider a following OpenAPI spec
```yaml
...
components:
x-codegen-enabled: true
schemas:
MainSchema:
type: object
properties:
fieldA:
$ref: '#/components/schemas/DepA'
nullable: true
fieldB:
$ref: '#/components/schemas/DepB'
fieldC:
$ref: '#/components/schemas/DepC'
DepB:
type: boolean
DepC:
type: integer
DepA:
type: string
```
Running code generation for the spec above produces the following artefacts
```ts
import { z } from 'zod';
export type MainSchema = z.infer<typeof MainSchema>;
export const MainSchema = z.object({
fieldA: DepA.nullable().optional(),
fieldB: DepB.optional(),
fieldC: DepC.optional(),
});
export type DepB = z.infer<typeof DepB>;
export const DepB = z.boolean();
export type DepC = z.infer<typeof DepC>;
export const DepC = z.number().int();
export type DepA = z.infer<typeof DepA>;
export const DepA = z.string();
```
which is not valid since dependencies are defined after the dependent `MainSchema`.
### After the fix
The fix takes into account that references chain represents a graph in a common case. When there are no cycles in references they represent a [Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph). In graph's terminology schema is a node and reference is an edge. There are `[topological sorting algorithms](https://en.wikipedia.org/wiki/Topological_sorting)` which order DAG nodes starting from zero incoming edges to the maximum number. There is an ability to take into account cycles so the result has sorted nodes which don't take part in cycles. References sorted this way have dependencies defined before dependent schemas.
After applying this fix and running code generation for the OpenAPI spec above we will get the following
```ts
import { z } from 'zod';
export type DepA = z.infer<typeof DepA>;
export const DepA = z.string();
export type DepB = z.infer<typeof DepB>;
export const DepB = z.boolean();
export type DepC = z.infer<typeof DepC>;
export const DepC = z.number().int();
export type MainSchema = z.infer<typeof MainSchema>;
export const MainSchema = z.object({
fieldA: DepA.nullable().optional(),
fieldB: DepB.optional(),
fieldC: DepC.optional(),
});
```
### Notes
- Implementation preserves original schemas order when possible.
Generally speaking topological sorting doesn't define relative order in a group of elements with the same number of incoming edges (dependencies in our case). It means the result ordering can vary. **To reduce the diff topological sorting implementation in this PR preserves original schemas order when possible**. When sorting is necessary schemas are places in dependencies discovery order. You can see that OpenAPI spec above has schemas ordered like `DepB`, `DepC` and `DepA` while prodiced TS file has them ordered `DepA`, `DepB` and `DepC` since it's the order the dependencies were discovered in the `MainSchema`.
- There are two way in how to implement topological sorting Depth-First Search (DFS) and Breadth-First Search (BFS). Implementation in this PR uses recursive DFS implementation since it allows to preserve the original order where possible and has better readability.
**Relates to:** https://github.com/elastic/kibana/issues/186066, https://github.com/elastic/kibana/pull/186221
## Summary
This PR fixes generated TS files for circular OpenAPI schemas when non circular (internal or external) schema is used.
## Details
https://github.com/elastic/kibana/pull/186221 added code generation support for circular schemas. Such schemas have input TS types generated which may depend on the other circular or non circular TS types. The problem appears when a circular schema uses a non circular schema. Generated code expects an input type for used schemas exist but it's not a case non circular schemas.
Let's consider a following OpenAPI spec with a self circular schema and a field referencing `NonEmptyString` schema
```yaml
...
components:
x-codegen-enabled: true
schemas:
SelfCircular:
type: object
properties:
circularField:
$ref: '#/components/schemas/SelfCircular'
stringField:
$ref: '../model/primitives.schema.yaml#/components/schemas/NonEmptyString'
```
where a generated TS file looks like
```ts
import type { ZodTypeDef } from 'zod';
import { z } from 'zod';
import { NonEmptyString } from '../model/primitives.gen';
export interface SelfCircular {
circularField?: SelfCircular;
stringField?: NonEmptyString;
}
export interface SelfCircularInput {
circularField?: SelfCircularInput;
stringField?: NonEmptyStringInput;
}
export const SelfCircular: z.ZodType<SelfCircular, ZodTypeDef, SelfCircularInput> = z.object({
circularField: z.lazy(() => SelfCircular).optional(),
stringField: NonEmptyString.optional(),
});
```
You can notice the generated TS file contains usage of `NonEmptyStringInput` which doesn't exist.
**After applying the fix the generated TS file looks like**
```ts
import type { ZodTypeDef } from 'zod';
import { z } from 'zod';
import { NonEmptyString } from '../model/primitives.gen';
export interface SelfCircular {
circularField?: SelfCircular;
stringField?: NonEmptyString;
}
export interface SelfCircularInput {
circularField?: SelfCircularInput;
stringField?: NonEmptyString;
}
export const SelfCircular: z.ZodType<SelfCircular, ZodTypeDef, SelfCircularInput> = z.object({
circularField: z.lazy(() => SelfCircular).optional(),
stringField: NonEmptyString.optional(),
});
```
**Addresses:** https://github.com/elastic/kibana/issues/183661
## Summary
This PR adds missing OpenAPI specs for the Alert Index API endpoints available in ESS
- `POST /api/detection_engine/index`
- `GET /api/detection_engine/index`
- `DELETE /api/detection_engine/index`
**Addresses:** https://github.com/elastic/kibana/issues/186066
## Summary
This PR adds `kbn-openapi-generator` support for local OpenAPI circular references.
## Details
Circular references represent a problem for OpenAPI adoption since lack of support forces engineers to disable code generation and manually define Zod schemas and related TS types. This PR brings `kbn-openapi-generator` support for local OpenAPI circular references. Local references means references to schemas inside one document which looks like `#/components/schemas/MySchema`. It should cover the majority of cases when circular references support id required.
The feature is implemented by detecting local circular references and applying a different generation template for involved in cycles inspired by [Zod recursive types docs section](https://github.com/colinhacks/zod?tab=readme-ov-file#recursive-types) for schemas. It involves TS types generation since TS isn't able to properly infer types in this case and hinting circular Zod schemas via `z.ZodType<>`. On top of that circular schema usages are wrapped in `z.lazy()`.
## What's not implemented?
- Multi-file circular references aren't supported
## How to test
Security Solution doesn't have OpenAPI circular schemas yet. But generator's behavior has been verified on the following specs
- Self-recursive
Spec:
```yaml
components:
x-codegen-enabled: true
schemas:
SelfRecursive:
type: object
properties:
fieldA:
$ref: '#/components/schemas/SelfRecursive'
```
Generated output:
```ts
import type { ZodTypeDef } from 'zod';
import { z } from 'zod';
export interface SchemaA {
fieldA?: SchemaA;
}
export interface SchemaAInput {
fieldA?: SchemaAInput;
}
export const SchemaA: z.ZodType<SchemaA, ZodTypeDef, SchemaAInput> = z.object({
fieldA: z.lazy(() => SchemaA).optional(),
});
```
- Two circular schemas
Spec:
```yaml
components:
x-codegen-enabled: true
schemas:
SchemaA:
type: object
properties:
recursiveFieldB:
$ref: '#/components/schemas/SchemaB'
SchemaB:
type: object
properties:
recursiveFieldA:
$ref: '#/components/schemas/SchemaA'
```
Generated output:
```ts
import type { ZodTypeDef } from 'zod';
import { z } from 'zod';
export interface SchemaA {
recursiveFieldB?: SchemaB;
}
export interface SchemaAInput {
recursiveFieldB?: SchemaBInput;
}
export const SchemaA: z.ZodType<SchemaA, ZodTypeDef, SchemaAInput> = z.object({
recursiveFieldB: z.lazy(() => SchemaB).optional(),
});
export interface SchemaB {
recursiveFieldA?: SchemaA;
}
export interface SchemaBInput {
recursiveFieldA?: SchemaAInput;
}
export const SchemaB: z.ZodType<SchemaB, ZodTypeDef, SchemaBInput> = z.object({
recursiveFieldA: z.lazy(() => SchemaA).optional(),
});
```
- More complex example with circular with `anyOf` and non circular schemas
Spec:
```yaml
components:
x-codegen-enabled: true
schemas:
SchemaA:
anyOf:
- $ref: '#/components/schemas/SchemaB'
- $ref: '#/components/schemas/SchemaC'
- type: string
enum: [valueA, valueB]
SchemaB:
type: object
properties:
fieldB:
$ref: '#/components/schemas/SchemaC'
defaultableField:
type: number
default: 1
required: [fieldB]
SchemaC:
type: object
properties:
fieldC:
$ref: '#/components/schemas/SchemaA'
SchemaZ:
anyOf:
- type: object
properties:
fieldZ:
type: string
defField:
type: number
default: 1
required: [fieldZ]
- type: object
properties:
fieldX:
type: boolean
required: [fieldX]
```
Generated output:
```ts
import type { ZodTypeDef } from 'zod';
import { z } from 'zod';
export type SchemaA = SchemaB | SchemaC | 'valueA' | 'valueB';
export type SchemaAInput = SchemaBInput | SchemaCInput | 'valueA' | 'valueB';
export const SchemaA: z.ZodType<SchemaA, ZodTypeDef, SchemaAInput> = z.union([
z.lazy(() => SchemaB),
z.lazy(() => SchemaC),
z.enum(['valueA', 'valueB']),
]);
export interface SchemaB {
fieldB: SchemaC;
defaultableField: number;
}
export interface SchemaBInput {
fieldB: SchemaCInput;
defaultableField?: number;
}
export const SchemaB: z.ZodType<SchemaB, ZodTypeDef, SchemaBInput> = z.object({
fieldB: z.lazy(() => SchemaC),
defaultableField: z.number().optional().default(1),
});
export interface SchemaC {
fieldC?: SchemaA;
}
export interface SchemaCInput {
fieldC?: SchemaAInput;
}
export const SchemaC: z.ZodType<SchemaC, ZodTypeDef, SchemaCInput> = z.object({
fieldC: z.lazy(() => SchemaA).optional(),
});
export type SchemaZ = z.infer<typeof SchemaZ>;
export const SchemaZ = z.union([
z.object({
fieldZ: z.string(),
defField: z.number().optional().default(1),
}),
z.object({
fieldX: z.boolean(),
}),
]);
```
- Real life example provided by @bhapas while working on OpenAPI specs for Integration Assistant APIs
Spec:
```yaml
components:
x-codegen-enabled: true
schemas:
ESProcessorItem:
type: object
description: Processor item for the Elasticsearch processor.
additionalProperties:
$ref: '#/components/schemas/ESProcessorOptions'
ESProcessorOptions:
type: object
description: Processor options for the Elasticsearch processor.
properties:
on_failure:
type: array
items:
$ref: '#/components/schemas/ESProcessorItem'
description: An array of items to execute if the processor fails.
ignore_failure:
type: boolean
description: If true, the processor continues to the next processor if the current processor fails.
ignore_missing:
type: boolean
description: If true, the processor continues to the next processor if the field is missing.
if:
type: string
description: Conditionally execute the processor.
tag:
type: string
description: A tag to assign to the document after processing.
additionalProperties: true
```
Generated output:
```ts
import type { ZodTypeDef } from 'zod';
import { z } from 'zod';
/**
* Processor item for the Elasticsearch processor.
*/
export interface ESProcessorItem {
[key: string]: ESProcessorOptions;
}
export interface ESProcessorItemInput {
[key: string]: ESProcessorOptionsInput;
}
export const ESProcessorItem: z.ZodType<ESProcessorItem, ZodTypeDef, ESProcessorItemInput> = z
.object({})
.catchall(z.lazy(() => ESProcessorOptions));
/**
* Processor options for the Elasticsearch processor.
*/
export interface ESProcessorOptions {
/**
* An array of items to execute if the processor fails.
*/
on_failure?: ESProcessorItem[];
/**
* If true, the processor continues to the next processor if the current processor fails.
*/
ignore_failure?: boolean;
/**
* If true, the processor continues to the next processor if the field is missing.
*/
ignore_missing?: boolean;
/**
* Conditionally execute the processor.
*/
if?: string;
/**
* A tag to assign to the document after processing.
*/
tag?: string;
[key: string]: unknown;
}
export interface ESProcessorOptionsInput {
/**
* An array of items to execute if the processor fails.
*/
on_failure?: ESProcessorItemInput[];
/**
* If true, the processor continues to the next processor if the current processor fails.
*/
ignore_failure?: boolean;
/**
* If true, the processor continues to the next processor if the field is missing.
*/
ignore_missing?: boolean;
/**
* Conditionally execute the processor.
*/
if?: string;
/**
* A tag to assign to the document after processing.
*/
tag?: string;
[key: string]: unknown;
}
export const ESProcessorOptions: z.ZodType<
ESProcessorOptions,
ZodTypeDef,
ESProcessorOptionsInput
> = z
.object({
/**
* An array of items to execute if the processor fails.
*/
on_failure: z.array(z.lazy(() => ESProcessorItem)).optional(),
/**
* If true, the processor continues to the next processor if the current processor fails.
*/
ignore_failure: z.boolean().optional(),
/**
* If true, the processor continues to the next processor if the field is missing.
*/
ignore_missing: z.boolean().optional(),
/**
* Conditionally execute the processor.
*/
if: z.string().optional(),
/**
* A tag to assign to the document after processing.
*/
tag: z.string().optional(),
})
.catchall(z.unknown());
```
## Summary
This PR moves disclaimer comment section to the top of the generated
files to support `import/order` ESlint rule.
## Details
`kbn-openapi-generator` generates `<schema-name>.gen.ts` files for each
encountered schema with enabled code generation. The generate file
contains imports of the referenced schema from the other generated
files. Everything works until there is a reference to a package or
another plugin. Consider an example below where we have a generated file
with an import from `kbn-openapi-common` package (this package doesn't
exist in reality and used for clarity but the same can be shown with
cross plugin references)
```ts
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { z } from 'zod';
/*
* NOTICE: Do not edit this file manually.
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
*
* info:
* title: Shared Alert Primitives Schema
* version: not applicable
*/
import { NonEmptyString } from '@kbn/openapi-common/primitives';
/**
* A list of alerts ids.
*/
export type AlertIds = z.infer<typeof AlertIds>;
export const AlertIds = z.array(NonEmptyString).min(1);
```
If `import/order` rule is enabled for this file linting with fixing will
fail with an error `8:1 error There should be no empty line within
import group import/order` since auto-fix can't fix the file due to the
comment between imports. Linting with auto-fixing is a part of code
generation process which means code generation will fail in that case.
For example lists plugin has `import/order` rule enabled.
The problem is fixed by moving disclaimer (NOTICE) section to the top
just right above the first import. Since the whole file is
auto-generated it makes sense.
**Resolves: https://github.com/elastic/kibana/issues/180121**
**Resolves: https://github.com/elastic/kibana/issues/180122**
**Resolves: https://github.com/elastic/kibana/issues/180124**
## Summary
As part of the preparatory changes for the work in Milestone 3, we want
to add the new `rule_source` field to the API schema.
- Added `rule_source` as an **optional** property to `RuleResponse`, by
introducing it as an optional property in the `ResponseFields` schema.
- For now, all endpoints should return `undefined` for the `rule_source`
field.
- Added `rule_source` as an **optional** property to `RuleToImport`,
which defines the schema of required and accepted fields when importing
a rule.
- For now, the new `rule_source` field should be ignored in the endpoint
logic.
- Added the `ruleSource` field to the `BaseRuleParams` schema, as an
optional field.
- Implemented a Zod transformation from `snake_case` to `camelCase` for
object keys to reduce code duplication.
## Summary
Closes https://github.com/elastic/kibana/issues/181948
We do not support URLs in `$ref` fields but we were previously exiting
with success and generating an invalid .ts file. This makes the error
more explicit.
I have also removed the reference which drew our attention to this bug.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Pablo Machado <pablo.nevesmachado@elastic.co>
## Summary
- Extended the code generation package to allow for bundling of OpenAPI
specifications before passing them to a template. Here's an example of
how the new `bundle` option can be used:
```ts
await generate({
title: 'API client for tests',
rootDir: SECURITY_SOLUTION_ROOT,
sourceGlob: './**/*.schema.yaml',
templateName: 'api_client_supertest',
skipLinting: true,
bundle: {
outFile: join(REPO_ROOT,
'x-pack/test/api_integration/services/security_solution_api.gen.ts'),
},
})
```
- Added a new template
(`packages/kbn-openapi-generator/src/template_service/templates/api_client_supertest.handlebars`)
that allows to generate an API client to be used in integrational tests.
The template outputs a collection of all Security Solution APIs that is
later added to `FtrProviderContext` for easy access in test files.
- The generated client is located in
`x-pack/test/api_integration/services/security_solution_api.gen.ts`
- It is now used in some detection engine API integration tests.
Before:
```ts
const { body } = await supertest
.post(DETECTION_ENGINE_RULES_URL)
.set('kbn-xsrf', 'true')
.set(ELASTIC_HTTP_VERSION_HEADER, '2023-10-31')
.send(rule)
.expect(200);
```
After:
```ts
const securitySolutionApi = getService('securitySolutionApi');
const { body } = await securitySolutionApi.createRule({ body: rule
}).expect(200);
```
- All API methods of the generated client now have correct types defined
for body, query, and param arguments
## Summary
Follow up PR from https://github.com/elastic/kibana/pull/174317 with the
following fixes/enhancements to `kbn-openapi-generator`:
* Fix extraneous `.`'s in paths causing generated files to not be
written to disk
* Updates `README.md` for latest method of adding CI actions
* Adds `info` details to generated metadata comment for more easily
tracing back to source schema
* Moves assistant `*.schema.yaml` files from `elastic_assistant` plugin
to `kbn-elastic-assistant-common` package
> [!NOTE]
> This PR includes a manual run of the `kbn-elastic-assistant-common`
package `yarn openapi:generate` script as a reference example. Since
this PR also updates the generation template to include the `info`
metadata, CI will run the generator for the other consumers
(`security_solution` & `osquery`) automatically, and commit those
updates to this PR. <img width="16"
src="https://user-images.githubusercontent.com/2946766/160040365-b1b8bb8a-d2d7-4187-b9b9-04817f8e2ae5.gif"
/>
### Test instructions
You can test against the `kbn-elastic-assistant-common` package using
either the main CLI script from kibana root:
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
or via the yarn command:
```
cd x-pack/packages/kbn-elastic-assistant-common/
yarn openapi-generate
```
### Checklist
Delete any items that are not applicable to this PR.
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/security-team/issues/8099**
## Summary
This PR adds an a command to lint OpenAPI specs defined in Security
Solution plugin.
## Details
We have a number of OpenAPI specs defined in Security Solution plugin.
While `@kbn/openapi-generator` package processes the specs and makes
sure the specs are parsable and processable we don't have proper specs
linter set up.
This PR introduces OpenAPI specs linting by leveraging [Redocly
CLI](https://github.com/Redocly/redocly-cli)'s `lint` command with a
custom configuration placed in `@kbn/openapi-generator` package .
Configuration includes reasonable best practices by using [built-in
Redocly rules](https://redocly.com/docs/cli/rules/built-in-rules/).
The lint utility fulfil the following requirements
- Validates yaml files against OpenAPI specification. It supports `3.0`
and `3.1`.
- Validates `x-modify` property to have only `partial`, `required` or
`requiredOptional` values.
- Checks for reasonable best practices and displays a warning message
when it's not met. Reasonable best practices are based on the
[recommended
ruleset](https://redocly.com/docs/cli/rules/recommended/#recommended-ruleset).
The lint utility has been incorporated into the existing OpenAPI
generator, and linting is performed before generation.
### Tool selection
[Swagger CLI](https://github.com/APIDevTools/swagger-cli) is a well
known tool to validate OpenAPI specs. On November 15th 2023 the repo has
been archived with a message in README
> ⚠️ Swagger CLI has been deprecated, due to the maintenance burnden of
trying to keep up with expectations of a huge userbase with little to no
pull requests or support. [Redocly
CLI](https://redocly.com/redocly-cli/) covers all of the same
functionality, and has more advanced linting with custom rules, and we
highly recommend using that instead. They have conveniently provided a
[migration
guide](https://redocly.com/docs/cli/guides/migrate-from-swagger-cli/)
for existing Swagger CLI users. Read the review of [Redocly CLI from
APIs You Won't Hate](https://apisyouwonthate.com/blog/redocly-cli/).
Taking it into account choice falls on **Redocly CLI**.
## How to test?
Change directory to Security Solution plugin's root and run linting by
using the following commands from Kibana root
```sh
cd x-pack/plugins/security_solution
yarn openapi:generate
```
---------
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
**Epics:** https://github.com/elastic/security-team/issues/8058,
https://github.com/elastic/security-team/issues/6726 (internal)
**Partially addresses:**
https://github.com/elastic/security-team/issues/7991 (internal)
## Summary
The main benefit of this PR is shown in `rule_request_schema.test.ts`,
where the error messages are now more accurate and concise. With regular
unions, `zod` has to try validating the input against all schemas in the
union and reports the errors from every schema in the union. Switching
to discriminated unions, with `type` as the discriminator, allows `zod`
to pick the right rule type schema from the union and only validate
against that rule type. This means the error message reports that either
the discriminator is invalid, in any case where `type` is not valid, or
if `type` is valid but another field is wrong for that type of rule then
the error message is the validation result from only that rule type.
To make it possible to use discriminated unions, we need to switch from
using zod's `.and()` for intersections to `.merge()` because `.and()`
returns an intersection type that is incompatible with discriminated
unions in zod. Similarly, we need to remove the usage of `.transform()`
because it returns a ZodEffect that is incompatible with `.merge()`.
Instead of using `.transform()` to turn properties from optional to
possibly undefined, we can use `requiredOptional` explicitly in specific
places to convert the types. Similarly, the `RequiredOptional` type can
be used on the return type of conversion functions between API and
internal schemas to enforce that all properties are explicitly specified
in the conversion.
Future work:
- better alignment of codegen with OpenAPI definitions of anyOf/oneOf.
https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/#oneof
oneOf requires that the input match exactly one schema from the list,
which is different from z.union. anyOf should be z.union, oneOf should
be z.discriminatedUnion
- flatten the schema structure further to avoid `Type instantiation is
excessively deep and possibly infinite`. Seems to be a common issue with
zod (https://github.com/microsoft/TypeScript/issues/34933) Limiting the
number of `.merge` and other zod operations needed to build a particular
schema object seems to help resolve the error. Combining
`ResponseRequiredFields` and `ResponseOptionalFields` into a single
object rather than merging them solved the immediate problem. However,
we may still be near the depth limit. Changing `RuleResponse` as seen
below also solved the problem in testing, and may give us more headroom
for future changes if we apply techniques like this here and in other
places. The difference here is that `SharedResponseProps` is only
intersected with the type specific schemas after they're combined in a
discriminated union, whereas in `main` we merge `SharedResponseProps`
with each individual schema then merge them all together.
- combine other Required and Optional schemas, like
QueryRuleRequiredFields and QueryRuleOptionalFields
```ts
export type RuleResponse = z.infer<typeof RuleResponse>;
export const RuleResponse = SharedResponseProps.and(z.discriminatedUnion('type', [
EqlRuleResponseFields,
QueryRuleResponseFields,
SavedQueryRuleResponseFields,
ThresholdRuleResponseFields,
ThreatMatchRuleResponseFields,
MachineLearningRuleResponseFields,
NewTermsRuleResponseFields,
EsqlRuleResponseFields,
]));
```
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
❓ Are there tests for @kbn/openapi-generator? I couldn't find them ❓
Our team had a `@timestamp` field in an API response, this caused the
zod schema to break because the key was not being escaped in the object
e.g:
```js
export const AssetCriticalityRecord = CreateAssetCriticalityRecord.and(z.object({
// ❌ invalid JS
@timestamp: z.string().datetime(),
}));
```
With the fix, the `@timestamp` key is surrounded by quotes.
```js
export const AssetCriticalityRecord = CreateAssetCriticalityRecord.and(z.object({
// ✅
'@timestamp': z.string().datetime(),
}));
```
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
**Related to: https://github.com/elastic/security-team/issues/7491**
## Summary
Migrated remaining public Detection Engine endpoints to OpenAPI schema
and code generation:
- `POST /api/detection_engine/rules/_bulk_action`
- `GET /api/detection_engine/rules/_find`
Also completed the migration of internal APIs:
- `GET /internal/detection_engine/rules/{ruleId}/execution/events`
- `GET /internal/detection_engine/rules/{ruleId}/execution/results`
### Other notable changes
- Changed how we compose Zod error messages for unions, see
`packages/kbn-zod-helpers/src/stringify_zod_error.ts`. Now we are trying
to list the validation errors of all union members but limiting the
total number of validation errors displayed to users.
- Addressed some remaining `TODO
https://github.com/elastic/security-team/issues/7491`
- Removed dependencies of the risk engine and timelines on detection
engine schemas
- Removed outdated legacy rule schemas that are no longer in use
- Added new schema helpers that work with query params:
`BooleanFromString` and `ArrayFromString`
**Parent meta ticket:
https://github.com/elastic/security-team/issues/7491**
Resolves: https://github.com/elastic/security-team/issues/7582
Resolves: https://github.com/elastic/security-team/issues/7580
Resolves: https://github.com/elastic/security-team/issues/7581
## Summary
This PR migrates the rules schema to OpenAPI, Zod, and code generation.
The following APIs now have complete OpenAPI specifications and are
enabled for code generation:
| Method | Endpoint | OpenAPI spec | Fully migrated |
| ------ |
---------------------------------------------------------------- |
------------ | -------------- |
| POST | /api/detection_engine/rules | ✅ | ✅ |
| GET | /api/detection_engine/rules | ✅ | ✅ |
| PUT | /api/detection_engine/rules | ✅ | ✅ |
| PATCH | /api/detection_engine/rules | ✅ | ✅ |
| DELETE | /api/detection_engine/rules | ✅ | ✅ |
| POST | /api/detection_engine/rules/\_bulk_create | ✅ | ✅ |
| PUT | /api/detection_engine/rules/\_bulk_update | ✅ | ✅ |
| PATCH | /api/detection_engine/rules/\_bulk_update | ✅ | ✅ |
| DELETE | /api/detection_engine/rules/\_bulk_delete | ✅ | ✅ |
| POST | /api/detection_engine/rules/\_bulk_delete | ✅ | ✅ |
### Rule schemas are now forward-compatible
We now allow extra fields in schemas for forward compatibility, but we
remove them from the payload during parsing. So from now on, extra
fields are simply ignored and won't lead to validation errors.
## Summary
Security Solution writes e2e and other tests using Cypress. In the past,
these tests, if they failed on a tracked branch, couldn't be easily
skipped. They also weren't run in parallelized jobs. For primarily these
reasons, they didn't run on most Kibana PRs.
This PR moves these Cypress tests back to the main PR pipeline. Tests
that fail on tracked branches create (or update) Github issues which can
be used with the skip-test github workflow script to easily skip the
failing tests. The pipeline steps are parallelized and run in under 40
minutes.
### Open Questions
- [ ] Should this PR enable Serverless Security Defend Workflows Cypress
Tests @patrykkopycinski
### Some buildkite pipelines that used to run only on Security PRs now
run on all PRs:
These steps run on all PRs with these changes
- Security Solution Cypress Tests (general tests that haven't been
organized into an area team)
- Explore tests
- Investigations Tests
- Defend Workflows Tests
- Defend Workflows Serverless
- Threat Intel Tests
- OS Query Tests
- Security Solution Burning Changed Specs (these run only recently
changed specs a few extra times)
- Security Solution OpenAPI codegen
- OSQuery burning
- OSQuery Serverless
<details>
<summary><b>And these already run on all PRs</b></summary>
- Serverless Security Cypress Tests
- Serverless Explore tests
- Serverless Investigations Tests
</details>
### Security Cypress tests run in the main `on merge` pipeline instead
of the `on merge unsupported ftrs` pipeline:
These steps run in the `on merge` pipeline with these changes:
- Security Solution Cypress Tests
- Explore Cypress Tests
- Investigations Cypress Tests
- Defend Workflows Cypress Tests
- Defend Workflows Serverless Cypress Tests
- Threat Intelligence Cypress Tests
- Osquery Cypress Tests
<details>
<summary><b>and these already run on the `on merge`
pipeline</b></summary>
- Serverless Security Cypress Tests
- Serverless Explore - Security Solution Cypress Tests
- Serverless Investigations - Security Solution Cypress Tests
</details>
### Additional work to be done:
We need to consolidate build steps, enhance test skipping to support
Cypress-grep flags, avoid out-of-memory errors in cypress, enhance
parallelization, improve Cypress reporting, and probably other things.
These are tracked separately. Reach out to me if you need details.
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
