mirror of
https://github.com/elastic/kibana.git
synced 2025-04-24 01:38:56 -04:00
[SecuritySolution][SIEM migrations] Implement background task API (#197997)
## Summary It implements the background task to execute the rule migrations and the API to manage them. It also contains a basic implementation of the langGraph agent workflow that will perform the migration using generative AI. > [!NOTE] > This feature needs `siemMigrationsEnabled` experimental flag enabled to work. Otherwise, the new API routes won't be registered, and the `SiemRuleMigrationsService` _setup_ won't be called. So no migration task code can be reached, and no data stream/template will be installed to ES. ### The rule migration task implementation: - Retrieve a batch of N rule migration documents (50 rules initially, we may change that later) with `status: pending`. - Update those documents to `status: processing`. - Execute the migration for each of the N migrations in parallel. - If there is any error update the document with `status: error`. - For each rule migration that finishes we set the result to the storage, and also update `status: finished`. - When all the batch of rules is finished the task will check if there are still migration documents with `status: pending` if so it will process the next batch with a delay (10 seconds initially, we may change that later). - If the task is stopped (via API call or server shut-down), we do a bulk update for all the `status: processing` documents back to `status: pending`. ### Task API - `POST /internal/siem_migrations/rules` (implemented [here](https://github.com/elastic/security-team/issues/10654)) -> Creates the migration on the backend and stores the original rules. It returns the `migration_id` - `GET /internal/siem_migrations/rules/stats` -> Retrieves the stats for all the existing migrations, aggregated by `migration_id`. - `GET /internal/siem_migrations/rules/{migration_id}` -> Retrieves all the migration rule documents of a specific migration. - `PUT /internal/siem_migrations/rules/{migration_id}/start` -> Starts the background task for a specific migration. - `GET /internal/siem_migrations/rules/{migration_id}/stats` -> Retrieves the stats of a specific migration task. The UI will do polling to this endpoint. - `PUT /internal/siem_migrations/rules/{migration_id}/stop` -> Stops the execution of a specific migration running task. When a migration is stopped, the executing task is aborted and all the rules in the batch being processed are moved back to pending, all finished rules will remain stored. When the Kibana server shuts down all the running migrations are stopped automatically. To resume the migration we can call `{migration_id}/start` again and it will take it from the same rules batch it was left. #### Stats (UI polling) response example: ``` { "status": "running", "rules": { "total": 34, "finished": 20, "pending": 4, "processing": 10, "failed": 0 }, "last_updated_at": "2024-10-29T15:04:49.618Z" } ``` ### LLM agent Graph The initial implementation of the agent graph that is executed per rule:  The first node tries to match the original rule with an Elastic prebuilt rule. If it does not succeed, the second node will try to translate the query as a custom rule using the ES|QL knowledge base, this composes previous PoCs: - https://github.com/elastic/kibana/pull/193900 - https://github.com/elastic/kibana/pull/196651 ## Testing locally Enable the flag ``` xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled'] ``` cURL request examples: <details> <summary>Rules migration `create` POST request</summary> ``` curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '[ { "id": "f8c325ea-506e-4105-8ccf-da1492e90115", "vendor": "splunk", "title": "Linux Auditd Add User Account Type", "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.", "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *", "query_language":"spl", "mitre_attack_ids": [ "T1136" ] }, { "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90", "vendor": "splunk", "title": "Linux Auditd Change File Owner To Root", "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.", "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`", "query_language": "spl", "mitre_attack_ids": [ "T1222" ] } ]' ``` </details> <details> <summary>Rules migration `start` task request</summary> - Assuming the connector `azureOpenAiGPT4o` is already created in the local environment. - Using the {{`migration_id`}} from the first POST request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '{ "connectorId": "azureOpenAiGPT4o" }' ``` </details> <details> <summary>Rules migration `stop` task request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration task `stats` request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration rules documents request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration all stats request</summary> ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Sergi Massaneda
GitHub
parent
f410085ffc
commit
cc66320e97
51 changed files with 2346 additions and 202 deletions
|
|
@ -365,7 +365,16 @@ import type {
|
|||
import type {
|
||||
CreateRuleMigrationRequestBodyInput,
|
||||
CreateRuleMigrationResponse,
|
||||
GetAllStatsRuleMigrationResponse,
|
||||
GetRuleMigrationRequestParamsInput,
|
||||
GetRuleMigrationResponse,
|
||||
GetRuleMigrationStatsRequestParamsInput,
|
||||
GetRuleMigrationStatsResponse,
|
||||
StartRuleMigrationRequestParamsInput,
|
||||
StartRuleMigrationRequestBodyInput,
|
||||
StartRuleMigrationResponse,
|
||||
StopRuleMigrationRequestParamsInput,
|
||||
StopRuleMigrationResponse,
|
||||
} from '../siem_migrations/model/api/rules/rules_migration.gen';
|
||||
|
||||
export interface ClientOptions {
|
||||
|
|
@ -1238,6 +1247,21 @@ finalize it.
|
|||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Retrieves the rule migrations stats for all migrations stored in the system
|
||||
*/
|
||||
async getAllStatsRuleMigration() {
|
||||
this.log.info(`${new Date().toISOString()} Calling API GetAllStatsRuleMigration`);
|
||||
return this.kbnClient
|
||||
.request<GetAllStatsRuleMigrationResponse>({
|
||||
path: '/internal/siem_migrations/rules/stats',
|
||||
headers: {
|
||||
[ELASTIC_HTTP_VERSION_HEADER]: '1',
|
||||
},
|
||||
method: 'GET',
|
||||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Get the asset criticality record for a specific entity.
|
||||
*/
|
||||
|
|
@ -1431,13 +1455,28 @@ finalize it.
|
|||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Retrieves the rule migrations stored in the system
|
||||
* Retrieves the rule documents stored in the system given the rule migration id
|
||||
*/
|
||||
async getRuleMigration() {
|
||||
async getRuleMigration(props: GetRuleMigrationProps) {
|
||||
this.log.info(`${new Date().toISOString()} Calling API GetRuleMigration`);
|
||||
return this.kbnClient
|
||||
.request<GetRuleMigrationResponse>({
|
||||
path: '/internal/siem_migrations/rules',
|
||||
path: replaceParams('/internal/siem_migrations/rules/{migration_id}', props.params),
|
||||
headers: {
|
||||
[ELASTIC_HTTP_VERSION_HEADER]: '1',
|
||||
},
|
||||
method: 'GET',
|
||||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Retrieves the stats of a SIEM rules migration using the migration id provided
|
||||
*/
|
||||
async getRuleMigrationStats(props: GetRuleMigrationStatsProps) {
|
||||
this.log.info(`${new Date().toISOString()} Calling API GetRuleMigrationStats`);
|
||||
return this.kbnClient
|
||||
.request<GetRuleMigrationStatsResponse>({
|
||||
path: replaceParams('/internal/siem_migrations/rules/{migration_id}/stats', props.params),
|
||||
headers: {
|
||||
[ELASTIC_HTTP_VERSION_HEADER]: '1',
|
||||
},
|
||||
|
|
@ -1973,6 +2012,22 @@ detection engine rules.
|
|||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Starts a SIEM rules migration using the migration id provided
|
||||
*/
|
||||
async startRuleMigration(props: StartRuleMigrationProps) {
|
||||
this.log.info(`${new Date().toISOString()} Calling API StartRuleMigration`);
|
||||
return this.kbnClient
|
||||
.request<StartRuleMigrationResponse>({
|
||||
path: replaceParams('/internal/siem_migrations/rules/{migration_id}/start', props.params),
|
||||
headers: {
|
||||
[ELASTIC_HTTP_VERSION_HEADER]: '1',
|
||||
},
|
||||
method: 'PUT',
|
||||
body: props.body,
|
||||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
async stopEntityEngine(props: StopEntityEngineProps) {
|
||||
this.log.info(`${new Date().toISOString()} Calling API StopEntityEngine`);
|
||||
return this.kbnClient
|
||||
|
|
@ -1985,6 +2040,21 @@ detection engine rules.
|
|||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Stops a running SIEM rules migration using the migration id provided
|
||||
*/
|
||||
async stopRuleMigration(props: StopRuleMigrationProps) {
|
||||
this.log.info(`${new Date().toISOString()} Calling API StopRuleMigration`);
|
||||
return this.kbnClient
|
||||
.request<StopRuleMigrationResponse>({
|
||||
path: replaceParams('/internal/siem_migrations/rules/{migration_id}/stop', props.params),
|
||||
headers: {
|
||||
[ELASTIC_HTTP_VERSION_HEADER]: '1',
|
||||
},
|
||||
method: 'PUT',
|
||||
})
|
||||
.catch(catchAxiosErrorFormatAndThrow);
|
||||
}
|
||||
/**
|
||||
* Suggests user profiles.
|
||||
*/
|
||||
|
|
@ -2221,6 +2291,12 @@ export interface GetRuleExecutionResultsProps {
|
|||
query: GetRuleExecutionResultsRequestQueryInput;
|
||||
params: GetRuleExecutionResultsRequestParamsInput;
|
||||
}
|
||||
export interface GetRuleMigrationProps {
|
||||
params: GetRuleMigrationRequestParamsInput;
|
||||
}
|
||||
export interface GetRuleMigrationStatsProps {
|
||||
params: GetRuleMigrationStatsRequestParamsInput;
|
||||
}
|
||||
export interface GetTimelineProps {
|
||||
query: GetTimelineRequestQueryInput;
|
||||
}
|
||||
|
|
@ -2297,9 +2373,16 @@ export interface SetAlertTagsProps {
|
|||
export interface StartEntityEngineProps {
|
||||
params: StartEntityEngineRequestParamsInput;
|
||||
}
|
||||
export interface StartRuleMigrationProps {
|
||||
params: StartRuleMigrationRequestParamsInput;
|
||||
body: StartRuleMigrationRequestBodyInput;
|
||||
}
|
||||
export interface StopEntityEngineProps {
|
||||
params: StopEntityEngineRequestParamsInput;
|
||||
}
|
||||
export interface StopRuleMigrationProps {
|
||||
params: StopRuleMigrationRequestParamsInput;
|
||||
}
|
||||
export interface SuggestUserProfilesProps {
|
||||
query: SuggestUserProfilesRequestQueryInput;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,9 +8,24 @@
|
|||
export const SIEM_MIGRATIONS_PATH = '/internal/siem_migrations' as const;
|
||||
export const SIEM_RULE_MIGRATIONS_PATH = `${SIEM_MIGRATIONS_PATH}/rules` as const;
|
||||
|
||||
export enum SiemMigrationsStatus {
|
||||
export const SIEM_RULE_MIGRATIONS_ALL_STATS_PATH = `${SIEM_RULE_MIGRATIONS_PATH}/stats` as const;
|
||||
export const SIEM_RULE_MIGRATIONS_GET_PATH = `${SIEM_RULE_MIGRATIONS_PATH}/{migration_id}` as const;
|
||||
export const SIEM_RULE_MIGRATIONS_START_PATH =
|
||||
`${SIEM_RULE_MIGRATIONS_PATH}/{migration_id}/start` as const;
|
||||
export const SIEM_RULE_MIGRATIONS_STATS_PATH =
|
||||
`${SIEM_RULE_MIGRATIONS_PATH}/{migration_id}/stats` as const;
|
||||
export const SIEM_RULE_MIGRATIONS_STOP_PATH =
|
||||
`${SIEM_RULE_MIGRATIONS_PATH}/{migration_id}/stop` as const;
|
||||
|
||||
export enum SiemMigrationStatus {
|
||||
PENDING = 'pending',
|
||||
PROCESSING = 'processing',
|
||||
FINISHED = 'finished',
|
||||
ERROR = 'error',
|
||||
COMPLETED = 'completed',
|
||||
FAILED = 'failed',
|
||||
}
|
||||
|
||||
export enum SiemMigrationRuleTranslationResult {
|
||||
FULL = 'full',
|
||||
PARTIAL = 'partial',
|
||||
UNTRANSLATABLE = 'untranslatable',
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,13 @@
|
|||
|
||||
import { z } from '@kbn/zod';
|
||||
|
||||
import { OriginalRule, RuleMigration } from '../../rule_migration.gen';
|
||||
import {
|
||||
OriginalRule,
|
||||
RuleMigrationAllTaskStats,
|
||||
RuleMigration,
|
||||
RuleMigrationTaskStats,
|
||||
} from '../../rule_migration.gen';
|
||||
import { ConnectorId, LangSmithOptions } from '../common.gen';
|
||||
|
||||
export type CreateRuleMigrationRequestBody = z.infer<typeof CreateRuleMigrationRequestBody>;
|
||||
export const CreateRuleMigrationRequestBody = z.array(OriginalRule);
|
||||
|
|
@ -30,5 +36,60 @@ export const CreateRuleMigrationResponse = z.object({
|
|||
migration_id: z.string(),
|
||||
});
|
||||
|
||||
export type GetAllStatsRuleMigrationResponse = z.infer<typeof GetAllStatsRuleMigrationResponse>;
|
||||
export const GetAllStatsRuleMigrationResponse = RuleMigrationAllTaskStats;
|
||||
|
||||
export type GetRuleMigrationRequestParams = z.infer<typeof GetRuleMigrationRequestParams>;
|
||||
export const GetRuleMigrationRequestParams = z.object({
|
||||
migration_id: z.string(),
|
||||
});
|
||||
export type GetRuleMigrationRequestParamsInput = z.input<typeof GetRuleMigrationRequestParams>;
|
||||
|
||||
export type GetRuleMigrationResponse = z.infer<typeof GetRuleMigrationResponse>;
|
||||
export const GetRuleMigrationResponse = z.array(RuleMigration);
|
||||
|
||||
export type GetRuleMigrationStatsRequestParams = z.infer<typeof GetRuleMigrationStatsRequestParams>;
|
||||
export const GetRuleMigrationStatsRequestParams = z.object({
|
||||
migration_id: z.string(),
|
||||
});
|
||||
export type GetRuleMigrationStatsRequestParamsInput = z.input<
|
||||
typeof GetRuleMigrationStatsRequestParams
|
||||
>;
|
||||
|
||||
export type GetRuleMigrationStatsResponse = z.infer<typeof GetRuleMigrationStatsResponse>;
|
||||
export const GetRuleMigrationStatsResponse = RuleMigrationTaskStats;
|
||||
|
||||
export type StartRuleMigrationRequestParams = z.infer<typeof StartRuleMigrationRequestParams>;
|
||||
export const StartRuleMigrationRequestParams = z.object({
|
||||
migration_id: z.string(),
|
||||
});
|
||||
export type StartRuleMigrationRequestParamsInput = z.input<typeof StartRuleMigrationRequestParams>;
|
||||
|
||||
export type StartRuleMigrationRequestBody = z.infer<typeof StartRuleMigrationRequestBody>;
|
||||
export const StartRuleMigrationRequestBody = z.object({
|
||||
connector_id: ConnectorId,
|
||||
langsmith_options: LangSmithOptions.optional(),
|
||||
});
|
||||
export type StartRuleMigrationRequestBodyInput = z.input<typeof StartRuleMigrationRequestBody>;
|
||||
|
||||
export type StartRuleMigrationResponse = z.infer<typeof StartRuleMigrationResponse>;
|
||||
export const StartRuleMigrationResponse = z.object({
|
||||
/**
|
||||
* Indicates the migration has been started. `false` means the migration does not need to be started.
|
||||
*/
|
||||
started: z.boolean(),
|
||||
});
|
||||
|
||||
export type StopRuleMigrationRequestParams = z.infer<typeof StopRuleMigrationRequestParams>;
|
||||
export const StopRuleMigrationRequestParams = z.object({
|
||||
migration_id: z.string(),
|
||||
});
|
||||
export type StopRuleMigrationRequestParamsInput = z.input<typeof StopRuleMigrationRequestParams>;
|
||||
|
||||
export type StopRuleMigrationResponse = z.infer<typeof StopRuleMigrationResponse>;
|
||||
export const StopRuleMigrationResponse = z.object({
|
||||
/**
|
||||
* Indicates the migration has been stopped.
|
||||
*/
|
||||
stopped: z.boolean(),
|
||||
});
|
||||
|
|
|
|||
|
|
@ -10,8 +10,7 @@ paths:
|
|||
x-codegen-enabled: true
|
||||
description: Creates a new SIEM rules migration using the original vendor rules provided
|
||||
tags:
|
||||
- SIEM Migrations
|
||||
- Rule Migrations
|
||||
- SIEM Rule Migrations
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
|
|
@ -33,20 +32,146 @@ paths:
|
|||
migration_id:
|
||||
type: string
|
||||
description: The migration id created.
|
||||
|
||||
/internal/siem_migrations/rules/stats:
|
||||
get:
|
||||
summary: Retrieves rule migrations
|
||||
operationId: GetRuleMigration
|
||||
summary: Retrieves the stats for all rule migrations
|
||||
operationId: GetAllStatsRuleMigration
|
||||
x-codegen-enabled: true
|
||||
description: Retrieves the rule migrations stored in the system
|
||||
description: Retrieves the rule migrations stats for all migrations stored in the system
|
||||
tags:
|
||||
- SIEM Migrations
|
||||
- Rule Migrations
|
||||
- SIEM Rule Migrations
|
||||
responses:
|
||||
200:
|
||||
description: Indicates rule migrations have been retrieved correctly.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigrationAllTaskStats'
|
||||
|
||||
/internal/siem_migrations/rules/{migration_id}:
|
||||
get:
|
||||
summary: Retrieves all the rules of a migration
|
||||
operationId: GetRuleMigration
|
||||
x-codegen-enabled: true
|
||||
description: Retrieves the rule documents stored in the system given the rule migration id
|
||||
tags:
|
||||
- SIEM Rule Migrations
|
||||
parameters:
|
||||
- name: migration_id
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
description: The migration id to start
|
||||
responses:
|
||||
200:
|
||||
description: Indicates rule migration have been retrieved correctly.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: array
|
||||
items:
|
||||
$ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigration'
|
||||
204:
|
||||
description: Indicates the migration id was not found.
|
||||
|
||||
/internal/siem_migrations/rules/{migration_id}/start:
|
||||
put:
|
||||
summary: Starts a rule migration
|
||||
operationId: StartRuleMigration
|
||||
x-codegen-enabled: true
|
||||
description: Starts a SIEM rules migration using the migration id provided
|
||||
tags:
|
||||
- SIEM Rule Migrations
|
||||
parameters:
|
||||
- name: migration_id
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
description: The migration id to start
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
required:
|
||||
- connector_id
|
||||
properties:
|
||||
connector_id:
|
||||
$ref: '../common.schema.yaml#/components/schemas/ConnectorId'
|
||||
langsmith_options:
|
||||
$ref: '../common.schema.yaml#/components/schemas/LangSmithOptions'
|
||||
responses:
|
||||
200:
|
||||
description: Indicates the migration start request has been processed successfully.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
required:
|
||||
- started
|
||||
properties:
|
||||
started:
|
||||
type: boolean
|
||||
description: Indicates the migration has been started. `false` means the migration does not need to be started.
|
||||
204:
|
||||
description: Indicates the migration id was not found.
|
||||
|
||||
/internal/siem_migrations/rules/{migration_id}/stats:
|
||||
get:
|
||||
summary: Gets a rule migration task stats
|
||||
operationId: GetRuleMigrationStats
|
||||
x-codegen-enabled: true
|
||||
description: Retrieves the stats of a SIEM rules migration using the migration id provided
|
||||
tags:
|
||||
- SIEM Rule Migrations
|
||||
parameters:
|
||||
- name: migration_id
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
description: The migration id to start
|
||||
responses:
|
||||
200:
|
||||
description: Indicates the migration stats has been retrieved correctly.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '../../rule_migration.schema.yaml#/components/schemas/RuleMigrationTaskStats'
|
||||
204:
|
||||
description: Indicates the migration id was not found.
|
||||
|
||||
/internal/siem_migrations/rules/{migration_id}/stop:
|
||||
put:
|
||||
summary: Stops an existing rule migration
|
||||
operationId: StopRuleMigration
|
||||
x-codegen-enabled: true
|
||||
description: Stops a running SIEM rules migration using the migration id provided
|
||||
tags:
|
||||
- SIEM Rule Migrations
|
||||
parameters:
|
||||
- name: migration_id
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
description: The migration id to stop
|
||||
responses:
|
||||
200:
|
||||
description: Indicates migration task stop has been processed successfully.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
required:
|
||||
- stopped
|
||||
properties:
|
||||
stopped:
|
||||
type: boolean
|
||||
description: Indicates the migration has been stopped.
|
||||
204:
|
||||
description: Indicates the migration id was not found running.
|
||||
|
|
|
|||
|
|
@ -71,11 +71,11 @@ export const ElasticRule = z.object({
|
|||
/**
|
||||
* The translated elastic query.
|
||||
*/
|
||||
query: z.string(),
|
||||
query: z.string().optional(),
|
||||
/**
|
||||
* The translated elastic query language.
|
||||
*/
|
||||
query_language: z.literal('esql').default('esql'),
|
||||
query_language: z.literal('esql').optional(),
|
||||
/**
|
||||
* The Elastic prebuilt rule id matched.
|
||||
*/
|
||||
|
|
@ -99,16 +99,20 @@ export const RuleMigration = z.object({
|
|||
* The migration id.
|
||||
*/
|
||||
migration_id: z.string(),
|
||||
/**
|
||||
* The username of the user who created the migration.
|
||||
*/
|
||||
created_by: z.string(),
|
||||
original_rule: OriginalRule,
|
||||
elastic_rule: ElasticRule.optional(),
|
||||
/**
|
||||
* The translation state.
|
||||
* The rule translation result.
|
||||
*/
|
||||
translation_state: z.enum(['complete', 'partial', 'untranslatable']).optional(),
|
||||
translation_result: z.enum(['full', 'partial', 'untranslatable']).optional(),
|
||||
/**
|
||||
* The status of the rule migration.
|
||||
* The status of the rule migration process.
|
||||
*/
|
||||
status: z.enum(['pending', 'processing', 'finished', 'error']).default('pending'),
|
||||
status: z.enum(['pending', 'processing', 'completed', 'failed']).default('pending'),
|
||||
/**
|
||||
* The comments for the migration including a summary from the LLM in markdown.
|
||||
*/
|
||||
|
|
@ -122,3 +126,55 @@ export const RuleMigration = z.object({
|
|||
*/
|
||||
updated_by: z.string().optional(),
|
||||
});
|
||||
|
||||
/**
|
||||
* The rule migration task stats object.
|
||||
*/
|
||||
export type RuleMigrationTaskStats = z.infer<typeof RuleMigrationTaskStats>;
|
||||
export const RuleMigrationTaskStats = z.object({
|
||||
/**
|
||||
* Indicates if the migration task status.
|
||||
*/
|
||||
status: z.enum(['ready', 'running', 'stopped', 'finished']),
|
||||
/**
|
||||
* The rules migration stats.
|
||||
*/
|
||||
rules: z.object({
|
||||
/**
|
||||
* The total number of rules to migrate.
|
||||
*/
|
||||
total: z.number().int(),
|
||||
/**
|
||||
* The number of rules that are pending migration.
|
||||
*/
|
||||
pending: z.number().int(),
|
||||
/**
|
||||
* The number of rules that are being migrated.
|
||||
*/
|
||||
processing: z.number().int(),
|
||||
/**
|
||||
* The number of rules that have been migrated successfully.
|
||||
*/
|
||||
completed: z.number().int(),
|
||||
/**
|
||||
* The number of rules that have failed migration.
|
||||
*/
|
||||
failed: z.number().int(),
|
||||
}),
|
||||
/**
|
||||
* The moment of the last update.
|
||||
*/
|
||||
last_updated_at: z.string().optional(),
|
||||
});
|
||||
|
||||
export type RuleMigrationAllTaskStats = z.infer<typeof RuleMigrationAllTaskStats>;
|
||||
export const RuleMigrationAllTaskStats = z.array(
|
||||
RuleMigrationTaskStats.merge(
|
||||
z.object({
|
||||
/**
|
||||
* The migration id
|
||||
*/
|
||||
migration_id: z.string(),
|
||||
})
|
||||
)
|
||||
);
|
||||
|
|
|
|||
|
|
@ -48,8 +48,6 @@ components:
|
|||
description: The migrated elastic rule.
|
||||
required:
|
||||
- title
|
||||
- query
|
||||
- query_language
|
||||
properties:
|
||||
title:
|
||||
type: string
|
||||
|
|
@ -68,7 +66,6 @@ components:
|
|||
description: The translated elastic query language.
|
||||
enum:
|
||||
- esql
|
||||
default: esql
|
||||
prebuilt_rule_id:
|
||||
type: string
|
||||
description: The Elastic prebuilt rule id matched.
|
||||
|
|
@ -84,32 +81,36 @@ components:
|
|||
- migration_id
|
||||
- original_rule
|
||||
- status
|
||||
- created_by
|
||||
properties:
|
||||
"@timestamp":
|
||||
'@timestamp':
|
||||
type: string
|
||||
description: The moment of creation
|
||||
migration_id:
|
||||
type: string
|
||||
description: The migration id.
|
||||
created_by:
|
||||
type: string
|
||||
description: The username of the user who created the migration.
|
||||
original_rule:
|
||||
$ref: '#/components/schemas/OriginalRule'
|
||||
elastic_rule:
|
||||
$ref: '#/components/schemas/ElasticRule'
|
||||
translation_state:
|
||||
translation_result:
|
||||
type: string
|
||||
description: The translation state.
|
||||
enum:
|
||||
- complete
|
||||
description: The rule translation result.
|
||||
enum: # should match SiemMigrationRuleTranslationResult enum at ../constants.ts
|
||||
- full
|
||||
- partial
|
||||
- untranslatable
|
||||
status:
|
||||
type: string
|
||||
description: The status of the rule migration.
|
||||
description: The status of the rule migration process.
|
||||
enum: # should match SiemMigrationsStatus enum at ../constants.ts
|
||||
- pending
|
||||
- processing
|
||||
- finished
|
||||
- error
|
||||
- completed
|
||||
- failed
|
||||
default: pending
|
||||
comments:
|
||||
type: array
|
||||
|
|
@ -122,3 +123,60 @@ components:
|
|||
updated_by:
|
||||
type: string
|
||||
description: The user who last updated the migration
|
||||
|
||||
RuleMigrationTaskStats:
|
||||
type: object
|
||||
description: The rule migration task stats object.
|
||||
required:
|
||||
- status
|
||||
- rules
|
||||
properties:
|
||||
status:
|
||||
type: string
|
||||
description: Indicates if the migration task status.
|
||||
enum:
|
||||
- ready
|
||||
- running
|
||||
- stopped
|
||||
- finished
|
||||
rules:
|
||||
type: object
|
||||
description: The rules migration stats.
|
||||
required:
|
||||
- total
|
||||
- pending
|
||||
- processing
|
||||
- completed
|
||||
- failed
|
||||
properties:
|
||||
total:
|
||||
type: integer
|
||||
description: The total number of rules to migrate.
|
||||
pending:
|
||||
type: integer
|
||||
description: The number of rules that are pending migration.
|
||||
processing:
|
||||
type: integer
|
||||
description: The number of rules that are being migrated.
|
||||
completed:
|
||||
type: integer
|
||||
description: The number of rules that have been migrated successfully.
|
||||
failed:
|
||||
type: integer
|
||||
description: The number of rules that have failed migration.
|
||||
last_updated_at:
|
||||
type: string
|
||||
description: The moment of the last update.
|
||||
|
||||
RuleMigrationAllTaskStats:
|
||||
type: array
|
||||
items:
|
||||
allOf:
|
||||
- $ref: '#/components/schemas/RuleMigrationTaskStats'
|
||||
- type: object
|
||||
required:
|
||||
- migration_id
|
||||
properties:
|
||||
migration_id:
|
||||
type: string
|
||||
description: The migration id
|
||||
|
|
|
|||
|
|
@ -58,7 +58,8 @@
|
|||
"savedSearch",
|
||||
"unifiedDocViewer",
|
||||
"charts",
|
||||
"entityManager"
|
||||
"entityManager",
|
||||
"inference"
|
||||
],
|
||||
"optionalPlugins": [
|
||||
"encryptedSavedObjects",
|
||||
|
|
|
|||
|
|
@ -79,7 +79,8 @@ export const createMockClients = () => {
|
|||
internalFleetServices: {
|
||||
packages: packageServiceMock.createClient(),
|
||||
},
|
||||
siemMigrationsClient: siemMigrationsServiceMock.createClient(),
|
||||
siemRuleMigrationsClient: siemMigrationsServiceMock.createRulesClient(),
|
||||
getInferenceClient: jest.fn(),
|
||||
};
|
||||
};
|
||||
|
||||
|
|
@ -166,7 +167,8 @@ const createSecuritySolutionRequestContextMock = (
|
|||
getAuditLogger: jest.fn(() => mockAuditLogger),
|
||||
getDataViewsService: jest.fn(),
|
||||
getEntityStoreDataClient: jest.fn(() => clients.entityStoreDataClient),
|
||||
getSiemMigrationsClient: jest.fn(() => clients.siemMigrationsClient),
|
||||
getSiemRuleMigrationsClient: jest.fn(() => clients.siemRuleMigrationsClient),
|
||||
getInferenceClient: jest.fn(() => clients.getInferenceClient()),
|
||||
};
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -7,18 +7,16 @@
|
|||
|
||||
import { createRuleMigrationClient } from '../rules/__mocks__/mocks';
|
||||
|
||||
const createClient = () => ({ rules: createRuleMigrationClient() });
|
||||
|
||||
export const mockSetup = jest.fn().mockResolvedValue(undefined);
|
||||
export const mockCreateClient = jest.fn().mockReturnValue(createClient());
|
||||
export const mockCreateClient = jest.fn().mockReturnValue(createRuleMigrationClient());
|
||||
export const mockStop = jest.fn();
|
||||
|
||||
export const siemMigrationsServiceMock = {
|
||||
create: () =>
|
||||
jest.fn().mockImplementation(() => ({
|
||||
setup: mockSetup,
|
||||
createClient: mockCreateClient,
|
||||
createRulesClient: mockCreateClient,
|
||||
stop: mockStop,
|
||||
})),
|
||||
createClient: () => createClient(),
|
||||
createRulesClient: () => createRuleMigrationClient(),
|
||||
};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { siemMigrationsServiceMock } from './mocks';
|
||||
export const SiemMigrationsService = siemMigrationsServiceMock.create();
|
||||
|
|
@ -5,17 +5,56 @@
|
|||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { SiemRuleMigrationsClient } from '../types';
|
||||
|
||||
export const createRuleMigrationClient = (): SiemRuleMigrationsClient => ({
|
||||
export const createRuleMigrationDataClient = jest.fn().mockImplementation(() => ({
|
||||
create: jest.fn().mockResolvedValue({ success: true }),
|
||||
search: jest.fn().mockResolvedValue([]),
|
||||
getRules: jest.fn().mockResolvedValue([]),
|
||||
takePending: jest.fn().mockResolvedValue([]),
|
||||
saveFinished: jest.fn().mockResolvedValue({ success: true }),
|
||||
saveError: jest.fn().mockResolvedValue({ success: true }),
|
||||
releaseProcessing: jest.fn().mockResolvedValue({ success: true }),
|
||||
releaseProcessable: jest.fn().mockResolvedValue({ success: true }),
|
||||
getStats: jest.fn().mockResolvedValue({
|
||||
status: 'done',
|
||||
rules: {
|
||||
total: 1,
|
||||
finished: 1,
|
||||
processing: 0,
|
||||
pending: 0,
|
||||
failed: 0,
|
||||
},
|
||||
}),
|
||||
getAllStats: jest.fn().mockResolvedValue([]),
|
||||
}));
|
||||
|
||||
export const createRuleMigrationTaskClient = () => ({
|
||||
start: jest.fn().mockResolvedValue({ started: true }),
|
||||
stop: jest.fn().mockResolvedValue({ stopped: true }),
|
||||
getStats: jest.fn().mockResolvedValue({
|
||||
status: 'done',
|
||||
rules: {
|
||||
total: 1,
|
||||
finished: 1,
|
||||
processing: 0,
|
||||
pending: 0,
|
||||
failed: 0,
|
||||
},
|
||||
}),
|
||||
getAllStats: jest.fn().mockResolvedValue([]),
|
||||
});
|
||||
|
||||
export const createRuleMigrationClient = () => ({
|
||||
data: createRuleMigrationDataClient(),
|
||||
task: createRuleMigrationTaskClient(),
|
||||
});
|
||||
|
||||
export const MockSiemRuleMigrationsClient = jest.fn().mockImplementation(createRuleMigrationClient);
|
||||
|
||||
export const mockSetup = jest.fn();
|
||||
export const mockGetClient = jest.fn().mockReturnValue(createRuleMigrationClient());
|
||||
export const mockCreateClient = jest.fn().mockReturnValue(createRuleMigrationClient());
|
||||
export const mockStop = jest.fn();
|
||||
|
||||
export const MockSiemRuleMigrationsService = jest.fn().mockImplementation(() => ({
|
||||
setup: mockSetup,
|
||||
getClient: mockGetClient,
|
||||
createClient: mockCreateClient,
|
||||
stop: mockStop,
|
||||
}));
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { MockSiemRuleMigrationsClient } from './mocks';
|
||||
export const SiemRuleMigrationsClient = MockSiemRuleMigrationsClient;
|
||||
|
|
@ -8,14 +8,11 @@
|
|||
import type { IKibanaResponse, Logger } from '@kbn/core/server';
|
||||
import { buildRouteValidationWithZod } from '@kbn/zod-helpers';
|
||||
import { v4 as uuidV4 } from 'uuid';
|
||||
import type { RuleMigration } from '../../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
import type { CreateRuleMigrationResponse } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { CreateRuleMigrationRequestBody } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import {
|
||||
SIEM_RULE_MIGRATIONS_PATH,
|
||||
SiemMigrationsStatus,
|
||||
} from '../../../../../common/siem_migrations/constants';
|
||||
import { SIEM_RULE_MIGRATIONS_PATH } from '../../../../../common/siem_migrations/constants';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
import type { CreateRuleMigrationInput } from '../data_stream/rule_migrations_data_client';
|
||||
|
||||
export const registerSiemRuleMigrationsCreateRoute = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
|
|
@ -25,11 +22,7 @@ export const registerSiemRuleMigrationsCreateRoute = (
|
|||
.post({
|
||||
path: SIEM_RULE_MIGRATIONS_PATH,
|
||||
access: 'internal',
|
||||
security: {
|
||||
authz: {
|
||||
requiredPrivileges: ['securitySolution'],
|
||||
},
|
||||
},
|
||||
security: { authz: { requiredPrivileges: ['securitySolution'] } },
|
||||
})
|
||||
.addVersion(
|
||||
{
|
||||
|
|
@ -41,27 +34,22 @@ export const registerSiemRuleMigrationsCreateRoute = (
|
|||
async (context, req, res): Promise<IKibanaResponse<CreateRuleMigrationResponse>> => {
|
||||
const originalRules = req.body;
|
||||
try {
|
||||
const ctx = await context.resolve(['core', 'actions', 'securitySolution']);
|
||||
|
||||
const siemMigrationClient = ctx.securitySolution.getSiemMigrationsClient();
|
||||
const ctx = await context.resolve(['securitySolution']);
|
||||
const ruleMigrationsClient = ctx.securitySolution.getSiemRuleMigrationsClient();
|
||||
|
||||
const migrationId = uuidV4();
|
||||
const timestamp = new Date().toISOString();
|
||||
|
||||
const ruleMigrations = originalRules.map<RuleMigration>((originalRule) => ({
|
||||
'@timestamp': timestamp,
|
||||
const ruleMigrations = originalRules.map<CreateRuleMigrationInput>((originalRule) => ({
|
||||
migration_id: migrationId,
|
||||
original_rule: originalRule,
|
||||
status: SiemMigrationsStatus.PENDING,
|
||||
}));
|
||||
await siemMigrationClient.rules.create(ruleMigrations);
|
||||
|
||||
await ruleMigrationsClient.data.create(ruleMigrations);
|
||||
|
||||
return res.ok({ body: { migration_id: migrationId } });
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return res.badRequest({
|
||||
body: err.message,
|
||||
});
|
||||
return res.badRequest({ body: err.message });
|
||||
}
|
||||
}
|
||||
);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,47 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { IKibanaResponse, Logger } from '@kbn/core/server';
|
||||
import { buildRouteValidationWithZod } from '@kbn/zod-helpers';
|
||||
import type { GetRuleMigrationResponse } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { GetRuleMigrationRequestParams } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { SIEM_RULE_MIGRATIONS_GET_PATH } from '../../../../../common/siem_migrations/constants';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
|
||||
export const registerSiemRuleMigrationsGetRoute = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
logger: Logger
|
||||
) => {
|
||||
router.versioned
|
||||
.get({
|
||||
path: SIEM_RULE_MIGRATIONS_GET_PATH,
|
||||
access: 'internal',
|
||||
security: { authz: { requiredPrivileges: ['securitySolution'] } },
|
||||
})
|
||||
.addVersion(
|
||||
{
|
||||
version: '1',
|
||||
validate: {
|
||||
request: { params: buildRouteValidationWithZod(GetRuleMigrationRequestParams) },
|
||||
},
|
||||
},
|
||||
async (context, req, res): Promise<IKibanaResponse<GetRuleMigrationResponse>> => {
|
||||
const migrationId = req.params.migration_id;
|
||||
try {
|
||||
const ctx = await context.resolve(['securitySolution']);
|
||||
const ruleMigrationsClient = ctx.securitySolution.getSiemRuleMigrationsClient();
|
||||
|
||||
const migrationRules = await ruleMigrationsClient.data.getRules(migrationId);
|
||||
|
||||
return res.ok({ body: migrationRules });
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return res.badRequest({ body: err.message });
|
||||
}
|
||||
}
|
||||
);
|
||||
};
|
||||
|
|
@ -8,10 +8,20 @@
|
|||
import type { Logger } from '@kbn/core/server';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
import { registerSiemRuleMigrationsCreateRoute } from './create';
|
||||
import { registerSiemRuleMigrationsGetRoute } from './get';
|
||||
import { registerSiemRuleMigrationsStartRoute } from './start';
|
||||
import { registerSiemRuleMigrationsStatsRoute } from './stats';
|
||||
import { registerSiemRuleMigrationsStopRoute } from './stop';
|
||||
import { registerSiemRuleMigrationsStatsAllRoute } from './stats_all';
|
||||
|
||||
export const registerSiemRuleMigrationsRoutes = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
logger: Logger
|
||||
) => {
|
||||
registerSiemRuleMigrationsCreateRoute(router, logger);
|
||||
registerSiemRuleMigrationsStatsAllRoute(router, logger);
|
||||
registerSiemRuleMigrationsGetRoute(router, logger);
|
||||
registerSiemRuleMigrationsStartRoute(router, logger);
|
||||
registerSiemRuleMigrationsStatsRoute(router, logger);
|
||||
registerSiemRuleMigrationsStopRoute(router, logger);
|
||||
};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,91 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { IKibanaResponse, Logger } from '@kbn/core/server';
|
||||
import { buildRouteValidationWithZod } from '@kbn/zod-helpers';
|
||||
import { APMTracer } from '@kbn/langchain/server/tracers/apm';
|
||||
import { getLangSmithTracer } from '@kbn/langchain/server/tracers/langsmith';
|
||||
import type { StartRuleMigrationResponse } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import {
|
||||
StartRuleMigrationRequestBody,
|
||||
StartRuleMigrationRequestParams,
|
||||
} from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { SIEM_RULE_MIGRATIONS_START_PATH } from '../../../../../common/siem_migrations/constants';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
|
||||
export const registerSiemRuleMigrationsStartRoute = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
logger: Logger
|
||||
) => {
|
||||
router.versioned
|
||||
.put({
|
||||
path: SIEM_RULE_MIGRATIONS_START_PATH,
|
||||
access: 'internal',
|
||||
security: { authz: { requiredPrivileges: ['securitySolution'] } },
|
||||
})
|
||||
.addVersion(
|
||||
{
|
||||
version: '1',
|
||||
validate: {
|
||||
request: {
|
||||
params: buildRouteValidationWithZod(StartRuleMigrationRequestParams),
|
||||
body: buildRouteValidationWithZod(StartRuleMigrationRequestBody),
|
||||
},
|
||||
},
|
||||
},
|
||||
async (context, req, res): Promise<IKibanaResponse<StartRuleMigrationResponse>> => {
|
||||
const migrationId = req.params.migration_id;
|
||||
const { langsmith_options: langsmithOptions, connector_id: connectorId } = req.body;
|
||||
|
||||
try {
|
||||
const ctx = await context.resolve([
|
||||
'core',
|
||||
'actions',
|
||||
'alerting',
|
||||
'securitySolution',
|
||||
'licensing',
|
||||
]);
|
||||
if (!ctx.licensing.license.hasAtLeast('enterprise')) {
|
||||
return res.forbidden({
|
||||
body: 'You must have a trial or enterprise license to use this feature',
|
||||
});
|
||||
}
|
||||
|
||||
const ruleMigrationsClient = ctx.securitySolution.getSiemRuleMigrationsClient();
|
||||
const inferenceClient = ctx.securitySolution.getInferenceClient();
|
||||
const actionsClient = ctx.actions.getActionsClient();
|
||||
const soClient = ctx.core.savedObjects.client;
|
||||
const rulesClient = ctx.alerting.getRulesClient();
|
||||
|
||||
const invocationConfig = {
|
||||
callbacks: [
|
||||
new APMTracer({ projectName: langsmithOptions?.project_name ?? 'default' }, logger),
|
||||
...getLangSmithTracer({ ...langsmithOptions, logger }),
|
||||
],
|
||||
};
|
||||
|
||||
const { exists, started } = await ruleMigrationsClient.task.start({
|
||||
migrationId,
|
||||
connectorId,
|
||||
invocationConfig,
|
||||
inferenceClient,
|
||||
actionsClient,
|
||||
soClient,
|
||||
rulesClient,
|
||||
});
|
||||
|
||||
if (!exists) {
|
||||
return res.noContent();
|
||||
}
|
||||
return res.ok({ body: { started } });
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return res.badRequest({ body: err.message });
|
||||
}
|
||||
}
|
||||
);
|
||||
};
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { IKibanaResponse, Logger } from '@kbn/core/server';
|
||||
import { buildRouteValidationWithZod } from '@kbn/zod-helpers';
|
||||
import type { GetRuleMigrationStatsResponse } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { GetRuleMigrationStatsRequestParams } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { SIEM_RULE_MIGRATIONS_STATS_PATH } from '../../../../../common/siem_migrations/constants';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
|
||||
export const registerSiemRuleMigrationsStatsRoute = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
logger: Logger
|
||||
) => {
|
||||
router.versioned
|
||||
.get({
|
||||
path: SIEM_RULE_MIGRATIONS_STATS_PATH,
|
||||
access: 'internal',
|
||||
security: { authz: { requiredPrivileges: ['securitySolution'] } },
|
||||
})
|
||||
.addVersion(
|
||||
{
|
||||
version: '1',
|
||||
validate: {
|
||||
request: { params: buildRouteValidationWithZod(GetRuleMigrationStatsRequestParams) },
|
||||
},
|
||||
},
|
||||
async (context, req, res): Promise<IKibanaResponse<GetRuleMigrationStatsResponse>> => {
|
||||
const migrationId = req.params.migration_id;
|
||||
try {
|
||||
const ctx = await context.resolve(['securitySolution']);
|
||||
const ruleMigrationsClient = ctx.securitySolution.getSiemRuleMigrationsClient();
|
||||
|
||||
const stats = await ruleMigrationsClient.task.getStats(migrationId);
|
||||
|
||||
return res.ok({ body: stats });
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return res.badRequest({ body: err.message });
|
||||
}
|
||||
}
|
||||
);
|
||||
};
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { IKibanaResponse, Logger } from '@kbn/core/server';
|
||||
import type { GetAllStatsRuleMigrationResponse } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { SIEM_RULE_MIGRATIONS_ALL_STATS_PATH } from '../../../../../common/siem_migrations/constants';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
|
||||
export const registerSiemRuleMigrationsStatsAllRoute = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
logger: Logger
|
||||
) => {
|
||||
router.versioned
|
||||
.get({
|
||||
path: SIEM_RULE_MIGRATIONS_ALL_STATS_PATH,
|
||||
access: 'internal',
|
||||
security: { authz: { requiredPrivileges: ['securitySolution'] } },
|
||||
})
|
||||
.addVersion(
|
||||
{ version: '1', validate: {} },
|
||||
async (context, req, res): Promise<IKibanaResponse<GetAllStatsRuleMigrationResponse>> => {
|
||||
try {
|
||||
const ctx = await context.resolve(['securitySolution']);
|
||||
const ruleMigrationsClient = ctx.securitySolution.getSiemRuleMigrationsClient();
|
||||
|
||||
const allStats = await ruleMigrationsClient.task.getAllStats();
|
||||
|
||||
return res.ok({ body: allStats });
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return res.badRequest({ body: err.message });
|
||||
}
|
||||
}
|
||||
);
|
||||
};
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { IKibanaResponse, Logger } from '@kbn/core/server';
|
||||
import { buildRouteValidationWithZod } from '@kbn/zod-helpers';
|
||||
import type { StopRuleMigrationResponse } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { StopRuleMigrationRequestParams } from '../../../../../common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { SIEM_RULE_MIGRATIONS_STOP_PATH } from '../../../../../common/siem_migrations/constants';
|
||||
import type { SecuritySolutionPluginRouter } from '../../../../types';
|
||||
|
||||
export const registerSiemRuleMigrationsStopRoute = (
|
||||
router: SecuritySolutionPluginRouter,
|
||||
logger: Logger
|
||||
) => {
|
||||
router.versioned
|
||||
.put({
|
||||
path: SIEM_RULE_MIGRATIONS_STOP_PATH,
|
||||
access: 'internal',
|
||||
security: { authz: { requiredPrivileges: ['securitySolution'] } },
|
||||
})
|
||||
.addVersion(
|
||||
{
|
||||
version: '1',
|
||||
validate: {
|
||||
request: { params: buildRouteValidationWithZod(StopRuleMigrationRequestParams) },
|
||||
},
|
||||
},
|
||||
async (context, req, res): Promise<IKibanaResponse<StopRuleMigrationResponse>> => {
|
||||
const migrationId = req.params.migration_id;
|
||||
try {
|
||||
const ctx = await context.resolve(['securitySolution']);
|
||||
const ruleMigrationsClient = ctx.securitySolution.getSiemRuleMigrationsClient();
|
||||
|
||||
const { exists, stopped } = await ruleMigrationsClient.task.stop(migrationId);
|
||||
|
||||
if (!exists) {
|
||||
return res.noContent();
|
||||
}
|
||||
return res.ok({ body: { stopped } });
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return res.badRequest({ body: err.message });
|
||||
}
|
||||
}
|
||||
);
|
||||
};
|
||||
|
|
@ -7,9 +7,9 @@
|
|||
|
||||
export const mockIndexName = 'mocked_data_stream_name';
|
||||
export const mockInstall = jest.fn().mockResolvedValue(undefined);
|
||||
export const mockInstallSpace = jest.fn().mockResolvedValue(mockIndexName);
|
||||
export const mockCreateClient = jest.fn().mockReturnValue({});
|
||||
|
||||
export const MockRuleMigrationsDataStream = jest.fn().mockImplementation(() => ({
|
||||
install: mockInstall,
|
||||
installSpace: mockInstallSpace,
|
||||
createClient: mockCreateClient,
|
||||
}));
|
||||
|
|
|
|||
|
|
@ -0,0 +1,275 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { AuthenticatedUser, ElasticsearchClient, Logger } from '@kbn/core/server';
|
||||
import assert from 'assert';
|
||||
import type {
|
||||
AggregationsFilterAggregate,
|
||||
AggregationsMaxAggregate,
|
||||
AggregationsStringTermsAggregate,
|
||||
AggregationsStringTermsBucket,
|
||||
QueryDslQueryContainer,
|
||||
SearchHit,
|
||||
SearchResponse,
|
||||
} from '@elastic/elasticsearch/lib/api/types';
|
||||
import type { StoredRuleMigration } from '../types';
|
||||
import { SiemMigrationStatus } from '../../../../../common/siem_migrations/constants';
|
||||
import type {
|
||||
RuleMigration,
|
||||
RuleMigrationTaskStats,
|
||||
} from '../../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
|
||||
export type CreateRuleMigrationInput = Omit<RuleMigration, '@timestamp' | 'status' | 'created_by'>;
|
||||
export type RuleMigrationDataStats = Omit<RuleMigrationTaskStats, 'status'>;
|
||||
export type RuleMigrationAllDataStats = Array<RuleMigrationDataStats & { migration_id: string }>;
|
||||
|
||||
export class RuleMigrationsDataClient {
|
||||
constructor(
|
||||
private dataStreamNamePromise: Promise<string>,
|
||||
private currentUser: AuthenticatedUser,
|
||||
private esClient: ElasticsearchClient,
|
||||
private logger: Logger
|
||||
) {}
|
||||
|
||||
/** Indexes an array of rule migrations to be processed */
|
||||
async create(ruleMigrations: CreateRuleMigrationInput[]): Promise<void> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
await this.esClient
|
||||
.bulk({
|
||||
refresh: 'wait_for',
|
||||
operations: ruleMigrations.flatMap((ruleMigration) => [
|
||||
{ create: { _index: index } },
|
||||
{
|
||||
...ruleMigration,
|
||||
'@timestamp': new Date().toISOString(),
|
||||
status: SiemMigrationStatus.PENDING,
|
||||
created_by: this.currentUser.username,
|
||||
},
|
||||
]),
|
||||
})
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error creating rule migrations: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
|
||||
/** Retrieves an array of rule documents of a specific migrations */
|
||||
async getRules(migrationId: string): Promise<StoredRuleMigration[]> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
const query = this.getFilterQuery(migrationId);
|
||||
|
||||
const storedRuleMigrations = await this.esClient
|
||||
.search<RuleMigration>({ index, query, sort: '_doc' })
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error searching getting rule migrations: ${error.message}`);
|
||||
throw error;
|
||||
})
|
||||
.then((response) => this.processHits(response.hits.hits));
|
||||
return storedRuleMigrations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieves `pending` rule migrations with the provided id and updates their status to `processing`.
|
||||
* This operation is not atomic at migration level:
|
||||
* - Multiple tasks can process different migrations simultaneously.
|
||||
* - Multiple tasks should not process the same migration simultaneously.
|
||||
*/
|
||||
async takePending(migrationId: string, size: number): Promise<StoredRuleMigration[]> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
const query = this.getFilterQuery(migrationId, SiemMigrationStatus.PENDING);
|
||||
|
||||
const storedRuleMigrations = await this.esClient
|
||||
.search<RuleMigration>({ index, query, sort: '_doc', size })
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error searching for rule migrations: ${error.message}`);
|
||||
throw error;
|
||||
})
|
||||
.then((response) =>
|
||||
this.processHits(response.hits.hits, { status: SiemMigrationStatus.PROCESSING })
|
||||
);
|
||||
|
||||
await this.esClient
|
||||
.bulk({
|
||||
refresh: 'wait_for',
|
||||
operations: storedRuleMigrations.flatMap(({ _id, _index, status }) => [
|
||||
{ update: { _id, _index } },
|
||||
{
|
||||
doc: {
|
||||
status,
|
||||
updated_by: this.currentUser.username,
|
||||
updated_at: new Date().toISOString(),
|
||||
},
|
||||
},
|
||||
]),
|
||||
})
|
||||
.catch((error) => {
|
||||
this.logger.error(
|
||||
`Error updating for rule migrations status to processing: ${error.message}`
|
||||
);
|
||||
throw error;
|
||||
});
|
||||
|
||||
return storedRuleMigrations;
|
||||
}
|
||||
|
||||
/** Updates one rule migration with the provided data and sets the status to `completed` */
|
||||
async saveFinished({ _id, _index, ...ruleMigration }: StoredRuleMigration): Promise<void> {
|
||||
const doc = {
|
||||
...ruleMigration,
|
||||
status: SiemMigrationStatus.COMPLETED,
|
||||
updated_by: this.currentUser.username,
|
||||
updated_at: new Date().toISOString(),
|
||||
};
|
||||
await this.esClient
|
||||
.update({ index: _index, id: _id, doc, refresh: 'wait_for' })
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error updating rule migration status to completed: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
|
||||
/** Updates one rule migration with the provided data and sets the status to `failed` */
|
||||
async saveError({ _id, _index, ...ruleMigration }: StoredRuleMigration): Promise<void> {
|
||||
const doc = {
|
||||
...ruleMigration,
|
||||
status: SiemMigrationStatus.FAILED,
|
||||
updated_by: this.currentUser.username,
|
||||
updated_at: new Date().toISOString(),
|
||||
};
|
||||
await this.esClient
|
||||
.update({ index: _index, id: _id, doc, refresh: 'wait_for' })
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error updating rule migration status to completed: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
|
||||
/** Updates all the rule migration with the provided id with status `processing` back to `pending` */
|
||||
async releaseProcessing(migrationId: string): Promise<void> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
const query = this.getFilterQuery(migrationId, SiemMigrationStatus.PROCESSING);
|
||||
const script = { source: `ctx._source['status'] = '${SiemMigrationStatus.PENDING}'` };
|
||||
await this.esClient.updateByQuery({ index, query, script, refresh: false }).catch((error) => {
|
||||
this.logger.error(`Error releasing rule migrations status to pending: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
|
||||
/** Updates all the rule migration with the provided id with status `processing` or `failed` back to `pending` */
|
||||
async releaseProcessable(migrationId: string): Promise<void> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
const query = this.getFilterQuery(migrationId, [
|
||||
SiemMigrationStatus.PROCESSING,
|
||||
SiemMigrationStatus.FAILED,
|
||||
]);
|
||||
const script = { source: `ctx._source['status'] = '${SiemMigrationStatus.PENDING}'` };
|
||||
await this.esClient.updateByQuery({ index, query, script, refresh: true }).catch((error) => {
|
||||
this.logger.error(`Error releasing rule migrations status to pending: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
|
||||
/** Retrieves the stats for the rule migrations with the provided id */
|
||||
async getStats(migrationId: string): Promise<RuleMigrationDataStats> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
const query = this.getFilterQuery(migrationId);
|
||||
const aggregations = {
|
||||
pending: { filter: { term: { status: SiemMigrationStatus.PENDING } } },
|
||||
processing: { filter: { term: { status: SiemMigrationStatus.PROCESSING } } },
|
||||
completed: { filter: { term: { status: SiemMigrationStatus.COMPLETED } } },
|
||||
failed: { filter: { term: { status: SiemMigrationStatus.FAILED } } },
|
||||
lastUpdatedAt: { max: { field: 'updated_at' } },
|
||||
};
|
||||
const result = await this.esClient
|
||||
.search({ index, query, aggregations, _source: false })
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error getting rule migrations stats: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
|
||||
const { pending, processing, completed, lastUpdatedAt, failed } = result.aggregations ?? {};
|
||||
return {
|
||||
rules: {
|
||||
total: this.getTotalHits(result),
|
||||
pending: (pending as AggregationsFilterAggregate)?.doc_count ?? 0,
|
||||
processing: (processing as AggregationsFilterAggregate)?.doc_count ?? 0,
|
||||
completed: (completed as AggregationsFilterAggregate)?.doc_count ?? 0,
|
||||
failed: (failed as AggregationsFilterAggregate)?.doc_count ?? 0,
|
||||
},
|
||||
last_updated_at: (lastUpdatedAt as AggregationsMaxAggregate)?.value_as_string,
|
||||
};
|
||||
}
|
||||
|
||||
/** Retrieves the stats for all the rule migrations aggregated by migration id */
|
||||
async getAllStats(): Promise<RuleMigrationAllDataStats> {
|
||||
const index = await this.dataStreamNamePromise;
|
||||
const aggregations = {
|
||||
migrationIds: {
|
||||
terms: { field: 'migration_id' },
|
||||
aggregations: {
|
||||
pending: { filter: { term: { status: SiemMigrationStatus.PENDING } } },
|
||||
processing: { filter: { term: { status: SiemMigrationStatus.PROCESSING } } },
|
||||
completed: { filter: { term: { status: SiemMigrationStatus.COMPLETED } } },
|
||||
failed: { filter: { term: { status: SiemMigrationStatus.FAILED } } },
|
||||
lastUpdatedAt: { max: { field: 'updated_at' } },
|
||||
},
|
||||
},
|
||||
};
|
||||
const result = await this.esClient
|
||||
.search({ index, aggregations, _source: false })
|
||||
.catch((error) => {
|
||||
this.logger.error(`Error getting all rule migrations stats: ${error.message}`);
|
||||
throw error;
|
||||
});
|
||||
|
||||
const migrationsAgg = result.aggregations?.migrationIds as AggregationsStringTermsAggregate;
|
||||
const buckets = (migrationsAgg?.buckets as AggregationsStringTermsBucket[]) ?? [];
|
||||
return buckets.map((bucket) => ({
|
||||
migration_id: bucket.key,
|
||||
rules: {
|
||||
total: bucket.doc_count,
|
||||
pending: bucket.pending?.doc_count ?? 0,
|
||||
processing: bucket.processing?.doc_count ?? 0,
|
||||
completed: bucket.completed?.doc_count ?? 0,
|
||||
failed: bucket.failed?.doc_count ?? 0,
|
||||
},
|
||||
last_updated_at: bucket.lastUpdatedAt?.value_as_string,
|
||||
}));
|
||||
}
|
||||
|
||||
private getFilterQuery(
|
||||
migrationId: string,
|
||||
status?: SiemMigrationStatus | SiemMigrationStatus[]
|
||||
): QueryDslQueryContainer {
|
||||
const filter: QueryDslQueryContainer[] = [{ term: { migration_id: migrationId } }];
|
||||
if (status) {
|
||||
if (Array.isArray(status)) {
|
||||
filter.push({ terms: { status } });
|
||||
} else {
|
||||
filter.push({ term: { status } });
|
||||
}
|
||||
}
|
||||
return { bool: { filter } };
|
||||
}
|
||||
|
||||
private processHits(
|
||||
hits: Array<SearchHit<RuleMigration>>,
|
||||
override: Partial<RuleMigration> = {}
|
||||
): StoredRuleMigration[] {
|
||||
return hits.map(({ _id, _index, _source }) => {
|
||||
assert(_id, 'RuleMigration document should have _id');
|
||||
assert(_source, 'RuleMigration document should have _source');
|
||||
return { ..._source, ...override, _id, _index };
|
||||
});
|
||||
}
|
||||
|
||||
private getTotalHits(response: SearchResponse) {
|
||||
return typeof response.hits.total === 'number'
|
||||
? response.hits.total
|
||||
: response.hits.total?.value ?? 0;
|
||||
}
|
||||
}
|
||||
|
|
@ -11,9 +11,19 @@ import type { InstallParams } from '@kbn/data-stream-adapter';
|
|||
import { DataStreamSpacesAdapter } from '@kbn/data-stream-adapter';
|
||||
import { elasticsearchServiceMock } from '@kbn/core-elasticsearch-server-mocks';
|
||||
import { loggerMock } from '@kbn/logging-mocks';
|
||||
import { loggingSystemMock } from '@kbn/core-logging-server-mocks';
|
||||
import { securityServiceMock } from '@kbn/core-security-server-mocks';
|
||||
|
||||
jest.mock('@kbn/data-stream-adapter');
|
||||
|
||||
// This mock is required to have a way to await the data stream name promise
|
||||
const mockDataStreamNamePromise = jest.fn();
|
||||
jest.mock('./rule_migrations_data_client', () => ({
|
||||
RuleMigrationsDataClient: jest.fn((dataStreamNamePromise: Promise<string>) => {
|
||||
mockDataStreamNamePromise.mockReturnValue(dataStreamNamePromise);
|
||||
}),
|
||||
}));
|
||||
|
||||
const MockedDataStreamSpacesAdapter = DataStreamSpacesAdapter as unknown as jest.MockedClass<
|
||||
typeof DataStreamSpacesAdapter
|
||||
>;
|
||||
|
|
@ -21,18 +31,21 @@ const MockedDataStreamSpacesAdapter = DataStreamSpacesAdapter as unknown as jest
|
|||
const esClient = elasticsearchServiceMock.createStart().client.asInternalUser;
|
||||
|
||||
describe('SiemRuleMigrationsDataStream', () => {
|
||||
const kibanaVersion = '8.16.0';
|
||||
const logger = loggingSystemMock.createLogger();
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
describe('constructor', () => {
|
||||
it('should create DataStreamSpacesAdapter', () => {
|
||||
new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
expect(MockedDataStreamSpacesAdapter).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('should create component templates', () => {
|
||||
new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
const [dataStreamSpacesAdapter] = MockedDataStreamSpacesAdapter.mock.instances;
|
||||
expect(dataStreamSpacesAdapter.setComponentTemplate).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ name: '.kibana.siem-rule-migrations' })
|
||||
|
|
@ -40,7 +53,7 @@ describe('SiemRuleMigrationsDataStream', () => {
|
|||
});
|
||||
|
||||
it('should create index templates', () => {
|
||||
new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
const [dataStreamSpacesAdapter] = MockedDataStreamSpacesAdapter.mock.instances;
|
||||
expect(dataStreamSpacesAdapter.setIndexTemplate).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ name: '.kibana.siem-rule-migrations' })
|
||||
|
|
@ -50,22 +63,20 @@ describe('SiemRuleMigrationsDataStream', () => {
|
|||
|
||||
describe('install', () => {
|
||||
it('should install data stream', async () => {
|
||||
const dataStream = new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
const params: InstallParams = {
|
||||
const dataStream = new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
const params: Omit<InstallParams, 'logger'> = {
|
||||
esClient,
|
||||
logger: loggerMock.create(),
|
||||
pluginStop$: new Subject(),
|
||||
};
|
||||
await dataStream.install(params);
|
||||
const [dataStreamSpacesAdapter] = MockedDataStreamSpacesAdapter.mock.instances;
|
||||
expect(dataStreamSpacesAdapter.install).toHaveBeenCalledWith(params);
|
||||
expect(dataStreamSpacesAdapter.install).toHaveBeenCalledWith(expect.objectContaining(params));
|
||||
});
|
||||
|
||||
it('should log error', async () => {
|
||||
const dataStream = new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
const params: InstallParams = {
|
||||
const dataStream = new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
const params: Omit<InstallParams, 'logger'> = {
|
||||
esClient,
|
||||
logger: loggerMock.create(),
|
||||
pluginStop$: new Subject(),
|
||||
};
|
||||
const [dataStreamSpacesAdapter] = MockedDataStreamSpacesAdapter.mock.instances;
|
||||
|
|
@ -73,13 +84,16 @@ describe('SiemRuleMigrationsDataStream', () => {
|
|||
(dataStreamSpacesAdapter.install as jest.Mock).mockRejectedValueOnce(error);
|
||||
|
||||
await dataStream.install(params);
|
||||
expect(params.logger.error).toHaveBeenCalledWith(expect.any(String), error);
|
||||
expect(logger.error).toHaveBeenCalledWith(expect.any(String), error);
|
||||
});
|
||||
});
|
||||
|
||||
describe('installSpace', () => {
|
||||
describe('createClient', () => {
|
||||
const currentUser = securityServiceMock.createMockAuthenticatedUser();
|
||||
const createClientParams = { spaceId: 'space1', currentUser, esClient };
|
||||
|
||||
it('should install space data stream', async () => {
|
||||
const dataStream = new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
const dataStream = new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
const params: InstallParams = {
|
||||
esClient,
|
||||
logger: loggerMock.create(),
|
||||
|
|
@ -89,19 +103,23 @@ describe('SiemRuleMigrationsDataStream', () => {
|
|||
(dataStreamSpacesAdapter.install as jest.Mock).mockResolvedValueOnce(undefined);
|
||||
|
||||
await dataStream.install(params);
|
||||
await dataStream.installSpace('space1');
|
||||
dataStream.createClient(createClientParams);
|
||||
await mockDataStreamNamePromise();
|
||||
|
||||
expect(dataStreamSpacesAdapter.getInstalledSpaceName).toHaveBeenCalledWith('space1');
|
||||
expect(dataStreamSpacesAdapter.installSpace).toHaveBeenCalledWith('space1');
|
||||
});
|
||||
|
||||
it('should not install space data stream if install not executed', async () => {
|
||||
const dataStream = new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
await expect(dataStream.installSpace('space1')).rejects.toThrowError();
|
||||
const dataStream = new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
await expect(async () => {
|
||||
dataStream.createClient(createClientParams);
|
||||
await mockDataStreamNamePromise();
|
||||
}).rejects.toThrowError();
|
||||
});
|
||||
|
||||
it('should throw error if main install had error', async () => {
|
||||
const dataStream = new RuleMigrationsDataStream({ kibanaVersion: '8.13.0' });
|
||||
const dataStream = new RuleMigrationsDataStream(logger, kibanaVersion);
|
||||
const params: InstallParams = {
|
||||
esClient,
|
||||
logger: loggerMock.create(),
|
||||
|
|
@ -112,7 +130,10 @@ describe('SiemRuleMigrationsDataStream', () => {
|
|||
(dataStreamSpacesAdapter.install as jest.Mock).mockRejectedValueOnce(error);
|
||||
await dataStream.install(params);
|
||||
|
||||
await expect(dataStream.installSpace('space1')).rejects.toThrowError(error);
|
||||
await expect(async () => {
|
||||
dataStream.createClient(createClientParams);
|
||||
await mockDataStreamNamePromise();
|
||||
}).rejects.toThrowError(error);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -6,51 +6,69 @@
|
|||
*/
|
||||
|
||||
import { DataStreamSpacesAdapter, type InstallParams } from '@kbn/data-stream-adapter';
|
||||
import type { AuthenticatedUser, ElasticsearchClient, Logger } from '@kbn/core/server';
|
||||
import { ruleMigrationsFieldMap } from './rule_migrations_field_map';
|
||||
import { RuleMigrationsDataClient } from './rule_migrations_data_client';
|
||||
|
||||
const TOTAL_FIELDS_LIMIT = 2500;
|
||||
|
||||
const DATA_STREAM_NAME = '.kibana.siem-rule-migrations';
|
||||
const ECS_COMPONENT_TEMPLATE_NAME = 'ecs';
|
||||
|
||||
interface RuleMigrationsDataStreamCreateClientParams {
|
||||
spaceId: string;
|
||||
currentUser: AuthenticatedUser;
|
||||
esClient: ElasticsearchClient;
|
||||
}
|
||||
|
||||
export class RuleMigrationsDataStream {
|
||||
private readonly dataStream: DataStreamSpacesAdapter;
|
||||
private readonly dataStreamAdapter: DataStreamSpacesAdapter;
|
||||
private installPromise?: Promise<void>;
|
||||
|
||||
constructor({ kibanaVersion }: { kibanaVersion: string }) {
|
||||
this.dataStream = new DataStreamSpacesAdapter(DATA_STREAM_NAME, {
|
||||
constructor(private logger: Logger, kibanaVersion: string) {
|
||||
this.dataStreamAdapter = new DataStreamSpacesAdapter(DATA_STREAM_NAME, {
|
||||
kibanaVersion,
|
||||
totalFieldsLimit: TOTAL_FIELDS_LIMIT,
|
||||
});
|
||||
this.dataStream.setComponentTemplate({
|
||||
this.dataStreamAdapter.setComponentTemplate({
|
||||
name: DATA_STREAM_NAME,
|
||||
fieldMap: ruleMigrationsFieldMap,
|
||||
});
|
||||
|
||||
this.dataStream.setIndexTemplate({
|
||||
this.dataStreamAdapter.setIndexTemplate({
|
||||
name: DATA_STREAM_NAME,
|
||||
componentTemplateRefs: [DATA_STREAM_NAME, ECS_COMPONENT_TEMPLATE_NAME],
|
||||
componentTemplateRefs: [DATA_STREAM_NAME],
|
||||
});
|
||||
}
|
||||
|
||||
async install(params: InstallParams) {
|
||||
async install(params: Omit<InstallParams, 'logger'>) {
|
||||
try {
|
||||
this.installPromise = this.dataStream.install(params);
|
||||
this.installPromise = this.dataStreamAdapter.install({ ...params, logger: this.logger });
|
||||
await this.installPromise;
|
||||
} catch (err) {
|
||||
params.logger.error(`Error installing siem rule migrations data stream. ${err.message}`, err);
|
||||
this.logger.error(`Error installing siem rule migrations data stream. ${err.message}`, err);
|
||||
}
|
||||
}
|
||||
|
||||
async installSpace(spaceId: string): Promise<string> {
|
||||
createClient({
|
||||
spaceId,
|
||||
currentUser,
|
||||
esClient,
|
||||
}: RuleMigrationsDataStreamCreateClientParams): RuleMigrationsDataClient {
|
||||
const dataStreamNamePromise = this.installSpace(spaceId);
|
||||
return new RuleMigrationsDataClient(dataStreamNamePromise, currentUser, esClient, this.logger);
|
||||
}
|
||||
|
||||
// Installs the data stream for the specific space. it will only install if it hasn't been installed yet.
|
||||
// The adapter stores the data stream name promise, it will return it directly when the data stream is known to be installed.
|
||||
private async installSpace(spaceId: string): Promise<string> {
|
||||
if (!this.installPromise) {
|
||||
throw new Error('Siem rule migrations data stream not installed');
|
||||
}
|
||||
// wait for install to complete, may reject if install failed, routes should handle this
|
||||
await this.installPromise;
|
||||
let dataStreamName = await this.dataStream.getInstalledSpaceName(spaceId);
|
||||
let dataStreamName = await this.dataStreamAdapter.getInstalledSpaceName(spaceId);
|
||||
if (!dataStreamName) {
|
||||
dataStreamName = await this.dataStream.installSpace(spaceId);
|
||||
dataStreamName = await this.dataStreamAdapter.installSpace(spaceId);
|
||||
}
|
||||
return dataStreamName;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ import type { RuleMigration } from '../../../../../common/siem_migrations/model/
|
|||
export const ruleMigrationsFieldMap: FieldMap<SchemaFieldMapKeys<RuleMigration>> = {
|
||||
'@timestamp': { type: 'date', required: false },
|
||||
migration_id: { type: 'keyword', required: true },
|
||||
created_by: { type: 'keyword', required: true },
|
||||
status: { type: 'keyword', required: true },
|
||||
original_rule: { type: 'nested', required: true },
|
||||
'original_rule.vendor': { type: 'keyword', required: true },
|
||||
|
|
@ -28,7 +29,7 @@ export const ruleMigrationsFieldMap: FieldMap<SchemaFieldMapKeys<RuleMigration>>
|
|||
'elastic_rule.severity': { type: 'keyword', required: false },
|
||||
'elastic_rule.prebuilt_rule_id': { type: 'keyword', required: false },
|
||||
'elastic_rule.id': { type: 'keyword', required: false },
|
||||
translation_state: { type: 'keyword', required: false },
|
||||
translation_result: { type: 'keyword', required: false },
|
||||
comments: { type: 'text', array: true, required: false },
|
||||
updated_at: { type: 'date', required: false },
|
||||
updated_by: { type: 'keyword', required: false },
|
||||
|
|
|
|||
|
|
@ -8,25 +8,28 @@ import {
|
|||
loggingSystemMock,
|
||||
elasticsearchServiceMock,
|
||||
httpServerMock,
|
||||
securityServiceMock,
|
||||
} from '@kbn/core/server/mocks';
|
||||
import { SiemRuleMigrationsService } from './siem_rule_migrations_service';
|
||||
import { Subject } from 'rxjs';
|
||||
import type { RuleMigration } from '../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
import {
|
||||
MockRuleMigrationsDataStream,
|
||||
mockInstall,
|
||||
mockInstallSpace,
|
||||
mockIndexName,
|
||||
mockCreateClient,
|
||||
} from './data_stream/__mocks__/mocks';
|
||||
import type { KibanaRequest } from '@kbn/core/server';
|
||||
import type { SiemRuleMigrationsCreateClientParams } from './types';
|
||||
|
||||
jest.mock('./data_stream/rule_migrations_data_stream');
|
||||
jest.mock('./task/rule_migrations_task_runner', () => ({
|
||||
RuleMigrationsTaskRunner: jest.fn(),
|
||||
}));
|
||||
|
||||
describe('SiemRuleMigrationsService', () => {
|
||||
let ruleMigrationsService: SiemRuleMigrationsService;
|
||||
const kibanaVersion = '8.16.0';
|
||||
|
||||
const esClusterClient = elasticsearchServiceMock.createClusterClient();
|
||||
const currentUser = securityServiceMock.createMockAuthenticatedUser();
|
||||
const logger = loggingSystemMock.createLogger();
|
||||
const pluginStop$ = new Subject<void>();
|
||||
|
||||
|
|
@ -36,7 +39,7 @@ describe('SiemRuleMigrationsService', () => {
|
|||
});
|
||||
|
||||
it('should instantiate the rule migrations data stream adapter', () => {
|
||||
expect(MockRuleMigrationsDataStream).toHaveBeenCalledWith({ kibanaVersion });
|
||||
expect(MockRuleMigrationsDataStream).toHaveBeenCalledWith(logger, kibanaVersion);
|
||||
});
|
||||
|
||||
describe('when setup is called', () => {
|
||||
|
|
@ -45,22 +48,26 @@ describe('SiemRuleMigrationsService', () => {
|
|||
|
||||
expect(mockInstall).toHaveBeenCalledWith({
|
||||
esClient: esClusterClient.asInternalUser,
|
||||
logger,
|
||||
pluginStop$,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('when getClient is called', () => {
|
||||
let request: KibanaRequest;
|
||||
describe('when createClient is called', () => {
|
||||
let createClientParams: SiemRuleMigrationsCreateClientParams;
|
||||
|
||||
beforeEach(() => {
|
||||
request = httpServerMock.createKibanaRequest();
|
||||
createClientParams = {
|
||||
spaceId: 'default',
|
||||
currentUser,
|
||||
request: httpServerMock.createKibanaRequest(),
|
||||
};
|
||||
});
|
||||
|
||||
describe('without setup', () => {
|
||||
it('should throw an error', () => {
|
||||
expect(() => {
|
||||
ruleMigrationsService.getClient({ spaceId: 'default', request });
|
||||
ruleMigrationsService.createClient(createClientParams);
|
||||
}).toThrowError('ES client not available, please call setup first');
|
||||
});
|
||||
});
|
||||
|
|
@ -71,44 +78,19 @@ describe('SiemRuleMigrationsService', () => {
|
|||
});
|
||||
|
||||
it('should call installSpace', () => {
|
||||
ruleMigrationsService.getClient({ spaceId: 'default', request });
|
||||
|
||||
expect(mockInstallSpace).toHaveBeenCalledWith('default');
|
||||
ruleMigrationsService.createClient(createClientParams);
|
||||
expect(mockCreateClient).toHaveBeenCalledWith({
|
||||
spaceId: createClientParams.spaceId,
|
||||
currentUser: createClientParams.currentUser,
|
||||
esClient: esClusterClient.asScoped().asCurrentUser,
|
||||
});
|
||||
});
|
||||
|
||||
it('should return a client with create and search methods after setup', () => {
|
||||
const client = ruleMigrationsService.getClient({ spaceId: 'default', request });
|
||||
it('should return data and task clients', () => {
|
||||
const client = ruleMigrationsService.createClient(createClientParams);
|
||||
|
||||
expect(client).toHaveProperty('create');
|
||||
expect(client).toHaveProperty('search');
|
||||
});
|
||||
|
||||
it('should call ES bulk create API with the correct parameters with create is called', async () => {
|
||||
const client = ruleMigrationsService.getClient({ spaceId: 'default', request });
|
||||
|
||||
const ruleMigrations = [{ migration_id: 'dummy_migration_id' } as RuleMigration];
|
||||
await client.create(ruleMigrations);
|
||||
|
||||
expect(esClusterClient.asScoped().asCurrentUser.bulk).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
body: [{ create: { _index: mockIndexName } }, { migration_id: 'dummy_migration_id' }],
|
||||
refresh: 'wait_for',
|
||||
})
|
||||
);
|
||||
});
|
||||
|
||||
it('should call ES search API with the correct parameters with search is called', async () => {
|
||||
const client = ruleMigrationsService.getClient({ spaceId: 'default', request });
|
||||
|
||||
const term = { migration_id: 'dummy_migration_id' };
|
||||
await client.search(term);
|
||||
|
||||
expect(esClusterClient.asScoped().asCurrentUser.search).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
index: mockIndexName,
|
||||
body: { query: { term } },
|
||||
})
|
||||
);
|
||||
expect(client).toHaveProperty('data');
|
||||
expect(client).toHaveProperty('task');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -5,52 +5,67 @@
|
|||
* 2.0.
|
||||
*/
|
||||
|
||||
import assert from 'assert';
|
||||
import type { IClusterClient, Logger } from '@kbn/core/server';
|
||||
import { RuleMigrationsDataStream } from './data_stream/rule_migrations_data_stream';
|
||||
import type {
|
||||
SiemRuleMigrationsClient,
|
||||
SiemRulesMigrationsSetupParams,
|
||||
SiemRuleMigrationsGetClientParams,
|
||||
SiemRuleMigrationsCreateClientParams,
|
||||
SiemRuleMigrationsClient,
|
||||
} from './types';
|
||||
import { RuleMigrationsTaskRunner } from './task/rule_migrations_task_runner';
|
||||
|
||||
export class SiemRuleMigrationsService {
|
||||
private dataStreamAdapter: RuleMigrationsDataStream;
|
||||
private rulesDataStream: RuleMigrationsDataStream;
|
||||
private esClusterClient?: IClusterClient;
|
||||
private taskRunner: RuleMigrationsTaskRunner;
|
||||
|
||||
constructor(private logger: Logger, kibanaVersion: string) {
|
||||
this.dataStreamAdapter = new RuleMigrationsDataStream({ kibanaVersion });
|
||||
this.rulesDataStream = new RuleMigrationsDataStream(this.logger, kibanaVersion);
|
||||
this.taskRunner = new RuleMigrationsTaskRunner(this.logger);
|
||||
}
|
||||
|
||||
setup({ esClusterClient, ...params }: SiemRulesMigrationsSetupParams) {
|
||||
this.esClusterClient = esClusterClient;
|
||||
const esClient = esClusterClient.asInternalUser;
|
||||
this.dataStreamAdapter.install({ ...params, esClient, logger: this.logger }).catch((err) => {
|
||||
|
||||
this.rulesDataStream.install({ ...params, esClient }).catch((err) => {
|
||||
this.logger.error(`Error installing data stream for rule migrations: ${err.message}`);
|
||||
throw err;
|
||||
});
|
||||
}
|
||||
|
||||
getClient({ spaceId, request }: SiemRuleMigrationsGetClientParams): SiemRuleMigrationsClient {
|
||||
if (!this.esClusterClient) {
|
||||
throw new Error('ES client not available, please call setup first');
|
||||
}
|
||||
// Installs the data stream for the specific space. it will only install if it hasn't been installed yet.
|
||||
// The adapter stores the data stream name promise, it will return it directly when the data stream is known to be installed.
|
||||
const dataStreamNamePromise = this.dataStreamAdapter.installSpace(spaceId);
|
||||
createClient({
|
||||
spaceId,
|
||||
currentUser,
|
||||
request,
|
||||
}: SiemRuleMigrationsCreateClientParams): SiemRuleMigrationsClient {
|
||||
assert(currentUser, 'Current user must be authenticated');
|
||||
assert(this.esClusterClient, 'ES client not available, please call setup first');
|
||||
|
||||
const esClient = this.esClusterClient.asScoped(request).asCurrentUser;
|
||||
const dataClient = this.rulesDataStream.createClient({ spaceId, currentUser, esClient });
|
||||
|
||||
return {
|
||||
create: async (ruleMigrations) => {
|
||||
const _index = await dataStreamNamePromise;
|
||||
return esClient.bulk({
|
||||
refresh: 'wait_for',
|
||||
body: ruleMigrations.flatMap((ruleMigration) => [{ create: { _index } }, ruleMigration]),
|
||||
});
|
||||
},
|
||||
search: async (term) => {
|
||||
const index = await dataStreamNamePromise;
|
||||
return esClient.search({ index, body: { query: { term } } });
|
||||
data: dataClient,
|
||||
task: {
|
||||
start: (params) => {
|
||||
return this.taskRunner.start({ ...params, currentUser, dataClient });
|
||||
},
|
||||
stop: (migrationId) => {
|
||||
return this.taskRunner.stop({ migrationId, dataClient });
|
||||
},
|
||||
getStats: async (migrationId) => {
|
||||
return this.taskRunner.getStats({ migrationId, dataClient });
|
||||
},
|
||||
getAllStats: async () => {
|
||||
return this.taskRunner.getAllStats({ dataClient });
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
stop() {
|
||||
this.taskRunner.stopAll();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,43 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { END, START, StateGraph } from '@langchain/langgraph';
|
||||
import { migrateRuleState } from './state';
|
||||
import type { MigrateRuleGraphParams, MigrateRuleState } from './types';
|
||||
import { getTranslateQueryNode } from './nodes/translate_query';
|
||||
import { getMatchPrebuiltRuleNode } from './nodes/match_prebuilt_rule';
|
||||
|
||||
export function getRuleMigrationAgent({
|
||||
model,
|
||||
inferenceClient,
|
||||
prebuiltRulesMap,
|
||||
connectorId,
|
||||
logger,
|
||||
}: MigrateRuleGraphParams) {
|
||||
const matchPrebuiltRuleNode = getMatchPrebuiltRuleNode({ model, prebuiltRulesMap, logger });
|
||||
const translationNode = getTranslateQueryNode({ inferenceClient, connectorId, logger });
|
||||
|
||||
const translateRuleGraph = new StateGraph(migrateRuleState)
|
||||
// Nodes
|
||||
.addNode('matchPrebuiltRule', matchPrebuiltRuleNode)
|
||||
.addNode('translation', translationNode)
|
||||
// Edges
|
||||
.addEdge(START, 'matchPrebuiltRule')
|
||||
.addConditionalEdges('matchPrebuiltRule', matchedPrebuiltRuleConditional)
|
||||
.addEdge('translation', END);
|
||||
|
||||
const graph = translateRuleGraph.compile();
|
||||
graph.name = 'Rule Migration Graph'; // Customizes the name displayed in LangSmith
|
||||
return graph;
|
||||
}
|
||||
|
||||
const matchedPrebuiltRuleConditional = (state: MigrateRuleState) => {
|
||||
if (state.elastic_rule?.prebuilt_rule_id) {
|
||||
return END;
|
||||
}
|
||||
return 'translation';
|
||||
};
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
export { getRuleMigrationAgent } from './graph';
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
export { getMatchPrebuiltRuleNode } from './match_prebuilt_rule';
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Logger } from '@kbn/core/server';
|
||||
import { StringOutputParser } from '@langchain/core/output_parsers';
|
||||
import type { ChatModel } from '../../../util/actions_client_chat';
|
||||
import type { GraphNode } from '../../types';
|
||||
import { filterPrebuiltRules, type PrebuiltRulesMapByName } from '../../../util/prebuilt_rules';
|
||||
import { MATCH_PREBUILT_RULE_PROMPT } from './prompts';
|
||||
|
||||
interface GetMatchPrebuiltRuleNodeParams {
|
||||
model: ChatModel;
|
||||
prebuiltRulesMap: PrebuiltRulesMapByName;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
export const getMatchPrebuiltRuleNode =
|
||||
({ model, prebuiltRulesMap }: GetMatchPrebuiltRuleNodeParams): GraphNode =>
|
||||
async (state) => {
|
||||
const mitreAttackIds = state.original_rule.mitre_attack_ids;
|
||||
if (!mitreAttackIds?.length) {
|
||||
return {};
|
||||
}
|
||||
const filteredPrebuiltRulesMap = filterPrebuiltRules(prebuiltRulesMap, mitreAttackIds);
|
||||
if (filteredPrebuiltRulesMap.size === 0) {
|
||||
return {};
|
||||
}
|
||||
|
||||
const outputParser = new StringOutputParser();
|
||||
const matchPrebuiltRule = MATCH_PREBUILT_RULE_PROMPT.pipe(model).pipe(outputParser);
|
||||
|
||||
const elasticSecurityRules = Array(filteredPrebuiltRulesMap.keys()).join('\n');
|
||||
const response = await matchPrebuiltRule.invoke({
|
||||
elasticSecurityRules,
|
||||
ruleTitle: state.original_rule.title,
|
||||
});
|
||||
const cleanResponse = response.trim();
|
||||
if (cleanResponse === 'no_match') {
|
||||
return {};
|
||||
}
|
||||
|
||||
const result = filteredPrebuiltRulesMap.get(cleanResponse);
|
||||
if (result != null) {
|
||||
return {
|
||||
elastic_rule: {
|
||||
title: result.rule.name,
|
||||
description: result.rule.description,
|
||||
prebuilt_rule_id: result.rule.rule_id,
|
||||
id: result.installedRuleId,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
return {};
|
||||
};
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { ChatPromptTemplate } from '@langchain/core/prompts';
|
||||
export const MATCH_PREBUILT_RULE_PROMPT = ChatPromptTemplate.fromMessages([
|
||||
[
|
||||
'system',
|
||||
`You are an expert assistant in Cybersecurity, your task is to help migrating a SIEM detection rule, from Splunk Security to Elastic Security.
|
||||
You will be provided with a Splunk Detection Rule name by the user, your goal is to try find an Elastic Detection Rule that covers the same threat, if any.
|
||||
The list of Elastic Detection Rules suggested is provided in the context below.
|
||||
|
||||
Guidelines:
|
||||
If there is no Elastic rule in the list that covers the same threat, answer only with the string: no_match
|
||||
If there is one Elastic rule in the list that covers the same threat, answer only with its name without any further explanation.
|
||||
If there are multiple rules in the list that cover the same threat, answer with the most specific of them, for example: "Linux User Account Creation" is more specific than "User Account Creation".
|
||||
|
||||
<ELASTIC_DETECTION_RULE_NAMES>
|
||||
{elasticSecurityRules}
|
||||
</ELASTIC_DETECTION_RULE_NAMES>
|
||||
`,
|
||||
],
|
||||
[
|
||||
'human',
|
||||
`The Splunk Detection Rule is:
|
||||
<<SPLUNK_RULE_TITLE>>
|
||||
{ruleTitle}
|
||||
<<SPLUNK_RULE_TITLE>>
|
||||
`,
|
||||
],
|
||||
['ai', 'Please find the answer below:'],
|
||||
]);
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Logger } from '@kbn/core/server';
|
||||
import { naturalLanguageToEsql, type InferenceClient } from '@kbn/inference-plugin/server';
|
||||
import { lastValueFrom } from 'rxjs';
|
||||
|
||||
export type EsqlKnowledgeBaseCaller = (input: string) => Promise<string>;
|
||||
|
||||
type GetEsqlTranslatorToolParams = (params: {
|
||||
inferenceClient: InferenceClient;
|
||||
connectorId: string;
|
||||
logger: Logger;
|
||||
}) => EsqlKnowledgeBaseCaller;
|
||||
|
||||
export const getEsqlKnowledgeBase: GetEsqlTranslatorToolParams =
|
||||
({ inferenceClient: client, connectorId, logger }) =>
|
||||
async (input: string) => {
|
||||
const { content } = await lastValueFrom(
|
||||
naturalLanguageToEsql({
|
||||
client,
|
||||
connectorId,
|
||||
input,
|
||||
logger: {
|
||||
debug: (source) => {
|
||||
logger.debug(typeof source === 'function' ? source() : source);
|
||||
},
|
||||
},
|
||||
})
|
||||
);
|
||||
return content;
|
||||
};
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
export { getTranslateQueryNode } from './translate_query';
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { MigrateRuleState } from '../../types';
|
||||
|
||||
export const getEsqlTranslationPrompt = (
|
||||
state: MigrateRuleState
|
||||
): string => `You are a helpful cybersecurity (SIEM) expert agent. Your task is to migrate "detection rules" from Splunk to Elastic Security.
|
||||
Below you will find Splunk rule information: the title, description and the SPL (Search Processing Language) query.
|
||||
Your goal is to translate the SPL query into an equivalent Elastic Security Query Language (ES|QL) query.
|
||||
|
||||
Guidelines:
|
||||
- Start the translation process by analyzing the SPL query and identifying the key components.
|
||||
- Always use logs* index pattern for the ES|QL translated query.
|
||||
- If, in the SPL query, you find a lookup list or macro that, based only on its name, you can not translate with confidence to ES|QL, mention it in the summary and
|
||||
add a placeholder in the query with the format [macro:<macro_name>(parameters)] or [lookup:<lookup_name>] including the [] keys, example: [macro:my_macro(first_param,second_param)] or [lookup:my_lookup].
|
||||
|
||||
The output will be parsed and should contain:
|
||||
- First, the ES|QL query inside an \`\`\`esql code block.
|
||||
- At the end, the summary of the translation process followed in markdown, starting with "## Migration Summary".
|
||||
|
||||
This is the Splunk rule information:
|
||||
|
||||
<<SPLUNK_RULE_TITLE>>
|
||||
${state.original_rule.title}
|
||||
<</SPLUNK_RULE_TITLE>>
|
||||
|
||||
<<SPLUNK_RULE_DESCRIPTION>>
|
||||
${state.original_rule.description}
|
||||
<</SPLUNK_RULE_DESCRIPTION>>
|
||||
|
||||
<<SPLUNK_RULE_QUERY_SLP>>
|
||||
${state.original_rule.query}
|
||||
<</SPLUNK_RULE_QUERY_SLP>>
|
||||
`;
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Logger } from '@kbn/core/server';
|
||||
import type { InferenceClient } from '@kbn/inference-plugin/server';
|
||||
import type { GraphNode } from '../../types';
|
||||
import { getEsqlKnowledgeBase } from './esql_knowledge_base_caller';
|
||||
import { getEsqlTranslationPrompt } from './prompt';
|
||||
import { SiemMigrationRuleTranslationResult } from '../../../../../../../../common/siem_migrations/constants';
|
||||
|
||||
interface GetTranslateQueryNodeParams {
|
||||
inferenceClient: InferenceClient;
|
||||
connectorId: string;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
export const getTranslateQueryNode = ({
|
||||
inferenceClient,
|
||||
connectorId,
|
||||
logger,
|
||||
}: GetTranslateQueryNodeParams): GraphNode => {
|
||||
const esqlKnowledgeBaseCaller = getEsqlKnowledgeBase({ inferenceClient, connectorId, logger });
|
||||
return async (state) => {
|
||||
const input = getEsqlTranslationPrompt(state);
|
||||
const response = await esqlKnowledgeBaseCaller(input);
|
||||
|
||||
const esqlQuery = response.match(/```esql\n([\s\S]*?)\n```/)?.[1] ?? '';
|
||||
const summary = response.match(/## Migration Summary[\s\S]*$/)?.[0] ?? '';
|
||||
|
||||
const translationResult = getTranslationResult(esqlQuery);
|
||||
|
||||
return {
|
||||
response,
|
||||
comments: [summary],
|
||||
translation_result: translationResult,
|
||||
elastic_rule: {
|
||||
title: state.original_rule.title,
|
||||
description: state.original_rule.description,
|
||||
severity: 'low',
|
||||
query: esqlQuery,
|
||||
query_language: 'esql',
|
||||
},
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
const getTranslationResult = (esqlQuery: string): SiemMigrationRuleTranslationResult => {
|
||||
if (esqlQuery.match(/\[(macro|lookup):[\s\S]*\]/)) {
|
||||
return SiemMigrationRuleTranslationResult.PARTIAL;
|
||||
}
|
||||
return SiemMigrationRuleTranslationResult.FULL;
|
||||
};
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { BaseMessage } from '@langchain/core/messages';
|
||||
import { Annotation, messagesStateReducer } from '@langchain/langgraph';
|
||||
import type {
|
||||
ElasticRule,
|
||||
OriginalRule,
|
||||
RuleMigration,
|
||||
} from '../../../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
import type { SiemMigrationRuleTranslationResult } from '../../../../../../common/siem_migrations/constants';
|
||||
|
||||
export const migrateRuleState = Annotation.Root({
|
||||
messages: Annotation<BaseMessage[]>({
|
||||
reducer: messagesStateReducer,
|
||||
default: () => [],
|
||||
}),
|
||||
original_rule: Annotation<OriginalRule>(),
|
||||
elastic_rule: Annotation<ElasticRule>({
|
||||
reducer: (state, action) => ({ ...state, ...action }),
|
||||
}),
|
||||
translation_result: Annotation<SiemMigrationRuleTranslationResult>(),
|
||||
comments: Annotation<RuleMigration['comments']>({
|
||||
reducer: (current, value) => (value ? (current ?? []).concat(value) : current),
|
||||
default: () => [],
|
||||
}),
|
||||
response: Annotation<string>(),
|
||||
});
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Logger } from '@kbn/core/server';
|
||||
import type { InferenceClient } from '@kbn/inference-plugin/server';
|
||||
import type { migrateRuleState } from './state';
|
||||
import type { ChatModel } from '../util/actions_client_chat';
|
||||
import type { PrebuiltRulesMapByName } from '../util/prebuilt_rules';
|
||||
|
||||
export type MigrateRuleState = typeof migrateRuleState.State;
|
||||
export type GraphNode = (state: MigrateRuleState) => Promise<Partial<MigrateRuleState>>;
|
||||
|
||||
export interface MigrateRuleGraphParams {
|
||||
inferenceClient: InferenceClient;
|
||||
model: ChatModel;
|
||||
connectorId: string;
|
||||
prebuiltRulesMap: PrebuiltRulesMapByName;
|
||||
logger: Logger;
|
||||
}
|
||||
|
|
@ -0,0 +1,285 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Logger } from '@kbn/core/server';
|
||||
import { AbortError, abortSignalToPromise } from '@kbn/kibana-utils-plugin/server';
|
||||
import type { RunnableConfig } from '@langchain/core/runnables';
|
||||
import type {
|
||||
RuleMigrationAllTaskStats,
|
||||
RuleMigrationTaskStats,
|
||||
} from '../../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
import type { RuleMigrationDataStats } from '../data_stream/rule_migrations_data_client';
|
||||
import type {
|
||||
RuleMigrationTaskStartParams,
|
||||
RuleMigrationTaskStartResult,
|
||||
RuleMigrationTaskStatsParams,
|
||||
RuleMigrationTaskStopParams,
|
||||
RuleMigrationTaskStopResult,
|
||||
RuleMigrationTaskPrepareParams,
|
||||
RuleMigrationTaskRunParams,
|
||||
MigrationAgent,
|
||||
RuleMigrationAllTaskStatsParams,
|
||||
} from './types';
|
||||
import { getRuleMigrationAgent } from './agent';
|
||||
import type { MigrateRuleState } from './agent/types';
|
||||
import { retrievePrebuiltRulesMap } from './util/prebuilt_rules';
|
||||
import { ActionsClientChat } from './util/actions_client_chat';
|
||||
|
||||
interface TaskLogger {
|
||||
info: (msg: string) => void;
|
||||
debug: (msg: string) => void;
|
||||
error: (msg: string, error: Error) => void;
|
||||
}
|
||||
const getTaskLogger = (logger: Logger): TaskLogger => {
|
||||
const prefix = '[ruleMigrationsTask]: ';
|
||||
return {
|
||||
info: (msg) => logger.info(`${prefix}${msg}`),
|
||||
debug: (msg) => logger.debug(`${prefix}${msg}`),
|
||||
error: (msg, error) => logger.error(`${prefix}${msg}: ${error.message}`),
|
||||
};
|
||||
};
|
||||
|
||||
const ITERATION_BATCH_SIZE = 50 as const;
|
||||
const ITERATION_SLEEP_SECONDS = 10 as const;
|
||||
|
||||
export class RuleMigrationsTaskRunner {
|
||||
private migrationsRunning: Map<string, { user: string; abortController: AbortController }>;
|
||||
private taskLogger: TaskLogger;
|
||||
|
||||
constructor(private logger: Logger) {
|
||||
this.migrationsRunning = new Map();
|
||||
this.taskLogger = getTaskLogger(logger);
|
||||
}
|
||||
|
||||
/** Starts a rule migration task */
|
||||
async start(params: RuleMigrationTaskStartParams): Promise<RuleMigrationTaskStartResult> {
|
||||
const { migrationId, dataClient } = params;
|
||||
if (this.migrationsRunning.has(migrationId)) {
|
||||
return { exists: true, started: false };
|
||||
}
|
||||
// Just in case some previous execution was interrupted without releasing
|
||||
await dataClient.releaseProcessable(migrationId);
|
||||
|
||||
const { rules } = await dataClient.getStats(migrationId);
|
||||
if (rules.total === 0) {
|
||||
return { exists: false, started: false };
|
||||
}
|
||||
if (rules.pending === 0) {
|
||||
return { exists: true, started: false };
|
||||
}
|
||||
|
||||
const abortController = new AbortController();
|
||||
|
||||
// Await the preparation to make sure the agent is created properly so the task can run
|
||||
const agent = await this.prepare({ ...params, abortController });
|
||||
|
||||
// not awaiting the `run` promise to execute the task in the background
|
||||
this.run({ ...params, agent, abortController }).catch((err) => {
|
||||
// All errors in the `run` method are already catch, this should never happen, but just in case
|
||||
this.taskLogger.error(`Unexpected error running the migration ID:${migrationId}`, err);
|
||||
});
|
||||
|
||||
return { exists: true, started: true };
|
||||
}
|
||||
|
||||
private async prepare({
|
||||
connectorId,
|
||||
inferenceClient,
|
||||
actionsClient,
|
||||
rulesClient,
|
||||
soClient,
|
||||
abortController,
|
||||
}: RuleMigrationTaskPrepareParams): Promise<MigrationAgent> {
|
||||
const prebuiltRulesMap = await retrievePrebuiltRulesMap({ soClient, rulesClient });
|
||||
|
||||
const actionsClientChat = new ActionsClientChat(connectorId, actionsClient, this.logger);
|
||||
const model = await actionsClientChat.createModel({
|
||||
signal: abortController.signal,
|
||||
temperature: 0.05,
|
||||
});
|
||||
|
||||
const agent = getRuleMigrationAgent({
|
||||
connectorId,
|
||||
model,
|
||||
inferenceClient,
|
||||
prebuiltRulesMap,
|
||||
logger: this.logger,
|
||||
});
|
||||
return agent;
|
||||
}
|
||||
|
||||
private async run({
|
||||
migrationId,
|
||||
agent,
|
||||
dataClient,
|
||||
currentUser,
|
||||
invocationConfig,
|
||||
abortController,
|
||||
}: RuleMigrationTaskRunParams): Promise<void> {
|
||||
if (this.migrationsRunning.has(migrationId)) {
|
||||
// This should never happen, but just in case
|
||||
throw new Error(`Task already running for migration ID:${migrationId} `);
|
||||
}
|
||||
this.taskLogger.info(`Starting migration ID:${migrationId}`);
|
||||
|
||||
this.migrationsRunning.set(migrationId, { user: currentUser.username, abortController });
|
||||
const config: RunnableConfig = {
|
||||
...invocationConfig,
|
||||
// signal: abortController.signal, // not working properly https://github.com/langchain-ai/langgraphjs/issues/319
|
||||
};
|
||||
|
||||
const abortPromise = abortSignalToPromise(abortController.signal);
|
||||
|
||||
try {
|
||||
const sleep = async (seconds: number) => {
|
||||
this.taskLogger.debug(`Sleeping ${seconds}s for migration ID:${migrationId}`);
|
||||
await Promise.race([
|
||||
new Promise((resolve) => setTimeout(resolve, seconds * 1000)),
|
||||
abortPromise.promise,
|
||||
]);
|
||||
};
|
||||
|
||||
let isDone: boolean = false;
|
||||
do {
|
||||
const ruleMigrations = await dataClient.takePending(migrationId, ITERATION_BATCH_SIZE);
|
||||
this.taskLogger.debug(
|
||||
`Processing ${ruleMigrations.length} rules for migration ID:${migrationId}`
|
||||
);
|
||||
|
||||
await Promise.all(
|
||||
ruleMigrations.map(async (ruleMigration) => {
|
||||
this.taskLogger.debug(
|
||||
`Starting migration of rule "${ruleMigration.original_rule.title}"`
|
||||
);
|
||||
try {
|
||||
const start = Date.now();
|
||||
|
||||
const ruleMigrationResult: MigrateRuleState = await Promise.race([
|
||||
agent.invoke({ original_rule: ruleMigration.original_rule }, config),
|
||||
abortPromise.promise, // workaround for the issue with the langGraph signal
|
||||
]);
|
||||
|
||||
const duration = (Date.now() - start) / 1000;
|
||||
this.taskLogger.debug(
|
||||
`Migration of rule "${ruleMigration.original_rule.title}" finished in ${duration}s`
|
||||
);
|
||||
|
||||
await dataClient.saveFinished({
|
||||
...ruleMigration,
|
||||
elastic_rule: ruleMigrationResult.elastic_rule,
|
||||
translation_result: ruleMigrationResult.translation_result,
|
||||
comments: ruleMigrationResult.comments,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof AbortError) {
|
||||
throw error;
|
||||
}
|
||||
this.taskLogger.error(
|
||||
`Error migrating rule "${ruleMigration.original_rule.title}"`,
|
||||
error
|
||||
);
|
||||
await dataClient.saveError({
|
||||
...ruleMigration,
|
||||
comments: [`Error migrating rule: ${error.message}`],
|
||||
});
|
||||
}
|
||||
})
|
||||
);
|
||||
|
||||
this.taskLogger.debug(`Batch processed successfully for migration ID:${migrationId}`);
|
||||
|
||||
const { rules } = await dataClient.getStats(migrationId);
|
||||
isDone = rules.pending === 0;
|
||||
if (!isDone) {
|
||||
await sleep(ITERATION_SLEEP_SECONDS);
|
||||
}
|
||||
} while (!isDone);
|
||||
|
||||
this.taskLogger.info(`Finished migration ID:${migrationId}`);
|
||||
} catch (error) {
|
||||
await dataClient.releaseProcessing(migrationId);
|
||||
|
||||
if (error instanceof AbortError) {
|
||||
this.taskLogger.info(`Abort signal received, stopping migration ID:${migrationId}`);
|
||||
return;
|
||||
} else {
|
||||
this.taskLogger.error(`Error processing migration ID:${migrationId}`, error);
|
||||
}
|
||||
} finally {
|
||||
this.migrationsRunning.delete(migrationId);
|
||||
abortPromise.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
/** Returns the stats of a migration */
|
||||
async getStats({
|
||||
migrationId,
|
||||
dataClient,
|
||||
}: RuleMigrationTaskStatsParams): Promise<RuleMigrationTaskStats> {
|
||||
const dataStats = await dataClient.getStats(migrationId);
|
||||
const status = this.getTaskStatus(migrationId, dataStats.rules);
|
||||
return { status, ...dataStats };
|
||||
}
|
||||
|
||||
/** Returns the stats of all migrations */
|
||||
async getAllStats({
|
||||
dataClient,
|
||||
}: RuleMigrationAllTaskStatsParams): Promise<RuleMigrationAllTaskStats> {
|
||||
const allDataStats = await dataClient.getAllStats();
|
||||
return allDataStats.map((dataStats) => {
|
||||
const status = this.getTaskStatus(dataStats.migration_id, dataStats.rules);
|
||||
return { status, ...dataStats };
|
||||
});
|
||||
}
|
||||
|
||||
private getTaskStatus(
|
||||
migrationId: string,
|
||||
dataStats: RuleMigrationDataStats['rules']
|
||||
): RuleMigrationTaskStats['status'] {
|
||||
if (this.migrationsRunning.has(migrationId)) {
|
||||
return 'running';
|
||||
}
|
||||
if (dataStats.pending === dataStats.total) {
|
||||
return 'ready';
|
||||
}
|
||||
if (dataStats.completed + dataStats.failed === dataStats.total) {
|
||||
return 'finished';
|
||||
}
|
||||
return 'stopped';
|
||||
}
|
||||
|
||||
/** Stops one running migration */
|
||||
async stop({
|
||||
migrationId,
|
||||
dataClient,
|
||||
}: RuleMigrationTaskStopParams): Promise<RuleMigrationTaskStopResult> {
|
||||
try {
|
||||
const migrationRunning = this.migrationsRunning.get(migrationId);
|
||||
if (migrationRunning) {
|
||||
migrationRunning.abortController.abort();
|
||||
return { exists: true, stopped: true };
|
||||
}
|
||||
|
||||
const { rules } = await dataClient.getStats(migrationId);
|
||||
if (rules.total > 0) {
|
||||
return { exists: true, stopped: false };
|
||||
}
|
||||
return { exists: false, stopped: false };
|
||||
} catch (err) {
|
||||
this.taskLogger.error(`Error stopping migration ID:${migrationId}`, err);
|
||||
return { exists: true, stopped: false };
|
||||
}
|
||||
}
|
||||
|
||||
/** Stops all running migrations */
|
||||
stopAll() {
|
||||
this.migrationsRunning.forEach((migrationRunning) => {
|
||||
migrationRunning.abortController.abort();
|
||||
});
|
||||
this.migrationsRunning.clear();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { AuthenticatedUser, SavedObjectsClientContract } from '@kbn/core/server';
|
||||
import type { RunnableConfig } from '@langchain/core/runnables';
|
||||
import type { InferenceClient } from '@kbn/inference-plugin/server';
|
||||
import type { ActionsClient } from '@kbn/actions-plugin/server';
|
||||
import type { RulesClient } from '@kbn/alerting-plugin/server';
|
||||
import type { RuleMigrationsDataClient } from '../data_stream/rule_migrations_data_client';
|
||||
import type { getRuleMigrationAgent } from './agent';
|
||||
|
||||
export type MigrationAgent = ReturnType<typeof getRuleMigrationAgent>;
|
||||
|
||||
export interface RuleMigrationTaskStartParams {
|
||||
migrationId: string;
|
||||
currentUser: AuthenticatedUser;
|
||||
connectorId: string;
|
||||
invocationConfig: RunnableConfig;
|
||||
inferenceClient: InferenceClient;
|
||||
actionsClient: ActionsClient;
|
||||
rulesClient: RulesClient;
|
||||
soClient: SavedObjectsClientContract;
|
||||
dataClient: RuleMigrationsDataClient;
|
||||
}
|
||||
|
||||
export interface RuleMigrationTaskPrepareParams {
|
||||
connectorId: string;
|
||||
inferenceClient: InferenceClient;
|
||||
actionsClient: ActionsClient;
|
||||
rulesClient: RulesClient;
|
||||
soClient: SavedObjectsClientContract;
|
||||
abortController: AbortController;
|
||||
}
|
||||
|
||||
export interface RuleMigrationTaskRunParams {
|
||||
migrationId: string;
|
||||
currentUser: AuthenticatedUser;
|
||||
invocationConfig: RunnableConfig;
|
||||
agent: MigrationAgent;
|
||||
dataClient: RuleMigrationsDataClient;
|
||||
abortController: AbortController;
|
||||
}
|
||||
|
||||
export interface RuleMigrationTaskStopParams {
|
||||
migrationId: string;
|
||||
dataClient: RuleMigrationsDataClient;
|
||||
}
|
||||
|
||||
export interface RuleMigrationTaskStatsParams {
|
||||
migrationId: string;
|
||||
dataClient: RuleMigrationsDataClient;
|
||||
}
|
||||
|
||||
export interface RuleMigrationAllTaskStatsParams {
|
||||
dataClient: RuleMigrationsDataClient;
|
||||
}
|
||||
|
||||
export interface RuleMigrationTaskStartResult {
|
||||
started: boolean;
|
||||
exists: boolean;
|
||||
}
|
||||
|
||||
export interface RuleMigrationTaskStopResult {
|
||||
stopped: boolean;
|
||||
exists: boolean;
|
||||
}
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { ActionsClientSimpleChatModel } from '@kbn/langchain/server';
|
||||
import {
|
||||
ActionsClientBedrockChatModel,
|
||||
ActionsClientChatOpenAI,
|
||||
ActionsClientChatVertexAI,
|
||||
} from '@kbn/langchain/server';
|
||||
import type { Logger } from '@kbn/core/server';
|
||||
import type { ActionsClient } from '@kbn/actions-plugin/server';
|
||||
import type { ActionsClientChatOpenAIParams } from '@kbn/langchain/server/language_models/chat_openai';
|
||||
import type { CustomChatModelInput as ActionsClientBedrockChatModelParams } from '@kbn/langchain/server/language_models/bedrock_chat';
|
||||
import type { CustomChatModelInput as ActionsClientChatVertexAIParams } from '@kbn/langchain/server/language_models/gemini_chat';
|
||||
import type { CustomChatModelInput as ActionsClientSimpleChatModelParams } from '@kbn/langchain/server/language_models/simple_chat_model';
|
||||
|
||||
export type ChatModel =
|
||||
| ActionsClientSimpleChatModel
|
||||
| ActionsClientChatOpenAI
|
||||
| ActionsClientBedrockChatModel
|
||||
| ActionsClientChatVertexAI;
|
||||
|
||||
export type ActionsClientChatModelClass =
|
||||
| typeof ActionsClientSimpleChatModel
|
||||
| typeof ActionsClientChatOpenAI
|
||||
| typeof ActionsClientBedrockChatModel
|
||||
| typeof ActionsClientChatVertexAI;
|
||||
|
||||
export type ChatModelParams = Partial<ActionsClientSimpleChatModelParams> &
|
||||
Partial<ActionsClientChatOpenAIParams> &
|
||||
Partial<ActionsClientBedrockChatModelParams> &
|
||||
Partial<ActionsClientChatVertexAIParams> & {
|
||||
/** Enables the streaming mode of the response, disabled by default */
|
||||
streaming?: boolean;
|
||||
};
|
||||
|
||||
const llmTypeDictionary: Record<string, string> = {
|
||||
[`.gen-ai`]: `openai`,
|
||||
[`.bedrock`]: `bedrock`,
|
||||
[`.gemini`]: `gemini`,
|
||||
};
|
||||
|
||||
export class ActionsClientChat {
|
||||
constructor(
|
||||
private readonly connectorId: string,
|
||||
private readonly actionsClient: ActionsClient,
|
||||
private readonly logger: Logger
|
||||
) {}
|
||||
|
||||
public async createModel(params?: ChatModelParams): Promise<ChatModel> {
|
||||
const connector = await this.actionsClient.get({ id: this.connectorId });
|
||||
if (!connector) {
|
||||
throw new Error(`Connector not found: ${this.connectorId}`);
|
||||
}
|
||||
|
||||
const llmType = this.getLLMType(connector.actionTypeId);
|
||||
const ChatModelClass = this.getLLMClass(llmType);
|
||||
|
||||
const model = new ChatModelClass({
|
||||
actionsClient: this.actionsClient,
|
||||
connectorId: this.connectorId,
|
||||
logger: this.logger,
|
||||
llmType,
|
||||
model: connector.config?.defaultModel,
|
||||
...params,
|
||||
streaming: params?.streaming ?? false, // disabling streaming by default, for some reason is enabled when omitted
|
||||
});
|
||||
return model;
|
||||
}
|
||||
|
||||
private getLLMType(actionTypeId: string): string | undefined {
|
||||
if (llmTypeDictionary[actionTypeId]) {
|
||||
return llmTypeDictionary[actionTypeId];
|
||||
}
|
||||
throw new Error(`Unknown LLM type for action type ID: ${actionTypeId}`);
|
||||
}
|
||||
|
||||
private getLLMClass(llmType?: string): ActionsClientChatModelClass {
|
||||
switch (llmType) {
|
||||
case 'bedrock':
|
||||
return ActionsClientBedrockChatModel;
|
||||
case 'gemini':
|
||||
return ActionsClientChatVertexAI;
|
||||
case 'openai':
|
||||
default:
|
||||
return ActionsClientChatOpenAI;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { savedObjectsClientMock } from '@kbn/core-saved-objects-api-server-mocks';
|
||||
import type { PrebuiltRulesMapByName } from './prebuilt_rules';
|
||||
import { filterPrebuiltRules, retrievePrebuiltRulesMap } from './prebuilt_rules';
|
||||
import { rulesClientMock } from '@kbn/alerting-plugin/server/mocks';
|
||||
|
||||
jest.mock(
|
||||
'../../../../detection_engine/prebuilt_rules/logic/rule_objects/prebuilt_rule_objects_client',
|
||||
() => ({ createPrebuiltRuleObjectsClient: jest.fn() })
|
||||
);
|
||||
jest.mock(
|
||||
'../../../../detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client',
|
||||
() => ({ createPrebuiltRuleAssetsClient: jest.fn() })
|
||||
);
|
||||
|
||||
const mitreAttackIds = 'T1234';
|
||||
const rule1 = {
|
||||
name: 'rule one',
|
||||
id: 'rule1',
|
||||
threat: [
|
||||
{
|
||||
framework: 'MITRE ATT&CK',
|
||||
technique: [{ id: mitreAttackIds, name: 'tactic one' }],
|
||||
},
|
||||
],
|
||||
};
|
||||
const rule2 = {
|
||||
name: 'rule two',
|
||||
id: 'rule2',
|
||||
};
|
||||
|
||||
const defaultRuleVersionsTriad = new Map<string, unknown>([
|
||||
['rule1', { target: rule1 }],
|
||||
['rule2', { target: rule2, current: rule2 }],
|
||||
]);
|
||||
const mockFetchRuleVersionsTriad = jest.fn().mockResolvedValue(defaultRuleVersionsTriad);
|
||||
jest.mock(
|
||||
'../../../../detection_engine/prebuilt_rules/logic/rule_versions/fetch_rule_versions_triad',
|
||||
() => ({
|
||||
fetchRuleVersionsTriad: () => mockFetchRuleVersionsTriad(),
|
||||
})
|
||||
);
|
||||
|
||||
const defaultParams = {
|
||||
soClient: savedObjectsClientMock.create(),
|
||||
rulesClient: rulesClientMock.create(),
|
||||
};
|
||||
|
||||
describe('retrievePrebuiltRulesMap', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
describe('when prebuilt rule is installed', () => {
|
||||
it('should return isInstalled flag', async () => {
|
||||
const prebuiltRulesMap = await retrievePrebuiltRulesMap(defaultParams);
|
||||
expect(prebuiltRulesMap.size).toBe(2);
|
||||
expect(prebuiltRulesMap.get('rule one')).toEqual(
|
||||
expect.objectContaining({ installedRuleId: undefined })
|
||||
);
|
||||
expect(prebuiltRulesMap.get('rule two')).toEqual(
|
||||
expect.objectContaining({ installedRuleId: rule2.id })
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('filterPrebuiltRules', () => {
|
||||
let prebuiltRulesMap: PrebuiltRulesMapByName;
|
||||
|
||||
beforeEach(async () => {
|
||||
prebuiltRulesMap = await retrievePrebuiltRulesMap(defaultParams);
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
describe('when splunk rule contains empty mitreAttackIds', () => {
|
||||
it('should return empty rules map', async () => {
|
||||
const filteredPrebuiltRules = filterPrebuiltRules(prebuiltRulesMap, []);
|
||||
expect(filteredPrebuiltRules.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('when splunk rule does not match mitreAttackIds', () => {
|
||||
it('should return empty rules map', async () => {
|
||||
const filteredPrebuiltRules = filterPrebuiltRules(prebuiltRulesMap, [`${mitreAttackIds}_2`]);
|
||||
expect(filteredPrebuiltRules.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('when splunk rule contains matching mitreAttackIds', () => {
|
||||
it('should return the filtered rules map', async () => {
|
||||
const filteredPrebuiltRules = filterPrebuiltRules(prebuiltRulesMap, [mitreAttackIds]);
|
||||
expect(filteredPrebuiltRules.size).toBe(1);
|
||||
expect(filteredPrebuiltRules.get('rule one')).toEqual(
|
||||
expect.objectContaining({ rule: rule1 })
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { RulesClient } from '@kbn/alerting-plugin/server';
|
||||
import type { SavedObjectsClientContract } from '@kbn/core-saved-objects-api-server';
|
||||
import type { PrebuiltRuleAsset } from '../../../../detection_engine/prebuilt_rules';
|
||||
import { fetchRuleVersionsTriad } from '../../../../detection_engine/prebuilt_rules/logic/rule_versions/fetch_rule_versions_triad';
|
||||
import { createPrebuiltRuleObjectsClient } from '../../../../detection_engine/prebuilt_rules/logic/rule_objects/prebuilt_rule_objects_client';
|
||||
import { createPrebuiltRuleAssetsClient } from '../../../../detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client';
|
||||
|
||||
export interface PrebuiltRuleMapped {
|
||||
rule: PrebuiltRuleAsset;
|
||||
installedRuleId?: string;
|
||||
}
|
||||
|
||||
export type PrebuiltRulesMapByName = Map<string, PrebuiltRuleMapped>;
|
||||
|
||||
interface RetrievePrebuiltRulesParams {
|
||||
soClient: SavedObjectsClientContract;
|
||||
rulesClient: RulesClient;
|
||||
}
|
||||
|
||||
export const retrievePrebuiltRulesMap = async ({
|
||||
soClient,
|
||||
rulesClient,
|
||||
}: RetrievePrebuiltRulesParams): Promise<PrebuiltRulesMapByName> => {
|
||||
const ruleAssetsClient = createPrebuiltRuleAssetsClient(soClient);
|
||||
const ruleObjectsClient = createPrebuiltRuleObjectsClient(rulesClient);
|
||||
|
||||
const prebuiltRulesMap = await fetchRuleVersionsTriad({
|
||||
ruleAssetsClient,
|
||||
ruleObjectsClient,
|
||||
});
|
||||
const prebuiltRulesByName: PrebuiltRulesMapByName = new Map();
|
||||
prebuiltRulesMap.forEach((ruleVersions) => {
|
||||
const rule = ruleVersions.target || ruleVersions.current;
|
||||
if (rule) {
|
||||
prebuiltRulesByName.set(rule.name, {
|
||||
rule,
|
||||
installedRuleId: ruleVersions.current?.id,
|
||||
});
|
||||
}
|
||||
});
|
||||
return prebuiltRulesByName;
|
||||
};
|
||||
|
||||
export const filterPrebuiltRules = (
|
||||
prebuiltRulesByName: PrebuiltRulesMapByName,
|
||||
mitreAttackIds: string[]
|
||||
) => {
|
||||
const filteredPrebuiltRulesByName = new Map();
|
||||
if (mitreAttackIds?.length) {
|
||||
// If this rule has MITRE ATT&CK IDs, remove unrelated prebuilt rules
|
||||
prebuiltRulesByName.forEach(({ rule }, ruleName) => {
|
||||
const mitreAttackThreat = rule.threat?.filter(
|
||||
({ framework }) => framework === 'MITRE ATT&CK'
|
||||
);
|
||||
if (!mitreAttackThreat) {
|
||||
// If this rule has no MITRE ATT&CK reference we skip it
|
||||
return;
|
||||
}
|
||||
|
||||
const sameTechnique = mitreAttackThreat.find((threat) =>
|
||||
threat.technique?.some(({ id }) => mitreAttackIds?.includes(id))
|
||||
);
|
||||
|
||||
if (sameTechnique) {
|
||||
filteredPrebuiltRulesByName.set(ruleName, prebuiltRulesByName.get(ruleName));
|
||||
}
|
||||
});
|
||||
}
|
||||
return filteredPrebuiltRulesByName;
|
||||
};
|
||||
|
|
@ -5,10 +5,29 @@
|
|||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { BulkResponse, SearchResponse } from '@elastic/elasticsearch/lib/api/types';
|
||||
import type { IClusterClient, KibanaRequest } from '@kbn/core/server';
|
||||
import type {
|
||||
AuthenticatedUser,
|
||||
IClusterClient,
|
||||
KibanaRequest,
|
||||
SavedObjectsClientContract,
|
||||
} from '@kbn/core/server';
|
||||
import type { Subject } from 'rxjs';
|
||||
import type { RuleMigration } from '../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
import type { InferenceClient } from '@kbn/inference-plugin/server';
|
||||
import type { RunnableConfig } from '@langchain/core/runnables';
|
||||
import type { ActionsClient } from '@kbn/actions-plugin/server';
|
||||
import type { RulesClient } from '@kbn/alerting-plugin/server';
|
||||
import type {
|
||||
RuleMigration,
|
||||
RuleMigrationAllTaskStats,
|
||||
RuleMigrationTaskStats,
|
||||
} from '../../../../common/siem_migrations/model/rule_migration.gen';
|
||||
import type { RuleMigrationsDataClient } from './data_stream/rule_migrations_data_client';
|
||||
import type { RuleMigrationTaskStopResult, RuleMigrationTaskStartResult } from './task/types';
|
||||
|
||||
export interface StoredRuleMigration extends RuleMigration {
|
||||
_id: string;
|
||||
_index: string;
|
||||
}
|
||||
|
||||
export interface SiemRulesMigrationsSetupParams {
|
||||
esClusterClient: IClusterClient;
|
||||
|
|
@ -16,15 +35,28 @@ export interface SiemRulesMigrationsSetupParams {
|
|||
tasksTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface SiemRuleMigrationsGetClientParams {
|
||||
export interface SiemRuleMigrationsCreateClientParams {
|
||||
request: KibanaRequest;
|
||||
currentUser: AuthenticatedUser | null;
|
||||
spaceId: string;
|
||||
}
|
||||
|
||||
export interface RuleMigrationSearchParams {
|
||||
migration_id?: string;
|
||||
export interface SiemRuleMigrationsStartTaskParams {
|
||||
migrationId: string;
|
||||
connectorId: string;
|
||||
invocationConfig: RunnableConfig;
|
||||
inferenceClient: InferenceClient;
|
||||
actionsClient: ActionsClient;
|
||||
rulesClient: RulesClient;
|
||||
soClient: SavedObjectsClientContract;
|
||||
}
|
||||
|
||||
export interface SiemRuleMigrationsClient {
|
||||
create: (body: RuleMigration[]) => Promise<BulkResponse>;
|
||||
search: (params: RuleMigrationSearchParams) => Promise<SearchResponse>;
|
||||
data: RuleMigrationsDataClient;
|
||||
task: {
|
||||
start: (params: SiemRuleMigrationsStartTaskParams) => Promise<RuleMigrationTaskStartResult>;
|
||||
stop: (migrationId: string) => Promise<RuleMigrationTaskStopResult>;
|
||||
getStats: (migrationId: string) => Promise<RuleMigrationTaskStats>;
|
||||
getAllStats: () => Promise<RuleMigrationAllTaskStats>;
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,9 +8,15 @@ import {
|
|||
loggingSystemMock,
|
||||
elasticsearchServiceMock,
|
||||
httpServerMock,
|
||||
securityServiceMock,
|
||||
} from '@kbn/core/server/mocks';
|
||||
import { SiemMigrationsService } from './siem_migrations_service';
|
||||
import { MockSiemRuleMigrationsService, mockSetup, mockGetClient } from './rules/__mocks__/mocks';
|
||||
import {
|
||||
MockSiemRuleMigrationsService,
|
||||
mockSetup,
|
||||
mockCreateClient,
|
||||
mockStop,
|
||||
} from './rules/__mocks__/mocks';
|
||||
import type { ConfigType } from '../../config';
|
||||
|
||||
jest.mock('./rules/siem_rule_migrations_service');
|
||||
|
|
@ -25,6 +31,7 @@ describe('SiemMigrationsService', () => {
|
|||
let siemMigrationsService: SiemMigrationsService;
|
||||
const kibanaVersion = '8.16.0';
|
||||
|
||||
const currentUser = securityServiceMock.createMockAuthenticatedUser();
|
||||
const esClusterClient = elasticsearchServiceMock.createClusterClient();
|
||||
const logger = loggingSystemMock.createLogger();
|
||||
|
||||
|
|
@ -57,17 +64,22 @@ describe('SiemMigrationsService', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('when createClient is called', () => {
|
||||
describe('when createRulesClient is called', () => {
|
||||
it('should create rules client', async () => {
|
||||
const request = httpServerMock.createKibanaRequest();
|
||||
siemMigrationsService.createClient({ spaceId: 'default', request });
|
||||
expect(mockGetClient).toHaveBeenCalledWith({ spaceId: 'default', request });
|
||||
const createRulesClientParams = {
|
||||
spaceId: 'default',
|
||||
request: httpServerMock.createKibanaRequest(),
|
||||
currentUser,
|
||||
};
|
||||
siemMigrationsService.createRulesClient(createRulesClientParams);
|
||||
expect(mockCreateClient).toHaveBeenCalledWith(createRulesClientParams);
|
||||
});
|
||||
});
|
||||
|
||||
describe('when stop is called', () => {
|
||||
it('should trigger the pluginStop subject', async () => {
|
||||
siemMigrationsService.stop();
|
||||
expect(mockStop).toHaveBeenCalled();
|
||||
expect(mockReplaySubject$.next).toHaveBeenCalled();
|
||||
expect(mockReplaySubject$.complete).toHaveBeenCalled();
|
||||
});
|
||||
|
|
|
|||
|
|
@ -9,11 +9,8 @@ import type { Logger } from '@kbn/core/server';
|
|||
import { ReplaySubject, type Subject } from 'rxjs';
|
||||
import type { ConfigType } from '../../config';
|
||||
import { SiemRuleMigrationsService } from './rules/siem_rule_migrations_service';
|
||||
import type {
|
||||
SiemMigrationsClient,
|
||||
SiemMigrationsSetupParams,
|
||||
SiemMigrationsGetClientParams,
|
||||
} from './types';
|
||||
import type { SiemMigrationsSetupParams, SiemMigrationsCreateClientParams } from './types';
|
||||
import type { SiemRuleMigrationsClient } from './rules/types';
|
||||
|
||||
export class SiemMigrationsService {
|
||||
private pluginStop$: Subject<void>;
|
||||
|
|
@ -30,13 +27,12 @@ export class SiemMigrationsService {
|
|||
}
|
||||
}
|
||||
|
||||
createClient(params: SiemMigrationsGetClientParams): SiemMigrationsClient {
|
||||
return {
|
||||
rules: this.rules.getClient(params),
|
||||
};
|
||||
createRulesClient(params: SiemMigrationsCreateClientParams): SiemRuleMigrationsClient {
|
||||
return this.rules.createClient(params);
|
||||
}
|
||||
|
||||
stop() {
|
||||
this.rules.stop();
|
||||
this.pluginStop$.next();
|
||||
this.pluginStop$.complete();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,15 +6,11 @@
|
|||
*/
|
||||
|
||||
import type { IClusterClient } from '@kbn/core/server';
|
||||
import type { SiemRuleMigrationsClient, SiemRuleMigrationsGetClientParams } from './rules/types';
|
||||
import type { SiemRuleMigrationsCreateClientParams } from './rules/types';
|
||||
|
||||
export interface SiemMigrationsSetupParams {
|
||||
esClusterClient: IClusterClient;
|
||||
tasksTimeoutMs?: number;
|
||||
}
|
||||
|
||||
export type SiemMigrationsGetClientParams = SiemRuleMigrationsGetClientParams;
|
||||
|
||||
export interface SiemMigrationsClient {
|
||||
rules: SiemRuleMigrationsClient;
|
||||
}
|
||||
export type SiemMigrationsCreateClientParams = SiemRuleMigrationsCreateClientParams;
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ import type { SharePluginStart } from '@kbn/share-plugin/server';
|
|||
import type { GuidedOnboardingPluginSetup } from '@kbn/guided-onboarding-plugin/server';
|
||||
import type { PluginSetup as UnifiedSearchServerPluginSetup } from '@kbn/unified-search-plugin/server';
|
||||
import type { ElasticAssistantPluginStart } from '@kbn/elastic-assistant-plugin/server';
|
||||
import type { InferenceServerStart } from '@kbn/inference-plugin/server';
|
||||
import type { ProductFeaturesService } from './lib/product_features_service/product_features_service';
|
||||
import type { ExperimentalFeatures } from '../common';
|
||||
|
||||
|
|
@ -88,6 +89,7 @@ export interface SecuritySolutionPluginStartDependencies {
|
|||
telemetry?: TelemetryPluginStart;
|
||||
share: SharePluginStart;
|
||||
actions: ActionsPluginStartContract;
|
||||
inference: InferenceServerStart;
|
||||
}
|
||||
|
||||
export interface SecuritySolutionPluginSetup {
|
||||
|
|
|
|||
|
|
@ -168,10 +168,16 @@ export class RequestContextFactory implements IRequestContextFactory {
|
|||
})
|
||||
),
|
||||
|
||||
getSiemMigrationsClient: memoize(() =>
|
||||
siemMigrationsService.createClient({ request, spaceId: getSpaceId() })
|
||||
getSiemRuleMigrationsClient: memoize(() =>
|
||||
siemMigrationsService.createRulesClient({
|
||||
request,
|
||||
currentUser: coreContext.security.authc.getCurrentUser(),
|
||||
spaceId: getSpaceId(),
|
||||
})
|
||||
),
|
||||
|
||||
getInferenceClient: memoize(() => startPlugins.inference.getClient({ request })),
|
||||
|
||||
getExceptionListClient: () => {
|
||||
if (!lists) {
|
||||
return null;
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import type { AlertsClient, IRuleDataService } from '@kbn/rule-registry-plugin/s
|
|||
|
||||
import type { Readable } from 'stream';
|
||||
import type { AuditLogger } from '@kbn/security-plugin-types-server';
|
||||
import type { InferenceClient } from '@kbn/inference-plugin/server';
|
||||
import type { DataViewsService } from '@kbn/data-views-plugin/common';
|
||||
import type { Immutable } from '../common/endpoint/types';
|
||||
import { AppClient } from './client';
|
||||
|
|
@ -36,7 +37,7 @@ import type { RiskScoreDataClient } from './lib/entity_analytics/risk_score/risk
|
|||
import type { AssetCriticalityDataClient } from './lib/entity_analytics/asset_criticality';
|
||||
import type { IDetectionRulesClient } from './lib/detection_engine/rule_management/logic/detection_rules_client/detection_rules_client_interface';
|
||||
import type { EntityStoreDataClient } from './lib/entity_analytics/entity_store/entity_store_data_client';
|
||||
import type { SiemMigrationsClient } from './lib/siem_migrations/types';
|
||||
import type { SiemRuleMigrationsClient } from './lib/siem_migrations/rules/types';
|
||||
export { AppClient };
|
||||
|
||||
export interface SecuritySolutionApiRequestHandlerContext {
|
||||
|
|
@ -60,7 +61,8 @@ export interface SecuritySolutionApiRequestHandlerContext {
|
|||
getRiskScoreDataClient: () => RiskScoreDataClient;
|
||||
getAssetCriticalityDataClient: () => AssetCriticalityDataClient;
|
||||
getEntityStoreDataClient: () => EntityStoreDataClient;
|
||||
getSiemMigrationsClient: () => SiemMigrationsClient;
|
||||
getSiemRuleMigrationsClient: () => SiemRuleMigrationsClient;
|
||||
getInferenceClient: () => InferenceClient;
|
||||
}
|
||||
|
||||
export type SecuritySolutionRequestHandlerContext = CustomRequestHandlerContext<{
|
||||
|
|
|
|||
|
|
@ -228,5 +228,6 @@
|
|||
"@kbn/data-stream-adapter",
|
||||
"@kbn/core-lifecycle-server",
|
||||
"@kbn/core-user-profile-common",
|
||||
"@kbn/langchain",
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -95,6 +95,8 @@ import {
|
|||
GetRuleExecutionResultsRequestQueryInput,
|
||||
GetRuleExecutionResultsRequestParamsInput,
|
||||
} from '@kbn/security-solution-plugin/common/api/detection_engine/rule_monitoring/rule_execution_logs/get_rule_execution_results/get_rule_execution_results_route.gen';
|
||||
import { GetRuleMigrationRequestParamsInput } from '@kbn/security-solution-plugin/common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { GetRuleMigrationStatsRequestParamsInput } from '@kbn/security-solution-plugin/common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { GetTimelineRequestQueryInput } from '@kbn/security-solution-plugin/common/api/timeline/get_timeline/get_timeline_route.gen';
|
||||
import { GetTimelinesRequestQueryInput } from '@kbn/security-solution-plugin/common/api/timeline/get_timelines/get_timelines_route.gen';
|
||||
import { ImportRulesRequestQueryInput } from '@kbn/security-solution-plugin/common/api/detection_engine/rule_management/import_rules/import_rules_route.gen';
|
||||
|
|
@ -127,7 +129,12 @@ import { SetAlertAssigneesRequestBodyInput } from '@kbn/security-solution-plugin
|
|||
import { SetAlertsStatusRequestBodyInput } from '@kbn/security-solution-plugin/common/api/detection_engine/signals/set_signal_status/set_signals_status_route.gen';
|
||||
import { SetAlertTagsRequestBodyInput } from '@kbn/security-solution-plugin/common/api/detection_engine/alert_tags/set_alert_tags/set_alert_tags.gen';
|
||||
import { StartEntityEngineRequestParamsInput } from '@kbn/security-solution-plugin/common/api/entity_analytics/entity_store/engine/start.gen';
|
||||
import {
|
||||
StartRuleMigrationRequestParamsInput,
|
||||
StartRuleMigrationRequestBodyInput,
|
||||
} from '@kbn/security-solution-plugin/common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { StopEntityEngineRequestParamsInput } from '@kbn/security-solution-plugin/common/api/entity_analytics/entity_store/engine/stop.gen';
|
||||
import { StopRuleMigrationRequestParamsInput } from '@kbn/security-solution-plugin/common/siem_migrations/model/api/rules/rules_migration.gen';
|
||||
import { SuggestUserProfilesRequestQueryInput } from '@kbn/security-solution-plugin/common/api/detection_engine/users/suggest_user_profiles_route.gen';
|
||||
import { TriggerRiskScoreCalculationRequestBodyInput } from '@kbn/security-solution-plugin/common/api/entity_analytics/risk_engine/entity_calculation_route.gen';
|
||||
import { UpdateRuleRequestBodyInput } from '@kbn/security-solution-plugin/common/api/detection_engine/rule_management/crud/update_rule/update_rule_route.gen';
|
||||
|
|
@ -782,6 +789,16 @@ finalize it.
|
|||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
|
||||
.query(props.query);
|
||||
},
|
||||
/**
|
||||
* Retrieves the rule migrations stats for all migrations stored in the system
|
||||
*/
|
||||
getAllStatsRuleMigration(kibanaSpace: string = 'default') {
|
||||
return supertest
|
||||
.get(routeWithNamespace('/internal/siem_migrations/rules/stats', kibanaSpace))
|
||||
.set('kbn-xsrf', 'true')
|
||||
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
|
||||
},
|
||||
/**
|
||||
* Get the asset criticality record for a specific entity.
|
||||
*/
|
||||
|
|
@ -939,11 +956,31 @@ finalize it.
|
|||
.query(props.query);
|
||||
},
|
||||
/**
|
||||
* Retrieves the rule migrations stored in the system
|
||||
* Retrieves the rule documents stored in the system given the rule migration id
|
||||
*/
|
||||
getRuleMigration(kibanaSpace: string = 'default') {
|
||||
getRuleMigration(props: GetRuleMigrationProps, kibanaSpace: string = 'default') {
|
||||
return supertest
|
||||
.get(routeWithNamespace('/internal/siem_migrations/rules', kibanaSpace))
|
||||
.get(
|
||||
routeWithNamespace(
|
||||
replaceParams('/internal/siem_migrations/rules/{migration_id}', props.params),
|
||||
kibanaSpace
|
||||
)
|
||||
)
|
||||
.set('kbn-xsrf', 'true')
|
||||
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
|
||||
},
|
||||
/**
|
||||
* Retrieves the stats of a SIEM rules migration using the migration id provided
|
||||
*/
|
||||
getRuleMigrationStats(props: GetRuleMigrationStatsProps, kibanaSpace: string = 'default') {
|
||||
return supertest
|
||||
.get(
|
||||
routeWithNamespace(
|
||||
replaceParams('/internal/siem_migrations/rules/{migration_id}/stats', props.params),
|
||||
kibanaSpace
|
||||
)
|
||||
)
|
||||
.set('kbn-xsrf', 'true')
|
||||
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
|
||||
|
|
@ -1314,6 +1351,22 @@ detection engine rules.
|
|||
.set(ELASTIC_HTTP_VERSION_HEADER, '2023-10-31')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
|
||||
},
|
||||
/**
|
||||
* Starts a SIEM rules migration using the migration id provided
|
||||
*/
|
||||
startRuleMigration(props: StartRuleMigrationProps, kibanaSpace: string = 'default') {
|
||||
return supertest
|
||||
.put(
|
||||
routeWithNamespace(
|
||||
replaceParams('/internal/siem_migrations/rules/{migration_id}/start', props.params),
|
||||
kibanaSpace
|
||||
)
|
||||
)
|
||||
.set('kbn-xsrf', 'true')
|
||||
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
|
||||
.send(props.body as object);
|
||||
},
|
||||
stopEntityEngine(props: StopEntityEngineProps, kibanaSpace: string = 'default') {
|
||||
return supertest
|
||||
.post(
|
||||
|
|
@ -1326,6 +1379,21 @@ detection engine rules.
|
|||
.set(ELASTIC_HTTP_VERSION_HEADER, '2023-10-31')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
|
||||
},
|
||||
/**
|
||||
* Stops a running SIEM rules migration using the migration id provided
|
||||
*/
|
||||
stopRuleMigration(props: StopRuleMigrationProps, kibanaSpace: string = 'default') {
|
||||
return supertest
|
||||
.put(
|
||||
routeWithNamespace(
|
||||
replaceParams('/internal/siem_migrations/rules/{migration_id}/stop', props.params),
|
||||
kibanaSpace
|
||||
)
|
||||
)
|
||||
.set('kbn-xsrf', 'true')
|
||||
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
|
||||
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
|
||||
},
|
||||
/**
|
||||
* Suggests user profiles.
|
||||
*/
|
||||
|
|
@ -1544,6 +1612,12 @@ export interface GetRuleExecutionResultsProps {
|
|||
query: GetRuleExecutionResultsRequestQueryInput;
|
||||
params: GetRuleExecutionResultsRequestParamsInput;
|
||||
}
|
||||
export interface GetRuleMigrationProps {
|
||||
params: GetRuleMigrationRequestParamsInput;
|
||||
}
|
||||
export interface GetRuleMigrationStatsProps {
|
||||
params: GetRuleMigrationStatsRequestParamsInput;
|
||||
}
|
||||
export interface GetTimelineProps {
|
||||
query: GetTimelineRequestQueryInput;
|
||||
}
|
||||
|
|
@ -1616,9 +1690,16 @@ export interface SetAlertTagsProps {
|
|||
export interface StartEntityEngineProps {
|
||||
params: StartEntityEngineRequestParamsInput;
|
||||
}
|
||||
export interface StartRuleMigrationProps {
|
||||
params: StartRuleMigrationRequestParamsInput;
|
||||
body: StartRuleMigrationRequestBodyInput;
|
||||
}
|
||||
export interface StopEntityEngineProps {
|
||||
params: StopEntityEngineRequestParamsInput;
|
||||
}
|
||||
export interface StopRuleMigrationProps {
|
||||
params: StopRuleMigrationRequestParamsInput;
|
||||
}
|
||||
export interface SuggestUserProfilesProps {
|
||||
query: SuggestUserProfilesRequestQueryInput;
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue