github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 17:28:26 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Detection Engine] Adds Alert Suppression to ML Rules (#181926)

Browse source 
## Summary
This PR introduces Alert Suppression for ML Detection Rules. This
feature is behaviorally similar to alerting suppression for other
Detection Engine Rule types, and nearly identical to the analogous
features for EQL rules.

There are some additional UI behaviors introduced here as well, mainly
intended to cover the shortcomings discovered in
https://github.com/elastic/kibana/issues/183100. Those behaviors are:

1. Populating the suppression field list with fields from the anomaly
index(es).
1. Disabling the suppression UI if no selected ML jobs are running
(because we cannot populate the list of fields on which they'll be
suppressing).
1. Warning the user if _some_ selected ML jobs are not running (because
the list of suppression fields may be incomplete).

See screenshots below for more info.

### Intermediate Serverless Deployment
As per the "intermediate deployment" requirements for serverless, while
the schema (and declared alert SO mappings) will be extended to allow
this functionality, the user-facing features are currently hidden behind
a feature flag. Once this is merged and released, we can issue a "final"
deployment in which the feature flag is enabled, and the feature
effectively released.


## Screenshots
* Overview of new UI fields
<img width="1044" alt="Screenshot 2024-05-16 at 3 22 02 PM"
src="8c07700d-5860-4d1e-a701-eac84fc35558">
* Example of Anomaly fields in suppression combobox
<img width="881" alt="Screenshot 2024-06-06 at 5 14 17 PM"
src="9aa6ed99-1e02-44a0-ad1b-785136510d68">
* Suppression disabled due to no jobs running
<img width="668" alt="Screenshot 2024-06-17 at 11 23 39 PM"
src="a8636a52-31bd-4579-9bcd-d59d93c26984">
* Warning due to not all jobs running
<img width="776" alt="Screenshot 2024-06-17 at 11 26 16 PM"
src="f44c2400-570e-4fde-adce-e5841a2de08d">

## Steps to Review
1. Review the Test Plan for an overview of behavior
2. Review Integration tests for an overview of implementation and edge
cases
3. Review Cypress tests for an overview of UX changes
4. Testing on [Demo
Instance](https://rylnd-pr-181926-ml-rule-alert-suppression.kbndev.co/)
(elastic/changeme)
1. This instance has the relevant feature flag enabled, has some sample
auditbeat data, as well as the [anomalies archive
data](https://github.com/elastic/kibana/tree/main/x-pack/test/functional/es_archives/security_solution/anomalies)
for the purposes of exercising an ML rule against "real" anomalies
    1. There are a few example rules in the default space:
1. A simple [query
rule](f6f5960d-7e4b-40c1-ae15-501112822130)
against auditbeat data
1. An [ML
rule](9122669e-b2e1-41ce-af25-eeae15aa9ece)
with per-execution suppression on both `by_field_name` and
`by_field_value` (which ends up not actually suppressing anything)
1. An [ML
rule](0aabc280-00bd-42d4-82e6-65997c751797)
with per-execution suppression on `by_field_name` (which suppresses all
anomalies into a single alert)

## Related Issues
- This feature was temporarily blocked by
https://github.com/elastic/kibana/issues/183100, but those changes are
now in this PR.

## Checklist
- [x] Functional changes are hidden behind a feature flag. If not
hidden, the PR explains why these changes are being implemented in a
long-living feature branch.
- [x] Functional changes are covered with a test plan and automated
tests.
    * [Test Plan](https://github.com/elastic/security-team/pull/9279)
- [x] Stability of new and changed tests is verified using the [Flaky
Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner) in
both ESS and Serverless. By default, use 200 runs for ESS and 200 runs
for Serverless.
* [ESS - Cypress x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6449)
* [Serverless - Cypress x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6450)
* [ESS - API x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6447)
* [Serverless - API x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6448)
- [ ] Comprehensive manual testing is done by two engineers: the PR
author and one of the PR reviewers. Changes are tested in both ESS and
Serverless.
- [ ] Mapping changes are accompanied by a technical design document. It
can be a GitHub issue or an RFC explaining the changes. The design
document is shared with and approved by the appropriate teams and
individual stakeholders.
- [ ] (OPTIONAL) OpenAPI specs changes include detailed descriptions and
examples of usage and are ready to be released on
https://docs.elastic.co/api-reference. NOTE: This is optional because at
the moment we don't have yet any OpenAPI specs that would be fully
"documented" and "GA-ready" for publishing on
https://docs.elastic.co/api-reference.
- [ ] Functional changes are communicated to the Docs team. A ticket is
opened in https://github.com/elastic/security-docs using the [Internal
documentation request (Elastic
employees)](https://github.com/elastic/security-docs/issues/new?assignees=&labels=&projects=&template=docs-request-internal.yaml&title=%5BRequest%5D+)
template. The following information is included: feature flags used,
target ESS version, planned timing for ESS and Serverless releases.

---------

Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Ryland Herrick 2024-07-02 14:33:11 -05:00 • committed by GitHub
parent e654e46466
commit 2aa94a27f0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
51 changed files with 2504 additions and 223 deletions
Download patch file Download diff file Expand all files Collapse all files

21
packages/kbn-es/src/serverless_resources/project_roles/security/roles.yml
View file

@ -35,6 +35,7 @@ viewer:
- '.fleet-actions*'
- 'risk-score.risk-score-*'
- '.asset-criticality.asset-criticality-*'
- '.ml-anomalies-*'
privileges:
- read
applications:
@ -100,6 +101,10 @@ editor:
- 'read'
- 'write'
allow_restricted_indices: false
- names:
- '.ml-anomalies-*'
privileges:
- read
applications:
- application: 'kibana-.kibana'
privileges:
@ -154,6 +159,7 @@ t1_analyst:
- '.fleet-actions*'
- risk-score.risk-score-*
- .asset-criticality.asset-criticality-*
- '.ml-anomalies-*'
privileges:
- read
applications:
@ -201,6 +207,7 @@ t2_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- '.ml-anomalies-*'
privileges:
- read
- names:
@ -262,6 +269,7 @@ t3_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- '.ml-anomalies-*'
privileges:
- read
applications:
@ -331,6 +339,7 @@ threat_intelligence_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- '.ml-anomalies-*'
privileges:
- read
applications:
@ -389,6 +398,7 @@ rule_author:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- '.ml-anomalies-*'
privileges:
- read
applications:
@ -453,6 +463,7 @@ soc_manager:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- '.ml-anomalies-*'
privileges:
- read
applications:
@ -513,6 +524,7 @@ detections_admin:
- metrics-endpoint.metadata_current_*
- .fleet-agents*
- .fleet-actions*
- '.ml-anomalies-*'
privileges:
- read
- names:
@ -570,6 +582,10 @@ platform_engineer:
privileges:
- read
- write
- names:
- '.ml-anomalies-*'
privileges:
- read
applications:
- application: 'kibana-.kibana'
privileges:
@ -620,6 +636,7 @@ endpoint_operations_analyst:
- .lists*
- .items*
- risk-score.risk-score-*
- '.ml-anomalies-*'
privileges:
- read
- names:
@ -710,6 +727,10 @@ endpoint_policy_manager:
- read
- write
- manage
- names:
- '.ml-anomalies-*'
privileges:
- read
applications:
- application: 'kibana-.kibana'
privileges:

46
x-pack/plugins/alerting/server/integration_tests/__snapshots__/serverless_upgrade_and_rollback_checks.test.ts.snap generated
View file

@ -7127,6 +7127,52 @@ Object {
},
Object {
"properties": Object {
"alertSuppression": Object {
"additionalProperties": false,
"properties": Object {
"duration": Object {
"additionalProperties": false,
"properties": Object {
"unit": Object {
"enum": Array [
"s",
"m",
"h",
],
"type": "string",
},
"value": Object {
"minimum": 1,
"type": "integer",
},
},
"required": Array [
"value",
"unit",
],
"type": "object",
},
"groupBy": Object {
"items": Object {
"type": "string",
},
"maxItems": 3,
"minItems": 1,
"type": "array",
},
"missingFieldsStrategy": Object {
"enum": Array [
"doNotSuppress",
"suppress",
],
"type": "string",
},
},
"required": Array [
"groupBy",
],
"type": "object",
},
"anomalyThreshold": Object {
"minimum": 0,
"type": "integer",

1
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_request_schema.test.ts
View file

@ -1272,6 +1272,7 @@ describe('rules schema', () => {
{ ruleType: 'saved_query', ruleMock: getCreateSavedQueryRulesSchemaMock() },
{ ruleType: 'eql', ruleMock: getCreateEqlRuleSchemaMock() },
{ ruleType: 'new_terms', ruleMock: getCreateNewTermsRulesSchemaMock() },
{ ruleType: 'machine_learning', ruleMock: getCreateMachineLearningRulesSchemaMock() },
];
cases.forEach(({ ruleType, ruleMock }) => {

17
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts
View file

@ -468,14 +468,25 @@ export const MachineLearningRuleRequiredFields = z.object({
machine_learning_job_id: MachineLearningJobId,
});
export type MachineLearningRuleOptionalFields = z.infer<typeof MachineLearningRuleOptionalFields>;
export const MachineLearningRuleOptionalFields = z.object({
alert_suppression: AlertSuppression.optional(),
});
export type MachineLearningRulePatchFields = z.infer<typeof MachineLearningRulePatchFields>;
export const MachineLearningRulePatchFields = MachineLearningRuleRequiredFields.partial();
export const MachineLearningRulePatchFields = MachineLearningRuleRequiredFields.partial().merge(
MachineLearningRuleOptionalFields
);
export type MachineLearningRuleResponseFields = z.infer<typeof MachineLearningRuleResponseFields>;
export const MachineLearningRuleResponseFields = MachineLearningRuleRequiredFields;
export const MachineLearningRuleResponseFields = MachineLearningRuleRequiredFields.merge(
MachineLearningRuleOptionalFields
);
export type MachineLearningRuleCreateFields = z.infer<typeof MachineLearningRuleCreateFields>;
export const MachineLearningRuleCreateFields = MachineLearningRuleRequiredFields;
export const MachineLearningRuleCreateFields = MachineLearningRuleRequiredFields.merge(
MachineLearningRuleOptionalFields
);
export type MachineLearningRule = z.infer<typeof MachineLearningRule>;
export const MachineLearningRule = SharedResponseProps.merge(MachineLearningRuleResponseFields);

9
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.schema.yaml
View file

@ -686,18 +686,27 @@ components:
- machine_learning_job_id
- anomaly_threshold
MachineLearningRuleOptionalFields:
type: object
properties:
alert_suppression:
$ref: './common_attributes.schema.yaml#/components/schemas/AlertSuppression'
MachineLearningRulePatchFields:
allOf:
- $ref: '#/components/schemas/MachineLearningRuleRequiredFields'
x-modify: partial
- $ref: '#/components/schemas/MachineLearningRuleOptionalFields'
MachineLearningRuleResponseFields:
allOf:
- $ref: '#/components/schemas/MachineLearningRuleRequiredFields'
- $ref: '#/components/schemas/MachineLearningRuleOptionalFields'
MachineLearningRuleCreateFields:
allOf:
- $ref: '#/components/schemas/MachineLearningRuleRequiredFields'
- $ref: '#/components/schemas/MachineLearningRuleOptionalFields'
MachineLearningRule:
allOf:

1
x-pack/plugins/security_solution/common/detection_engine/constants.ts
View file

@ -47,6 +47,7 @@ export const SUPPRESSIBLE_ALERT_RULES: Type[] = [
'new_terms',
'threat_match',
'eql',
'machine_learning',
];
export const SUPPRESSIBLE_ALERT_RULES_GA: Type[] = ['saved_query', 'query'];

16
x-pack/plugins/security_solution/common/detection_engine/utils.test.ts
View file

@ -236,9 +236,7 @@ describe('Alert Suppression Rules', () => {
expect(isSuppressibleAlertRule('threat_match')).toBe(true);
expect(isSuppressibleAlertRule('new_terms')).toBe(true);
expect(isSuppressibleAlertRule('eql')).toBe(true);
// Rule types that don't support alert suppression:
expect(isSuppressibleAlertRule('machine_learning')).toBe(false);
expect(isSuppressibleAlertRule('machine_learning')).toBe(true);
});
test('should return false for an unknown rule type', () => {
@ -273,9 +271,7 @@ describe('Alert Suppression Rules', () => {
expect(isSuppressionRuleConfiguredWithDuration('threat_match')).toBe(true);
expect(isSuppressionRuleConfiguredWithDuration('new_terms')).toBe(true);
expect(isSuppressionRuleConfiguredWithDuration('eql')).toBe(true);
// Rule types that don't support alert suppression:
expect(isSuppressionRuleConfiguredWithDuration('machine_learning')).toBe(false);
expect(isSuppressionRuleConfiguredWithDuration('machine_learning')).toBe(true);
});
test('should return false for an unknown rule type', () => {
@ -294,9 +290,7 @@ describe('Alert Suppression Rules', () => {
expect(isSuppressionRuleConfiguredWithGroupBy('threat_match')).toBe(true);
expect(isSuppressionRuleConfiguredWithGroupBy('new_terms')).toBe(true);
expect(isSuppressionRuleConfiguredWithGroupBy('eql')).toBe(true);
// Rule types that don't support alert suppression:
expect(isSuppressionRuleConfiguredWithGroupBy('machine_learning')).toBe(false);
expect(isSuppressionRuleConfiguredWithGroupBy('machine_learning')).toBe(true);
});
test('should return false for a threshold rule type', () => {
@ -320,9 +314,7 @@ describe('Alert Suppression Rules', () => {
expect(isSuppressionRuleConfiguredWithMissingFields('threat_match')).toBe(true);
expect(isSuppressionRuleConfiguredWithMissingFields('new_terms')).toBe(true);
expect(isSuppressionRuleConfiguredWithMissingFields('eql')).toBe(true);
// Rule types that don't support alert suppression:
expect(isSuppressionRuleConfiguredWithMissingFields('machine_learning')).toBe(false);
expect(isSuppressionRuleConfiguredWithMissingFields('machine_learning')).toBe(true);
});
test('should return false for a threshold rule type', () => {

5
x-pack/plugins/security_solution/common/experimental_features.ts
View file

@ -175,6 +175,11 @@ export const allowedExperimentalValues = Object.freeze({
*/
riskEnginePrivilegesRouteEnabled: true,
/**
* Enables alerts suppression for machine learning rules
*/
alertSuppressionForMachineLearningRuleEnabled: false,
/**
* Enables experimental Experimental S1 integration data to be available in Analyzer
*/

62
x-pack/plugins/security_solution/public/common/components/ml/hooks/use_ml_rule_config.ts Normal file
View file

@ -0,0 +1,62 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { useMemo } from 'react';
import type { DataViewFieldBase } from '@kbn/es-query';
import { getTermsAggregationFields } from '../../../../detection_engine/rule_creation_ui/components/step_define_rule/utils';
import { useRuleFields } from '../../../../detection_engine/rule_management/logic/use_rule_fields';
import type { BrowserField } from '../../../containers/source';
import { useMlCapabilities } from './use_ml_capabilities';
import { useMlRuleValidations } from './use_ml_rule_validations';
import { hasMlAdminPermissions } from '../../../../../common/machine_learning/has_ml_admin_permissions';
import { hasMlLicense } from '../../../../../common/machine_learning/has_ml_license';
export interface UseMlRuleConfigReturn {
hasMlAdminPermissions: boolean;
hasMlLicense: boolean;
mlFields: DataViewFieldBase[];
mlFieldsLoading: boolean;
mlSuppressionFields: BrowserField[];
noMlJobsStarted: boolean;
someMlJobsStarted: boolean;
}
/**
* This hook is used to retrieve the various configurations and status needed for creating/editing an ML Rule in the Detection Engine UI. It composes several other ML hooks.
*
* @param machineLearningJobId The ID(s) of the ML job to retrieve the configuration for
*
* @returns {UseMlRuleConfigReturn} An object containing the various configurations and statuses needed for creating/editing an ML Rule
*
*/
export const useMLRuleConfig = ({
machineLearningJobId,
}: {
machineLearningJobId: string[];
}): UseMlRuleConfigReturn => {
const mlCapabilities = useMlCapabilities();
const { someJobsStarted: someMlJobsStarted, noJobsStarted: noMlJobsStarted } =
useMlRuleValidations({ machineLearningJobId });
const { loading: mlFieldsLoading, fields: mlFields } = useRuleFields({
machineLearningJobId,
});
const mlSuppressionFields = useMemo(
() => getTermsAggregationFields(mlFields as BrowserField[]),
[mlFields]
);
return {
hasMlAdminPermissions: hasMlAdminPermissions(mlCapabilities),
hasMlLicense: hasMlLicense(mlCapabilities),
mlFields,
mlFieldsLoading,
mlSuppressionFields,
noMlJobsStarted,
someMlJobsStarted,
};
};

102
x-pack/plugins/security_solution/public/common/components/ml/hooks/use_ml_rule_validations.test.ts Normal file
View file

@ -0,0 +1,102 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { renderHook } from '@testing-library/react-hooks';
import { TestProviders } from '../../../mock';
import { buildMockJobsSummary, getJobsSummaryResponseMock } from '../../ml_popover/api.mock';
import { useInstalledSecurityJobs } from './use_installed_security_jobs';
import { useMlRuleValidations } from './use_ml_rule_validations';
jest.mock('./use_installed_security_jobs');
describe('useMlRuleValidations', () => {
const machineLearningJobId = ['test_job', 'test_job_2'];
beforeEach(() => {
(useInstalledSecurityJobs as jest.Mock).mockReturnValue({
loading: true,
jobs: [],
});
});
it('returns loading state from inner hook', () => {
const { result, rerender } = renderHook(() => useMlRuleValidations({ machineLearningJobId }), {
wrapper: TestProviders,
});
expect(result.current).toEqual(expect.objectContaining({ loading: true }));
(useInstalledSecurityJobs as jest.Mock).mockReturnValueOnce({
loading: false,
jobs: [],
});
rerender();
expect(result.current).toEqual(expect.objectContaining({ loading: false }));
});
it('returns no jobs started when no jobs are started', () => {
const { result } = renderHook(() => useMlRuleValidations({ machineLearningJobId }), {
wrapper: TestProviders,
});
expect(result.current).toEqual(
expect.objectContaining({ noJobsStarted: true, someJobsStarted: false })
);
});
it('returns some jobs started when some jobs are started', () => {
(useInstalledSecurityJobs as jest.Mock).mockReturnValueOnce({
loading: false,
jobs: getJobsSummaryResponseMock([
buildMockJobsSummary({
id: machineLearningJobId[0],
jobState: 'opened',
datafeedState: 'started',
}),
buildMockJobsSummary({
id: machineLearningJobId[1],
}),
]),
});
const { result } = renderHook(() => useMlRuleValidations({ machineLearningJobId }), {
wrapper: TestProviders,
});
expect(result.current).toEqual(
expect.objectContaining({ noJobsStarted: false, someJobsStarted: true })
);
});
it('returns neither "no jobs started" nor "some jobs started" when all jobs are started', () => {
(useInstalledSecurityJobs as jest.Mock).mockReturnValueOnce({
loading: false,
jobs: getJobsSummaryResponseMock([
buildMockJobsSummary({
id: machineLearningJobId[0],
jobState: 'opened',
datafeedState: 'started',
}),
buildMockJobsSummary({
id: machineLearningJobId[1],
jobState: 'opened',
datafeedState: 'started',
}),
]),
});
const { result } = renderHook(() => useMlRuleValidations({ machineLearningJobId }), {
wrapper: TestProviders,
});
expect(result.current).toEqual(
expect.objectContaining({ noJobsStarted: false, someJobsStarted: false })
);
});
});

41
x-pack/plugins/security_solution/public/common/components/ml/hooks/use_ml_rule_validations.ts Normal file
View file

@ -0,0 +1,41 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { isJobStarted } from '../../../../../common/machine_learning/helpers';
import { useInstalledSecurityJobs } from './use_installed_security_jobs';
export interface UseMlRuleValidationsParams {
machineLearningJobId: string[] | undefined;
}
export interface UseMlRuleValidationsReturn {
loading: boolean;
noJobsStarted: boolean;
someJobsStarted: boolean;
}
/**
* Hook to encapsulate some of our validation checks for ML rules.
*
* @param machineLearningJobId the ML Job IDs of the rule
* @returns validation state about the rule, relative to its ML jobs.
*/
export const useMlRuleValidations = ({
machineLearningJobId,
}: UseMlRuleValidationsParams): UseMlRuleValidationsReturn => {
const { jobs: installedJobs, loading } = useInstalledSecurityJobs();
const ruleMlJobs = installedJobs.filter((installedJob) =>
(machineLearningJobId ?? []).includes(installedJob.id)
);
const numberOfRuleMlJobsStarted = ruleMlJobs.filter((job) =>
isJobStarted(job.jobState, job.datafeedState)
).length;
const noMlJobsStarted = numberOfRuleMlJobsStarted === 0;
const someMlJobsStarted = !noMlJobsStarted && numberOfRuleMlJobsStarted !== ruleMlJobs.length;
return { loading, noJobsStarted: noMlJobsStarted, someJobsStarted: someMlJobsStarted };
};

10
x-pack/plugins/security_solution/public/common/components/ml_popover/api.mock.ts
View file

@ -100,6 +100,16 @@ export const mockJobsSummaryResponse: MlSummaryJob[] = [
},
];
export const getJobsSummaryResponseMock = (additionalJobs: MlSummaryJob[]): MlSummaryJob[] => [
...mockJobsSummaryResponse,
...additionalJobs,
];
export const buildMockJobsSummary = (overrides: Partial<MlSummaryJob>): MlSummaryJob => ({
...mockJobsSummaryResponse[0],
...overrides,
});
export const mockGetModuleResponse: Module[] = [
{
id: 'security_linux_v3',

13
x-pack/plugins/security_solution/public/common/components/ml_popover/hooks/use_security_jobs_helpers.tsx
View file

@ -6,6 +6,7 @@
*/
import type { MlSummaryJob } from '@kbn/ml-plugin/public';
import { isSecurityJob } from '../../../../../common/machine_learning/is_security_job';
import type {
AugmentedSecurityJobFields,
Module,
@ -111,13 +112,11 @@ export const getInstalledJobs = (
moduleJobs: SecurityJob[],
compatibleModuleIds: string[]
): SecurityJob[] =>
jobSummaryData
.filter(({ groups }) => groups.includes('siem') || groups.includes('security'))
.map<SecurityJob>((jobSummary) => ({
...jobSummary,
...getAugmentedFields(jobSummary.id, moduleJobs, compatibleModuleIds),
isInstalled: true,
}));
jobSummaryData.filter(isSecurityJob).map((jobSummary) => ({
...jobSummary,
...getAugmentedFields(jobSummary.id, moduleJobs, compatibleModuleIds),
isInstalled: true,
}));
/**
* Combines installed jobs + moduleSecurityJobs that don't overlap and sorts by name asc

73
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/description_step/index.test.tsx
View file

@ -14,7 +14,6 @@ import {
buildListItems,
getDescriptionItem,
} from '.';
import type { Type } from '@kbn/securitysolution-io-ts-alerting-types';
import { FilterManager, UI_SETTINGS } from '@kbn/data-plugin/public';
import type { Filter } from '@kbn/es-query';
@ -575,7 +574,6 @@ describe('description_step', () => {
});
describe('alert suppression', () => {
const ruleTypesWithoutSuppression: Type[] = ['machine_learning'];
const suppressionFields = {
groupByDuration: {
unit: 'm',
@ -587,23 +585,6 @@ describe('description_step', () => {
suppressionMissingFields: 'suppress',
};
describe('groupByDuration', () => {
ruleTypesWithoutSuppression.forEach((ruleType) => {
test(`should be empty if rule is ${ruleType}`, () => {
const result: ListItems[] = getDescriptionItem(
'groupByDuration',
'label',
{
ruleType,
...suppressionFields,
},
mockFilterManager,
mockLicenseService
);
expect(result).toEqual([]);
});
});
['query', 'saved_query'].forEach((ruleType) => {
test(`should be empty if groupByFields empty for ${ruleType} rule`, () => {
const result: ListItems[] = getDescriptionItem(
@ -686,22 +667,21 @@ describe('description_step', () => {
});
describe('groupByFields', () => {
[...ruleTypesWithoutSuppression, 'threshold'].forEach((ruleType) => {
test(`should be empty if rule is ${ruleType}`, () => {
const result: ListItems[] = getDescriptionItem(
'groupByFields',
'label',
{
ruleType,
...suppressionFields,
},
mockFilterManager,
mockLicenseService
);
test(`should be empty if rule type is 'threshold'`, () => {
const result: ListItems[] = getDescriptionItem(
'groupByFields',
'label',
{
ruleType: 'threshold',
...suppressionFields,
},
mockFilterManager,
mockLicenseService
);
expect(result).toEqual([]);
});
expect(result).toEqual([]);
});
['query', 'saved_query'].forEach((ruleType) => {
test(`should return item for ${ruleType} rule`, () => {
const result: ListItems[] = getDescriptionItem(
@ -720,22 +700,21 @@ describe('description_step', () => {
});
describe('suppressionMissingFields', () => {
[...ruleTypesWithoutSuppression, 'threshold'].forEach((ruleType) => {
test(`should be empty if rule is ${ruleType}`, () => {
const result: ListItems[] = getDescriptionItem(
'suppressionMissingFields',
'label',
{
ruleType,
...suppressionFields,
},
mockFilterManager,
mockLicenseService
);
test(`should be empty if rule type is 'threshold'`, () => {
const result: ListItems[] = getDescriptionItem(
'suppressionMissingFields',
'label',
{
ruleType: 'threshold',
...suppressionFields,
},
mockFilterManager,
mockLicenseService
);
expect(result).toEqual([]);
});
expect(result).toEqual([]);
});
['query', 'saved_query'].forEach((ruleType) => {
test(`should return item for ${ruleType} rule`, () => {
const result: ListItems[] = getDescriptionItem(

108
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/index.tsx
View file

@ -36,9 +36,6 @@ import { FormattedMessage } from '@kbn/i18n-react';
import { useSetFieldValueWithCallback } from '../../../../common/utils/use_set_field_value_cb';
import { useRuleFromTimeline } from '../../../../detections/containers/detection_engine/rules/use_rule_from_timeline';
import { isMlRule } from '../../../../../common/machine_learning/helpers';
import { hasMlAdminPermissions } from '../../../../../common/machine_learning/has_ml_admin_permissions';
import { hasMlLicense } from '../../../../../common/machine_learning/has_ml_license';
import { useMlCapabilities } from '../../../../common/components/ml/hooks/use_ml_capabilities';
import type { EqlOptionsSelected, FieldsEqlOptions } from '../../../../../common/search_strategy';
import { filterRuleFieldsForType, getStepDataDataSource } from '../../pages/rule_creation/helpers';
import type {
@ -105,6 +102,7 @@ import { useAllEsqlRuleFields } from '../../hooks';
import { useAlertSuppression } from '../../../rule_management/logic/use_alert_suppression';
import { AiAssistant } from '../ai_assistant';
import { RelatedIntegrations } from '../../../rule_creation/components/related_integrations';
import { useMLRuleConfig } from '../../../../common/components/ml/hooks/use_ml_rule_config';
const CommonUseField = getUseField({ component: Field });
@ -169,41 +167,53 @@ const IntendedRuleTypeEuiFormRow = styled(RuleTypeEuiFormRow)`
// eslint-disable-next-line complexity
const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
browserFields,
dataSourceType,
defaultSavedQuery,
enableThresholdSuppression,
form,
groupByFields,
index,
indexPattern,
indicesConfig,
isIndexPatternLoading,
isLoading,
isQueryBarValid,
isUpdateView = false,
kibanaDataViews,
indicesConfig,
threatIndicesConfig,
defaultSavedQuery,
form,
optionsSelected,
setOptionsSelected,
indexPattern,
isIndexPatternLoading,
browserFields,
isQueryBarValid,
queryBarSavedId,
queryBarTitle,
ruleType,
setIsQueryBarValid,
setIsThreatQueryBarValid,
ruleType,
index,
threatIndex,
groupByFields,
dataSourceType,
setOptionsSelected,
shouldLoadQueryDynamically,
queryBarTitle,
queryBarSavedId,
threatIndex,
threatIndicesConfig,
thresholdFields,
enableThresholdSuppression,
}) => {
const queryClient = useQueryClient();
const { isSuppressionEnabled: isAlertSuppressionEnabled } = useAlertSuppression(ruleType);
const mlCapabilities = useMlCapabilities();
const [openTimelineSearch, setOpenTimelineSearch] = useState(false);
const [indexModified, setIndexModified] = useState(false);
const [threatIndexModified, setThreatIndexModified] = useState(false);
const license = useLicense();
const [{ machineLearningJobId }] = useFormData<DefineStepRule>({
form,
watch: ['machineLearningJobId'],
});
const {
hasMlAdminPermissions,
hasMlLicense,
mlFieldsLoading,
mlSuppressionFields,
noMlJobsStarted,
someMlJobsStarted,
} = useMLRuleConfig({ machineLearningJobId });
const esqlQueryRef = useRef<DefineStepRule['queryBar'] | undefined>(undefined);
const isAlertSuppressionLicenseValid = license.isAtLeast(MINIMUM_LICENSE_FOR_SUPPRESSION);
@ -474,6 +484,24 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
isEqlSequenceQuery(queryBar?.query?.query as string) &&
groupByFields.length === 0;
const isSuppressionGroupByDisabled =
!isAlertSuppressionLicenseValid ||
areSuppressionFieldsDisabledBySequence ||
isEsqlSuppressionLoading ||
(isMlRule(ruleType) && (noMlJobsStarted || mlFieldsLoading || !mlSuppressionFields.length));
const suppressionGroupByDisabledText = areSuppressionFieldsDisabledBySequence
? i18n.EQL_SEQUENCE_SUPPRESSION_DISABLE_TOOLTIP
: isMlRule(ruleType) && noMlJobsStarted
? i18n.MACHINE_LEARNING_SUPPRESSION_DISABLED_LABEL
: alertSuppressionUpsellingMessage;
const suppressionGroupByFields = isEsqlRule(ruleType)
? esqlSuppressionFields
: isMlRule(ruleType)
? mlSuppressionFields
: termsAggregationFields;
/**
* Component that allows selection of suppression intervals disabled:
* - if suppression license is not valid(i.e. less than platinum)
@ -868,10 +896,10 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
() => ({
describedByIds: ['detectionEngineStepDefineRuleType'],
isUpdateView,
hasValidLicense: hasMlLicense(mlCapabilities),
isMlAdmin: hasMlAdminPermissions(mlCapabilities),
hasValidLicense: hasMlLicense,
isMlAdmin: hasMlAdminPermissions,
}),
[isUpdateView, mlCapabilities]
[hasMlAdminPermissions, hasMlLicense, isUpdateView]
);
return (
@ -1078,22 +1106,22 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
</EuiText>
}
>
<UseField
path="groupByFields"
component={MultiSelectFieldsAutocomplete}
componentProps={{
browserFields: isEsqlRule(ruleType)
? esqlSuppressionFields
: termsAggregationFields,
isDisabled:
!isAlertSuppressionLicenseValid ||
areSuppressionFieldsDisabledBySequence ||
isEsqlSuppressionLoading,
disabledText: areSuppressionFieldsDisabledBySequence
? i18n.EQL_SEQUENCE_SUPPRESSION_DISABLE_TOOLTIP
: alertSuppressionUpsellingMessage,
}}
/>
<>
<UseField
path="groupByFields"
component={MultiSelectFieldsAutocomplete}
componentProps={{
browserFields: suppressionGroupByFields,
isDisabled: isSuppressionGroupByDisabled,
disabledText: suppressionGroupByDisabledText,
}}
/>
{someMlJobsStarted && (
<EuiText size="xs" color="warning">
{i18n.MACHINE_LEARNING_SUPPRESSION_INCOMPLETE_LABEL}
</EuiText>
)}
</>
</RuleTypeEuiFormRow>
<IntendedRuleTypeEuiFormRow

15
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/translations.tsx
View file

@ -234,6 +234,21 @@ export const EQL_SEQUENCE_SUPPRESSION_GROUPBY_VALIDATION_TEXT = i18n.translate(
}
);
export const MACHINE_LEARNING_SUPPRESSION_DISABLED_LABEL = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.machineLearningSuppressionDisabledLabel',
{
defaultMessage: 'To enable alert suppression, start relevant Machine Learning jobs.',
}
);
export const MACHINE_LEARNING_SUPPRESSION_INCOMPLETE_LABEL = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.machineLearningSuppressionIncompleteLabel',
{
defaultMessage:
'This list of fields might be incomplete as some Machine Learning jobs are not running. Start all relevant jobs for a complete list.',
}
);
export const GROUP_BY_TECH_PREVIEW_LABEL_APPEND = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.groupByFieldsTechPreviewLabelAppend',
{

10
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/use_experimental_feature_fields_transform.ts
View file

@ -8,7 +8,7 @@
import { useCallback } from 'react';
import type { DefineStepRule } from '../../../../detections/pages/detection_engine/rules/types';
import { useIsExperimentalFeatureEnabled } from '../../../../common/hooks/use_experimental_features';
import { isEsqlRule } from '../../../../../common/detection_engine/utils';
import { isEsqlRule, isMlRule } from '../../../../../common/detection_engine/utils';
/**
* transforms DefineStepRule fields according to experimental feature flags
@ -16,6 +16,9 @@ import { isEsqlRule } from '../../../../../common/detection_engine/utils';
export const useExperimentalFeatureFieldsTransform = <T extends Partial<DefineStepRule>>(): ((
fields: T
) => T) => {
const isAlertSuppressionForMachineLearningRuleEnabled = useIsExperimentalFeatureEnabled(
'alertSuppressionForMachineLearningRuleEnabled'
);
const isAlertSuppressionForEsqlRuleEnabled = useIsExperimentalFeatureEnabled(
'alertSuppressionForEsqlRuleEnabled'
);
@ -23,7 +26,8 @@ export const useExperimentalFeatureFieldsTransform = <T extends Partial<DefineSt
const transformer = useCallback(
(fields: T) => {
const isSuppressionDisabled =
isEsqlRule(fields.ruleType) && !isAlertSuppressionForEsqlRuleEnabled;
(isMlRule(fields.ruleType) && !isAlertSuppressionForMachineLearningRuleEnabled) ||
(isEsqlRule(fields.ruleType) && !isAlertSuppressionForEsqlRuleEnabled);
// reset any alert suppression values hidden behind feature flag
if (isSuppressionDisabled) {
@ -38,7 +42,7 @@ export const useExperimentalFeatureFieldsTransform = <T extends Partial<DefineSt
return fields;
},
[isAlertSuppressionForEsqlRuleEnabled]
[isAlertSuppressionForEsqlRuleEnabled, isAlertSuppressionForMachineLearningRuleEnabled]
);
return transformer;

26
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/pages/rule_creation/helpers.test.ts
View file

@ -587,6 +587,32 @@ describe('helpers', () => {
expect(result).toEqual(expected);
});
it('returns suppression fields for machine_learning rules', () => {
const mockStepData: DefineStepRule = {
...mockData,
ruleType: 'machine_learning',
machineLearningJobId: ['some_jobert_id'],
anomalyThreshold: 44,
groupByFields: ['event.type'],
groupByRadioSelection: GroupByOptions.PerTimePeriod,
groupByDuration: { value: 10, unit: 'm' },
};
const result = formatDefineStepData(mockStepData);
const expected: DefineStepRuleJson = {
machine_learning_job_id: ['some_jobert_id'],
anomaly_threshold: 44,
type: 'machine_learning',
alert_suppression: {
group_by: ['event.type'],
duration: { value: 10, unit: 'm' },
missing_fields_strategy: 'suppress',
},
};
expect(result).toEqual(expect.objectContaining(expected));
});
});
describe('formatScheduleStepData', () => {

1
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/pages/rule_creation/helpers.ts
View file

@ -439,6 +439,7 @@ export const formatDefineStepData = (defineStepData: DefineStepRule): DefineStep
? {
anomaly_threshold: ruleFields.anomalyThreshold,
machine_learning_job_id: ruleFields.machineLearningJobId,
...alertSuppressionFields,
}
: isThresholdFields(ruleFields)
? {

38
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_alert_suppression.test.tsx
View file

@ -37,18 +37,38 @@ describe('useAlertSuppression', () => {
expect(result.current.isSuppressionEnabled).toBe(false);
});
it('should return isSuppressionEnabled false if ES|QL Feature Flag is disabled', () => {
const { result } = renderHook(() => useAlertSuppression('esql'));
describe('ML rules', () => {
it('is true if the feature flag is enabled', () => {
jest
.spyOn(useIsExperimentalFeatureEnabledMock, 'useIsExperimentalFeatureEnabled')
.mockReset()
.mockReturnValue(true);
const { result } = renderHook(() => useAlertSuppression('machine_learning'));
expect(result.current.isSuppressionEnabled).toBe(false);
expect(result.current.isSuppressionEnabled).toBe(true);
});
it('is false if the feature flag is disabled', () => {
const { result } = renderHook(() => useAlertSuppression('machine_learning'));
expect(result.current.isSuppressionEnabled).toBe(false);
});
});
it('should return isSuppressionEnabled true if ES|QL Feature Flag is enabled', () => {
jest
.spyOn(useIsExperimentalFeatureEnabledMock, 'useIsExperimentalFeatureEnabled')
.mockImplementation((flag) => flag === 'alertSuppressionForEsqlRuleEnabled');
const { result } = renderHook(() => useAlertSuppression('esql'));
describe('ES|QL rules', () => {
it('should return isSuppressionEnabled false if ES|QL Feature Flag is disabled', () => {
const { result } = renderHook(() => useAlertSuppression('esql'));
expect(result.current.isSuppressionEnabled).toBe(true);
expect(result.current.isSuppressionEnabled).toBe(false);
});
it('should return isSuppressionEnabled true if ES|QL Feature Flag is enabled', () => {
jest
.spyOn(useIsExperimentalFeatureEnabledMock, 'useIsExperimentalFeatureEnabled')
.mockImplementation((flag) => flag === 'alertSuppressionForEsqlRuleEnabled');
const { result } = renderHook(() => useAlertSuppression('esql'));
expect(result.current.isSuppressionEnabled).toBe(true);
});
});
});

15
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_alert_suppression.tsx
View file

@ -6,7 +6,7 @@
*/
import { useCallback } from 'react';
import type { Type } from '@kbn/securitysolution-io-ts-alerting-types';
import { isSuppressibleAlertRule } from '../../../../common/detection_engine/utils';
import { isMlRule, isSuppressibleAlertRule } from '../../../../common/detection_engine/utils';
import { useIsExperimentalFeatureEnabled } from '../../../common/hooks/use_experimental_features';
export interface UseAlertSuppressionReturn {
@ -14,6 +14,9 @@ export interface UseAlertSuppressionReturn {
}
export const useAlertSuppression = (ruleType: Type | undefined): UseAlertSuppressionReturn => {
const isAlertSuppressionForMachineLearningRuleEnabled = useIsExperimentalFeatureEnabled(
'alertSuppressionForMachineLearningRuleEnabled'
);
const isAlertSuppressionForEsqlRuleEnabled = useIsExperimentalFeatureEnabled(
'alertSuppressionForEsqlRuleEnabled'
);
@ -27,8 +30,16 @@ export const useAlertSuppression = (ruleType: Type | undefined): UseAlertSuppres
return isSuppressibleAlertRule(ruleType) && isAlertSuppressionForEsqlRuleEnabled;
}
if (isMlRule(ruleType)) {
return isSuppressibleAlertRule(ruleType) && isAlertSuppressionForMachineLearningRuleEnabled;
}
return isSuppressibleAlertRule(ruleType);
}, [ruleType, isAlertSuppressionForEsqlRuleEnabled]);
}, [
isAlertSuppressionForEsqlRuleEnabled,
isAlertSuppressionForMachineLearningRuleEnabled,
ruleType,
]);
return {
isSuppressionEnabled: isSuppressionEnabledForRuleType(),

36
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_rule_fields.ts Normal file
View file

@ -0,0 +1,36 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { DataViewFieldBase } from '@kbn/es-query';
import { useRuleIndices } from './use_rule_indices';
import { useFetchIndex } from '../../../common/containers/source';
interface UseRuleFieldParams {
machineLearningJobId?: string[];
indexPattern?: string[];
}
interface UseRuleFieldsReturn {
loading: boolean;
fields: DataViewFieldBase[];
}
export const useRuleFields = ({
machineLearningJobId,
indexPattern,
}: UseRuleFieldParams): UseRuleFieldsReturn => {
const { ruleIndices } = useRuleIndices(machineLearningJobId, indexPattern);
const [
loading,
{
indexPatterns: { fields },
},
] = useFetchIndex(ruleIndices);
return { loading, fields };
};

52
x-pack/plugins/security_solution/public/detections/components/alerts_table/actions.tsx
View file

@ -30,6 +30,7 @@ import {
TIMESTAMP,
} from '@kbn/rule-data-utils';
import type { Type as RuleType } from '@kbn/securitysolution-io-ts-alerting-types';
import { lastValueFrom } from 'rxjs';
import type { EcsSecurityExtension as Ecs } from '@kbn/securitysolution-ecs';
import type { DataTableModel } from '@kbn/securitysolution-data-table';
@ -42,7 +43,13 @@ import {
ALERT_NEW_TERMS,
ALERT_RULE_INDICES,
} from '../../../../common/field_maps/field_names';
import { isEqlRule, isEsqlRule } from '../../../../common/detection_engine/utils';
import {
isEqlRule,
isEsqlRule,
isMlRule,
isNewTermsRule,
isThresholdRule,
} from '../../../../common/detection_engine/utils';
import type { TimelineResult } from '../../../../common/api/timeline';
import { TimelineId } from '../../../../common/types/timeline';
import { TimelineStatus, TimelineType } from '../../../../common/api/timeline';
@ -266,31 +273,16 @@ export const isEqlAlertWithGroupId = (ecsData: Ecs): boolean => {
return isEql && groupId?.length > 0;
};
export const isThresholdAlert = (ecsData: Ecs): boolean => {
const getRuleType = (ecsData: Ecs): RuleType | undefined => {
const ruleType = getField(ecsData, ALERT_RULE_TYPE);
return (
ruleType === 'threshold' ||
(Array.isArray(ruleType) && ruleType.length > 0 && ruleType[0] === 'threshold')
);
return Array.isArray(ruleType) ? ruleType[0] : ruleType;
};
export const isEqlAlert = (ecsData: Ecs): boolean => {
const ruleType = getField(ecsData, ALERT_RULE_TYPE);
return isEqlRule(ruleType) || (Array.isArray(ruleType) && isEqlRule(ruleType[0]));
};
export const isEsqlAlert = (ecsData: Ecs): boolean => {
const ruleType = getField(ecsData, ALERT_RULE_TYPE);
return isEsqlRule(ruleType) || (Array.isArray(ruleType) && isEsqlRule(ruleType[0]));
};
export const isNewTermsAlert = (ecsData: Ecs): boolean => {
const ruleType = getField(ecsData, ALERT_RULE_TYPE);
return (
ruleType === 'new_terms' ||
(Array.isArray(ruleType) && ruleType.length > 0 && ruleType[0] === 'new_terms')
);
};
const isNewTermsAlert = (ecsData: Ecs): boolean => isNewTermsRule(getRuleType(ecsData));
const isEsqlAlert = (ecsData: Ecs): boolean => isEsqlRule(getRuleType(ecsData));
const isEqlAlert = (ecsData: Ecs): boolean => isEqlRule(getRuleType(ecsData));
const isThresholdAlert = (ecsData: Ecs): boolean => isThresholdRule(getRuleType(ecsData));
const isMlAlert = (ecsData: Ecs): boolean => isMlRule(getRuleType(ecsData));
const isSuppressedAlert = (ecsData: Ecs): boolean => {
return getField(ecsData, ALERT_SUPPRESSION_DOCS_COUNT) != null;
@ -1035,7 +1027,12 @@ export const sendAlertToTimelineAction = async ({
getExceptionFilter
);
// The Query field should remain unpopulated with the suppressed EQL/ES|QL alert.
} else if (isSuppressedAlert(ecsData) && !isEqlAlert(ecsData) && !isEsqlAlert(ecsData)) {
} else if (
isSuppressedAlert(ecsData) &&
!isEqlAlert(ecsData) &&
!isEsqlAlert(ecsData) &&
!isMlAlert(ecsData)
) {
return createSuppressedTimeline(
ecsData,
createTimeline,
@ -1106,7 +1103,12 @@ export const sendAlertToTimelineAction = async ({
} else if (isNewTermsAlert(ecsData)) {
return createNewTermsTimeline(ecsData, createTimeline, noteContent, {}, getExceptionFilter);
// The Query field should remain unpopulated with the suppressed EQL/ES|QL alert.
} else if (isSuppressedAlert(ecsData) && !isEqlAlert(ecsData) && !isEsqlAlert(ecsData)) {
} else if (
isSuppressedAlert(ecsData) &&
!isEqlAlert(ecsData) &&
!isEsqlAlert(ecsData) &&
!isMlAlert(ecsData)
) {
return createSuppressedTimeline(ecsData, createTimeline, noteContent, {}, getExceptionFilter);
} else {
let { dataProviders, filters } = buildTimelineDataProviderOrFilter(

18
x-pack/plugins/security_solution/scripts/endpoint/common/roles_users/serverless/es_serverless_resources/roles.yml
View file

@ -53,6 +53,7 @@ viewer:
- ".fleet-actions*"
- "risk-score.risk-score-*"
- ".asset-criticality.asset-criticality-*"
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -119,6 +120,10 @@ editor:
- "read"
- "write"
allow_restricted_indices: false
- names:
- ".ml-anomalies-*"
privileges:
- read
applications:
- application: "kibana-.kibana"
privileges:
@ -174,6 +179,7 @@ t1_analyst:
- ".fleet-actions*"
- risk-score.risk-score-*
- .asset-criticality.asset-criticality-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -222,6 +228,7 @@ t2_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
- names:
@ -284,6 +291,7 @@ t3_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -349,6 +357,7 @@ threat_intelligence_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -408,6 +417,7 @@ rule_author:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -473,6 +483,7 @@ soc_manager:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -534,6 +545,7 @@ detections_admin:
- metrics-endpoint.metadata_current_*
- .fleet-agents*
- .fleet-actions*
- ".ml-anomalies-*"
privileges:
- read
- names:
@ -592,6 +604,10 @@ platform_engineer:
privileges:
- read
- write
- names:
- ".ml-anomalies-*"
privileges:
- read
applications:
- application: "kibana-.kibana"
privileges:
@ -643,6 +659,7 @@ endpoint_operations_analyst:
- .lists*
- .items*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
- names:
@ -711,6 +728,7 @@ endpoint_policy_manager:
- packetbeat-*
- winlogbeat-*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
- names:

117
x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.test.ts
View file

@ -110,6 +110,51 @@ describe('rule_converters', () => {
});
});
describe('machine learning rules', () => {
test('should accept machine learning params when existing rule type is machine learning', () => {
const patchParams = {
anomaly_threshold: 5,
};
const rule = getMlRuleParams();
const patchedParams = patchTypeSpecificSnakeToCamel(patchParams, rule);
expect(patchedParams).toEqual(
expect.objectContaining({
anomalyThreshold: 5,
})
);
});
test('should reject invalid machine learning params when existing rule type is machine learning', () => {
const patchParams = {
anomaly_threshold: 'invalid',
} as PatchRuleRequestBody;
const rule = getMlRuleParams();
expect(() => patchTypeSpecificSnakeToCamel(patchParams, rule)).toThrowError(
'anomaly_threshold: Expected number, received string'
);
});
it('accepts suppression params', () => {
const patchParams = {
alert_suppression: {
group_by: ['agent.name'],
missing_fields_strategy: 'suppress' as const,
},
};
const rule = getMlRuleParams();
const patchedParams = patchTypeSpecificSnakeToCamel(patchParams, rule);
expect(patchedParams).toEqual(
expect.objectContaining({
alertSuppression: {
groupBy: ['agent.name'],
missingFieldsStrategy: 'suppress',
},
})
);
});
});
test('should accept threat match params when existing rule type is threat match', () => {
const patchParams = {
threat_indicator_path: 'my.indicator',
@ -298,29 +343,6 @@ describe('rule_converters', () => {
);
});
test('should accept machine learning params when existing rule type is machine learning', () => {
const patchParams = {
anomaly_threshold: 5,
};
const rule = getMlRuleParams();
const patchedParams = patchTypeSpecificSnakeToCamel(patchParams, rule);
expect(patchedParams).toEqual(
expect.objectContaining({
anomalyThreshold: 5,
})
);
});
test('should reject invalid machine learning params when existing rule type is machine learning', () => {
const patchParams = {
anomaly_threshold: 'invalid',
} as PatchRuleRequestBody;
const rule = getMlRuleParams();
expect(() => patchTypeSpecificSnakeToCamel(patchParams, rule)).toThrowError(
'anomaly_threshold: Expected number, received string'
);
});
test('should accept new terms params when existing rule type is new terms', () => {
const patchParams = {
new_terms_fields: ['event.new_field'],
@ -344,6 +366,7 @@ describe('rule_converters', () => {
);
});
});
describe('typeSpecificCamelToSnake', () => {
describe('EQL', () => {
test('should accept EQL params when existing rule type is EQL', () => {
@ -396,6 +419,54 @@ describe('rule_converters', () => {
);
});
});
describe('machine learning rules', () => {
it('accepts normal params', () => {
const params = {
anomalyThreshold: 74,
machineLearningJobId: ['job-1'],
};
const ruleParams = { ...getMlRuleParams(), ...params };
const transformedParams = typeSpecificCamelToSnake(ruleParams);
expect(transformedParams).toEqual(
expect.objectContaining({
anomaly_threshold: 74,
machine_learning_job_id: ['job-1'],
})
);
});
it('accepts suppression params', () => {
const params = {
anomalyThreshold: 74,
machineLearningJobId: ['job-1'],
alertSuppression: {
groupBy: ['event.type'],
duration: {
value: 10,
unit: 'm',
} as AlertSuppressionDuration,
missingFieldsStrategy: 'suppress' as AlertSuppressionMissingFieldsStrategy,
},
};
const ruleParams = { ...getMlRuleParams(), ...params };
const transformedParams = typeSpecificCamelToSnake(ruleParams);
expect(transformedParams).toEqual(
expect.objectContaining({
anomaly_threshold: 74,
machine_learning_job_id: ['job-1'],
alert_suppression: {
group_by: ['event.type'],
duration: {
value: 10,
unit: 'm',
},
missing_fields_strategy: 'suppress',
},
})
);
});
});
});
describe('commonParamsCamelToSnake', () => {

4
x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.ts
View file

@ -191,6 +191,7 @@ export const typeSpecificSnakeToCamel = (
type: params.type,
anomalyThreshold: params.anomaly_threshold,
machineLearningJobId: normalizeMachineLearningJobIds(params.machine_learning_job_id),
alertSuppression: convertAlertSuppressionToCamel(params.alert_suppression),
};
}
case 'new_terms': {
@ -338,6 +339,8 @@ const patchMachineLearningParams = (
machineLearningJobId: params.machine_learning_job_id
? normalizeMachineLearningJobIds(params.machine_learning_job_id)
: existingRule.machineLearningJobId,
alertSuppression:
convertAlertSuppressionToCamel(params.alert_suppression) ?? existingRule.alertSuppression,
};
};
@ -706,6 +709,7 @@ export const typeSpecificCamelToSnake = (
type: params.type,
anomaly_threshold: params.anomalyThreshold,
machine_learning_job_id: params.machineLearningJobId,
alert_suppression: convertAlertSuppressionToSnake(params.alertSuppression),
};
}
case 'new_terms': {

1
x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema/model/rule_schemas.ts
View file

@ -268,6 +268,7 @@ export const MachineLearningSpecificRuleParams = z.object({
type: z.literal('machine_learning'),
anomalyThreshold: AnomalyThreshold,
machineLearningJobId: z.array(z.string()),
alertSuppression: AlertSuppressionCamel.optional(),
});
export type MachineLearningRuleParams = BaseRuleParams & MachineLearningSpecificRuleParams;

39
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/create_ml_alert_type.ts
View file

@ -11,13 +11,15 @@ import { DEFAULT_APP_CATEGORIES } from '@kbn/core-application-common';
import { SERVER_APP_ID } from '../../../../../common/constants';
import { MachineLearningRuleParams } from '../../rule_schema';
import { getIsAlertSuppressionActive } from '../utils/get_is_alert_suppression_active';
import { mlExecutor } from './ml';
import type { CreateRuleOptions, SecurityAlertType } from '../types';
import type { CreateRuleOptions, SecurityAlertType, WrapSuppressedHits } from '../types';
import { wrapSuppressedAlerts } from '../utils/wrap_suppressed_alerts';
export const createMlAlertType = (
createOptions: CreateRuleOptions
): SecurityAlertType<MachineLearningRuleParams, {}, {}, 'default'> => {
const { ml } = createOptions;
const { experimentalFeatures, ml, licensing } = createOptions;
return {
id: ML_RULE_TYPE_ID,
name: 'Machine Learning Rule',
@ -56,11 +58,39 @@ export const createMlAlertType = (
wrapHits,
exceptionFilter,
unprocessedExceptions,
mergeStrategy,
alertTimestampOverride,
publicBaseUrl,
alertWithSuppression,
primaryTimestamp,
secondaryTimestamp,
},
services,
spaceId,
state,
} = execOptions;
const isAlertSuppressionActive = await getIsAlertSuppressionActive({
alertSuppression: completeRule.ruleParams.alertSuppression,
isFeatureDisabled: !experimentalFeatures.alertSuppressionForMachineLearningRuleEnabled,
licensing,
});
const wrapSuppressedHits: WrapSuppressedHits = (events, buildReasonMessage) =>
wrapSuppressedAlerts({
events,
spaceId,
completeRule,
mergeStrategy,
indicesToQuery: [],
buildReasonMessage,
alertTimestampOverride,
ruleExecutionLogger,
publicBaseUrl,
primaryTimestamp,
secondaryTimestamp,
});
const result = await mlExecutor({
completeRule,
tuple,
@ -72,6 +102,11 @@ export const createMlAlertType = (
wrapHits,
exceptionFilter,
unprocessedExceptions,
wrapSuppressedHits,
alertTimestampOverride,
alertWithSuppression,
isAlertSuppressionActive,
experimentalFeatures,
});
return { ...result, state };
},

60
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/ml.test.ts
View file

@ -9,6 +9,7 @@ import dateMath from '@kbn/datemath';
import type { RuleExecutorServicesMock } from '@kbn/alerting-plugin/server/mocks';
import { alertsMock } from '@kbn/alerting-plugin/server/mocks';
import { mlExecutor } from './ml';
import type { ExperimentalFeatures } from '../../../../../common';
import { getCompleteRuleMock, getMlRuleParams } from '../../rule_schema/mocks';
import { getListClientMock } from '@kbn/lists-plugin/server/services/lists/list_client.mock';
import { findMlSignals } from './find_ml_signals';
@ -21,6 +22,7 @@ jest.mock('./find_ml_signals');
jest.mock('./bulk_create_ml_signals');
describe('ml_executor', () => {
let mockExperimentalFeatures: jest.Mocked<ExperimentalFeatures>;
let jobsSummaryMock: jest.Mock;
let forceStartDatafeedsMock: jest.Mock;
let stopDatafeedsMock: jest.Mock;
@ -37,6 +39,7 @@ describe('ml_executor', () => {
const listClient = getListClientMock();
beforeEach(() => {
mockExperimentalFeatures = {} as jest.Mocked<ExperimentalFeatures>;
jobsSummaryMock = jest.fn();
mlMock = mlPluginServerMock.createSetupContract();
mlMock.jobServiceProvider.mockReturnValue({
@ -59,7 +62,7 @@ describe('ml_executor', () => {
});
(bulkCreateMlSignals as jest.Mock).mockResolvedValue({
success: true,
bulkCreateDuration: 0,
bulkCreateDuration: 21,
createdItemsCount: 0,
errors: [],
createdItems: [],
@ -80,6 +83,11 @@ describe('ml_executor', () => {
wrapHits: jest.fn(),
exceptionFilter: undefined,
unprocessedExceptions: [],
wrapSuppressedHits: jest.fn(),
alertTimestampOverride: undefined,
alertWithSuppression: jest.fn(),
isAlertSuppressionActive: true,
experimentalFeatures: mockExperimentalFeatures,
})
).rejects.toThrow('ML plugin unavailable during rule execution');
});
@ -97,6 +105,11 @@ describe('ml_executor', () => {
wrapHits: jest.fn(),
exceptionFilter: undefined,
unprocessedExceptions: [],
wrapSuppressedHits: jest.fn(),
alertTimestampOverride: undefined,
alertWithSuppression: jest.fn(),
isAlertSuppressionActive: true,
experimentalFeatures: mockExperimentalFeatures,
});
expect(ruleExecutionLogger.warn).toHaveBeenCalled();
expect(ruleExecutionLogger.warn.mock.calls[0][0]).toContain(
@ -125,6 +138,11 @@ describe('ml_executor', () => {
wrapHits: jest.fn(),
exceptionFilter: undefined,
unprocessedExceptions: [],
wrapSuppressedHits: jest.fn(),
alertTimestampOverride: undefined,
alertWithSuppression: jest.fn(),
isAlertSuppressionActive: true,
experimentalFeatures: mockExperimentalFeatures,
});
expect(ruleExecutionLogger.warn).toHaveBeenCalled();
expect(ruleExecutionLogger.warn.mock.calls[0][0]).toContain(
@ -149,9 +167,49 @@ describe('ml_executor', () => {
wrapHits: jest.fn(),
exceptionFilter: undefined,
unprocessedExceptions: [],
wrapSuppressedHits: jest.fn(),
alertTimestampOverride: undefined,
alertWithSuppression: jest.fn(),
isAlertSuppressionActive: true,
experimentalFeatures: mockExperimentalFeatures,
});
expect(result.userError).toEqual(true);
expect(result.success).toEqual(false);
expect(result.errors).toEqual(['my_test_job_name missing']);
});
it('returns some timing information as part of the result', async () => {
// ensure our mock corresponds to the job that the rule uses
jobsSummaryMock.mockResolvedValue(
mlCompleteRule.ruleParams.machineLearningJobId.map((jobId) => ({
id: jobId,
jobState: 'opened',
datafeedState: 'started',
}))
);
const result = await mlExecutor({
completeRule: mlCompleteRule,
tuple,
ml: mlMock,
services: alertServices,
ruleExecutionLogger,
listClient,
bulkCreate: jest.fn(),
wrapHits: jest.fn(),
exceptionFilter: undefined,
unprocessedExceptions: [],
wrapSuppressedHits: jest.fn(),
alertTimestampOverride: undefined,
alertWithSuppression: jest.fn(),
isAlertSuppressionActive: true,
experimentalFeatures: mockExperimentalFeatures,
});
expect(result).toEqual(
expect.objectContaining({
bulkCreateTimes: expect.arrayContaining([expect.any(Number)]),
})
);
});
});

84
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/ml/ml.ts
View file

@ -8,6 +8,7 @@
/* eslint require-atomic-updates: ["error", { "allowProperties": true }] */
import type { KibanaRequest } from '@kbn/core/server';
import type { SuppressedAlertService } from '@kbn/rule-registry-plugin/server';
import type { ExceptionListItemSchema } from '@kbn/securitysolution-io-ts-list-types';
import type {
AlertInstanceContext,
@ -17,11 +18,12 @@ import type {
import type { ListClient } from '@kbn/lists-plugin/server';
import type { Filter } from '@kbn/es-query';
import { isJobStarted } from '../../../../../common/machine_learning/helpers';
import type { ExperimentalFeatures } from '../../../../../common/experimental_features';
import type { CompleteRule, MachineLearningRuleParams } from '../../rule_schema';
import { bulkCreateMlSignals } from './bulk_create_ml_signals';
import { filterEventsAgainstList } from '../utils/large_list_filters/filter_events_against_list';
import { findMlSignals } from './find_ml_signals';
import type { BulkCreate, RuleRangeTuple, WrapHits } from '../types';
import type { BulkCreate, RuleRangeTuple, WrapHits, WrapSuppressedHits } from '../types';
import {
addToSearchAfterReturn,
createErrorsFromShard,
@ -33,6 +35,26 @@ import type { SetupPlugins } from '../../../../plugin';
import { withSecuritySpan } from '../../../../utils/with_security_span';
import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring';
import type { AnomalyResults } from '../../../machine_learning';
import { bulkCreateSuppressedAlertsInMemory } from '../utils/bulk_create_suppressed_alerts_in_memory';
import { buildReasonMessageForMlAlert } from '../utils/reason_formatters';
interface MachineLearningRuleExecutorParams {
completeRule: CompleteRule<MachineLearningRuleParams>;
tuple: RuleRangeTuple;
ml: SetupPlugins['ml'];
listClient: ListClient;
services: RuleExecutorServices<AlertInstanceState, AlertInstanceContext, 'default'>;
ruleExecutionLogger: IRuleExecutionLogForExecutors;
bulkCreate: BulkCreate;
wrapHits: WrapHits;
exceptionFilter: Filter | undefined;
unprocessedExceptions: ExceptionListItemSchema[];
wrapSuppressedHits: WrapSuppressedHits;
alertTimestampOverride: Date | undefined;
alertWithSuppression: SuppressedAlertService;
isAlertSuppressionActive: boolean;
experimentalFeatures: ExperimentalFeatures;
}
export const mlExecutor = async ({
completeRule,
@ -45,18 +67,12 @@ export const mlExecutor = async ({
wrapHits,
exceptionFilter,
unprocessedExceptions,
}: {
completeRule: CompleteRule<MachineLearningRuleParams>;
tuple: RuleRangeTuple;
ml: SetupPlugins['ml'];
listClient: ListClient;
services: RuleExecutorServices<AlertInstanceState, AlertInstanceContext, 'default'>;
ruleExecutionLogger: IRuleExecutionLogForExecutors;
bulkCreate: BulkCreate;
wrapHits: WrapHits;
exceptionFilter: Filter | undefined;
unprocessedExceptions: ExceptionListItemSchema[];
}) => {
isAlertSuppressionActive,
wrapSuppressedHits,
alertTimestampOverride,
alertWithSuppression,
experimentalFeatures,
}: MachineLearningRuleExecutorParams) => {
const result = createSearchAfterReturnType();
const ruleParams = completeRule.ruleParams;
@ -120,6 +136,7 @@ export const mlExecutor = async ({
return result;
}
// TODO we add the max_signals warning _before_ filtering the anomalies against the exceptions list. Is that correct?
if (
anomalyResults.hits.total &&
typeof anomalyResults.hits.total !== 'number' &&
@ -140,17 +157,36 @@ export const mlExecutor = async ({
ruleExecutionLogger.debug(`Found ${anomalyCount} signals from ML anomalies`);
}
const createResult = await bulkCreateMlSignals({
anomalyHits: filteredAnomalyHits,
completeRule,
services,
ruleExecutionLogger,
id: completeRule.alertId,
signalsIndex: ruleParams.outputIndex,
bulkCreate,
wrapHits,
});
addToSearchAfterReturn({ current: result, next: createResult });
if (anomalyCount && isAlertSuppressionActive) {
await bulkCreateSuppressedAlertsInMemory({
enrichedEvents: filteredAnomalyHits,
toReturn: result,
wrapHits,
bulkCreate,
services,
buildReasonMessage: buildReasonMessageForMlAlert,
ruleExecutionLogger,
tuple,
alertSuppression: completeRule.ruleParams.alertSuppression,
wrapSuppressedHits,
alertTimestampOverride,
alertWithSuppression,
experimentalFeatures,
});
} else {
const createResult = await bulkCreateMlSignals({
anomalyHits: filteredAnomalyHits,
completeRule,
services,
ruleExecutionLogger,
id: completeRule.alertId,
signalsIndex: ruleParams.outputIndex,
bulkCreate,
wrapHits,
});
addToSearchAfterReturn({ current: result, next: createResult });
}
const shardFailures = anomalyResults._shards.failures ?? [];
const searchErrors = createErrorsFromShard({
errors: shardFailures,

4
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/types.ts
View file

@ -37,7 +37,7 @@ import type { LicensingPluginSetup } from '@kbn/licensing-plugin/server';
import type { RuleResponseAction } from '../../../../common/api/detection_engine/model/rule_response_actions';
import type { ConfigType } from '../../../config';
import type { SetupPlugins } from '../../../plugin';
import type { CompleteRule, EqlRuleParams, RuleParams, ThreatRuleParams } from '../rule_schema';
import type { CompleteRule, RuleParams } from '../rule_schema';
import type { ExperimentalFeatures } from '../../../../common/experimental_features';
import type { ITelemetryEventsSender } from '../../telemetry/sender';
import type { IRuleExecutionLogForExecutors, IRuleMonitoringService } from '../rule_monitoring';
@ -401,5 +401,3 @@ export interface OverrideBodyQuery {
_source?: estypes.SearchSourceConfig;
fields?: estypes.Fields;
}
export type RuleWithInMemorySuppression = ThreatRuleParams | EqlRuleParams;

11
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/wrap_suppressed_alerts.ts
View file

@ -9,14 +9,19 @@ import objectHash from 'object-hash';
import { TIMESTAMP } from '@kbn/rule-data-utils';
import type { SuppressionFieldsLatest } from '@kbn/rule-registry-plugin/common/schemas';
import type { RuleWithInMemorySuppression, SignalSourceHit } from '../types';
import type { SignalSourceHit } from '../types';
import type {
BaseFieldsLatest,
WrappedFieldsLatest,
} from '../../../../../common/api/detection_engine/model/alerts';
import type { ConfigType } from '../../../../config';
import type { CompleteRule } from '../../rule_schema';
import type {
CompleteRule,
EqlRuleParams,
MachineLearningRuleParams,
ThreatRuleParams,
} from '../../rule_schema';
import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring';
import { buildBulkBody } from '../factories/utils/build_bulk_body';
import { getSuppressionAlertFields, getSuppressionTerms } from './suppression_utils';
@ -24,6 +29,8 @@ import { generateId } from './utils';
import type { BuildReasonMessage } from './reason_formatters';
type RuleWithInMemorySuppression = ThreatRuleParams | EqlRuleParams | MachineLearningRuleParams;
/**
* wraps suppressed alerts
* creates instanceId hash, which is used to search on time interval alerts

36
x-pack/test/common/utils/security_solution/detections_response/delete_all_anomalies.ts Normal file
View file

@ -0,0 +1,36 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { ToolingLog } from '@kbn/tooling-log';
import type { Client } from '@elastic/elasticsearch';
import { countDownTest } from './count_down_test';
export const deleteAllAnomalies = async (
log: ToolingLog,
es: Client,
index: string[] = ['.ml-anomalies-*']
): Promise<void> => {
await countDownTest(
async () => {
await es.deleteByQuery({
index,
body: {
query: {
match_all: {},
},
},
refresh: true,
});
return {
passed: true,
};
},
'deleteAllAnomalies',
log
);
};

1
x-pack/test/common/utils/security_solution/detections_response/index.ts
View file

@ -7,6 +7,7 @@
export * from './rules';
export * from './alerts';
export * from './delete_all_anomalies';
export * from './count_down_test';
export * from './route_with_namespace';
export * from './wait_for';

9
x-pack/test/functional/es_archives/security_solution/anomalies/mappings.json
View file

@ -2,22 +2,21 @@
"type": "index",
"value": {
"aliases": {
".ml-anomalies-.write-linux_anomalous_network_activity_ecs": {
".ml-anomalies-.write-v3_linux_anomalous_network_activity": {
"is_hidden": true
},
".ml-anomalies-linux_anomalous_network_activity_ecs": {
".ml-anomalies-v3_linux_anomalous_network_activity": {
"filter": {
"term": {
"job_id": {
"boost": 1,
"value": "linux_anomalous_network_activity_ecs"
"value": "v3_linux_anomalous_network_activity"
}
}
},
"is_hidden": true
}
},
"index": ".ml-anomalies-custom-linux_anomalous_network_activity_ecs",
"index": ".ml-anomalies-custom-v3_linux_anomalous_network_activity",
"mappings": {
"_meta": {
"version": "8.0.0"

1
x-pack/test/security_solution_api_integration/config/ess/config.base.ts
View file

@ -84,6 +84,7 @@ export function createTestConfig(options: CreateTestConfigOptions, testFiles?: s
'riskScoringPersistence',
'riskScoringRoutesEnabled',
'bulkCustomHighlightedFieldsEnabled',
'alertSuppressionForMachineLearningRuleEnabled',
'manualRuleRunEnabled',
])}`,
'--xpack.task_manager.poll_interval=1000',

1
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/configs/serverless.config.ts
View file

@ -19,6 +19,7 @@ export default createTestConfig({
])}`, // See tests within the file "ignore_fields.ts" which use these values in "alertIgnoreFields"
`--xpack.securitySolution.enableExperimental=${JSON.stringify([
'bulkCustomHighlightedFieldsEnabled',
'alertSuppressionForMachineLearningRuleEnabled',
'alertSuppressionForEsqlRuleEnabled',
])}`,
],

1
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/index.ts
View file

@ -14,6 +14,7 @@ export default ({ loadTestFile }: FtrProviderContext): void => {
loadTestFile(require.resolve('./esql'));
loadTestFile(require.resolve('./esql_suppression'));
loadTestFile(require.resolve('./machine_learning'));
loadTestFile(require.resolve('./machine_learning_alert_suppression'));
loadTestFile(require.resolve('./new_terms'));
loadTestFile(require.resolve('./new_terms_alert_suppression'));
loadTestFile(require.resolve('./saved_query'));

4
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/machine_learning.ts
View file

@ -151,7 +151,7 @@ export default ({ getService }: FtrProviderContext) => {
[SPACE_IDS]: ['default'],
[ALERT_SEVERITY]: 'critical',
[ALERT_RISK_SCORE]: 50,
[ALERT_RULE_PARAMETERS]: {
[ALERT_RULE_PARAMETERS]: expect.objectContaining({
anomaly_threshold: 30,
author: [],
description: 'Test ML rule description',
@ -174,7 +174,7 @@ export default ({ getService }: FtrProviderContext) => {
to: 'now',
type: 'machine_learning',
version: 1,
},
}),
[ALERT_DEPTH]: 1,
[ALERT_REASON]: `event with process store, by root on mothra created critical alert Test ML rule.`,
[ALERT_ORIGINAL_TIME]: expect.any(String),

1106
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/machine_learning_alert_suppression.ts Normal file
View file

File diff suppressed because it is too large Load diff

3
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/machine_learning/machine_learning_setup.ts
View file

@ -6,6 +6,7 @@
*/
import type SuperTest from 'supertest';
import { ML_GROUP_ID } from '@kbn/security-solution-plugin/common/constants';
import { getCommonRequestHeader } from '../../../../../functional/services/ml/common_api';
export const executeSetupModuleRequest = async ({
@ -22,7 +23,7 @@ export const executeSetupModuleRequest = async ({
.set(getCommonRequestHeader('1'))
.send({
prefix: '',
groups: ['auditbeat'],
groups: [ML_GROUP_ID],
indexPatternName: 'auditbeat-*',
startDatafeed: false,
useDedicatedIndex: true,

10
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/remove_server_generated_properties_including_rule_id.ts
View file

@ -7,7 +7,10 @@
import type { RuleResponse } from '@kbn/security-solution-plugin/common/api/detection_engine';
import { removeServerGeneratedProperties } from './remove_server_generated_properties';
import {
removeServerGeneratedProperties,
type RuleWithoutServerGeneratedProperties,
} from './remove_server_generated_properties';
/**
* This will remove server generated properties such as date times, etc... including the rule_id
@ -15,9 +18,8 @@ import { removeServerGeneratedProperties } from './remove_server_generated_prope
*/
export const removeServerGeneratedPropertiesIncludingRuleId = (
rule: RuleResponse
): Partial<RuleResponse> => {
): Omit<RuleWithoutServerGeneratedProperties, 'rule_id'> => {
const ruleWithRemovedProperties = removeServerGeneratedProperties(rule);
// eslint-disable-next-line @typescript-eslint/naming-convention
const { rule_id, ...additionalRuledIdRemoved } = ruleWithRemovedProperties;
const { rule_id: _, ...additionalRuledIdRemoved } = ruleWithRemovedProperties;
return additionalRuledIdRemoved;
};

1
x-pack/test/security_solution_cypress/config.ts
View file

@ -47,6 +47,7 @@ export default async function ({ readConfigFile }: FtrConfigProviderContext) {
`--xpack.securitySolution.enableExperimental=${JSON.stringify([
'alertSuppressionForEsqlRuleEnabled',
'bulkCustomHighlightedFieldsEnabled',
'alertSuppressionForMachineLearningRuleEnabled',
'manualRuleRunEnabled',
])}`,
// mock cloud to enable the guided onboarding tour in e2e tests

7
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/common_flows_suppression_serverless_essentials.cy.ts
View file

@ -15,6 +15,7 @@ import { login } from '../../../../tasks/login';
import { visit } from '../../../../tasks/navigation';
import {
ALERT_SUPPRESSION_FIELDS_INPUT,
MACHINE_LEARNING_TYPE,
THRESHOLD_ENABLE_SUPPRESSION_CHECKBOX,
} from '../../../../screens/create_new_rule';
import { CREATE_RULE_URL } from '../../../../urls/navigation';
@ -22,7 +23,7 @@ import { CREATE_RULE_URL } from '../../../../urls/navigation';
describe(
'Detection rules, Alert Suppression for Essentials tier',
{
// skipped in MKI as it depends on feature flag alertSuppressionForEsqlRuleEnabled
// skipped in MKI as it depends on feature flag alertSuppressionForEsqlRuleEnabled, alertSuppressionForMachineLearningRuleEnabled
tags: ['@serverless', '@skipInServerlessMKI'],
env: {
ftrConfig: {
@ -35,6 +36,7 @@ describe(
kbnServerArgs: [
`--xpack.securitySolution.enableExperimental=${JSON.stringify([
'alertSuppressionForEsqlRuleEnabled',
'alertSuppressionForMachineLearningRuleEnabled',
])}`,
],
},
@ -60,6 +62,9 @@ describe(
selectEsqlRuleType();
cy.get(ALERT_SUPPRESSION_FIELDS_INPUT).should('be.enabled');
// ML Rules require Complete tier
cy.get(MACHINE_LEARNING_TYPE).get('button').should('be.disabled');
});
}
);

4
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/common_flows_supression_ess_basic.cy.ts
View file

@ -8,6 +8,7 @@
import {
THRESHOLD_ENABLE_SUPPRESSION_CHECKBOX,
ALERT_SUPPRESSION_DURATION_INPUT,
MACHINE_LEARNING_TYPE,
} from '../../../../screens/create_new_rule';
import {
@ -52,6 +53,9 @@ describe(
selectEsqlRuleType();
openSuppressionFieldsTooltipAndCheckLicense();
// ML Rules require Platinum license
cy.get(MACHINE_LEARNING_TYPE).get('button').should('be.disabled');
selectThresholdRuleType();
cy.get(THRESHOLD_ENABLE_SUPPRESSION_CHECKBOX).should('be.disabled');
cy.get(THRESHOLD_ENABLE_SUPPRESSION_CHECKBOX).parent().trigger('mouseover');

198
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/machine_learning_rule_suppression.cy.ts Normal file
View file

@ -0,0 +1,198 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { getMachineLearningRule } from '../../../../objects/rule';
import { TOOLTIP } from '../../../../screens/common';
import {
ALERT_SUPPRESSION_FIELDS,
ALERT_SUPPRESSION_FIELDS_INPUT,
} from '../../../../screens/create_new_rule';
import {
DEFINITION_DETAILS,
DETAILS_TITLE,
SUPPRESS_BY_DETAILS,
SUPPRESS_FOR_DETAILS,
SUPPRESS_MISSING_FIELD,
} from '../../../../screens/rule_details';
import {
executeSetupModuleRequest,
forceStartDatafeeds,
forceStopAndCloseJob,
} from '../../../../support/machine_learning';
import {
continueFromDefineStep,
fillAlertSuppressionFields,
fillDefineMachineLearningRule,
selectMachineLearningRuleType,
selectAlertSuppressionPerInterval,
setAlertSuppressionDuration,
selectDoNotSuppressForMissingFields,
skipScheduleRuleAction,
fillAboutRuleMinimumAndContinue,
createRuleWithoutEnabling,
} from '../../../../tasks/create_new_rule';
import { login } from '../../../../tasks/login';
import { visit } from '../../../../tasks/navigation';
import { getDetails } from '../../../../tasks/rule_details';
import { CREATE_RULE_URL } from '../../../../urls/navigation';
describe(
'Machine Learning Detection Rules - Creation',
{
// Skipped in MKI as tests depend on feature flag alertSuppressionForMachineLearningRuleEnabled
tags: ['@ess', '@serverless', '@skipInServerlessMKI'],
env: {
ftrConfig: {
kbnServerArgs: [
`--xpack.securitySolution.enableExperimental=${JSON.stringify([
'alertSuppressionForMachineLearningRuleEnabled',
])}`,
],
},
},
},
() => {
let mlRule: ReturnType<typeof getMachineLearningRule>;
const jobId = 'v3_linux_anomalous_network_activity';
const suppressByFields = ['by_field_name', 'by_field_value'];
beforeEach(() => {
login();
visit(CREATE_RULE_URL);
});
describe('with Alert Suppression', () => {
describe('when no ML jobs have run', () => {
before(() => {
const machineLearningJobIds = ([] as string[]).concat(
getMachineLearningRule().machine_learning_job_id
);
// ensure no ML jobs are started before the suite
machineLearningJobIds.forEach((j) => forceStopAndCloseJob({ jobId: j }));
});
beforeEach(() => {
mlRule = getMachineLearningRule();
selectMachineLearningRuleType();
fillDefineMachineLearningRule(mlRule);
});
it('disables the suppression fields and displays a message', () => {
cy.get(ALERT_SUPPRESSION_FIELDS_INPUT).should('be.disabled');
cy.get(ALERT_SUPPRESSION_FIELDS_INPUT).realHover();
cy.get(TOOLTIP).should(
'contain.text',
'To enable alert suppression, start relevant Machine Learning jobs.'
);
});
});
describe('when ML jobs have run', () => {
before(() => {
cy.task('esArchiverLoad', { archiveName: '../auditbeat/hosts', type: 'ftr' });
executeSetupModuleRequest({ moduleName: 'security_linux_v3' });
forceStartDatafeeds({ jobIds: [jobId] });
cy.task('esArchiverLoad', { archiveName: 'anomalies', type: 'ftr' });
});
after(() => {
cy.task('esArchiverUnload', { archiveName: 'anomalies', type: 'ftr' });
cy.task('esArchiverUnload', { archiveName: '../auditbeat/hosts', type: 'ftr' });
});
describe('when not all jobs are running', () => {
beforeEach(() => {
mlRule = getMachineLearningRule();
selectMachineLearningRuleType();
fillDefineMachineLearningRule(mlRule);
});
it('displays a warning message on the suppression fields', () => {
cy.get(ALERT_SUPPRESSION_FIELDS_INPUT).should('be.enabled');
cy.get(ALERT_SUPPRESSION_FIELDS).should(
'contain.text',
'This list of fields might be incomplete as some Machine Learning jobs are not running. Start all relevant jobs for a complete list.'
);
});
});
describe('when all jobs are running', () => {
beforeEach(() => {
mlRule = getMachineLearningRule({ machine_learning_job_id: [jobId] });
selectMachineLearningRuleType();
fillDefineMachineLearningRule(mlRule);
});
it('allows a rule with per-execution suppression to be created and displayed', () => {
fillAlertSuppressionFields(suppressByFields);
continueFromDefineStep();
// ensures details preview works correctly
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(SUPPRESS_BY_DETAILS).should('have.text', suppressByFields.join(''));
getDetails(SUPPRESS_FOR_DETAILS).should('have.text', 'One rule execution');
getDetails(SUPPRESS_MISSING_FIELD).should(
'have.text',
'Suppress and group alerts for events with missing fields'
);
// suppression functionality should be under Tech Preview
cy.contains(DETAILS_TITLE, SUPPRESS_FOR_DETAILS).contains('Technical Preview');
});
fillAboutRuleMinimumAndContinue(mlRule);
skipScheduleRuleAction();
createRuleWithoutEnabling();
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(SUPPRESS_BY_DETAILS).should('have.text', suppressByFields.join(''));
getDetails(SUPPRESS_FOR_DETAILS).should('have.text', 'One rule execution');
getDetails(SUPPRESS_MISSING_FIELD).should(
'have.text',
'Suppress and group alerts for events with missing fields'
);
});
});
it('allows a rule with interval suppression to be created and displayed', () => {
fillAlertSuppressionFields(suppressByFields);
selectAlertSuppressionPerInterval();
setAlertSuppressionDuration(45, 'm');
selectDoNotSuppressForMissingFields();
continueFromDefineStep();
// ensures details preview works correctly
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(SUPPRESS_BY_DETAILS).should('have.text', suppressByFields.join(''));
getDetails(SUPPRESS_FOR_DETAILS).should('have.text', '45m');
getDetails(SUPPRESS_MISSING_FIELD).should(
'have.text',
'Do not suppress alerts for events with missing fields'
);
// suppression functionality should be under Tech Preview
cy.contains(DETAILS_TITLE, SUPPRESS_FOR_DETAILS).contains('Technical Preview');
});
fillAboutRuleMinimumAndContinue(mlRule);
skipScheduleRuleAction();
createRuleWithoutEnabling();
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(SUPPRESS_BY_DETAILS).should('have.text', suppressByFields.join(''));
getDetails(SUPPRESS_FOR_DETAILS).should('have.text', '45m');
getDetails(SUPPRESS_MISSING_FIELD).should(
'have.text',
'Do not suppress alerts for events with missing fields'
);
});
});
});
});
});
}
);

178
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_edit/machine_learning_rule.cy.ts Normal file
View file

@ -0,0 +1,178 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { getMachineLearningRule } from '../../../../objects/rule';
import {
ALERT_SUPPRESSION_DURATION_INPUT,
ALERT_SUPPRESSION_FIELDS,
ALERT_SUPPRESSION_MISSING_FIELDS_DO_NOT_SUPPRESS,
} from '../../../../screens/create_new_rule';
import {
DEFINITION_DETAILS,
DETAILS_TITLE,
SUPPRESS_BY_DETAILS,
SUPPRESS_FOR_DETAILS,
SUPPRESS_MISSING_FIELD,
} from '../../../../screens/rule_details';
import {
executeSetupModuleRequest,
forceStartDatafeeds,
forceStopAndCloseJob,
} from '../../../../support/machine_learning';
import { editFirstRule } from '../../../../tasks/alerts_detection_rules';
import { deleteAlertsAndRules } from '../../../../tasks/api_calls/common';
import { createRule } from '../../../../tasks/api_calls/rules';
import {
clearAlertSuppressionFields,
fillAlertSuppressionFields,
selectAlertSuppressionPerInterval,
selectAlertSuppressionPerRuleExecution,
setAlertSuppressionDuration,
} from '../../../../tasks/create_new_rule';
import { saveEditedRule } from '../../../../tasks/edit_rule';
import { login } from '../../../../tasks/login';
import { visit } from '../../../../tasks/navigation';
import { assertDetailsNotExist, getDetails } from '../../../../tasks/rule_details';
import { RULES_MANAGEMENT_URL } from '../../../../urls/rules_management';
describe(
'Machine Learning Detection Rules - Editing',
{
// Skipping in MKI as it depends on feature flag alertSuppressionForMachineLearningRuleEnabled
tags: ['@ess', '@serverless', '@skipInServerlessMKI'],
env: {
ftrConfig: {
kbnServerArgs: [
`--xpack.securitySolution.enableExperimental=${JSON.stringify([
'alertSuppressionForMachineLearningRuleEnabled',
])}`,
],
},
},
},
() => {
let mlRule: ReturnType<typeof getMachineLearningRule>;
const suppressByFields = ['by_field_name', 'by_field_value'];
const jobId = 'v3_linux_anomalous_network_activity';
before(() => {
const machineLearningJobIds = ([] as string[]).concat(
getMachineLearningRule().machine_learning_job_id
);
// ensure no ML jobs are started before the test
machineLearningJobIds.forEach((j) => forceStopAndCloseJob({ jobId: j }));
});
beforeEach(() => {
login();
deleteAlertsAndRules();
cy.task('esArchiverLoad', { archiveName: '../auditbeat/hosts', type: 'ftr' });
executeSetupModuleRequest({ moduleName: 'security_linux_v3' });
forceStartDatafeeds({ jobIds: [jobId] });
cy.task('esArchiverLoad', { archiveName: 'anomalies', type: 'ftr' });
});
describe('without Alert Suppression', () => {
beforeEach(() => {
mlRule = getMachineLearningRule({ machine_learning_job_id: [jobId] });
createRule(mlRule);
visit(RULES_MANAGEMENT_URL);
editFirstRule();
});
it('allows editing of a rule to add suppression configuration', () => {
fillAlertSuppressionFields(suppressByFields);
selectAlertSuppressionPerInterval();
setAlertSuppressionDuration(2, 'h');
saveEditedRule();
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(SUPPRESS_BY_DETAILS).should('have.text', suppressByFields.join(''));
getDetails(SUPPRESS_FOR_DETAILS).should('have.text', '2h');
getDetails(SUPPRESS_MISSING_FIELD).should(
'have.text',
'Suppress and group alerts for events with missing fields'
);
// suppression functionality should be under Tech Preview
cy.contains(DETAILS_TITLE, SUPPRESS_FOR_DETAILS).contains('Technical Preview');
});
});
});
describe('with Alert Suppression', () => {
beforeEach(() => {
mlRule = {
...getMachineLearningRule({ machine_learning_job_id: [jobId] }),
alert_suppression: {
group_by: suppressByFields,
duration: { value: 360, unit: 's' },
missing_fields_strategy: 'doNotSuppress',
},
};
createRule(mlRule);
visit(RULES_MANAGEMENT_URL);
editFirstRule();
});
it('allows editing of a rule to change its suppression configuration', () => {
// check saved suppression settings
cy.get(ALERT_SUPPRESSION_DURATION_INPUT)
.eq(0)
.should('be.enabled')
.should('have.value', 360);
cy.get(ALERT_SUPPRESSION_DURATION_INPUT)
.eq(1)
.should('be.enabled')
.should('have.value', 's');
cy.get(ALERT_SUPPRESSION_FIELDS).should('contain', suppressByFields.join(''));
cy.get(ALERT_SUPPRESSION_MISSING_FIELDS_DO_NOT_SUPPRESS).should('be.checked');
// set new duration first to overcome some flaky racing conditions during form save
setAlertSuppressionDuration(2, 'h');
selectAlertSuppressionPerRuleExecution();
saveEditedRule();
// check execution duration has changed
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(SUPPRESS_FOR_DETAILS).should('have.text', 'One rule execution');
});
});
it('allows editing of a rule to remove suppression configuration', () => {
// check saved suppression settings
cy.get(ALERT_SUPPRESSION_DURATION_INPUT)
.eq(0)
.should('be.enabled')
.should('have.value', 360);
cy.get(ALERT_SUPPRESSION_DURATION_INPUT)
.eq(1)
.should('be.enabled')
.should('have.value', 's');
cy.get(ALERT_SUPPRESSION_FIELDS).should('contain', suppressByFields.join(''));
cy.get(ALERT_SUPPRESSION_MISSING_FIELDS_DO_NOT_SUPPRESS).should('be.checked');
// set new duration first to overcome some flaky racing conditions during form save
setAlertSuppressionDuration(2, 'h');
clearAlertSuppressionFields();
saveEditedRule();
// check suppression is now absent
cy.get(DEFINITION_DETAILS).within(() => {
assertDetailsNotExist(SUPPRESS_FOR_DETAILS);
assertDetailsNotExist(SUPPRESS_BY_DETAILS);
});
});
});
}
);

28
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/prebuilt_rules_preview.cy.ts
View file

@ -220,9 +220,17 @@ describe('Detection rules, Prebuilt Rules Installation and Update workflow', ()
type: 'machine_learning',
anomaly_threshold: 65,
machine_learning_job_id: ['auth_high_count_logon_events', 'auth_high_count_logon_fails'],
alert_suppression: {
group_by: ['host.name'],
duration: { unit: 'm', value: 5 },
missing_fields_strategy: 'suppress',
},
}),
['security-rule.query', 'security-rule.language']
) as typeof CUSTOM_QUERY_INDEX_PATTERN_RULE;
) as Omit<
ReturnType<typeof createRuleAssetSavedObject>,
'security-rule.query' | 'security-rule.language'
>;
const THRESHOLD_RULE_INDEX_PATTERN = createRuleAssetSavedObject({
name: 'Threshold index pattern rule',
@ -500,24 +508,30 @@ describe('Detection rules, Prebuilt Rules Installation and Update workflow', ()
});
it('Machine learning rule properties', function () {
clickAddElasticRulesButton();
openRuleInstallPreview(MACHINE_LEARNING_RULE['security-rule'].name);
assertCommonPropertiesShown(commonProperties);
const {
name,
alert_suppression: alertSuppression,
anomaly_threshold: anomalyThreshold,
machine_learning_job_id: machineLearningJobIds,
} = MACHINE_LEARNING_RULE['security-rule'] as {
name: string;
anomaly_threshold: number;
machine_learning_job_id: string[];
alert_suppression: AlertSuppression;
};
clickAddElasticRulesButton();
openRuleInstallPreview(name);
assertCommonPropertiesShown(commonProperties);
assertMachineLearningPropertiesShown(
anomalyThreshold,
machineLearningJobIds,
this.mlModules
);
assertAlertSuppressionPropertiesShown(alertSuppression);
});
it('Threshold rule properties', () => {

64
x-pack/test/security_solution_cypress/cypress/support/machine_learning.ts
View file

@ -5,8 +5,72 @@
* 2.0.
*/
import { ML_GROUP_ID } from '@kbn/security-solution-plugin/common/constants';
import { rootRequest } from '../tasks/api_calls/common';
/**
*
* Calls the internal ML Module API to set up a module, which installs the jobs
* contained in that module.
* @param moduleName the name of the ML module to set up
* @returns the response from the setup module request
*/
export const executeSetupModuleRequest = ({ moduleName }: { moduleName: string }) =>
rootRequest({
headers: {
'elastic-api-version': 1,
},
method: 'POST',
url: `/internal/ml/modules/setup/${moduleName}`,
failOnStatusCode: true,
body: {
prefix: '',
groups: [ML_GROUP_ID],
indexPatternName: 'auditbeat-*',
startDatafeed: false,
useDedicatedIndex: true,
applyToAllSpaces: true,
},
});
/**
*
* Calls the internal ML Jobs API to force start the datafeeds for the given job IDs. Necessary to get them in the "started" state for the purposes of the detection engine
* @param jobIds the job IDs for which to force start datafeeds
* @returns the response from the force start datafeeds request
*/
export const forceStartDatafeeds = ({ jobIds }: { jobIds: string[] }) =>
rootRequest({
headers: {
'elastic-api-version': 1,
},
method: 'POST',
url: '/internal/ml/jobs/force_start_datafeeds',
failOnStatusCode: true,
body: {
datafeedIds: jobIds.map((jobId) => `datafeed-${jobId}`),
start: new Date().getUTCMilliseconds(),
},
});
/**
* Calls the internal ML Jobs API to stop the datafeeds for the given job IDs.
* @param jobIds the job IDs for which to stop datafeeds
* @returns the response from the stop datafeeds request
*/
export const stopDatafeeds = ({ jobIds }: { jobIds: string[] }) =>
rootRequest({
headers: {
'elastic-api-version': 1,
},
method: 'POST',
url: '/internal/ml/jobs/stop_datafeeds',
failOnStatusCode: true,
body: {
datafeedIds: jobIds.map((jobId) => `datafeed-${jobId}`),
},
});
/**
* Calls the internal ML Jobs API to force stop the datafeed of, and force close
* the job with the given ID.

1
x-pack/test/security_solution_cypress/serverless_config.ts
View file

@ -37,6 +37,7 @@ export default async function ({ readConfigFile }: FtrConfigProviderContext) {
`--xpack.securitySolution.enableExperimental=${JSON.stringify([
'alertSuppressionForEsqlRuleEnabled',
'bulkCustomHighlightedFieldsEnabled',
'alertSuppressionForMachineLearningRuleEnabled',
'manualRuleRunEnabled',
])}`,
],

18
x-pack/test_serverless/shared/lib/security/kibana_roles/project_controller_security_roles.yml
View file

@ -34,6 +34,7 @@ viewer:
- ".fleet-actions*"
- "risk-score.risk-score-*"
- ".asset-criticality.asset-criticality-*"
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -100,6 +101,10 @@ editor:
- "read"
- "write"
allow_restricted_indices: false
- names:
- ".ml-anomalies-*"
privileges:
- read
applications:
- application: "kibana-.kibana"
privileges:
@ -155,6 +160,7 @@ t1_analyst:
- ".fleet-actions*"
- risk-score.risk-score-*
- .asset-criticality.asset-criticality-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -203,6 +209,7 @@ t2_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
- names:
@ -265,6 +272,7 @@ t3_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -330,6 +338,7 @@ threat_intelligence_analyst:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -389,6 +398,7 @@ rule_author:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -454,6 +464,7 @@ soc_manager:
- .fleet-agents*
- .fleet-actions*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
applications:
@ -515,6 +526,7 @@ detections_admin:
- metrics-endpoint.metadata_current_*
- .fleet-agents*
- .fleet-actions*
- ".ml-anomalies-*"
privileges:
- read
- names:
@ -573,6 +585,10 @@ platform_engineer:
privileges:
- read
- write
- names:
- ".ml-anomalies-*"
privileges:
- read
applications:
- application: "kibana-.kibana"
privileges:
@ -624,6 +640,7 @@ endpoint_operations_analyst:
- .lists*
- .items*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
- names:
@ -692,6 +709,7 @@ endpoint_policy_manager:
- packetbeat-*
- winlogbeat-*
- risk-score.risk-score-*
- ".ml-anomalies-*"
privileges:
- read
- names: