mirror of
https://github.com/elastic/kibana.git
synced 2025-04-23 17:28:26 -04:00
[Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450)
## Summary - related [epic](https://github.com/elastic/security-team/issues/6196) - introduces new ES|QL rule type in Technical Preview Stage - historical POC architecture [document](https://docs.google.com/document/d/1hcKzNrDEIrmoWwWoqas1YZ-bd8Kk5NRjJNSUaCvSntM/edit#heading=h.gheuu8zcz481)(internal link). Some of the information there can be outdated, but might be useful for historical context of some tech decision. In future, detailed technical documentation will be added ### UI ES|QL query component introduced in rule edit/creation form Rule name override supports values returned from ES|QL query As agreed on Adv. correlation WG, we don't introduce similar possibility for risk score/severity override at this point <details> <summary>How it looks like in UI</summary> <img width="2082" alt="Screenshot 2023-09-21 at 11 52 59" src="14c94e36
-ca90-496d-a7a5-4a31899d25b6"> <img width="2079" alt="Screenshot 2023-09-21 at 11 53 14" src="9abd53ec
-a0f4-4481-8b1f-4ecccdc5feae"> <img width="2072" alt="Screenshot 2023-09-21 at 12 14 17" src="58e4f9eb
-c15f-4849-bba0-bc1b92e8c945"> </details> ### Context We introduced concept of Aggregating and Non-aggregating rules for ES|QL. It depends on, whether STATS..BY command used in query **Aggregating rule** - is a rule that uses [stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html) grouping commands. So, its result can not be matched to a particular document in ES. This can lead to possibly duplicated alerts, since we are using document `id` to deduplicate alerts. We are going to introduce suppression for all rule types in future, that would help to mitigate this case ``` FROM logs* | STATS count = COUNT(host.name) BY host.name | SORT host.name ``` **Non-aggregating rule** - is a rule that does not use [stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html) grouping commands. Each row in result can be tracked to a source document in ES. For this type of rule operator \`[metadata _id, _index, _version]\` is required to be used after defining index source. This would allow deduplicate alerts and link them with the source document. ``` FROM logs* [metadata _id, _index, _version] | WHERE event.id == "test" | LIMIT 10 ``` ### Serverless Feature Flag ES|QL won't be available for Serverless as for 8.11 release, so it will be hidden by Security experimental feature flag `esqlRulesDisabled`. All UI changes will be hidden (it's mostly Form creation) and rule type won't be registered, which prevents rule to be created, returned in search if it exists or execute. ### Test envs - Serverless qa, [admin link to project](https://admin.qa.cld.elstc.co/projects/security/ef79684f92d64f27b69e1b04de86eb1a), disabled there - internal [link](https://elastic.slack.com/archives/C03E8TR26HE/p1693848029955229) to test env for Stateful ### Rule schema changes introduces value `esql` to `type` property introduces value `esql` to `language` property ### Tests coverage - cypress tests (as per 27/09/2023 added cypress tests for rule creation/edit/details,bulk_edit)) - functional tests for rule execution(exceptions, overrides, preview and actual rule execution) - functional tests for bulk_edit #### Flaky test runner - [cypress esql tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3233#_), non failed of added ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
parent
d35fa69138
commit
b03b2fd477
125 changed files with 3889 additions and 112 deletions
1
.github/CODEOWNERS
vendored
1
.github/CODEOWNERS
vendored
|
@ -1274,6 +1274,7 @@ x-pack/plugins/cloud_integrations/cloud_full_story/server/config.ts @elastic/kib
|
|||
/x-pack/plugins/security_solution/common/risk_engine @elastic/security-detection-engine
|
||||
|
||||
/x-pack/plugins/security_solution/public/common/components/sourcerer @elastic/security-detection-engine
|
||||
/x-pack/plugins/security_solution/public/detection_engine/rule_creation @elastic/security-detection-engine
|
||||
/x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui @elastic/security-detection-engine
|
||||
/x-pack/plugins/security_solution/public/detection_engine/rule_exceptions @elastic/security-detection-engine
|
||||
/x-pack/plugins/security_solution/public/detections/containers/detection_engine/lists @elastic/security-detection-engine
|
||||
|
|
|
@ -36,6 +36,7 @@ xpack.fleet.internal.registry.spec.max: '3.0'
|
|||
# Serverless security specific options
|
||||
xpack.securitySolution.enableExperimental:
|
||||
- discoverInTimeline
|
||||
- esqlRulesDisabled
|
||||
|
||||
xpack.ml.ad.enabled: true
|
||||
xpack.ml.dfa.enabled: true
|
||||
|
|
|
@ -843,6 +843,9 @@ export const getDocLinks = ({ kibanaBranch }: GetDocLinkOptions): DocLinks => {
|
|||
synthetics: {
|
||||
featureRoles: `${ELASTIC_WEBSITE_URL}guide/en/observability/${DOC_LINK_VERSION}/synthetics-feature-roles.html`,
|
||||
},
|
||||
esql: {
|
||||
statsBy: `${ELASTICSEARCH_DOCS}esql-stats-by.html`,
|
||||
},
|
||||
telemetry: {
|
||||
settings: `${KIBANA_DOCS}telemetry-settings-kbn.html`,
|
||||
},
|
||||
|
|
|
@ -601,6 +601,9 @@ export interface DocLinks {
|
|||
readonly synthetics: {
|
||||
readonly featureRoles: string;
|
||||
};
|
||||
readonly esql: {
|
||||
readonly statsBy: string;
|
||||
};
|
||||
readonly telemetry: {
|
||||
readonly settings: string;
|
||||
};
|
||||
|
|
|
@ -8,7 +8,7 @@
|
|||
|
||||
import * as t from 'io-ts';
|
||||
|
||||
export const language = t.keyof({ eql: null, kuery: null, lucene: null });
|
||||
export const language = t.keyof({ eql: null, kuery: null, lucene: null, esql: null });
|
||||
export type Language = t.TypeOf<typeof language>;
|
||||
|
||||
export const languageOrUndefined = t.union([language, t.undefined]);
|
||||
|
|
|
@ -16,6 +16,7 @@ export const type = t.keyof({
|
|||
threshold: null,
|
||||
threat_match: null,
|
||||
new_terms: null,
|
||||
esql: null,
|
||||
});
|
||||
export type Type = t.TypeOf<typeof type>;
|
||||
|
||||
|
|
|
@ -16,6 +16,7 @@ export const SIGNALS_ID = `siem.signals` as const;
|
|||
*/
|
||||
const RULE_TYPE_PREFIX = `siem` as const;
|
||||
export const EQL_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.eqlRule` as const;
|
||||
export const ESQL_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.esqlRule` as const;
|
||||
export const INDICATOR_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.indicatorRule` as const;
|
||||
export const ML_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.mlRule` as const;
|
||||
export const QUERY_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.queryRule` as const;
|
||||
|
|
|
@ -8,6 +8,7 @@
|
|||
|
||||
import {
|
||||
EQL_RULE_TYPE_ID,
|
||||
ESQL_RULE_TYPE_ID,
|
||||
INDICATOR_RULE_TYPE_ID,
|
||||
ML_RULE_TYPE_ID,
|
||||
NEW_TERMS_RULE_TYPE_ID,
|
||||
|
@ -21,6 +22,7 @@ import {
|
|||
*/
|
||||
export const ruleTypeMappings = {
|
||||
eql: EQL_RULE_TYPE_ID,
|
||||
esql: ESQL_RULE_TYPE_ID,
|
||||
machine_learning: ML_RULE_TYPE_ID,
|
||||
query: QUERY_RULE_TYPE_ID,
|
||||
saved_query: SAVED_QUERY_RULE_TYPE_ID,
|
||||
|
|
|
@ -9,3 +9,4 @@
|
|||
export * from './src/add_remove_id_to_item';
|
||||
export * from './src/transform_data_to_ndjson';
|
||||
export * from './src/path_validations';
|
||||
export * from './src/esql';
|
||||
|
|
|
@ -0,0 +1,65 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import { computeIsESQLQueryAggregating } from './compute_if_esql_query_aggregating';
|
||||
|
||||
describe('computeIsESQLQueryAggregating', () => {
|
||||
describe('Aggregating query', () => {
|
||||
it('should detect aggregating with STATS only', () => {
|
||||
expect(computeIsESQLQueryAggregating('from packetbeat* | stats count(agent.name)')).toBe(
|
||||
true
|
||||
);
|
||||
});
|
||||
it('should detect aggregating with STATS..BY', () => {
|
||||
expect(
|
||||
computeIsESQLQueryAggregating('from packetbeat* | stats count(agent.name) by agent.name')
|
||||
).toBe(true);
|
||||
});
|
||||
it('should detect aggregating with multiple spaces', () => {
|
||||
expect(
|
||||
computeIsESQLQueryAggregating(
|
||||
'from packetbeat* | stats count(agent.name) by agent.name'
|
||||
)
|
||||
).toBe(true);
|
||||
});
|
||||
it('should detect aggregating case agnostic', () => {
|
||||
expect(
|
||||
computeIsESQLQueryAggregating('from packetbeat* | STATS count(agent.name) BY agent.name')
|
||||
).toBe(true);
|
||||
|
||||
expect(computeIsESQLQueryAggregating('from packetbeat* | STATS count(agent.name)')).toBe(
|
||||
true
|
||||
);
|
||||
});
|
||||
|
||||
it('should detect aggregating for multiple aggregation', () => {
|
||||
expect(
|
||||
computeIsESQLQueryAggregating(
|
||||
'from packetbeat* | stats count(agent.name) by agent.name | stats distinct=count_distinct(agent.name)'
|
||||
)
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
||||
describe('Non-aggregating query', () => {
|
||||
it('should detect non-aggregating', () => {
|
||||
expect(computeIsESQLQueryAggregating('from packetbeat* | keep agent.*')).toBe(false);
|
||||
});
|
||||
|
||||
it('should detect non-aggregating if word stats is part of variable name', () => {
|
||||
expect(
|
||||
computeIsESQLQueryAggregating('from packetbeat* | keep agent.some_stats | limit 10')
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('should detect non-aggregating if word stats is part of a path', () => {
|
||||
expect(computeIsESQLQueryAggregating('from packetbeat* | keep agent.stats | limit 10')).toBe(
|
||||
false
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
|
@ -0,0 +1,16 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
/**
|
||||
* compute if esqlQuery is aggregating/grouping, i.e. using STATS...BY command
|
||||
* @param esqlQuery
|
||||
* @returns boolean
|
||||
*/
|
||||
export const computeIsESQLQueryAggregating = (esqlQuery: string): boolean => {
|
||||
return /\|\s+stats\s/i.test(esqlQuery);
|
||||
};
|
|
@ -0,0 +1,37 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
import { getIndexListFromEsqlQuery } from './get_index_list_from_esql_query';
|
||||
import { getIndexPatternFromESQLQuery } from '@kbn/es-query';
|
||||
|
||||
jest.mock('@kbn/es-query', () => {
|
||||
return {
|
||||
getIndexPatternFromESQLQuery: jest.fn(),
|
||||
};
|
||||
});
|
||||
|
||||
const getIndexPatternFromESQLQueryMock = getIndexPatternFromESQLQuery as jest.Mock;
|
||||
|
||||
describe('getIndexListFromEsqlQuery', () => {
|
||||
it('should return empty array if index string is empty', () => {
|
||||
getIndexPatternFromESQLQueryMock.mockReturnValue('');
|
||||
expect(getIndexListFromEsqlQuery('')).toEqual([]);
|
||||
});
|
||||
it('should return single item array if one index present', () => {
|
||||
getIndexPatternFromESQLQueryMock.mockReturnValue('test-1');
|
||||
expect(getIndexListFromEsqlQuery('From test-1')).toEqual(['test-1']);
|
||||
});
|
||||
it('should return array if index string has multiple indices', () => {
|
||||
getIndexPatternFromESQLQueryMock.mockReturnValue('test-1,test-2');
|
||||
expect(getIndexListFromEsqlQuery('From test-1,test-2')).toEqual(['test-1', 'test-2']);
|
||||
});
|
||||
it('should trim spaces in index names', () => {
|
||||
getIndexPatternFromESQLQueryMock.mockReturnValue('test-1 , test-2 ');
|
||||
expect(getIndexListFromEsqlQuery('From test-1, test-2 ')).toEqual(['test-1', 'test-2']);
|
||||
});
|
||||
});
|
|
@ -0,0 +1,28 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
import { getIndexPatternFromESQLQuery } from '@kbn/es-query';
|
||||
|
||||
/**
|
||||
* parses ES|QL query and returns array of indices
|
||||
*/
|
||||
export const getIndexListFromEsqlQuery = (query: string | undefined): string[] => {
|
||||
const indexString = getIndexPatternFromESQLQuery(query);
|
||||
|
||||
return getIndexListFromIndexString(indexString);
|
||||
};
|
||||
|
||||
/**
|
||||
* transforms sting of indices, separated by commas to array
|
||||
* index*, index2* => [index*, index2*]
|
||||
*/
|
||||
export const getIndexListFromIndexString = (indexString: string | undefined): string[] => {
|
||||
if (!indexString) {
|
||||
return [];
|
||||
}
|
||||
return indexString.split(',').map((index) => index.trim());
|
||||
};
|
10
packages/kbn-securitysolution-utils/src/esql/index.ts
Normal file
10
packages/kbn-securitysolution-utils/src/esql/index.ts
Normal file
|
@ -0,0 +1,10 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0 and the Server Side Public License, v 1; you may not use this file except
|
||||
* in compliance with, at your election, the Elastic License 2.0 or the Server
|
||||
* Side Public License, v 1.
|
||||
*/
|
||||
|
||||
export * from './compute_if_esql_query_aggregating';
|
||||
export * from './get_index_list_from_esql_query';
|
|
@ -11,7 +11,8 @@
|
|||
"**/*.ts"
|
||||
],
|
||||
"kbn_references": [
|
||||
"@kbn/i18n"
|
||||
"@kbn/i18n",
|
||||
"@kbn/es-query"
|
||||
],
|
||||
"exclude": [
|
||||
"target/**/*",
|
||||
|
|
|
@ -10,6 +10,7 @@ import { i18n } from '@kbn/i18n';
|
|||
import { DEFAULT_APP_CATEGORIES } from '@kbn/core-application-common';
|
||||
import {
|
||||
EQL_RULE_TYPE_ID,
|
||||
ESQL_RULE_TYPE_ID,
|
||||
INDICATOR_RULE_TYPE_ID,
|
||||
ML_RULE_TYPE_ID,
|
||||
NEW_TERMS_RULE_TYPE_ID,
|
||||
|
@ -29,6 +30,7 @@ import type { SecurityFeatureParams } from './types';
|
|||
|
||||
const SECURITY_RULE_TYPES = [
|
||||
LEGACY_NOTIFICATIONS_ID,
|
||||
ESQL_RULE_TYPE_ID,
|
||||
EQL_RULE_TYPE_ID,
|
||||
INDICATOR_RULE_TYPE_ID,
|
||||
ML_RULE_TYPE_ID,
|
||||
|
|
|
@ -11,6 +11,7 @@ import type {
|
|||
MachineLearningRuleUpdateProps,
|
||||
QueryRuleCreateProps,
|
||||
QueryRuleUpdateProps,
|
||||
EsqlRuleCreateProps,
|
||||
SavedQueryRuleCreateProps,
|
||||
ThreatMatchRuleCreateProps,
|
||||
ThresholdRuleCreateProps,
|
||||
|
@ -152,6 +153,23 @@ export const getCreateNewTermsRulesSchemaMock = (
|
|||
history_window_start: 'now-7d',
|
||||
});
|
||||
|
||||
export const getCreateEsqlRulesSchemaMock = (
|
||||
ruleId = 'rule-1',
|
||||
enabled = false
|
||||
): EsqlRuleCreateProps => ({
|
||||
description: 'Detecting root and admin users',
|
||||
enabled,
|
||||
name: 'Query with a rule id',
|
||||
query: 'from auditbeat-* | where user.name=="root" or user.name=="admin"',
|
||||
severity: 'high',
|
||||
type: 'esql',
|
||||
risk_score: 55,
|
||||
language: 'esql',
|
||||
rule_id: ruleId,
|
||||
interval: '5m',
|
||||
from: 'now-6m',
|
||||
});
|
||||
|
||||
export const getUpdateRulesSchemaMock = (
|
||||
id = '04128c15-0d1b-4716-a4c5-46997ac7f3bd'
|
||||
): QueryRuleUpdateProps => ({
|
||||
|
|
|
@ -20,6 +20,7 @@ import {
|
|||
getCreateThresholdRulesSchemaMock,
|
||||
getCreateRulesSchemaMockWithDataView,
|
||||
getCreateMachineLearningRulesSchemaMock,
|
||||
getCreateEsqlRulesSchemaMock,
|
||||
} from './rule_request_schema.mock';
|
||||
import { buildResponseRuleSchema } from './build_rule_schemas';
|
||||
|
||||
|
@ -1206,6 +1207,38 @@ describe('rules schema', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('esql rule type', () => {
|
||||
it('should validate correct payload', () => {
|
||||
const payload = getCreateEsqlRulesSchemaMock();
|
||||
const decoded = RuleCreateProps.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
expect(getPaths(left(message.errors))).toEqual([]);
|
||||
expect(message.schema).toEqual(payload);
|
||||
});
|
||||
it('should not validate index property', () => {
|
||||
const payload = { ...getCreateEsqlRulesSchemaMock(), index: ['test*'] };
|
||||
const decoded = RuleCreateProps.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "index,["test*"]"']);
|
||||
});
|
||||
it('should not validate data_view_id property', () => {
|
||||
const payload = { ...getCreateEsqlRulesSchemaMock(), data_view_id: 'test' };
|
||||
const decoded = RuleCreateProps.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "data_view_id"']);
|
||||
});
|
||||
it('should not validate filters property', () => {
|
||||
const payload = { ...getCreateEsqlRulesSchemaMock(), filters: [] };
|
||||
const decoded = RuleCreateProps.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "filters,[]"']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('data_view_id', () => {
|
||||
test('validates when "data_view_id" and index are defined', () => {
|
||||
const payload = { ...getCreateRulesSchemaMockWithDataView(), index: ['auditbeat-*'] };
|
||||
|
|
|
@ -8,6 +8,7 @@
|
|||
import { DEFAULT_INDICATOR_SOURCE_PATH } from '../../../../constants';
|
||||
import type {
|
||||
EqlRule,
|
||||
EsqlRule,
|
||||
MachineLearningRule,
|
||||
QueryRule,
|
||||
SavedQueryRule,
|
||||
|
@ -103,6 +104,15 @@ export const getRulesMlSchemaMock = (anchorDate: string = ANCHOR_DATE): MachineL
|
|||
};
|
||||
};
|
||||
|
||||
export const getEsqlRuleSchemaMock = (anchorDate: string = ANCHOR_DATE): EsqlRule => {
|
||||
return {
|
||||
...getResponseBaseParams(anchorDate),
|
||||
query: 'from auditbeat* | limit 10',
|
||||
type: 'esql',
|
||||
language: 'esql',
|
||||
};
|
||||
};
|
||||
|
||||
export const getThreatMatchingSchemaMock = (anchorDate: string = ANCHOR_DATE): ThreatMatchRule => {
|
||||
return {
|
||||
...getResponseBaseParams(anchorDate),
|
||||
|
|
|
@ -16,6 +16,7 @@ import {
|
|||
getSavedQuerySchemaMock,
|
||||
getThreatMatchingSchemaMock,
|
||||
getRulesEqlSchemaMock,
|
||||
getEsqlRuleSchemaMock,
|
||||
} from './rule_response_schema.mock';
|
||||
|
||||
describe('Rule response schema', () => {
|
||||
|
@ -167,6 +168,41 @@ describe('Rule response schema', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('esql rule type', () => {
|
||||
test('it should NOT validate a type of "esql" with "index" defined', () => {
|
||||
const payload = { ...getEsqlRuleSchemaMock(), index: ['logs-*'] };
|
||||
|
||||
const decoded = RuleResponse.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "index,["logs-*"]"']);
|
||||
expect(message.schema).toEqual({});
|
||||
});
|
||||
|
||||
test('it should NOT validate a type of "esql" with "filters" defined', () => {
|
||||
const payload = { ...getEsqlRuleSchemaMock(), filters: [] };
|
||||
|
||||
const decoded = RuleResponse.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "filters,[]"']);
|
||||
expect(message.schema).toEqual({});
|
||||
});
|
||||
|
||||
test('it should NOT validate a type of "esql" with a "saved_id" dependent', () => {
|
||||
const payload = { ...getEsqlRuleSchemaMock(), saved_id: 'id' };
|
||||
|
||||
const decoded = RuleResponse.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "saved_id"']);
|
||||
expect(message.schema).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe('data_view_id', () => {
|
||||
test('it should validate a type of "query" with "data_view_id" defined', () => {
|
||||
const payload = { ...getRulesSchemaMock(), data_view_id: 'logs-*' };
|
||||
|
@ -231,6 +267,17 @@ describe('Rule response schema', () => {
|
|||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "data_view_id"']);
|
||||
expect(message.schema).toEqual({});
|
||||
});
|
||||
|
||||
test('it should NOT validate a type of "esql" with "data_view_id" defined', () => {
|
||||
const payload = { ...getEsqlRuleSchemaMock(), data_view_id: 'logs-*' };
|
||||
|
||||
const decoded = RuleResponse.decode(payload);
|
||||
const checked = exactCheck(payload, decoded);
|
||||
const message = pipe(checked, foldLeftRight);
|
||||
|
||||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "data_view_id"']);
|
||||
expect(message.schema).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe('investigation_fields', () => {
|
||||
|
|
|
@ -213,6 +213,7 @@ export enum QueryLanguage {
|
|||
'kuery' = 'kuery',
|
||||
'lucene' = 'lucene',
|
||||
'eql' = 'eql',
|
||||
'esql' = 'esql',
|
||||
}
|
||||
|
||||
export type KqlQueryLanguage = t.TypeOf<typeof KqlQueryLanguage>;
|
||||
|
@ -253,6 +254,37 @@ export const EqlRulePatchProps = t.intersection([SharedPatchProps, eqlSchema.pat
|
|||
export type EqlPatchParams = t.TypeOf<typeof EqlPatchParams>;
|
||||
export const EqlPatchParams = eqlSchema.patch;
|
||||
|
||||
// -------------------------------------------------------------------------------------------------
|
||||
// ES|QL rule schema
|
||||
|
||||
export type EsqlQueryLanguage = t.TypeOf<typeof EsqlQueryLanguage>;
|
||||
export const EsqlQueryLanguage = t.literal('esql');
|
||||
|
||||
const esqlSchema = buildRuleSchemas({
|
||||
required: {
|
||||
type: t.literal('esql'),
|
||||
language: EsqlQueryLanguage,
|
||||
query: RuleQuery,
|
||||
},
|
||||
optional: {},
|
||||
defaultable: {},
|
||||
});
|
||||
|
||||
export type EsqlRule = t.TypeOf<typeof EsqlRule>;
|
||||
export const EsqlRule = t.intersection([SharedResponseProps, esqlSchema.response]);
|
||||
|
||||
export type EsqlRuleCreateProps = t.TypeOf<typeof EsqlRuleCreateProps>;
|
||||
export const EsqlRuleCreateProps = t.intersection([SharedCreateProps, esqlSchema.create]);
|
||||
|
||||
export type EsqlRuleUpdateProps = t.TypeOf<typeof EsqlRuleUpdateProps>;
|
||||
export const EsqlRuleUpdateProps = t.intersection([SharedUpdateProps, esqlSchema.create]);
|
||||
|
||||
export type EsqlRulePatchProps = t.TypeOf<typeof EsqlRulePatchProps>;
|
||||
export const EsqlRulePatchProps = t.intersection([SharedPatchProps, esqlSchema.patch]);
|
||||
|
||||
export type EsqlPatchParams = t.TypeOf<typeof EsqlPatchParams>;
|
||||
export const EsqlPatchParams = esqlSchema.patch;
|
||||
|
||||
// -------------------------------------------------------------------------------------------------
|
||||
// Indicator Match rule schema
|
||||
|
||||
|
@ -500,6 +532,7 @@ export const NewTermsPatchParams = newTermsSchema.patch;
|
|||
export type TypeSpecificCreateProps = t.TypeOf<typeof TypeSpecificCreateProps>;
|
||||
export const TypeSpecificCreateProps = t.union([
|
||||
eqlSchema.create,
|
||||
esqlSchema.create,
|
||||
threatMatchSchema.create,
|
||||
querySchema.create,
|
||||
savedQuerySchema.create,
|
||||
|
@ -511,6 +544,7 @@ export const TypeSpecificCreateProps = t.union([
|
|||
export type TypeSpecificPatchProps = t.TypeOf<typeof TypeSpecificPatchProps>;
|
||||
export const TypeSpecificPatchProps = t.union([
|
||||
eqlSchema.patch,
|
||||
esqlSchema.patch,
|
||||
threatMatchSchema.patch,
|
||||
querySchema.patch,
|
||||
savedQuerySchema.patch,
|
||||
|
@ -522,6 +556,7 @@ export const TypeSpecificPatchProps = t.union([
|
|||
export type TypeSpecificResponse = t.TypeOf<typeof TypeSpecificResponse>;
|
||||
export const TypeSpecificResponse = t.union([
|
||||
eqlSchema.response,
|
||||
esqlSchema.response,
|
||||
threatMatchSchema.response,
|
||||
querySchema.response,
|
||||
savedQuerySchema.response,
|
||||
|
|
|
@ -87,6 +87,14 @@ export const RuleEqlQuery = t.exact(
|
|||
})
|
||||
);
|
||||
|
||||
export type RuleEsqlQuery = t.TypeOf<typeof RuleEsqlQuery>;
|
||||
export const RuleEsqlQuery = t.exact(
|
||||
t.type({
|
||||
query: RuleQuery,
|
||||
language: t.literal('esql'),
|
||||
})
|
||||
);
|
||||
|
||||
// -------------------------------------------------------------------------------------------------
|
||||
// Rule schedule
|
||||
|
||||
|
|
|
@ -53,6 +53,7 @@ import {
|
|||
import {
|
||||
BuildingBlockObject,
|
||||
RuleEqlQuery,
|
||||
RuleEsqlQuery,
|
||||
InlineKqlQuery,
|
||||
RuleKqlQuery,
|
||||
RuleDataSource,
|
||||
|
@ -147,6 +148,17 @@ export const DiffableEqlFields = buildSchema({
|
|||
},
|
||||
});
|
||||
|
||||
export type DiffableEsqlFields = t.TypeOf<typeof DiffableEsqlFields>;
|
||||
export const DiffableEsqlFields = buildSchema({
|
||||
required: {
|
||||
type: t.literal('esql'),
|
||||
esql_query: RuleEsqlQuery, // NOTE: new field
|
||||
},
|
||||
// this is a new type of rule, no prebuilt rules created yet.
|
||||
// new properties might be added here during further rule type development
|
||||
optional: {},
|
||||
});
|
||||
|
||||
export type DiffableThreatMatchFields = t.TypeOf<typeof DiffableThreatMatchFields>;
|
||||
export const DiffableThreatMatchFields = buildSchema({
|
||||
required: {
|
||||
|
@ -232,6 +244,7 @@ export const DiffableRule = t.intersection([
|
|||
DiffableCustomQueryFields,
|
||||
DiffableSavedQueryFields,
|
||||
DiffableEqlFields,
|
||||
DiffableEsqlFields,
|
||||
DiffableThreatMatchFields,
|
||||
DiffableThresholdFields,
|
||||
DiffableMachineLearningFields,
|
||||
|
|
|
@ -10,6 +10,7 @@ import type {
|
|||
DiffableCommonFields,
|
||||
DiffableCustomQueryFields,
|
||||
DiffableEqlFields,
|
||||
DiffableEsqlFields,
|
||||
DiffableMachineLearningFields,
|
||||
DiffableNewTermsFields,
|
||||
DiffableSavedQueryFields,
|
||||
|
@ -24,6 +25,7 @@ export type CommonFieldsDiff = FieldsDiff<DiffableCommonFields>;
|
|||
export type CustomQueryFieldsDiff = FieldsDiff<DiffableCustomQueryFields>;
|
||||
export type SavedQueryFieldsDiff = FieldsDiff<DiffableSavedQueryFields>;
|
||||
export type EqlFieldsDiff = FieldsDiff<DiffableEqlFields>;
|
||||
export type EsqlFieldsDiff = FieldsDiff<DiffableEsqlFields>;
|
||||
export type ThreatMatchFieldsDiff = FieldsDiff<DiffableThreatMatchFields>;
|
||||
export type ThresholdFieldsDiff = FieldsDiff<DiffableThresholdFields>;
|
||||
export type MachineLearningFieldsDiff = FieldsDiff<DiffableMachineLearningFields>;
|
||||
|
@ -45,6 +47,7 @@ export type RuleFieldsDiff = CommonFieldsDiff &
|
|||
| CustomQueryFieldsDiff
|
||||
| SavedQueryFieldsDiff
|
||||
| EqlFieldsDiff
|
||||
| EsqlFieldsDiff
|
||||
| ThreatMatchFieldsDiff
|
||||
| ThresholdFieldsDiff
|
||||
| MachineLearningFieldsDiff
|
||||
|
|
|
@ -444,6 +444,7 @@ export enum BulkActionsDryRunErrCode {
|
|||
IMMUTABLE = 'IMMUTABLE',
|
||||
MACHINE_LEARNING_AUTH = 'MACHINE_LEARNING_AUTH',
|
||||
MACHINE_LEARNING_INDEX_PATTERN = 'MACHINE_LEARNING_INDEX_PATTERN',
|
||||
ESQL_INDEX_PATTERN = 'ESQL_INDEX_PATTERN',
|
||||
}
|
||||
|
||||
export const RISKY_HOSTS_DOC_LINK =
|
||||
|
|
|
@ -12,6 +12,7 @@ import {
|
|||
normalizeMachineLearningJobIds,
|
||||
normalizeThresholdField,
|
||||
isMlRule,
|
||||
isEsqlRule,
|
||||
} from './utils';
|
||||
|
||||
import { hasLargeValueList } from '@kbn/securitysolution-list-utils';
|
||||
|
@ -135,6 +136,16 @@ describe('isMlRule', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('isEsqlRule', () => {
|
||||
test('it returns true if a ES|QL rule', () => {
|
||||
expect(isEsqlRule('esql')).toEqual(true);
|
||||
});
|
||||
|
||||
test('it returns false if not a ES|QL rule', () => {
|
||||
expect(isEsqlRule('query')).toEqual(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#hasEqlSequenceQuery', () => {
|
||||
describe('when a non-sequence query is passed', () => {
|
||||
const query = 'process where process.name == "regsvr32.exe"';
|
||||
|
|
|
@ -46,6 +46,7 @@ export const isThreatMatchRule = (ruleType: Type | undefined): boolean =>
|
|||
ruleType === 'threat_match';
|
||||
export const isMlRule = (ruleType: Type | undefined): boolean => ruleType === 'machine_learning';
|
||||
export const isNewTermsRule = (ruleType: Type | undefined): boolean => ruleType === 'new_terms';
|
||||
export const isEsqlRule = (ruleType: Type | undefined): boolean => ruleType === 'esql';
|
||||
|
||||
export const normalizeThresholdField = (
|
||||
thresholdField: string | string[] | null | undefined
|
||||
|
|
|
@ -110,6 +110,11 @@ export const allowedExperimentalValues = Object.freeze({
|
|||
* */
|
||||
discoverInTimeline: false,
|
||||
|
||||
/**
|
||||
* disables ES|QL rules
|
||||
*/
|
||||
esqlRulesDisabled: false,
|
||||
|
||||
/**
|
||||
* Enables Protection Updates tab in the Endpoint Policy Details page
|
||||
*/
|
||||
|
|
|
@ -47,6 +47,8 @@
|
|||
"files",
|
||||
"controls",
|
||||
"dataViewEditor",
|
||||
"savedObjectsManagement",
|
||||
"expressions",
|
||||
"stackConnectors",
|
||||
"discover",
|
||||
"notifications",
|
||||
|
|
|
@ -0,0 +1,30 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
import type { Query, AggregateQuery } from '@kbn/es-query';
|
||||
|
||||
/**
|
||||
* converts AggregateQuery type to Query
|
||||
* it needed because unified search bar emits 2 types of queries: Query and AggregateQuery
|
||||
* on security side we deal with one type only (Query), so we converge it to this type only
|
||||
*/
|
||||
export const convertToQueryType = (query: Query | AggregateQuery): Query => {
|
||||
if ('esql' in query) {
|
||||
return {
|
||||
query: query.esql,
|
||||
language: 'esql',
|
||||
};
|
||||
}
|
||||
|
||||
if ('sql' in query) {
|
||||
return {
|
||||
query: query.sql,
|
||||
language: 'sql',
|
||||
};
|
||||
}
|
||||
|
||||
return query;
|
||||
};
|
|
@ -8,15 +8,15 @@
|
|||
import React, { memo, useMemo, useCallback, useState, useEffect } from 'react';
|
||||
import deepEqual from 'fast-deep-equal';
|
||||
|
||||
import type { DataViewBase, Filter, Query, TimeRange } from '@kbn/es-query';
|
||||
import type { DataViewBase, Filter, Query, AggregateQuery, TimeRange } from '@kbn/es-query';
|
||||
import type { FilterManager, SavedQuery, SavedQueryTimeFilter } from '@kbn/data-plugin/public';
|
||||
import { TimeHistory } from '@kbn/data-plugin/public';
|
||||
import type { DataView } from '@kbn/data-views-plugin/public';
|
||||
import type { SearchBarProps } from '@kbn/unified-search-plugin/public';
|
||||
import { SearchBar } from '@kbn/unified-search-plugin/public';
|
||||
import { Storage } from '@kbn/kibana-utils-plugin/public';
|
||||
|
||||
import { useKibana } from '../../lib/kibana';
|
||||
import { convertToQueryType } from './convert_to_query_type';
|
||||
|
||||
export interface QueryBarComponentProps {
|
||||
dataTestSubj?: string;
|
||||
|
@ -64,18 +64,21 @@ export const QueryBar = memo<QueryBarComponentProps>(
|
|||
const { data } = useKibana().services;
|
||||
const [dataView, setDataView] = useState<DataView>();
|
||||
const onQuerySubmit = useCallback(
|
||||
(payload: { dateRange: TimeRange; query?: Query }) => {
|
||||
(payload: { dateRange: TimeRange; query?: Query | AggregateQuery }) => {
|
||||
if (payload.query != null && !deepEqual(payload.query, filterQuery)) {
|
||||
onSubmitQuery(payload.query);
|
||||
const payloadQuery = convertToQueryType(payload.query);
|
||||
|
||||
onSubmitQuery(payloadQuery);
|
||||
}
|
||||
},
|
||||
[filterQuery, onSubmitQuery]
|
||||
);
|
||||
|
||||
const onQueryChange = useCallback(
|
||||
(payload: { dateRange: TimeRange; query?: Query }) => {
|
||||
(payload: { dateRange: TimeRange; query?: Query | AggregateQuery }) => {
|
||||
if (onChangedQuery && payload.query != null && !deepEqual(payload.query, filterQuery)) {
|
||||
onChangedQuery(payload.query);
|
||||
const payloadQuery = convertToQueryType(payload.query);
|
||||
onChangedQuery(payloadQuery);
|
||||
}
|
||||
},
|
||||
[filterQuery, onChangedQuery]
|
||||
|
@ -109,11 +112,19 @@ export const QueryBar = memo<QueryBarComponentProps>(
|
|||
[filterManager]
|
||||
);
|
||||
|
||||
const isEsql = filterQuery?.language === 'esql';
|
||||
const query = useMemo(() => {
|
||||
if (isEsql && typeof filterQuery.query === 'string') {
|
||||
return { esql: filterQuery.query };
|
||||
}
|
||||
return filterQuery;
|
||||
}, [filterQuery, isEsql]);
|
||||
|
||||
useEffect(() => {
|
||||
let dv: DataView;
|
||||
if (isDataView(indexPattern)) {
|
||||
setDataView(indexPattern);
|
||||
} else {
|
||||
} else if (!isEsql) {
|
||||
const createDataView = async () => {
|
||||
dv = await data.dataViews.create({ title: indexPattern.title });
|
||||
setDataView(dv);
|
||||
|
@ -125,7 +136,7 @@ export const QueryBar = memo<QueryBarComponentProps>(
|
|||
data.dataViews.clearInstanceCache(dv?.id);
|
||||
}
|
||||
};
|
||||
}, [data.dataViews, indexPattern]);
|
||||
}, [data.dataViews, indexPattern, isEsql]);
|
||||
|
||||
const timeHistory = useMemo(() => new TimeHistory(new Storage(localStorage)), []);
|
||||
const arrDataView = useMemo(() => (dataView != null ? [dataView] : []), [dataView]);
|
||||
|
@ -138,7 +149,7 @@ export const QueryBar = memo<QueryBarComponentProps>(
|
|||
indexPatterns={arrDataView}
|
||||
isLoading={isLoading}
|
||||
isRefreshPaused={isRefreshPaused}
|
||||
query={filterQuery}
|
||||
query={query}
|
||||
onClearSavedQuery={onClearSavedQuery}
|
||||
onFiltersUpdated={onFiltersUpdated}
|
||||
onQueryChange={onQueryChange}
|
||||
|
|
|
@ -9,7 +9,7 @@ import type { CoreStart } from '@kbn/core/public';
|
|||
import type { StartPlugins } from '../../../types';
|
||||
|
||||
type GlobalServices = Pick<CoreStart, 'application' | 'http' | 'uiSettings' | 'notifications'> &
|
||||
Pick<StartPlugins, 'data' | 'unifiedSearch'>;
|
||||
Pick<StartPlugins, 'data' | 'unifiedSearch' | 'expressions'>;
|
||||
|
||||
export class KibanaServices {
|
||||
private static kibanaBranch?: string;
|
||||
|
@ -27,12 +27,21 @@ export class KibanaServices {
|
|||
prebuiltRulesPackageVersion,
|
||||
uiSettings,
|
||||
notifications,
|
||||
expressions,
|
||||
}: GlobalServices & {
|
||||
kibanaBranch: string;
|
||||
kibanaVersion: string;
|
||||
prebuiltRulesPackageVersion?: string;
|
||||
}) {
|
||||
this.services = { application, data, http, uiSettings, unifiedSearch, notifications };
|
||||
this.services = {
|
||||
application,
|
||||
data,
|
||||
http,
|
||||
uiSettings,
|
||||
unifiedSearch,
|
||||
notifications,
|
||||
expressions,
|
||||
};
|
||||
this.kibanaBranch = kibanaBranch;
|
||||
this.kibanaVersion = kibanaVersion;
|
||||
this.prebuiltRulesPackageVersion = prebuiltRulesPackageVersion;
|
||||
|
|
|
@ -0,0 +1,83 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import React, { useCallback } from 'react';
|
||||
import type { EuiComboBoxOptionOption } from '@elastic/eui';
|
||||
import { EuiFormRow, EuiComboBox } from '@elastic/eui';
|
||||
|
||||
import type { FieldHook } from '@kbn/es-ui-shared-plugin/static/forms/hook_form_lib';
|
||||
|
||||
import { useEsqlFieldOptions } from './use_esql_fields_options';
|
||||
const AS_PLAIN_TEXT = { asPlainText: true };
|
||||
const COMPONENT_WIDTH = 500;
|
||||
|
||||
interface AutocompleteFieldProps {
|
||||
dataTestSubj: string;
|
||||
field: FieldHook;
|
||||
idAria: string;
|
||||
isDisabled: boolean;
|
||||
fieldType: 'string';
|
||||
placeholder?: string;
|
||||
esqlQuery: string | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* autocomplete form component that works with ES|QL query
|
||||
* it receives query as one of the parameters, fetches available fields and convert them to
|
||||
* options, that populate autocomplete component
|
||||
*/
|
||||
export const EsqlAutocomplete: React.FC<AutocompleteFieldProps> = ({
|
||||
dataTestSubj,
|
||||
field,
|
||||
idAria,
|
||||
isDisabled,
|
||||
fieldType,
|
||||
placeholder,
|
||||
esqlQuery,
|
||||
}): JSX.Element => {
|
||||
const handleValuesChange = useCallback(
|
||||
([newOption]: EuiComboBoxOptionOption[]): void => {
|
||||
field.setValue(newOption?.label ?? '');
|
||||
},
|
||||
[field]
|
||||
);
|
||||
const { options, isLoading } = useEsqlFieldOptions(esqlQuery, fieldType);
|
||||
|
||||
const value = field?.value;
|
||||
const selectedOptions = typeof value === 'string' && value ? [{ label: value }] : [];
|
||||
|
||||
const isInvalid =
|
||||
typeof value === 'string' && value ? !options.some((option) => option.label === value) : false;
|
||||
|
||||
return (
|
||||
<EuiFormRow
|
||||
data-test-subj={dataTestSubj}
|
||||
describedByIds={idAria ? [idAria] : undefined}
|
||||
fullWidth
|
||||
helpText={field.helpText}
|
||||
label={field.label}
|
||||
labelAppend={field.labelAppend}
|
||||
>
|
||||
<EuiComboBox
|
||||
placeholder={placeholder ?? ''}
|
||||
options={options}
|
||||
selectedOptions={selectedOptions}
|
||||
onChange={handleValuesChange}
|
||||
isLoading={isLoading}
|
||||
isDisabled={isDisabled || isLoading}
|
||||
isClearable={false}
|
||||
singleSelection={AS_PLAIN_TEXT}
|
||||
data-test-subj="esqlAutocompleteComboBox"
|
||||
style={{ width: `${COMPONENT_WIDTH}px` }}
|
||||
fullWidth
|
||||
isInvalid={isInvalid}
|
||||
/>
|
||||
</EuiFormRow>
|
||||
);
|
||||
};
|
||||
|
||||
EsqlAutocomplete.displayName = 'EsqlAutocomplete';
|
|
@ -0,0 +1,8 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
export { EsqlAutocomplete } from './esql_autocomplete';
|
|
@ -0,0 +1,59 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { esqlToOptions } from './use_esql_fields_options';
|
||||
|
||||
describe('esqlToOptions', () => {
|
||||
it('should return empty array if data is undefined', () => {
|
||||
expect(esqlToOptions(undefined)).toEqual([]);
|
||||
});
|
||||
it('should return empty array if data is null', () => {
|
||||
expect(esqlToOptions(null)).toEqual([]);
|
||||
});
|
||||
it('should return empty array if data has error', () => {
|
||||
expect(esqlToOptions({ error: Error })).toEqual([]);
|
||||
});
|
||||
it('should transform all columns if fieldTYpe is not passed', () => {
|
||||
expect(
|
||||
esqlToOptions({
|
||||
type: 'datatable',
|
||||
rows: [],
|
||||
columns: [
|
||||
{ name: '@timestamp', id: '@timestamp', meta: { type: 'date' } },
|
||||
{ name: 'agent.build.original', id: 'agent.build.original', meta: { type: 'string' } },
|
||||
{ name: 'amqp.app-id', id: 'amqp.app-id', meta: { type: 'string' } },
|
||||
{ name: 'amqp.auto-delete', id: 'amqp.auto-delete', meta: { type: 'number' } },
|
||||
{ name: 'amqp.class-id', id: 'amqp.class-id', meta: { type: 'boolean' } },
|
||||
],
|
||||
})
|
||||
).toEqual([
|
||||
{ label: '@timestamp' },
|
||||
{ label: 'agent.build.original' },
|
||||
{ label: 'amqp.app-id' },
|
||||
{ label: 'amqp.auto-delete' },
|
||||
{ label: 'amqp.class-id' },
|
||||
]);
|
||||
});
|
||||
it('should transform only columns of exact fieldType', () => {
|
||||
expect(
|
||||
esqlToOptions(
|
||||
{
|
||||
type: 'datatable',
|
||||
rows: [],
|
||||
columns: [
|
||||
{ name: '@timestamp', id: '@timestamp', meta: { type: 'date' } },
|
||||
{ name: 'agent.build.original', id: 'agent.build.original', meta: { type: 'string' } },
|
||||
{ name: 'amqp.app-id', id: 'amqp.app-id', meta: { type: 'string' } },
|
||||
{ name: 'amqp.auto-delete', id: 'amqp.auto-delete', meta: { type: 'number' } },
|
||||
{ name: 'amqp.class-id', id: 'amqp.class-id', meta: { type: 'boolean' } },
|
||||
],
|
||||
},
|
||||
'string'
|
||||
)
|
||||
).toEqual([{ label: 'agent.build.original' }, { label: 'amqp.app-id' }]);
|
||||
});
|
||||
});
|
|
@ -0,0 +1,64 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
import { useMemo } from 'react';
|
||||
import type { EuiComboBoxOptionOption } from '@elastic/eui';
|
||||
import type { Datatable, ExpressionsStart } from '@kbn/expressions-plugin/public';
|
||||
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
|
||||
import { useKibana } from '@kbn/kibana-react-plugin/public';
|
||||
|
||||
import { getEsqlQueryConfig } from '../../logic/get_esql_query_config';
|
||||
import type { FieldType } from '../../logic/esql_validator';
|
||||
|
||||
export const esqlToOptions = (
|
||||
data: { error: unknown } | Datatable | undefined | null,
|
||||
fieldType?: FieldType
|
||||
) => {
|
||||
if (data && 'error' in data) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const options = (data?.columns ?? []).reduce<Array<{ label: string }>>((acc, { id, meta }) => {
|
||||
// if fieldType absent, we do not filter columns by type
|
||||
if (!fieldType || fieldType === meta.type) {
|
||||
acc.push({ label: id });
|
||||
}
|
||||
return acc;
|
||||
}, []);
|
||||
|
||||
return options;
|
||||
};
|
||||
|
||||
type UseEsqlFieldOptions = (
|
||||
esqlQuery: string | undefined,
|
||||
fieldType: FieldType
|
||||
) => {
|
||||
isLoading: boolean;
|
||||
options: Array<EuiComboBoxOptionOption<string>>;
|
||||
};
|
||||
|
||||
/**
|
||||
* fetches ES|QL fields and convert them to Combobox options
|
||||
*/
|
||||
export const useEsqlFieldOptions: UseEsqlFieldOptions = (esqlQuery, fieldType) => {
|
||||
const kibana = useKibana<{ expressions: ExpressionsStart }>();
|
||||
|
||||
const { expressions } = kibana.services;
|
||||
|
||||
const queryConfig = getEsqlQueryConfig({ esqlQuery, expressions });
|
||||
const { data, isLoading } = useQuery(queryConfig);
|
||||
|
||||
const options = useMemo(() => {
|
||||
return esqlToOptions(data, fieldType);
|
||||
}, [data, fieldType]);
|
||||
|
||||
return {
|
||||
options,
|
||||
isLoading,
|
||||
};
|
||||
};
|
|
@ -0,0 +1,49 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import React from 'react';
|
||||
import { EuiPopover, EuiText, EuiButtonIcon } from '@elastic/eui';
|
||||
import { Markdown } from '@kbn/kibana-react-plugin/public';
|
||||
import * as i18n from './translations';
|
||||
|
||||
import { useBoolState } from '../../../../common/hooks/use_bool_state';
|
||||
|
||||
import { useKibana } from '../../../../common/lib/kibana';
|
||||
|
||||
const POPOVER_WIDTH = 640;
|
||||
|
||||
/**
|
||||
* Icon and popover that gives hint to users how to get started with ES|QL rules
|
||||
*/
|
||||
const EsqlInfoIconComponent = () => {
|
||||
const { docLinks } = useKibana().services;
|
||||
|
||||
const [isPopoverOpen, , closePopover, togglePopover] = useBoolState();
|
||||
|
||||
const button = (
|
||||
<EuiButtonIcon iconType="iInCircle" onClick={togglePopover} aria-label={i18n.ARIA_LABEL} />
|
||||
);
|
||||
|
||||
return (
|
||||
<EuiPopover button={button} isOpen={isPopoverOpen} closePopover={closePopover}>
|
||||
<EuiText style={{ width: POPOVER_WIDTH }} size="s">
|
||||
<Markdown
|
||||
markdown={i18n.getTooltipContent(
|
||||
docLinks.links.esql.statsBy,
|
||||
// Docs team will provide actual link to a new page before release
|
||||
// For now, it's just a mock
|
||||
docLinks.links.esql.statsBy
|
||||
)}
|
||||
/>
|
||||
</EuiText>
|
||||
</EuiPopover>
|
||||
);
|
||||
};
|
||||
|
||||
export const EsqlInfoIcon = React.memo(EsqlInfoIconComponent);
|
||||
|
||||
EsqlInfoIcon.displayName = 'EsqlInfoIcon';
|
|
@ -0,0 +1,51 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { i18n } from '@kbn/i18n';
|
||||
|
||||
export const ARIA_LABEL = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlInfoAriaLabel',
|
||||
{
|
||||
defaultMessage: `Open help popover`,
|
||||
}
|
||||
);
|
||||
|
||||
export const getTooltipContent = (statsByLink: string, startUsingEsqlLink: string) =>
|
||||
i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlInfoTooltipContent',
|
||||
{
|
||||
defaultMessage: `
|
||||
### Aggregating rule
|
||||
Is a rule that uses {statsByLink} grouping commands. So, its result can not be matched with a particular document in ES.
|
||||
\`\`\`
|
||||
FROM logs*
|
||||
| STATS count = COUNT(host.name) BY host.name
|
||||
| SORT host.name
|
||||
\`\`\`
|
||||
|
||||
|
||||
### Non-aggregating rule
|
||||
Is a rule that does not use {statsByLink} grouping commands. Hence, each row in result can be tracked to a source document in ES. For this type of rule,
|
||||
please use operator \`[metadata _id, _index, _version]\` after defining index source. This would allow deduplicate alerts and link them with the source document.
|
||||
|
||||
Example
|
||||
|
||||
\`\`\`
|
||||
FROM logs* [metadata _id, _index, _version]
|
||||
| WHERE event.id == "test"
|
||||
| LIMIT 10
|
||||
\`\`\`
|
||||
|
||||
Please, ensure, metadata properties \`id\`, \`_index\`, \`_version\` are carried over through pipe operators.
|
||||
`,
|
||||
values: {
|
||||
statsByLink: `[STATS..BY](${statsByLink})`,
|
||||
// Docs team will provide actual link to a new page before release
|
||||
// startUsingEsqlLink: `[WIP: Get started using ES|QL rules](${startUsingEsqlLink})`,
|
||||
},
|
||||
}
|
||||
);
|
|
@ -0,0 +1,33 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { computeHasMetadataOperator } from './esql_validator';
|
||||
|
||||
describe('computeHasMetadataOperator', () => {
|
||||
it('should be false if query does not have operator', () => {
|
||||
expect(computeHasMetadataOperator('from test*')).toBe(false);
|
||||
expect(computeHasMetadataOperator('from test* [metadata]')).toBe(false);
|
||||
expect(computeHasMetadataOperator('from test* [metadata id]')).toBe(false);
|
||||
expect(computeHasMetadataOperator('from metadata*')).toBe(false);
|
||||
expect(computeHasMetadataOperator('from test* | keep metadata')).toBe(false);
|
||||
expect(computeHasMetadataOperator('from test* | eval x="[metadata _id]"')).toBe(false);
|
||||
});
|
||||
it('should be true if query has operator', () => {
|
||||
expect(computeHasMetadataOperator('from test* [metadata _id]')).toBe(true);
|
||||
expect(computeHasMetadataOperator('from test* [metadata _id, _index]')).toBe(true);
|
||||
expect(computeHasMetadataOperator('from test* [metadata _index, _id]')).toBe(true);
|
||||
expect(computeHasMetadataOperator('from test* [ metadata _id ]')).toBe(true);
|
||||
expect(computeHasMetadataOperator('from test* [ metadata _id] ')).toBe(true);
|
||||
expect(computeHasMetadataOperator('from test* [ metadata _id] | limit 10')).toBe(true);
|
||||
expect(
|
||||
computeHasMetadataOperator(`from packetbeat* [metadata
|
||||
|
||||
_id ]
|
||||
| limit 100`)
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
|
@ -0,0 +1,94 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { isEmpty } from 'lodash';
|
||||
|
||||
import { computeIsESQLQueryAggregating } from '@kbn/securitysolution-utils';
|
||||
|
||||
import { KibanaServices } from '../../../common/lib/kibana';
|
||||
import { securitySolutionQueryClient } from '../../../common/containers/query_client/query_client_provider';
|
||||
|
||||
import type { ValidationError, ValidationFunc } from '../../../shared_imports';
|
||||
import { isEsqlRule } from '../../../../common/detection_engine/utils';
|
||||
import type { DefineStepRule } from '../../../detections/pages/detection_engine/rules/types';
|
||||
import type { FieldValueQueryBar } from '../../../detections/components/rules/query_bar';
|
||||
import * as i18n from './translations';
|
||||
import { getEsqlQueryConfig } from './get_esql_query_config';
|
||||
export type FieldType = 'string';
|
||||
|
||||
export enum ERROR_CODES {
|
||||
INVALID_ESQL = 'ERR_INVALID_ESQL',
|
||||
ERR_MISSING_ID_FIELD_FROM_RESULT = 'ERR_MISSING_ID_FIELD_FROM_RESULT',
|
||||
}
|
||||
|
||||
const constructValidationError = (error: Error) => {
|
||||
return {
|
||||
code: ERROR_CODES.INVALID_ESQL,
|
||||
message: error?.message
|
||||
? i18n.esqlValidationErrorMessage(error.message)
|
||||
: i18n.ESQL_VALIDATION_UNKNOWN_ERROR,
|
||||
error,
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* checks whether query has [metadata _id] operator
|
||||
*/
|
||||
export const computeHasMetadataOperator = (esqlQuery: string) => {
|
||||
return /(?<!\|[\s\S.]*)\[\s*metadata[\s\S.]*_id[\s\S.]*\]/i.test(esqlQuery);
|
||||
};
|
||||
|
||||
/**
|
||||
* form validator for ES|QL queryBar
|
||||
*/
|
||||
export const esqlValidator = async (
|
||||
...args: Parameters<ValidationFunc>
|
||||
): Promise<ValidationError<ERROR_CODES> | void | undefined> => {
|
||||
const [{ value, formData }] = args;
|
||||
const { query: queryValue } = value as FieldValueQueryBar;
|
||||
const query = queryValue.query as string;
|
||||
const { ruleType } = formData as DefineStepRule;
|
||||
|
||||
const needsValidation = isEsqlRule(ruleType) && !isEmpty(query);
|
||||
if (!needsValidation) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const services = KibanaServices.get();
|
||||
|
||||
const isEsqlQueryAggregating = computeIsESQLQueryAggregating(query);
|
||||
|
||||
// non-aggregating query which does not have [metadata], is not a valid one
|
||||
if (!isEsqlQueryAggregating && !computeHasMetadataOperator(query)) {
|
||||
return {
|
||||
code: ERROR_CODES.ERR_MISSING_ID_FIELD_FROM_RESULT,
|
||||
message: i18n.ESQL_VALIDATION_MISSING_ID_IN_QUERY_ERROR,
|
||||
};
|
||||
}
|
||||
|
||||
const data = await securitySolutionQueryClient.fetchQuery(
|
||||
getEsqlQueryConfig({ esqlQuery: query, expressions: services.expressions })
|
||||
);
|
||||
|
||||
if (data && 'error' in data) {
|
||||
return constructValidationError(data.error);
|
||||
}
|
||||
|
||||
// check whether _id field is present in response
|
||||
const isIdFieldPresent = (data?.columns ?? []).find(({ id }) => '_id' === id);
|
||||
// for non-aggregating query, we want to disable queries w/o _id property returned in response
|
||||
if (!isEsqlQueryAggregating && !isIdFieldPresent) {
|
||||
return {
|
||||
code: ERROR_CODES.ERR_MISSING_ID_FIELD_FROM_RESULT,
|
||||
message: i18n.ESQL_VALIDATION_MISSING_ID_IN_QUERY_ERROR,
|
||||
};
|
||||
}
|
||||
} catch (error) {
|
||||
return constructValidationError(error);
|
||||
}
|
||||
};
|
|
@ -0,0 +1,38 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { fetchFieldsFromESQL } from '@kbn/text-based-editor';
|
||||
import type { ExpressionsStart } from '@kbn/expressions-plugin/public';
|
||||
|
||||
/**
|
||||
* react-query configuration to be used to fetch ES|QL fields
|
||||
* it sets limit in query to 0, so we don't fetch unnecessary results, only fields
|
||||
*/
|
||||
export const getEsqlQueryConfig = ({
|
||||
esqlQuery,
|
||||
expressions,
|
||||
}: {
|
||||
esqlQuery: string | undefined;
|
||||
expressions: ExpressionsStart;
|
||||
}) => {
|
||||
const emptyResultsEsqlQuery = `${esqlQuery} | limit 0`;
|
||||
return {
|
||||
queryKey: [(esqlQuery ?? '').trim()],
|
||||
queryFn: async () => {
|
||||
if (!esqlQuery) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const res = await fetchFieldsFromESQL({ esql: emptyResultsEsqlQuery }, expressions);
|
||||
return res;
|
||||
} catch (e) {
|
||||
return { error: e };
|
||||
}
|
||||
},
|
||||
staleTime: 60 * 1000,
|
||||
};
|
||||
};
|
|
@ -0,0 +1,28 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { i18n } from '@kbn/i18n';
|
||||
|
||||
export const ESQL_VALIDATION_UNKNOWN_ERROR = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.esqlValidation.unknownError',
|
||||
{
|
||||
defaultMessage: 'Unknown error while validating ES|QL',
|
||||
}
|
||||
);
|
||||
|
||||
export const esqlValidationErrorMessage = (message: string) =>
|
||||
i18n.translate('xpack.securitySolution.detectionEngine.esqlValidation.errorMessage', {
|
||||
values: { message },
|
||||
defaultMessage: 'Error validating ES|QL: "{message}"',
|
||||
});
|
||||
|
||||
export const ESQL_VALIDATION_MISSING_ID_IN_QUERY_ERROR = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.esqlValidation.missingIdInQueryError',
|
||||
{
|
||||
defaultMessage: `For non-aggregating rules(that don't use STATS..BY function), please write query that returns _id field from [metadata _id, _version, _index] operator`,
|
||||
}
|
||||
);
|
|
@ -0,0 +1,9 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
export { useEsqlIndex } from './use_esql_index';
|
||||
export { useEsqlQueryForAboutStep } from './use_esql_query_for_about_step';
|
|
@ -0,0 +1,38 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
import { renderHook } from '@testing-library/react-hooks';
|
||||
|
||||
import { useEsqlIndex } from './use_esql_index';
|
||||
|
||||
const validEsqlQuery = 'from auditbeat* [metadata _id, _index, _version]';
|
||||
describe('useEsqlIndex', () => {
|
||||
it('should return empty array if isQueryReadEnabled is undefined', () => {
|
||||
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'esql', undefined));
|
||||
|
||||
expect(result.current).toEqual([]);
|
||||
});
|
||||
it('should return empty array if isQueryReadEnabled is false', () => {
|
||||
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'esql', false));
|
||||
|
||||
expect(result.current).toEqual([]);
|
||||
});
|
||||
it('should return empty array if rule type is not esql', () => {
|
||||
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'query', true));
|
||||
|
||||
expect(result.current).toEqual([]);
|
||||
});
|
||||
it('should return empty array if query is empty', () => {
|
||||
const { result } = renderHook(() => useEsqlIndex('', 'esql', true));
|
||||
|
||||
expect(result.current).toEqual([]);
|
||||
});
|
||||
it('should return parsed index array from a valid query', () => {
|
||||
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'esql', true));
|
||||
|
||||
expect(result.current).toEqual(['auditbeat*']);
|
||||
});
|
||||
});
|
|
@ -0,0 +1,38 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
import { useMemo } from 'react';
|
||||
import { getIndexPatternFromESQLQuery } from '@kbn/es-query';
|
||||
import { getIndexListFromIndexString } from '@kbn/securitysolution-utils';
|
||||
|
||||
import type { Query } from '@kbn/es-query';
|
||||
import type { Type } from '@kbn/securitysolution-io-ts-alerting-types';
|
||||
|
||||
import { isEsqlRule } from '../../../../common/detection_engine/utils';
|
||||
|
||||
/**
|
||||
* parses ES|QL query and returns memoized array of indices
|
||||
* @param query - ES|QL query to retrieve index from
|
||||
* @param ruleType - rule type value
|
||||
* @param isQueryReadEnabled - if not enabled, return empty array. Useful if we know form or query is not valid and we don't want to retrieve index
|
||||
* @returns
|
||||
*/
|
||||
export const useEsqlIndex = (
|
||||
query: Query['query'],
|
||||
ruleType: Type,
|
||||
isQueryReadEnabled: boolean | undefined
|
||||
) => {
|
||||
const indexString = useMemo(() => {
|
||||
if (!isQueryReadEnabled) {
|
||||
return '';
|
||||
}
|
||||
const esqlQuery = typeof query === 'string' && isEsqlRule(ruleType) ? query : undefined;
|
||||
return getIndexPatternFromESQLQuery(esqlQuery);
|
||||
}, [query, isQueryReadEnabled, ruleType]);
|
||||
|
||||
const index = useMemo(() => getIndexListFromIndexString(indexString), [indexString]);
|
||||
return index;
|
||||
};
|
|
@ -0,0 +1,38 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
import { useMemo } from 'react';
|
||||
import { RuleStep } from '../../../detections/pages/detection_engine/rules/types';
|
||||
import { isEsqlRule } from '../../../../common/detection_engine/utils';
|
||||
|
||||
import type { DefineStepRule } from '../../../detections/pages/detection_engine/rules/types';
|
||||
|
||||
interface UseEsqlQueryForAboutStepArgs {
|
||||
defineStepData: DefineStepRule;
|
||||
activeStep: RuleStep;
|
||||
}
|
||||
|
||||
/**
|
||||
* If about step not active, return query as undefined to prevent unnecessary re-renders
|
||||
* ES|QL query can change frequently when user types it, so we don't want it to trigger about form step re-render, when * this from step not active
|
||||
* When it is active, query would not change, as it can be edit only in define form step
|
||||
*/
|
||||
export const useEsqlQueryForAboutStep = ({
|
||||
defineStepData,
|
||||
activeStep,
|
||||
}: UseEsqlQueryForAboutStepArgs) => {
|
||||
const esqlQueryForAboutStep = useMemo(() => {
|
||||
if (activeStep !== RuleStep.aboutRule) {
|
||||
return undefined;
|
||||
}
|
||||
return typeof defineStepData.queryBar.query.query === 'string' &&
|
||||
isEsqlRule(defineStepData.ruleType)
|
||||
? defineStepData.queryBar.query.query
|
||||
: undefined;
|
||||
}, [defineStepData.queryBar.query.query, defineStepData.ruleType, activeStep]);
|
||||
|
||||
return esqlQueryForAboutStep;
|
||||
};
|
|
@ -7,7 +7,7 @@
|
|||
|
||||
/* eslint-disable complexity */
|
||||
|
||||
import { has, isEmpty } from 'lodash/fp';
|
||||
import { has, isEmpty, get } from 'lodash/fp';
|
||||
import type { Unit } from '@kbn/datemath';
|
||||
import moment from 'moment';
|
||||
import deepmerge from 'deepmerge';
|
||||
|
@ -151,6 +151,20 @@ type NewTermsRuleFields<T> = Omit<
|
|||
| 'threatMapping'
|
||||
| 'eqlOptions'
|
||||
>;
|
||||
type EsqlRuleFields<T> = Omit<
|
||||
T,
|
||||
| 'anomalyThreshold'
|
||||
| 'machineLearningJobId'
|
||||
| 'threshold'
|
||||
| 'threatIndex'
|
||||
| 'threatQueryBar'
|
||||
| 'threatMapping'
|
||||
| 'eqlOptions'
|
||||
| 'index'
|
||||
| 'newTermsFields'
|
||||
| 'historyWindowSize'
|
||||
| 'dataViewId'
|
||||
>;
|
||||
|
||||
const isMlFields = <T>(
|
||||
fields:
|
||||
|
@ -160,6 +174,7 @@ const isMlFields = <T>(
|
|||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| NewTermsRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
): fields is MlRuleFields<T> => has('anomalyThreshold', fields);
|
||||
|
||||
const isThresholdFields = <T>(
|
||||
|
@ -170,6 +185,7 @@ const isThresholdFields = <T>(
|
|||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| NewTermsRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
): fields is ThresholdRuleFields<T> => has('threshold', fields);
|
||||
|
||||
const isThreatMatchFields = <T>(
|
||||
|
@ -180,6 +196,7 @@ const isThreatMatchFields = <T>(
|
|||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| NewTermsRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
): fields is ThreatMatchRuleFields<T> => has('threatIndex', fields);
|
||||
|
||||
const isNewTermsFields = <T>(
|
||||
|
@ -190,6 +207,7 @@ const isNewTermsFields = <T>(
|
|||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| NewTermsRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
): fields is NewTermsRuleFields<T> => has('newTermsFields', fields);
|
||||
|
||||
const isEqlFields = <T>(
|
||||
|
@ -200,8 +218,20 @@ const isEqlFields = <T>(
|
|||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| NewTermsRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
): fields is EqlQueryRuleFields<T> => has('eqlOptions', fields);
|
||||
|
||||
const isEsqlFields = <T>(
|
||||
fields:
|
||||
| QueryRuleFields<T>
|
||||
| EqlQueryRuleFields<T>
|
||||
| MlRuleFields<T>
|
||||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| NewTermsRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
): fields is EsqlRuleFields<T> => get('queryBar.query.language', fields) === 'esql';
|
||||
|
||||
export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
|
||||
fields: T,
|
||||
type: Type
|
||||
|
@ -211,6 +241,7 @@ export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
|
|||
| MlRuleFields<T>
|
||||
| ThresholdRuleFields<T>
|
||||
| ThreatMatchRuleFields<T>
|
||||
| EsqlRuleFields<T>
|
||||
| NewTermsRuleFields<T> => {
|
||||
switch (type) {
|
||||
case 'machine_learning':
|
||||
|
@ -279,6 +310,7 @@ export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
|
|||
...eqlRuleFields
|
||||
} = fields;
|
||||
return eqlRuleFields;
|
||||
|
||||
case 'new_terms':
|
||||
const {
|
||||
anomalyThreshold: ___a,
|
||||
|
@ -291,6 +323,23 @@ export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
|
|||
...newTermsRuleFields
|
||||
} = fields;
|
||||
return newTermsRuleFields;
|
||||
|
||||
case 'esql':
|
||||
const {
|
||||
anomalyThreshold: _esql_a,
|
||||
machineLearningJobId: _esql_m,
|
||||
threshold: _esql_t,
|
||||
threatIndex: _esql_removedThreatIndex,
|
||||
threatQueryBar: _esql_removedThreatQueryBar,
|
||||
threatMapping: _esql_removedThreatMapping,
|
||||
newTermsFields: _esql_removedNewTermsFields,
|
||||
historyWindowSize: _esql_removedHistoryWindowSize,
|
||||
eqlOptions: _esql__eqlOptions,
|
||||
index: _esql_index,
|
||||
dataViewId: _esql_dataViewId,
|
||||
...esqlRuleFields
|
||||
} = fields;
|
||||
return esqlRuleFields;
|
||||
}
|
||||
assertUnreachable(type);
|
||||
};
|
||||
|
@ -420,6 +469,11 @@ export const formatDefineStepData = (defineStepData: DefineStepRule): DefineStep
|
|||
new_terms_fields: ruleFields.newTermsFields,
|
||||
history_window_start: `now-${ruleFields.historyWindowSize}`,
|
||||
}
|
||||
: isEsqlFields(ruleFields) && !('index' in ruleFields)
|
||||
? {
|
||||
language: ruleFields.queryBar?.query?.language,
|
||||
query: ruleFields.queryBar?.query?.query as string,
|
||||
}
|
||||
: {
|
||||
...(ruleFields.groupByFields.length > 0
|
||||
? {
|
||||
|
@ -454,7 +508,7 @@ export const formatDefineStepData = (defineStepData: DefineStepRule): DefineStep
|
|||
return {
|
||||
...baseFields,
|
||||
...typeFields,
|
||||
data_view_id: ruleFields.dataViewId,
|
||||
...('dataViewId' in ruleFields ? { data_view_id: ruleFields.dataViewId } : {}),
|
||||
};
|
||||
};
|
||||
|
||||
|
|
|
@ -20,8 +20,13 @@ import React, { memo, useCallback, useRef, useState, useMemo, useEffect } from '
|
|||
import styled from 'styled-components';
|
||||
|
||||
import type { DataViewListItem } from '@kbn/data-views-plugin/common';
|
||||
|
||||
import { useAppToasts } from '../../../../common/hooks/use_app_toasts';
|
||||
import { isMlRule, isThreatMatchRule } from '../../../../../common/detection_engine/utils';
|
||||
import {
|
||||
isMlRule,
|
||||
isThreatMatchRule,
|
||||
isEsqlRule,
|
||||
} from '../../../../../common/detection_engine/utils';
|
||||
import { useCreateRule } from '../../../rule_management/logic';
|
||||
import type { RuleCreateProps } from '../../../../../common/api/detection_engine/model/rule_schema';
|
||||
import { useListsConfig } from '../../../../detections/containers/detection_engine/lists/use_lists_config';
|
||||
|
@ -62,6 +67,7 @@ import {
|
|||
import type { DefineStepRule } from '../../../../detections/pages/detection_engine/rules/types';
|
||||
import { RuleStep } from '../../../../detections/pages/detection_engine/rules/types';
|
||||
import { formatRule } from './helpers';
|
||||
import { useEsqlIndex, useEsqlQueryForAboutStep } from '../../hooks';
|
||||
import * as i18n from './translations';
|
||||
import { SecurityPageName } from '../../../../app/types';
|
||||
import {
|
||||
|
@ -190,6 +196,11 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
[defineStepData.ruleType]
|
||||
);
|
||||
|
||||
const isEsqlRuleValue = useMemo(
|
||||
() => isEsqlRule(defineStepData.ruleType),
|
||||
[defineStepData.ruleType]
|
||||
);
|
||||
|
||||
const [openSteps, setOpenSteps] = useState({
|
||||
[RuleStep.defineRule]: false,
|
||||
[RuleStep.aboutRule]: false,
|
||||
|
@ -206,11 +217,22 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
const [isQueryBarValid, setIsQueryBarValid] = useState(false);
|
||||
const [isThreatQueryBarValid, setIsThreatQueryBarValid] = useState(false);
|
||||
|
||||
const esqlQueryForAboutStep = useEsqlQueryForAboutStep({ defineStepData, activeStep });
|
||||
const esqlIndex = useEsqlIndex(
|
||||
defineStepData.queryBar.query.query,
|
||||
ruleType,
|
||||
defineStepForm.isValid
|
||||
);
|
||||
const memoizedIndex = useMemo(
|
||||
() => (isEsqlRuleValue ? esqlIndex : defineStepData.index),
|
||||
[defineStepData.index, esqlIndex, isEsqlRuleValue]
|
||||
);
|
||||
|
||||
const isPreviewDisabled = getIsRulePreviewDisabled({
|
||||
ruleType,
|
||||
isQueryBarValid,
|
||||
isThreatQueryBarValid,
|
||||
index: defineStepData.index,
|
||||
index: memoizedIndex,
|
||||
dataViewId: defineStepData.dataViewId,
|
||||
dataSourceType: defineStepData.dataSourceType,
|
||||
threatIndex: defineStepData.threatIndex,
|
||||
|
@ -250,7 +272,7 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
}, [dataViews]);
|
||||
const { indexPattern, isIndexPatternLoading, browserFields } = useRuleIndexPattern({
|
||||
dataSourceType: defineStepData.dataSourceType,
|
||||
index: defineStepData.index,
|
||||
index: memoizedIndex,
|
||||
dataViewId: defineStepData.dataViewId,
|
||||
});
|
||||
|
||||
|
@ -494,7 +516,7 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
setIsQueryBarValid={setIsQueryBarValid}
|
||||
setIsThreatQueryBarValid={setIsThreatQueryBarValid}
|
||||
ruleType={defineStepData.ruleType}
|
||||
index={defineStepData.index}
|
||||
index={memoizedIndex}
|
||||
threatIndex={defineStepData.threatIndex}
|
||||
groupByFields={defineStepData.groupByFields}
|
||||
dataSourceType={defineStepData.dataSourceType}
|
||||
|
@ -518,7 +540,7 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
defineRuleNextStep,
|
||||
defineStepData.dataSourceType,
|
||||
defineStepData.groupByFields,
|
||||
defineStepData.index,
|
||||
memoizedIndex,
|
||||
defineStepData.queryBar.saved_id,
|
||||
defineStepData.queryBar.title,
|
||||
defineStepData.ruleType,
|
||||
|
@ -575,12 +597,13 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
<StepAboutRule
|
||||
ruleType={defineStepData.ruleType}
|
||||
machineLearningJobId={defineStepData.machineLearningJobId}
|
||||
index={defineStepData.index}
|
||||
index={memoizedIndex}
|
||||
dataViewId={defineStepData.dataViewId}
|
||||
timestampOverride={aboutStepData.timestampOverride}
|
||||
isLoading={isCreateRuleLoading || loading}
|
||||
isActive={activeStep === RuleStep.aboutRule}
|
||||
form={aboutStepForm}
|
||||
esqlQuery={esqlQueryForAboutStep}
|
||||
/>
|
||||
|
||||
<NextStep
|
||||
|
@ -598,12 +621,13 @@ const CreateRulePageComponent: React.FC = () => {
|
|||
aboutStepForm,
|
||||
activeStep,
|
||||
defineStepData.dataViewId,
|
||||
defineStepData.index,
|
||||
memoizedIndex,
|
||||
defineStepData.machineLearningJobId,
|
||||
defineStepData.ruleType,
|
||||
isCreateRuleLoading,
|
||||
loading,
|
||||
memoAboutStepReadOnly,
|
||||
esqlQueryForAboutStep,
|
||||
]
|
||||
);
|
||||
const memoAboutStepExtraAction = useMemo(
|
||||
|
|
|
@ -22,6 +22,8 @@ import React, { memo, useCallback, useEffect, useMemo, useRef, useState } from '
|
|||
import { useParams } from 'react-router-dom';
|
||||
|
||||
import type { DataViewListItem } from '@kbn/data-views-plugin/common';
|
||||
|
||||
import { isEsqlRule } from '../../../../../common/detection_engine/utils';
|
||||
import { RulePreview } from '../../../../detections/components/rules/rule_preview';
|
||||
import { getIsRulePreviewDisabled } from '../../../../detections/components/rules/rule_preview/helpers';
|
||||
import type { RuleUpdateProps } from '../../../../../common/api/detection_engine/model/rule_schema';
|
||||
|
@ -64,6 +66,7 @@ import { useStartTransaction } from '../../../../common/lib/apm/use_start_transa
|
|||
import { SINGLE_RULE_ACTIONS } from '../../../../common/lib/apm/user_actions';
|
||||
import { useGetSavedQuery } from '../../../../detections/pages/detection_engine/rules/use_get_saved_query';
|
||||
import { useRuleForms, useRuleIndexPattern } from '../form';
|
||||
import { useEsqlIndex, useEsqlQueryForAboutStep } from '../../hooks';
|
||||
import { CustomHeaderPageMemo } from '..';
|
||||
|
||||
const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
|
||||
|
@ -144,11 +147,24 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
|
|||
actionsStepDefault: ruleActionsData,
|
||||
});
|
||||
|
||||
const esqlQueryForAboutStep = useEsqlQueryForAboutStep({ defineStepData, activeStep });
|
||||
const esqlIndex = useEsqlIndex(
|
||||
defineStepData.queryBar.query.query,
|
||||
defineStepData.ruleType,
|
||||
// allow to compute index from query only when query is valid or user switched to another tab
|
||||
// to prevent multiple data view initiations with partly typed index names
|
||||
defineStepForm.isValid || activeStep !== RuleStep.defineRule
|
||||
);
|
||||
const memoizedIndex = useMemo(
|
||||
() => (isEsqlRule(defineStepData.ruleType) ? esqlIndex : defineStepData.index),
|
||||
[defineStepData.index, esqlIndex, defineStepData.ruleType]
|
||||
);
|
||||
|
||||
const isPreviewDisabled = getIsRulePreviewDisabled({
|
||||
ruleType: defineStepData.ruleType,
|
||||
isQueryBarValid,
|
||||
isThreatQueryBarValid,
|
||||
index: defineStepData.index,
|
||||
index: memoizedIndex,
|
||||
dataViewId: defineStepData.dataViewId,
|
||||
dataSourceType: defineStepData.dataSourceType,
|
||||
threatIndex: defineStepData.threatIndex,
|
||||
|
@ -198,7 +214,7 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
|
|||
|
||||
const { indexPattern, isIndexPatternLoading, browserFields } = useRuleIndexPattern({
|
||||
dataSourceType: defineStepData.dataSourceType,
|
||||
index: defineStepData.index,
|
||||
index: memoizedIndex,
|
||||
dataViewId: defineStepData.dataViewId,
|
||||
});
|
||||
|
||||
|
@ -236,7 +252,7 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
|
|||
setIsQueryBarValid={setIsQueryBarValid}
|
||||
setIsThreatQueryBarValid={setIsThreatQueryBarValid}
|
||||
ruleType={defineStepData.ruleType}
|
||||
index={defineStepData.index}
|
||||
index={memoizedIndex}
|
||||
threatIndex={defineStepData.threatIndex}
|
||||
groupByFields={defineStepData.groupByFields}
|
||||
dataSourceType={defineStepData.dataSourceType}
|
||||
|
@ -270,10 +286,11 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
|
|||
isActive={activeStep === RuleStep.aboutRule}
|
||||
ruleType={defineStepData.ruleType}
|
||||
machineLearningJobId={defineStepData.machineLearningJobId}
|
||||
index={defineStepData.index}
|
||||
index={memoizedIndex}
|
||||
dataViewId={defineStepData.dataViewId}
|
||||
timestampOverride={aboutStepData.timestampOverride}
|
||||
form={aboutStepForm}
|
||||
esqlQuery={esqlQueryForAboutStep}
|
||||
key="aboutStep"
|
||||
/>
|
||||
)}
|
||||
|
@ -365,6 +382,8 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
|
|||
actionsStepData,
|
||||
actionMessageParams,
|
||||
actionsStepForm,
|
||||
memoizedIndex,
|
||||
esqlQueryForAboutStep,
|
||||
]
|
||||
);
|
||||
|
||||
|
|
|
@ -7,6 +7,7 @@
|
|||
|
||||
import { useEffect, useState, useMemo, useCallback } from 'react';
|
||||
import type { DataViewBase } from '@kbn/es-query';
|
||||
import { getIndexListFromEsqlQuery } from '@kbn/securitysolution-utils';
|
||||
|
||||
import type { FieldSpec, DataViewSpec } from '@kbn/data-views-plugin/common';
|
||||
|
||||
|
@ -39,6 +40,10 @@ export const useFetchIndexPatterns = (rules: Rule[] | null): ReturnUseFetchExcep
|
|||
() => rules != null && isSingleRule && rules[0].type === 'machine_learning',
|
||||
[isSingleRule, rules]
|
||||
);
|
||||
const isEsqlRule = useMemo(
|
||||
() => rules != null && isSingleRule && rules[0].type === 'esql',
|
||||
[isSingleRule, rules]
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
const fetchAndSetActiveSpace = async () => {
|
||||
|
@ -59,13 +64,14 @@ export const useFetchIndexPatterns = (rules: Rule[] | null): ReturnUseFetchExcep
|
|||
[isSingleRule, rules, activeSpaceId]
|
||||
);
|
||||
|
||||
const memoNonDataViewIndexPatterns = useMemo(
|
||||
() =>
|
||||
!memoDataViewId && rules != null && isSingleRule && rules[0].index != null
|
||||
? rules[0].index
|
||||
: [],
|
||||
[memoDataViewId, isSingleRule, rules]
|
||||
);
|
||||
const memoNonDataViewIndexPatterns = useMemo(() => {
|
||||
if (isEsqlRule) {
|
||||
return getIndexListFromEsqlQuery(rules?.[0].query);
|
||||
}
|
||||
return !memoDataViewId && rules != null && isSingleRule && rules[0].index != null
|
||||
? rules[0].index
|
||||
: [];
|
||||
}, [memoDataViewId, isSingleRule, rules, isEsqlRule]);
|
||||
|
||||
// Index pattern logic for ML
|
||||
const memoMlJobIds = useMemo(
|
||||
|
|
|
@ -201,6 +201,8 @@ const getRuleTypeDescription = (ruleType: Type) => {
|
|||
return descriptionStepI18n.THRESHOLD_TYPE_DESCRIPTION;
|
||||
case 'eql':
|
||||
return descriptionStepI18n.EQL_TYPE_DESCRIPTION;
|
||||
case 'esql':
|
||||
return descriptionStepI18n.ESQL_TYPE_DESCRIPTION;
|
||||
case 'threat_match':
|
||||
return descriptionStepI18n.THREAT_MATCH_TYPE_DESCRIPTION;
|
||||
case 'new_terms':
|
||||
|
@ -358,7 +360,7 @@ const prepareDefinitionSectionListItems = (
|
|||
}
|
||||
}
|
||||
|
||||
if ('filters' in rule && rule.filters && rule.filters.length > 0) {
|
||||
if ('filters' in rule && 'data_view_id' in rule && rule.filters?.length) {
|
||||
definitionSectionListItems.push({
|
||||
title: savedQuery
|
||||
? descriptionStepI18n.SAVED_QUERY_FILTERS_LABEL
|
||||
|
|
|
@ -6,10 +6,15 @@
|
|||
*/
|
||||
|
||||
import { useMemo } from 'react';
|
||||
import { getIndexListFromEsqlQuery } from '@kbn/securitysolution-utils';
|
||||
import { useSecurityJobs } from '../../../common/components/ml_popover/hooks/use_security_jobs';
|
||||
import { useGetInstalledJob } from '../../../common/components/ml/hooks/use_get_jobs';
|
||||
|
||||
export const useRuleIndices = (machineLearningJobId?: string[], defaultRuleIndices?: string[]) => {
|
||||
export const useRuleIndices = (
|
||||
machineLearningJobId?: string[],
|
||||
defaultRuleIndices?: string[],
|
||||
esqlQuery?: string
|
||||
) => {
|
||||
const memoMlJobIds = useMemo(() => machineLearningJobId ?? [], [machineLearningJobId]);
|
||||
const { loading: mlSecurityJobLoading, jobs } = useSecurityJobs();
|
||||
const memoSelectedMlJobs = useMemo(
|
||||
|
@ -32,10 +37,12 @@ export const useRuleIndices = (machineLearningJobId?: string[], defaultRuleIndic
|
|||
const memoRuleIndices = useMemo(() => {
|
||||
if (memoMlIndices.length > 0) {
|
||||
return memoMlIndices;
|
||||
} else if (esqlQuery) {
|
||||
return getIndexListFromEsqlQuery(esqlQuery);
|
||||
} else {
|
||||
return defaultRuleIndices ?? [];
|
||||
}
|
||||
}, [defaultRuleIndices, memoMlIndices]);
|
||||
}, [defaultRuleIndices, esqlQuery, memoMlIndices]);
|
||||
|
||||
return {
|
||||
mlJobLoading: mlSecurityJobLoading || mlInstalledJobLoading,
|
||||
|
|
|
@ -63,6 +63,10 @@ describe('Component BulkEditRuleErrorsList', () => {
|
|||
BulkActionsDryRunErrCode.MACHINE_LEARNING_INDEX_PATTERN,
|
||||
"2 custom machine learning rules (these rules don't have index patterns)",
|
||||
],
|
||||
[
|
||||
BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN,
|
||||
"2 custom ES|QL rules (these rules don't have index patterns)",
|
||||
],
|
||||
[
|
||||
BulkActionsDryRunErrCode.MACHINE_LEARNING_AUTH,
|
||||
"2 machine learning rules can't be edited (test failure)",
|
||||
|
|
|
@ -56,6 +56,16 @@ const BulkEditRuleErrorItem = ({
|
|||
/>
|
||||
</li>
|
||||
);
|
||||
case BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN:
|
||||
return (
|
||||
<li key={message}>
|
||||
<FormattedMessage
|
||||
id="xpack.securitySolution.detectionEngine.rules.allRules.bulkActions.esqlRulesIndexEditDescription"
|
||||
defaultMessage="{rulesCount, plural, =1 {# custom ES|QL rule} other {# custom ES|QL rules}} (these rules don't have index patterns)"
|
||||
values={{ rulesCount }}
|
||||
/>
|
||||
</li>
|
||||
);
|
||||
default:
|
||||
return (
|
||||
<li key={message}>
|
||||
|
|
|
@ -63,6 +63,16 @@ describe('prepareSearchParams', () => {
|
|||
excludeRuleTypes: ['machine_learning'],
|
||||
},
|
||||
],
|
||||
[
|
||||
BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN,
|
||||
{
|
||||
filter: '',
|
||||
tags: [],
|
||||
showCustomRules: false,
|
||||
showElasticRules: false,
|
||||
excludeRuleTypes: ['esql'],
|
||||
},
|
||||
],
|
||||
[
|
||||
BulkActionsDryRunErrCode.IMMUTABLE,
|
||||
{
|
||||
|
|
|
@ -49,6 +49,12 @@ export const prepareSearchParams = ({
|
|||
excludeRuleTypes: [...(modifiedFilterOptions.excludeRuleTypes ?? []), 'machine_learning'],
|
||||
};
|
||||
break;
|
||||
case BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN:
|
||||
modifiedFilterOptions = {
|
||||
...modifiedFilterOptions,
|
||||
excludeRuleTypes: [...(modifiedFilterOptions.excludeRuleTypes ?? []), 'esql'],
|
||||
};
|
||||
break;
|
||||
}
|
||||
});
|
||||
|
||||
|
|
|
@ -6,11 +6,11 @@
|
|||
*/
|
||||
|
||||
import React from 'react';
|
||||
import { EuiIcon, EuiToolTip, EuiBetaBadge } from '@elastic/eui';
|
||||
import { EuiIcon, EuiToolTip } from '@elastic/eui';
|
||||
|
||||
import type { LicenseService } from '../../../../../common/license';
|
||||
import { minimumLicenseForSuppression } from '../../../../../common/api/detection_engine/model/rule_schema';
|
||||
|
||||
import { TechnicalPreviewBadge } from '../technical_preview_badge';
|
||||
import * as i18n from './translations';
|
||||
|
||||
interface TechnicalPreviewBadgeProps {
|
||||
|
@ -18,14 +18,12 @@ interface TechnicalPreviewBadgeProps {
|
|||
license: LicenseService;
|
||||
}
|
||||
|
||||
export const TechnicalPreviewBadge = ({ label, license }: TechnicalPreviewBadgeProps) => (
|
||||
export const AlertSuppressionTechnicalPreviewBadge = ({
|
||||
label,
|
||||
license,
|
||||
}: TechnicalPreviewBadgeProps) => (
|
||||
<>
|
||||
{label}
|
||||
<EuiBetaBadge
|
||||
label={i18n.ALERT_SUPPRESSION_TECHNICAL_PREVIEW}
|
||||
style={{ verticalAlign: 'middle', marginLeft: '8px' }}
|
||||
size="s"
|
||||
/>
|
||||
<TechnicalPreviewBadge label={label} />
|
||||
{!license.isAtLeast(minimumLicenseForSuppression) && (
|
||||
<EuiToolTip position="top" content={i18n.ALERT_SUPPRESSION_INSUFFICIENT_LICENSE}>
|
||||
<EuiIcon type={'warning'} size="l" color="#BD271E" style={{ marginLeft: '8px' }} />
|
|
@ -46,7 +46,8 @@ import type {
|
|||
import { GroupByOptions } from '../../../pages/detection_engine/rules/types';
|
||||
import { defaultToEmptyTag } from '../../../../common/components/empty_value';
|
||||
import { ThreatEuiFlexGroup } from './threat_description';
|
||||
import { TechnicalPreviewBadge } from './technical_preview_badge';
|
||||
import { AlertSuppressionTechnicalPreviewBadge } from './alert_suppression_technical_preview_badge';
|
||||
import { TechnicalPreviewBadge } from '../technical_preview_badge';
|
||||
import type { LicenseService } from '../../../../../common/license';
|
||||
import { AlertSuppressionMissingFieldsStrategy } from '../../../../../common/api/detection_engine/model/rule_schema';
|
||||
const NoteDescriptionContainer = styled(EuiFlexItem)`
|
||||
|
@ -449,6 +450,14 @@ export const buildRuleTypeDescription = (label: string, ruleType: Type): ListIte
|
|||
},
|
||||
];
|
||||
}
|
||||
case 'esql': {
|
||||
return [
|
||||
{
|
||||
title: label,
|
||||
description: <TechnicalPreviewBadge label={i18n.ESQL_TYPE_DESCRIPTION} />,
|
||||
},
|
||||
];
|
||||
}
|
||||
default:
|
||||
return assertUnreachable(ruleType);
|
||||
}
|
||||
|
@ -571,7 +580,7 @@ export const buildAlertSuppressionDescription = (
|
|||
</EuiFlexGroup>
|
||||
);
|
||||
|
||||
const title = <TechnicalPreviewBadge label={label} license={license} />;
|
||||
const title = <AlertSuppressionTechnicalPreviewBadge label={label} license={license} />;
|
||||
return [
|
||||
{
|
||||
title,
|
||||
|
@ -591,7 +600,7 @@ export const buildAlertSuppressionWindowDescription = (
|
|||
? `${value.value}${value.unit}`
|
||||
: i18n.ALERT_SUPPRESSION_PER_RULE_EXECUTION;
|
||||
|
||||
const title = <TechnicalPreviewBadge label={label} license={license} />;
|
||||
const title = <AlertSuppressionTechnicalPreviewBadge label={label} license={license} />;
|
||||
return [
|
||||
{
|
||||
title,
|
||||
|
@ -614,7 +623,7 @@ export const buildAlertSuppressionMissingFieldsDescription = (
|
|||
? i18n.ALERT_SUPPRESSION_SUPPRESS_ON_MISSING_FIELDS
|
||||
: i18n.ALERT_SUPPRESSION_DO_NOT_SUPPRESS_ON_MISSING_FIELDS;
|
||||
|
||||
const title = <TechnicalPreviewBadge label={label} license={license} />;
|
||||
const title = <AlertSuppressionTechnicalPreviewBadge label={label} license={license} />;
|
||||
return [
|
||||
{
|
||||
title,
|
||||
|
|
|
@ -91,6 +91,13 @@ export const NEW_TERMS_TYPE_DESCRIPTION = i18n.translate(
|
|||
}
|
||||
);
|
||||
|
||||
export const ESQL_TYPE_DESCRIPTION = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.esqlRuleTypeDescription',
|
||||
{
|
||||
defaultMessage: 'ES|QL',
|
||||
}
|
||||
);
|
||||
|
||||
export const THRESHOLD_RESULTS_ALL = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.ruleDescription.thresholdResultsAllDescription',
|
||||
{
|
||||
|
@ -134,13 +141,6 @@ export const ALERT_SUPPRESSION_INSUFFICIENT_LICENSE = i18n.translate(
|
|||
}
|
||||
);
|
||||
|
||||
export const ALERT_SUPPRESSION_TECHNICAL_PREVIEW = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.ruleDescription.alertSuppressionTechnicalPreview',
|
||||
{
|
||||
defaultMessage: 'Technical Preview',
|
||||
}
|
||||
);
|
||||
|
||||
export const ALERT_SUPPRESSION_PER_RULE_EXECUTION = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.ruleDescription.alertSuppressionPerRuleExecution',
|
||||
{
|
||||
|
|
|
@ -113,6 +113,39 @@ const isNewTermsPreviewDisabled = (newTermsFields: string[]): boolean => {
|
|||
return newTermsFields.length === 0 || newTermsFields.length > MAX_NUMBER_OF_NEW_TERMS_FIELDS;
|
||||
};
|
||||
|
||||
const isEsqlPreviewDisabled = ({
|
||||
isQueryBarValid,
|
||||
queryBar,
|
||||
}: {
|
||||
queryBar: FieldValueQueryBar;
|
||||
isQueryBarValid: boolean;
|
||||
}): boolean => {
|
||||
return !isQueryBarValid || isEmpty(queryBar.query.query);
|
||||
};
|
||||
|
||||
const isThreatMatchPreviewDisabled = ({
|
||||
isThreatQueryBarValid,
|
||||
threatIndex,
|
||||
threatMapping,
|
||||
}: {
|
||||
threatIndex: string[];
|
||||
threatMapping: ThreatMapping;
|
||||
isThreatQueryBarValid: boolean;
|
||||
}): boolean => {
|
||||
if (!isThreatQueryBarValid || !threatIndex.length || !threatMapping) {
|
||||
return true;
|
||||
} else if (
|
||||
!threatMapping.length ||
|
||||
!threatMapping[0].entries?.length ||
|
||||
!threatMapping[0].entries[0].field ||
|
||||
!threatMapping[0].entries[0].value
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
};
|
||||
|
||||
export const getIsRulePreviewDisabled = ({
|
||||
ruleType,
|
||||
isQueryBarValid,
|
||||
|
@ -138,6 +171,9 @@ export const getIsRulePreviewDisabled = ({
|
|||
queryBar: FieldValueQueryBar;
|
||||
newTermsFields: string[];
|
||||
}) => {
|
||||
if (ruleType === 'esql') {
|
||||
return isEsqlPreviewDisabled({ isQueryBarValid, queryBar });
|
||||
}
|
||||
if (
|
||||
!isQueryBarValid ||
|
||||
(dataSourceType === DataSourceType.DataView && !dataViewId) ||
|
||||
|
@ -146,14 +182,11 @@ export const getIsRulePreviewDisabled = ({
|
|||
return true;
|
||||
}
|
||||
if (ruleType === 'threat_match') {
|
||||
if (!isThreatQueryBarValid || !threatIndex.length || !threatMapping) return true;
|
||||
if (
|
||||
!threatMapping.length ||
|
||||
!threatMapping[0].entries?.length ||
|
||||
!threatMapping[0].entries[0].field ||
|
||||
!threatMapping[0].entries[0].value
|
||||
)
|
||||
return true;
|
||||
return isThreatMatchPreviewDisabled({
|
||||
threatIndex,
|
||||
threatMapping,
|
||||
isThreatQueryBarValid,
|
||||
});
|
||||
}
|
||||
if (ruleType === 'machine_learning') {
|
||||
return machineLearningJobId.length === 0;
|
||||
|
|
|
@ -10,8 +10,15 @@ import { mount, shallow } from 'enzyme';
|
|||
|
||||
import { SelectRuleType } from '.';
|
||||
import { TestProviders, useFormFieldMock } from '../../../../common/mock';
|
||||
import { useIsExperimentalFeatureEnabled } from '../../../../common/hooks/use_experimental_features';
|
||||
|
||||
jest.mock('../../../../common/lib/kibana');
|
||||
|
||||
jest.mock('../../../../common/hooks/use_experimental_features', () => ({
|
||||
useIsExperimentalFeatureEnabled: jest.fn().mockReturnValue(false),
|
||||
}));
|
||||
const useIsExperimentalFeatureEnabledMock = useIsExperimentalFeatureEnabled as jest.Mock;
|
||||
|
||||
describe('SelectRuleType', () => {
|
||||
it('renders correctly', () => {
|
||||
const Component = () => {
|
||||
|
@ -51,6 +58,7 @@ describe('SelectRuleType', () => {
|
|||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeTruthy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeTruthy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeTruthy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeTruthy();
|
||||
});
|
||||
|
||||
it('renders only the card selected when in update mode of "eql"', () => {
|
||||
|
@ -71,6 +79,7 @@ describe('SelectRuleType', () => {
|
|||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeTruthy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
|
||||
});
|
||||
|
||||
it('renders only the card selected when in update mode of "machine_learning', () => {
|
||||
|
@ -91,6 +100,7 @@ describe('SelectRuleType', () => {
|
|||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
|
||||
});
|
||||
|
||||
it('renders only the card selected when in update mode of "query', () => {
|
||||
|
@ -111,6 +121,7 @@ describe('SelectRuleType', () => {
|
|||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
|
||||
});
|
||||
|
||||
it('renders only the card selected when in update mode of "threshold"', () => {
|
||||
|
@ -131,6 +142,7 @@ describe('SelectRuleType', () => {
|
|||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeTruthy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
|
||||
});
|
||||
|
||||
it('renders only the card selected when in update mode of "threat_match', () => {
|
||||
|
@ -151,6 +163,47 @@ describe('SelectRuleType', () => {
|
|||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeTruthy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
|
||||
});
|
||||
|
||||
it('renders only the card selected when in update mode of "esql"', () => {
|
||||
const field = useFormFieldMock<unknown>({ value: 'esql' });
|
||||
const wrapper = mount(
|
||||
<TestProviders>
|
||||
<SelectRuleType
|
||||
describedByIds={[]}
|
||||
field={field}
|
||||
isUpdateView={true}
|
||||
hasValidLicense={true}
|
||||
isMlAdmin={true}
|
||||
/>
|
||||
</TestProviders>
|
||||
);
|
||||
expect(wrapper.find('[data-test-subj="customRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="machineLearningRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeTruthy();
|
||||
});
|
||||
|
||||
it('should not render "esql" rule type if its feature disabled', () => {
|
||||
useIsExperimentalFeatureEnabledMock.mockReturnValue(false);
|
||||
const Component = () => {
|
||||
const field = useFormFieldMock();
|
||||
|
||||
return (
|
||||
<SelectRuleType
|
||||
field={field}
|
||||
describedByIds={[]}
|
||||
isUpdateView={false}
|
||||
hasValidLicense={true}
|
||||
isMlAdmin={true}
|
||||
/>
|
||||
);
|
||||
};
|
||||
const wrapper = shallow(<Component />);
|
||||
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
@ -16,10 +16,13 @@ import {
|
|||
isQueryRule,
|
||||
isThreatMatchRule,
|
||||
isNewTermsRule,
|
||||
isEsqlRule,
|
||||
} from '../../../../../common/detection_engine/utils';
|
||||
import type { FieldHook } from '../../../../shared_imports';
|
||||
import * as i18n from './translations';
|
||||
import { MlCardDescription } from './ml_card_description';
|
||||
import { TechnicalPreviewBadge } from '../technical_preview_badge';
|
||||
import { useIsExperimentalFeatureEnabled } from '../../../../common/hooks/use_experimental_features';
|
||||
|
||||
interface SelectRuleTypeProps {
|
||||
describedByIds: string[];
|
||||
|
@ -44,6 +47,9 @@ export const SelectRuleType: React.FC<SelectRuleTypeProps> = memo(
|
|||
const setThreshold = useCallback(() => setType('threshold'), [setType]);
|
||||
const setThreatMatch = useCallback(() => setType('threat_match'), [setType]);
|
||||
const setNewTerms = useCallback(() => setType('new_terms'), [setType]);
|
||||
const setEsql = useCallback(() => setType('esql'), [setType]);
|
||||
|
||||
const isEsqlFeatureEnabled = !useIsExperimentalFeatureEnabled('esqlRulesDisabled');
|
||||
|
||||
const eqlSelectableConfig = useMemo(
|
||||
() => ({
|
||||
|
@ -94,6 +100,14 @@ export const SelectRuleType: React.FC<SelectRuleTypeProps> = memo(
|
|||
[ruleType, setNewTerms]
|
||||
);
|
||||
|
||||
const esqlSelectableConfig = useMemo(
|
||||
() => ({
|
||||
onClick: setEsql,
|
||||
isSelected: isEsqlRule(ruleType),
|
||||
}),
|
||||
[ruleType, setEsql]
|
||||
);
|
||||
|
||||
return (
|
||||
<EuiFormRow
|
||||
fullWidth
|
||||
|
@ -181,6 +195,19 @@ export const SelectRuleType: React.FC<SelectRuleTypeProps> = memo(
|
|||
/>
|
||||
</EuiFlexItem>
|
||||
)}
|
||||
{isEsqlFeatureEnabled && (!isUpdateView || esqlSelectableConfig.isSelected) && (
|
||||
<EuiFlexItem>
|
||||
<EuiCard
|
||||
data-test-subj="esqlRuleType"
|
||||
title={<TechnicalPreviewBadge label={i18n.ESQL_TYPE_TITLE} />}
|
||||
titleSize="xs"
|
||||
description={i18n.ESQL_TYPE_DESCRIPTION}
|
||||
icon={<EuiIcon type="logoElasticsearch" size="l" />}
|
||||
selectable={esqlSelectableConfig}
|
||||
layout="horizontal"
|
||||
/>
|
||||
</EuiFlexItem>
|
||||
)}
|
||||
</EuiFlexGrid>
|
||||
</EuiFormRow>
|
||||
);
|
||||
|
|
|
@ -22,6 +22,20 @@ export const EQL_TYPE_DESCRIPTION = i18n.translate(
|
|||
}
|
||||
);
|
||||
|
||||
export const ESQL_TYPE_TITLE = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.ruleTypeField.esqlTypeTitle',
|
||||
{
|
||||
defaultMessage: 'ES|QL',
|
||||
}
|
||||
);
|
||||
|
||||
export const ESQL_TYPE_DESCRIPTION = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.ruleTypeField.esqlTypeDescription',
|
||||
{
|
||||
defaultMessage: 'Use The Elasticsearch Query Language (ES|QL) to search or aggregate events',
|
||||
}
|
||||
);
|
||||
|
||||
export const QUERY_TYPE_TITLE = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.ruleTypeField.queryTypeTitle',
|
||||
{
|
||||
|
|
|
@ -12,7 +12,8 @@ import styled from 'styled-components';
|
|||
|
||||
import type { DataViewBase } from '@kbn/es-query';
|
||||
import type { Severity, Type } from '@kbn/securitysolution-io-ts-alerting-types';
|
||||
import { isThreatMatchRule } from '../../../../../common/detection_engine/utils';
|
||||
|
||||
import { isThreatMatchRule, isEsqlRule } from '../../../../../common/detection_engine/utils';
|
||||
import type { RuleStepProps, AboutStepRule } from '../../../pages/detection_engine/rules/types';
|
||||
import { AddItem } from '../add_item_form';
|
||||
import { StepRuleDescription } from '../description_step';
|
||||
|
@ -33,6 +34,7 @@ import { useFetchIndex } from '../../../../common/containers/source';
|
|||
import { DEFAULT_INDICATOR_SOURCE_PATH } from '../../../../../common/constants';
|
||||
import { useKibana } from '../../../../common/lib/kibana';
|
||||
import { useRuleIndices } from '../../../../detection_engine/rule_management/logic/use_rule_indices';
|
||||
import { EsqlAutocomplete } from '../../../../detection_engine/rule_creation/components/esql_autocomplete';
|
||||
import { MultiSelectFieldsAutocomplete } from '../multi_select_fields';
|
||||
|
||||
const CommonUseField = getUseField({ component: Field });
|
||||
|
@ -44,7 +46,7 @@ interface StepAboutRuleProps extends RuleStepProps {
|
|||
dataViewId: string | undefined;
|
||||
timestampOverride: string;
|
||||
form: FormHook<AboutStepRule>;
|
||||
|
||||
esqlQuery?: string | undefined;
|
||||
// TODO: https://github.com/elastic/kibana/issues/161456
|
||||
// The About step page contains EuiRange component which does not work properly within memoized parents.
|
||||
// EUI team suggested not to memoize EuiRange/EuiDualRange: https://github.com/elastic/eui/issues/6846
|
||||
|
@ -83,10 +85,12 @@ const StepAboutRuleComponent: FC<StepAboutRuleProps> = ({
|
|||
isUpdateView = false,
|
||||
isLoading,
|
||||
form,
|
||||
esqlQuery,
|
||||
}) => {
|
||||
const { data } = useKibana().services;
|
||||
|
||||
const isThreatMatchRuleValue = useMemo(() => isThreatMatchRule(ruleType), [ruleType]);
|
||||
const isEsqlRuleValue = useMemo(() => isEsqlRule(ruleType), [ruleType]);
|
||||
|
||||
const { ruleIndices } = useRuleIndices(machineLearningJobId, index);
|
||||
|
||||
|
@ -327,18 +331,33 @@ const StepAboutRuleComponent: FC<StepAboutRuleProps> = ({
|
|||
</>
|
||||
)}
|
||||
<EuiSpacer size="l" />
|
||||
<UseField
|
||||
path="ruleNameOverride"
|
||||
component={AutocompleteField}
|
||||
componentProps={{
|
||||
dataTestSubj: 'detectionEngineStepAboutRuleRuleNameOverride',
|
||||
fieldType: 'string',
|
||||
idAria: 'detectionEngineStepAboutRuleRuleNameOverride',
|
||||
indices: indexPattern,
|
||||
isDisabled: isLoading || indexPatternLoading,
|
||||
placeholder: '',
|
||||
}}
|
||||
/>
|
||||
{isEsqlRuleValue ? (
|
||||
<UseField
|
||||
path="ruleNameOverride"
|
||||
component={EsqlAutocomplete}
|
||||
componentProps={{
|
||||
dataTestSubj: 'detectionEngineStepAboutRuleRuleNameOverrideForEsqlRuleType',
|
||||
idAria: 'detectionEngineStepAboutRuleRuleNameOverrideForEsqlRuleType',
|
||||
esqlQuery,
|
||||
fieldType: 'string',
|
||||
isDisabled: isLoading,
|
||||
}}
|
||||
/>
|
||||
) : (
|
||||
<UseField
|
||||
path="ruleNameOverride"
|
||||
component={AutocompleteField}
|
||||
componentProps={{
|
||||
dataTestSubj: 'detectionEngineStepAboutRuleRuleNameOverride',
|
||||
fieldType: 'string',
|
||||
idAria: 'detectionEngineStepAboutRuleRuleNameOverride',
|
||||
indices: indexPattern,
|
||||
isDisabled: isLoading || indexPatternLoading,
|
||||
placeholder: '',
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
|
||||
<EuiSpacer size="l" />
|
||||
<UseField
|
||||
path="timestampOverride"
|
||||
|
|
|
@ -18,7 +18,7 @@ import {
|
|||
EuiRadioGroup,
|
||||
} from '@elastic/eui';
|
||||
import type { FC } from 'react';
|
||||
import React, { memo, useCallback, useState, useEffect, useMemo } from 'react';
|
||||
import React, { memo, useCallback, useState, useEffect, useMemo, useRef } from 'react';
|
||||
|
||||
import styled from 'styled-components';
|
||||
import { i18n as i18nCore } from '@kbn/i18n';
|
||||
|
@ -54,6 +54,7 @@ import { PickTimeline } from '../pick_timeline';
|
|||
import { StepContentWrapper } from '../step_content_wrapper';
|
||||
import { ThresholdInput } from '../threshold_input';
|
||||
import { SuppressionInfoIcon } from '../suppression_info_icon';
|
||||
import { EsqlInfoIcon } from '../../../../detection_engine/rule_creation/components/esql_info_icon';
|
||||
import { Field, Form, getUseField, UseField, UseMultiFields } from '../../../../shared_imports';
|
||||
import type { FormHook } from '../../../../shared_imports';
|
||||
import { schema } from './schema';
|
||||
|
@ -65,6 +66,7 @@ import {
|
|||
isThreatMatchRule,
|
||||
isThresholdRule,
|
||||
isQueryRule,
|
||||
isEsqlRule,
|
||||
} from '../../../../../common/detection_engine/utils';
|
||||
import { EqlQueryBar } from '../eql_query_bar';
|
||||
import { DataViewSelector } from '../data_view_selector';
|
||||
|
@ -174,6 +176,8 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
|
|||
const [threatIndexModified, setThreatIndexModified] = useState(false);
|
||||
const license = useLicense();
|
||||
|
||||
const esqlQueryRef = useRef<DefineStepRule['queryBar'] | undefined>(undefined);
|
||||
|
||||
const { getFields, reset, setFieldValue } = form;
|
||||
|
||||
const setRuleTypeCallback = useSetFieldValueWithCallback({
|
||||
|
@ -312,6 +316,43 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
|
|||
}
|
||||
}, [ruleType, previousRuleType, getFields]);
|
||||
|
||||
/**
|
||||
* ensures when user switches between rule types, written ES|QL query is not getting lost
|
||||
* additional work is required in this code area, as currently switching to EQL will result in query lose
|
||||
* https://github.com/elastic/kibana/issues/166933
|
||||
*/
|
||||
useEffect(() => {
|
||||
const { queryBar: currentQuery } = getFields();
|
||||
if (currentQuery == null) {
|
||||
return;
|
||||
}
|
||||
|
||||
const currentQueryValue = currentQuery.value as DefineStepRule['queryBar'];
|
||||
|
||||
// sets ES|QL query to a default value or earlier added one, when switching to ES|QL rule type
|
||||
if (isEsqlRule(ruleType)) {
|
||||
if (previousRuleType && !isEsqlRule(previousRuleType)) {
|
||||
currentQuery.reset({
|
||||
defaultValue: esqlQueryRef.current ?? defaultCustomQuery.forEsqlRules,
|
||||
});
|
||||
}
|
||||
// otherwise reset it to default values of other rule types
|
||||
} else if (isEsqlRule(previousRuleType)) {
|
||||
// sets ES|QL query value to reference, so it can be used when user switch back from one rule type to another
|
||||
if (currentQueryValue?.query?.language === 'esql') {
|
||||
esqlQueryRef.current = currentQueryValue;
|
||||
}
|
||||
|
||||
const defaultValue = isThreatMatchRule(ruleType)
|
||||
? defaultCustomQuery.forThreatMatchRules
|
||||
: defaultCustomQuery.forNormalRules;
|
||||
|
||||
currentQuery.reset({
|
||||
defaultValue,
|
||||
});
|
||||
}
|
||||
}, [ruleType, previousRuleType, getFields]);
|
||||
|
||||
// if saved query failed to load:
|
||||
// - reset shouldLoadFormDynamically to false, as non existent query cannot be used for loading and execution
|
||||
const handleSavedQueryError = useCallback(() => {
|
||||
|
@ -570,6 +611,45 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
|
|||
handleResetIndices,
|
||||
]);
|
||||
|
||||
const queryBarProps = useMemo(
|
||||
() =>
|
||||
({
|
||||
idAria: 'detectionEngineStepDefineRuleQueryBar',
|
||||
indexPattern,
|
||||
isDisabled: isLoading,
|
||||
isLoading,
|
||||
dataTestSubj: 'detectionEngineStepDefineRuleQueryBar',
|
||||
onValidityChange: setIsQueryBarValid,
|
||||
} as QueryBarDefineRuleProps),
|
||||
[indexPattern, isLoading, setIsQueryBarValid]
|
||||
);
|
||||
|
||||
const esqlQueryBarConfig = useMemo(
|
||||
() => ({
|
||||
...schema.queryBar,
|
||||
label: i18n.ESQL_QUERY,
|
||||
labelAppend: <EsqlInfoIcon />,
|
||||
}),
|
||||
[]
|
||||
);
|
||||
|
||||
const EsqlQueryBarMemo = useMemo(
|
||||
() => (
|
||||
<UseField
|
||||
key="QueryBarDefineRule"
|
||||
path="queryBar"
|
||||
config={esqlQueryBarConfig}
|
||||
component={QueryBarDefineRule}
|
||||
componentProps={{
|
||||
...queryBarProps,
|
||||
dataTestSubj: 'detectionEngineStepDefineRuleEsqlQueryBar',
|
||||
idAria: 'detectionEngineStepDefineRuleEsqlQueryBar',
|
||||
}}
|
||||
/>
|
||||
),
|
||||
[queryBarProps, esqlQueryBarConfig]
|
||||
);
|
||||
|
||||
const QueryBarMemo = useMemo(
|
||||
() => (
|
||||
<UseField
|
||||
|
@ -688,8 +768,10 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
|
|||
/>
|
||||
<RuleTypeEuiFormRow $isVisible={!isMlRule(ruleType)} fullWidth>
|
||||
<>
|
||||
<EuiSpacer size="s" />
|
||||
{DataSource}
|
||||
<StyledVisibleContainer isVisible={!isEsqlRule(ruleType)}>
|
||||
<EuiSpacer size="s" />
|
||||
{DataSource}
|
||||
</StyledVisibleContainer>
|
||||
<EuiSpacer size="s" />
|
||||
{isEqlRule(ruleType) ? (
|
||||
<UseField
|
||||
|
@ -715,6 +797,8 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
|
|||
label: i18n.EQL_QUERY_BAR_LABEL,
|
||||
}}
|
||||
/>
|
||||
) : isEsqlRule(ruleType) ? (
|
||||
EsqlQueryBarMemo
|
||||
) : (
|
||||
QueryBarMemo
|
||||
)}
|
||||
|
|
|
@ -17,7 +17,7 @@ import {
|
|||
customValidators,
|
||||
} from '../../../../common/components/threat_match/helpers';
|
||||
import {
|
||||
isEqlRule,
|
||||
isEsqlRule,
|
||||
isNewTermsRule,
|
||||
isQueryRule,
|
||||
isThreatMatchRule,
|
||||
|
@ -31,15 +31,16 @@ import { FIELD_TYPES, fieldValidators } from '../../../../shared_imports';
|
|||
import type { DefineStepRule } from '../../../pages/detection_engine/rules/types';
|
||||
import { DataSourceType } from '../../../pages/detection_engine/rules/types';
|
||||
import { debounceAsync, eqlValidator } from '../eql_query_bar/validators';
|
||||
import { esqlValidator } from '../../../../detection_engine/rule_creation/logic/esql_validator';
|
||||
import {
|
||||
CUSTOM_QUERY_REQUIRED,
|
||||
EQL_QUERY_REQUIRED,
|
||||
INVALID_CUSTOM_QUERY,
|
||||
INDEX_HELPER_TEXT,
|
||||
THREAT_MATCH_INDEX_HELPER_TEXT,
|
||||
THREAT_MATCH_REQUIRED,
|
||||
THREAT_MATCH_EMPTIES,
|
||||
} from './translations';
|
||||
import { getQueryRequiredMessage } from './utils';
|
||||
|
||||
export const schema: FormSchema<DefineStepRule> = {
|
||||
index: {
|
||||
|
@ -60,7 +61,9 @@ export const schema: FormSchema<DefineStepRule> = {
|
|||
): ReturnType<ValidationFunc<{}, ERROR_CODE>> | undefined => {
|
||||
const [{ formData }] = args;
|
||||
const skipValidation =
|
||||
isMlRule(formData.ruleType) || formData.dataSourceType !== DataSourceType.IndexPatterns;
|
||||
isMlRule(formData.ruleType) ||
|
||||
isEsqlRule(formData.ruleType) ||
|
||||
formData.dataSourceType !== DataSourceType.IndexPatterns;
|
||||
|
||||
if (skipValidation) {
|
||||
return;
|
||||
|
@ -129,6 +132,7 @@ export const schema: FormSchema<DefineStepRule> = {
|
|||
},
|
||||
eqlOptions: {},
|
||||
queryBar: {
|
||||
fieldsToValidateOnChange: ['queryBar'],
|
||||
validations: [
|
||||
{
|
||||
validator: (
|
||||
|
@ -150,7 +154,7 @@ export const schema: FormSchema<DefineStepRule> = {
|
|||
// https://github.com/elastic/kibana/issues/159060
|
||||
return undefined;
|
||||
}
|
||||
const message = isEqlRule(formData.ruleType) ? EQL_QUERY_REQUIRED : CUSTOM_QUERY_REQUIRED;
|
||||
const message = getQueryRequiredMessage(formData.ruleType);
|
||||
return { code: 'ERR_FIELD_MISSING', path, message };
|
||||
},
|
||||
},
|
||||
|
@ -181,6 +185,9 @@ export const schema: FormSchema<DefineStepRule> = {
|
|||
{
|
||||
validator: debounceAsync(eqlValidator, 300),
|
||||
},
|
||||
{
|
||||
validator: debounceAsync(esqlValidator, 300),
|
||||
},
|
||||
],
|
||||
},
|
||||
ruleType: {
|
||||
|
|
|
@ -21,6 +21,13 @@ export const EQL_QUERY_REQUIRED = i18n.translate(
|
|||
}
|
||||
);
|
||||
|
||||
export const ESQL_QUERY_REQUIRED = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlQueryFieldRequiredError',
|
||||
{
|
||||
defaultMessage: 'An ES|QL query is required.',
|
||||
}
|
||||
);
|
||||
|
||||
export const INVALID_CUSTOM_QUERY = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.customQueryFieldInvalidError',
|
||||
{
|
||||
|
@ -172,6 +179,13 @@ export const ALERT_SUPPRESSION_MISSING_FIELDS_DO_NOT_SUPPRESS_OPTION = i18n.tran
|
|||
}
|
||||
);
|
||||
|
||||
export const ESQL_QUERY = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlQueryLabel',
|
||||
{
|
||||
defaultMessage: 'ES|QL query',
|
||||
}
|
||||
);
|
||||
|
||||
export const GROUP_BY_FIELD_LICENSE_WARNING = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.groupBy.licenseWarning',
|
||||
{
|
||||
|
|
|
@ -5,8 +5,14 @@
|
|||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Type } from '@kbn/securitysolution-io-ts-alerting-types';
|
||||
|
||||
import type { BrowserField } from '../../../../common/containers/source';
|
||||
|
||||
import { CUSTOM_QUERY_REQUIRED, EQL_QUERY_REQUIRED, ESQL_QUERY_REQUIRED } from './translations';
|
||||
|
||||
import { isEqlRule, isEsqlRule } from '../../../../../common/detection_engine/utils';
|
||||
|
||||
/**
|
||||
* Filters out fields, that are not supported in terms aggregation.
|
||||
* Terms aggregation supports limited number of types:
|
||||
|
@ -19,3 +25,18 @@ export const getTermsAggregationFields = (fields: BrowserField[]): BrowserField[
|
|||
|
||||
return fields.filter((field) => field.aggregatable === true && allowedTypesSet.has(field.type));
|
||||
};
|
||||
|
||||
/**
|
||||
* return query is required message depends on a rule type
|
||||
*/
|
||||
export const getQueryRequiredMessage = (ruleType: Type) => {
|
||||
if (isEsqlRule(ruleType)) {
|
||||
return ESQL_QUERY_REQUIRED;
|
||||
}
|
||||
|
||||
if (isEqlRule(ruleType)) {
|
||||
return EQL_QUERY_REQUIRED;
|
||||
}
|
||||
|
||||
return CUSTOM_QUERY_REQUIRED;
|
||||
};
|
||||
|
|
|
@ -0,0 +1,26 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import React from 'react';
|
||||
import { EuiBetaBadge } from '@elastic/eui';
|
||||
|
||||
import * as i18n from './translations';
|
||||
|
||||
interface TechnicalPreviewBadgeProps {
|
||||
label: string;
|
||||
}
|
||||
|
||||
export const TechnicalPreviewBadge = ({ label }: TechnicalPreviewBadgeProps) => (
|
||||
<>
|
||||
{label}
|
||||
<EuiBetaBadge
|
||||
label={i18n.TECHNICAL_PREVIEW}
|
||||
style={{ verticalAlign: 'middle', marginLeft: '8px' }}
|
||||
size="s"
|
||||
/>
|
||||
</>
|
||||
);
|
|
@ -0,0 +1,15 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { i18n } from '@kbn/i18n';
|
||||
|
||||
export const TECHNICAL_PREVIEW = i18n.translate(
|
||||
'xpack.securitySolution.detectionEngine.ruleDescription.TechnicalPreviewLabel',
|
||||
{
|
||||
defaultMessage: 'Technical Preview',
|
||||
}
|
||||
);
|
|
@ -359,6 +359,7 @@ const commonRuleParamsKeys = [
|
|||
'version',
|
||||
];
|
||||
const queryRuleParams = ['index', 'filters', 'language', 'query', 'saved_id', 'response_actions'];
|
||||
const esqlRuleParams = ['filters', 'language', 'query', 'response_actions'];
|
||||
const machineLearningRuleParams = ['anomaly_threshold', 'machine_learning_job_id'];
|
||||
const thresholdRuleParams = ['threshold', ...queryRuleParams];
|
||||
|
||||
|
@ -379,6 +380,8 @@ const getRuleSpecificRuleParamKeys = (ruleType: Type) => {
|
|||
return machineLearningRuleParams;
|
||||
case 'threshold':
|
||||
return thresholdRuleParams;
|
||||
case 'esql':
|
||||
return esqlRuleParams;
|
||||
case 'new_terms':
|
||||
case 'threat_match':
|
||||
case 'query':
|
||||
|
|
|
@ -132,4 +132,9 @@ const threatQueryBarDefaultValue: DefineStepRule['queryBar'] = {
|
|||
export const defaultCustomQuery = {
|
||||
forNormalRules: stepDefineDefaultValue.queryBar,
|
||||
forThreatMatchRules: threatQueryBarDefaultValue,
|
||||
forEsqlRules: {
|
||||
query: { query: '', language: 'esql' },
|
||||
filters: [],
|
||||
saved_id: null,
|
||||
},
|
||||
};
|
||||
|
|
|
@ -49,6 +49,7 @@ import type { CloudExperimentsPluginStart } from '@kbn/cloud-experiments-plugin/
|
|||
import type { GuidedOnboardingPluginStart } from '@kbn/guided-onboarding-plugin/public';
|
||||
import type { DataViewsServicePublic } from '@kbn/data-views-plugin/public';
|
||||
import type { ContentManagementPublicStart } from '@kbn/content-management-plugin/public';
|
||||
import type { ExpressionsStart } from '@kbn/expressions-plugin/public';
|
||||
|
||||
import type { RouteProps } from 'react-router-dom';
|
||||
import type { DiscoverStart } from '@kbn/discover-plugin/public';
|
||||
|
@ -132,6 +133,7 @@ export interface StartPlugins {
|
|||
fieldFormats: FieldFormatsStartCommon;
|
||||
discover: DiscoverStart;
|
||||
navigation: NavigationPublicPluginStart;
|
||||
expressions: ExpressionsStart;
|
||||
dataViewEditor: DataViewEditorStart;
|
||||
savedSearch: SavedSearchPublicPluginStart;
|
||||
}
|
||||
|
|
|
@ -14,6 +14,7 @@ import type {
|
|||
DiffableCommonFields,
|
||||
DiffableCustomQueryFields,
|
||||
DiffableEqlFields,
|
||||
DiffableEsqlFields,
|
||||
DiffableMachineLearningFields,
|
||||
DiffableNewTermsFields,
|
||||
DiffableRule,
|
||||
|
@ -23,6 +24,7 @@ import type {
|
|||
CommonFieldsDiff,
|
||||
CustomQueryFieldsDiff,
|
||||
EqlFieldsDiff,
|
||||
EsqlFieldsDiff,
|
||||
MachineLearningFieldsDiff,
|
||||
NewTermsFieldsDiff,
|
||||
RuleFieldsDiff,
|
||||
|
@ -142,6 +144,16 @@ export const calculateRuleFieldsDiff = (
|
|||
...calculateNewTermsFieldsDiff({ base_version, current_version, target_version }),
|
||||
};
|
||||
}
|
||||
case 'esql': {
|
||||
if (hasBaseVersion) {
|
||||
invariant(base_version.type === 'esql', BASE_TYPE_ERROR);
|
||||
}
|
||||
invariant(target_version.type === 'esql', TARGET_TYPE_ERROR);
|
||||
return {
|
||||
...commonFieldsDiff,
|
||||
...calculateEsqlFieldsDiff({ base_version, current_version, target_version }),
|
||||
};
|
||||
}
|
||||
default: {
|
||||
return assertUnreachable(current_version, 'Unhandled rule type');
|
||||
}
|
||||
|
@ -226,6 +238,17 @@ const eqlFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableEqlFields> = {
|
|||
tiebreaker_field: simpleDiffAlgorithm,
|
||||
};
|
||||
|
||||
const calculateEsqlFieldsDiff = (
|
||||
ruleVersions: ThreeVersionsOf<DiffableEsqlFields>
|
||||
): EsqlFieldsDiff => {
|
||||
return calculateFieldsDiffFor(ruleVersions, esqlFieldsDiffAlgorithms);
|
||||
};
|
||||
|
||||
const esqlFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableEsqlFields> = {
|
||||
type: simpleDiffAlgorithm,
|
||||
esql_query: simpleDiffAlgorithm,
|
||||
};
|
||||
|
||||
const calculateThreatMatchFieldsDiff = (
|
||||
ruleVersions: ThreeVersionsOf<DiffableThreatMatchFields>
|
||||
): ThreatMatchFieldsDiff => {
|
||||
|
|
|
@ -10,6 +10,8 @@ import { assertUnreachable } from '../../../../../../../common/utility_types';
|
|||
import type {
|
||||
EqlRule,
|
||||
EqlRuleCreateProps,
|
||||
EsqlRule,
|
||||
EsqlRuleCreateProps,
|
||||
MachineLearningRule,
|
||||
MachineLearningRuleCreateProps,
|
||||
NewTermsRule,
|
||||
|
@ -29,6 +31,7 @@ import type {
|
|||
DiffableCommonFields,
|
||||
DiffableCustomQueryFields,
|
||||
DiffableEqlFields,
|
||||
DiffableEsqlFields,
|
||||
DiffableMachineLearningFields,
|
||||
DiffableNewTermsFields,
|
||||
DiffableRule,
|
||||
|
@ -40,6 +43,7 @@ import { extractBuildingBlockObject } from './extract_building_block_object';
|
|||
import {
|
||||
extractInlineKqlQuery,
|
||||
extractRuleEqlQuery,
|
||||
extractRuleEsqlQuery,
|
||||
extractRuleKqlQuery,
|
||||
} from './extract_rule_data_query';
|
||||
import { extractRuleDataSource } from './extract_rule_data_source';
|
||||
|
@ -91,6 +95,11 @@ export const convertRuleToDiffable = (rule: RuleResponse | PrebuiltRuleAsset): D
|
|||
...commonFields,
|
||||
...extractDiffableNewTermsFieldsFromRuleObject(rule),
|
||||
};
|
||||
case 'esql':
|
||||
return {
|
||||
...commonFields,
|
||||
...extractDiffableEsqlFieldsFromRuleObject(rule),
|
||||
};
|
||||
default:
|
||||
return assertUnreachable(rule, 'Unhandled rule type');
|
||||
}
|
||||
|
@ -176,6 +185,15 @@ const extractDiffableEqlFieldsFromRuleObject = (
|
|||
};
|
||||
};
|
||||
|
||||
const extractDiffableEsqlFieldsFromRuleObject = (
|
||||
rule: EsqlRule | EsqlRuleCreateProps
|
||||
): DiffableEsqlFields => {
|
||||
return {
|
||||
type: rule.type,
|
||||
esql_query: extractRuleEsqlQuery(rule.query, rule.language),
|
||||
};
|
||||
};
|
||||
|
||||
const extractDiffableThreatMatchFieldsFromRuleObject = (
|
||||
rule: ThreatMatchRule | ThreatMatchRuleCreateProps
|
||||
): DiffableThreatMatchFields => {
|
||||
|
|
|
@ -7,6 +7,7 @@
|
|||
|
||||
import type {
|
||||
EqlQueryLanguage,
|
||||
EsqlQueryLanguage,
|
||||
KqlQueryLanguage,
|
||||
RuleFilterArray,
|
||||
RuleQuery,
|
||||
|
@ -14,6 +15,7 @@ import type {
|
|||
import type {
|
||||
InlineKqlQuery,
|
||||
RuleEqlQuery,
|
||||
RuleEsqlQuery,
|
||||
RuleKqlQuery,
|
||||
} from '../../../../../../../common/api/detection_engine/prebuilt_rules';
|
||||
import { KqlQueryType } from '../../../../../../../common/api/detection_engine/prebuilt_rules';
|
||||
|
@ -58,3 +60,13 @@ export const extractRuleEqlQuery = (
|
|||
filters: filters ?? [],
|
||||
};
|
||||
};
|
||||
|
||||
export const extractRuleEsqlQuery = (
|
||||
query: RuleQuery,
|
||||
language: EsqlQueryLanguage
|
||||
): RuleEsqlQuery => {
|
||||
return {
|
||||
query,
|
||||
language,
|
||||
};
|
||||
};
|
||||
|
|
|
@ -376,6 +376,41 @@ describe('ruleParamsModifier', () => {
|
|||
"Index patterns can't be overwritten. Machine learning rule doesn't have index patterns property"
|
||||
);
|
||||
});
|
||||
|
||||
test('should throw error on adding index pattern if rule is of ES|QL type', () => {
|
||||
expect(() =>
|
||||
ruleParamsModifier({ type: 'esql' } as RuleAlertType['params'], [
|
||||
{
|
||||
type: BulkActionEditType.add_index_patterns,
|
||||
value: ['my-index-*'],
|
||||
},
|
||||
])
|
||||
).toThrow("Index patterns can't be added. ES|QL rule doesn't have index patterns property");
|
||||
});
|
||||
|
||||
test('should throw error on deleting index pattern if rule is of ES|QL type', () => {
|
||||
expect(() =>
|
||||
ruleParamsModifier({ type: 'esql' } as RuleAlertType['params'], [
|
||||
{
|
||||
type: BulkActionEditType.delete_index_patterns,
|
||||
value: ['my-index-*'],
|
||||
},
|
||||
])
|
||||
).toThrow("Index patterns can't be deleted. ES|QL rule doesn't have index patterns property");
|
||||
});
|
||||
|
||||
test('should throw error on overwriting index pattern if rule is of ES|QL type', () => {
|
||||
expect(() =>
|
||||
ruleParamsModifier({ type: 'esql' } as RuleAlertType['params'], [
|
||||
{
|
||||
type: BulkActionEditType.set_index_patterns,
|
||||
value: ['my-index-*'],
|
||||
},
|
||||
])
|
||||
).toThrow(
|
||||
"Index patterns can't be overwritten. ES|QL rule doesn't have index patterns property"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('timeline', () => {
|
||||
|
|
|
@ -85,6 +85,10 @@ const applyBulkActionEditToRuleParams = (
|
|||
ruleParams.type !== 'machine_learning',
|
||||
"Index patterns can't be added. Machine learning rule doesn't have index patterns property"
|
||||
);
|
||||
invariant(
|
||||
ruleParams.type !== 'esql',
|
||||
"Index patterns can't be added. ES|QL rule doesn't have index patterns property"
|
||||
);
|
||||
|
||||
if (shouldSkipIndexPatternsBulkAction(ruleParams.index, ruleParams.dataViewId, action)) {
|
||||
isActionSkipped = true;
|
||||
|
@ -103,6 +107,10 @@ const applyBulkActionEditToRuleParams = (
|
|||
ruleParams.type !== 'machine_learning',
|
||||
"Index patterns can't be deleted. Machine learning rule doesn't have index patterns property"
|
||||
);
|
||||
invariant(
|
||||
ruleParams.type !== 'esql',
|
||||
"Index patterns can't be deleted. ES|QL rule doesn't have index patterns property"
|
||||
);
|
||||
|
||||
if (
|
||||
!action.overwrite_data_views &&
|
||||
|
@ -126,6 +134,10 @@ const applyBulkActionEditToRuleParams = (
|
|||
ruleParams.type !== 'machine_learning',
|
||||
"Index patterns can't be overwritten. Machine learning rule doesn't have index patterns property"
|
||||
);
|
||||
invariant(
|
||||
ruleParams.type !== 'esql',
|
||||
"Index patterns can't be overwritten. ES|QL rule doesn't have index patterns property"
|
||||
);
|
||||
|
||||
if (shouldSkipIndexPatternsBulkAction(ruleParams.index, ruleParams.dataViewId, action)) {
|
||||
isActionSkipped = true;
|
||||
|
|
|
@ -8,6 +8,7 @@
|
|||
import type { Type as RuleType } from '@kbn/securitysolution-io-ts-alerting-types';
|
||||
import { invariant } from '../../../../../../common/utils/invariant';
|
||||
import { isMlRule } from '../../../../../../common/machine_learning/helpers';
|
||||
import { isEsqlRule } from '../../../../../../common/detection_engine/utils';
|
||||
import { BulkActionsDryRunErrCode } from '../../../../../../common/constants';
|
||||
import type { BulkActionEditPayload } from '../../../../../../common/api/detection_engine/rule_management/bulk_actions/bulk_actions_route';
|
||||
import { BulkActionEditType } from '../../../../../../common/api/detection_engine/rule_management/bulk_actions/bulk_actions_route';
|
||||
|
@ -129,4 +130,15 @@ export const dryRunValidateBulkEditRule = async ({
|
|||
),
|
||||
BulkActionsDryRunErrCode.MACHINE_LEARNING_INDEX_PATTERN
|
||||
);
|
||||
|
||||
// if rule is es|ql, index pattern action can't be applied to it
|
||||
await throwDryRunError(
|
||||
() =>
|
||||
invariant(
|
||||
!isEsqlRule(rule.params.type) ||
|
||||
!edit.some((action) => isIndexPatternsBulkEditAction(action.type)),
|
||||
"ES|QL rule doesn't have index patterns"
|
||||
),
|
||||
BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN
|
||||
);
|
||||
};
|
||||
|
|
|
@ -7,6 +7,7 @@
|
|||
|
||||
import {
|
||||
EQL_RULE_TYPE_ID,
|
||||
ESQL_RULE_TYPE_ID,
|
||||
INDICATOR_RULE_TYPE_ID,
|
||||
ML_RULE_TYPE_ID,
|
||||
NEW_TERMS_RULE_TYPE_ID,
|
||||
|
@ -18,6 +19,7 @@ import {
|
|||
import { enrichFilterWithRuleTypeMapping } from './enrich_filter_with_rule_type_mappings';
|
||||
|
||||
const allAlertTypeIds = `alert.attributes.alertTypeId: ${EQL_RULE_TYPE_ID}
|
||||
OR alert.attributes.alertTypeId: ${ESQL_RULE_TYPE_ID}
|
||||
OR alert.attributes.alertTypeId: ${ML_RULE_TYPE_ID}
|
||||
OR alert.attributes.alertTypeId: ${QUERY_RULE_TYPE_ID}
|
||||
OR alert.attributes.alertTypeId: ${SAVED_QUERY_RULE_TYPE_ID}
|
||||
|
|
|
@ -29,6 +29,7 @@ import type {
|
|||
} from '../../../../../common/api/detection_engine/model/rule_schema';
|
||||
import {
|
||||
EqlPatchParams,
|
||||
EsqlPatchParams,
|
||||
MachineLearningPatchParams,
|
||||
NewTermsPatchParams,
|
||||
QueryPatchParams,
|
||||
|
@ -59,6 +60,8 @@ import type {
|
|||
BaseRuleParams,
|
||||
EqlRuleParams,
|
||||
EqlSpecificRuleParams,
|
||||
EsqlRuleParams,
|
||||
EsqlSpecificRuleParams,
|
||||
ThreatRuleParams,
|
||||
ThreatSpecificRuleParams,
|
||||
QueryRuleParams,
|
||||
|
@ -107,6 +110,13 @@ export const typeSpecificSnakeToCamel = (
|
|||
tiebreakerField: params.tiebreaker_field,
|
||||
};
|
||||
}
|
||||
case 'esql': {
|
||||
return {
|
||||
type: params.type,
|
||||
language: params.language,
|
||||
query: params.query,
|
||||
};
|
||||
}
|
||||
case 'threat_match': {
|
||||
return {
|
||||
type: params.type,
|
||||
|
@ -206,6 +216,17 @@ const patchEqlParams = (
|
|||
};
|
||||
};
|
||||
|
||||
const patchEsqlParams = (
|
||||
params: EsqlPatchParams,
|
||||
existingRule: EsqlRuleParams
|
||||
): EsqlSpecificRuleParams => {
|
||||
return {
|
||||
type: existingRule.type,
|
||||
language: params.language ?? existingRule.language,
|
||||
query: params.query ?? existingRule.query,
|
||||
};
|
||||
};
|
||||
|
||||
const patchThreatMatchParams = (
|
||||
params: ThreatMatchPatchParams,
|
||||
existingRule: ThreatRuleParams
|
||||
|
@ -341,6 +362,13 @@ export const patchTypeSpecificSnakeToCamel = (
|
|||
}
|
||||
return patchEqlParams(validated, existingRule);
|
||||
}
|
||||
case 'esql': {
|
||||
const [validated, error] = validateNonExact(params, EsqlPatchParams);
|
||||
if (validated == null) {
|
||||
throw parseValidationError(error);
|
||||
}
|
||||
return patchEsqlParams(validated, existingRule);
|
||||
}
|
||||
case 'threat_match': {
|
||||
const [validated, error] = validateNonExact(params, ThreatMatchPatchParams);
|
||||
if (validated == null) {
|
||||
|
@ -526,6 +554,13 @@ export const typeSpecificCamelToSnake = (params: TypeSpecificRuleParams): TypeSp
|
|||
tiebreaker_field: params.tiebreakerField,
|
||||
};
|
||||
}
|
||||
case 'esql': {
|
||||
return {
|
||||
type: params.type,
|
||||
language: params.language,
|
||||
query: params.query,
|
||||
};
|
||||
}
|
||||
case 'threat_match': {
|
||||
return {
|
||||
type: params.type,
|
||||
|
|
|
@ -53,6 +53,7 @@ import type {
|
|||
} from '../../../rule_types/types';
|
||||
import {
|
||||
createEqlAlertType,
|
||||
createEsqlAlertType,
|
||||
createIndicatorMatchAlertType,
|
||||
createMlAlertType,
|
||||
createQueryAlertType,
|
||||
|
@ -419,6 +420,27 @@ export const previewRulesRoute = async (
|
|||
}
|
||||
);
|
||||
break;
|
||||
case 'esql':
|
||||
if (config.experimentalFeatures.esqlRulesDisabled) {
|
||||
throw Error('ES|QL rule type is not supported');
|
||||
}
|
||||
const esqlAlertType = previewRuleTypeWrapper(createEsqlAlertType(ruleOptions));
|
||||
await runExecutors(
|
||||
esqlAlertType.executor,
|
||||
esqlAlertType.id,
|
||||
esqlAlertType.name,
|
||||
previewRuleParams,
|
||||
() => true,
|
||||
{
|
||||
create: alertInstanceFactoryStub,
|
||||
alertLimit: {
|
||||
getValue: () => 1000,
|
||||
setLimitReached: () => {},
|
||||
},
|
||||
done: () => ({ getRecoveredAlerts: () => [] }),
|
||||
}
|
||||
);
|
||||
break;
|
||||
case 'machine_learning':
|
||||
const mlAlertType = previewRuleTypeWrapper(createMlAlertType(ruleOptions));
|
||||
await runExecutors(
|
||||
|
|
|
@ -28,6 +28,7 @@ import {
|
|||
import {
|
||||
SIGNALS_ID,
|
||||
EQL_RULE_TYPE_ID,
|
||||
ESQL_RULE_TYPE_ID,
|
||||
INDICATOR_RULE_TYPE_ID,
|
||||
ML_RULE_TYPE_ID,
|
||||
QUERY_RULE_TYPE_ID,
|
||||
|
@ -141,6 +142,15 @@ export const eqlRuleParams = t.intersection([baseRuleParams, eqlSpecificRulePara
|
|||
export type EqlSpecificRuleParams = t.TypeOf<typeof eqlSpecificRuleParams>;
|
||||
export type EqlRuleParams = t.TypeOf<typeof eqlRuleParams>;
|
||||
|
||||
const esqlSpecificRuleParams = t.type({
|
||||
type: t.literal('esql'),
|
||||
language: t.literal('esql'),
|
||||
query: RuleQuery,
|
||||
});
|
||||
export const esqlRuleParams = t.intersection([baseRuleParams, esqlSpecificRuleParams]);
|
||||
export type EsqlSpecificRuleParams = t.TypeOf<typeof esqlSpecificRuleParams>;
|
||||
export type EsqlRuleParams = t.TypeOf<typeof esqlRuleParams>;
|
||||
|
||||
const threatSpecificRuleParams = t.type({
|
||||
type: t.literal('threat_match'),
|
||||
language: nonEqlLanguages,
|
||||
|
@ -244,6 +254,7 @@ export type NewTermsRuleParams = t.TypeOf<typeof newTermsRuleParams>;
|
|||
|
||||
export const typeSpecificRuleParams = t.union([
|
||||
eqlSpecificRuleParams,
|
||||
esqlSpecificRuleParams,
|
||||
threatSpecificRuleParams,
|
||||
querySpecificRuleParams,
|
||||
savedQuerySpecificRuleParams,
|
||||
|
@ -265,6 +276,7 @@ export interface CompleteRule<T extends RuleParams> {
|
|||
export const allRuleTypes = t.union([
|
||||
t.literal(SIGNALS_ID),
|
||||
t.literal(EQL_RULE_TYPE_ID),
|
||||
t.literal(ESQL_RULE_TYPE_ID),
|
||||
t.literal(INDICATOR_RULE_TYPE_ID),
|
||||
t.literal(ML_RULE_TYPE_ID),
|
||||
t.literal(QUERY_RULE_TYPE_ID),
|
||||
|
|
|
@ -16,6 +16,7 @@ import { buildExceptionFilter } from '@kbn/lists-plugin/server/services/exceptio
|
|||
import { technicalRuleFieldMap } from '@kbn/rule-registry-plugin/common/assets/field_maps/technical_rule_field_map';
|
||||
import type { FieldMap } from '@kbn/alerts-as-data-utils';
|
||||
import { parseScheduleDates } from '@kbn/securitysolution-io-ts-utils';
|
||||
import { getIndexListFromEsqlQuery } from '@kbn/securitysolution-utils';
|
||||
import type { FormatAlert } from '@kbn/alerting-plugin/server/types';
|
||||
import {
|
||||
checkPrivilegesFromEsClient,
|
||||
|
@ -24,6 +25,7 @@ import {
|
|||
hasReadIndexPrivileges,
|
||||
hasTimestampFields,
|
||||
isMachineLearningParams,
|
||||
isEsqlParams,
|
||||
} from './utils/utils';
|
||||
import { DEFAULT_MAX_SIGNALS, DEFAULT_SEARCH_AFTER_PAGE_SIZE } from '../../../../common/constants';
|
||||
import type { CreateSecurityRuleTypeWrapper } from './types';
|
||||
|
@ -209,13 +211,16 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper =
|
|||
|
||||
/**
|
||||
* Data Views Logic
|
||||
* Use of data views is supported for all rules other than ML.
|
||||
* Use of data views is supported for all rules other than ML and Esql.
|
||||
* Rules can define both a data view and index pattern, but on execution:
|
||||
* - Data view is used if it is defined
|
||||
* - Rule exits early if data view defined is not found (ie: it's been deleted)
|
||||
* - If no data view defined, falls to using existing index logic
|
||||
* Esql rules has index in query, which can be retrieved
|
||||
*/
|
||||
if (!isMachineLearningParams(params)) {
|
||||
if (isEsqlParams(params)) {
|
||||
inputIndex = getIndexListFromEsqlQuery(params.query);
|
||||
} else if (!isMachineLearningParams(params)) {
|
||||
try {
|
||||
const { index, runtimeMappings: dataViewRuntimeMappings } = await getInputIndex({
|
||||
index: params.index,
|
||||
|
@ -322,7 +327,7 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper =
|
|||
});
|
||||
}
|
||||
|
||||
if (!isMachineLearningParams(params)) {
|
||||
if (!isMachineLearningParams(params) && !isEsqlParams(params)) {
|
||||
inputIndexFields = await getFieldsForWildcard({
|
||||
index: inputIndex,
|
||||
dataViews: services.dataViews,
|
||||
|
|
|
@ -0,0 +1,65 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
|
||||
import type { Filter } from '@kbn/es-query';
|
||||
import type {
|
||||
RuleFilterArray,
|
||||
TimestampOverride,
|
||||
} from '../../../../../common/api/detection_engine/model/rule_schema';
|
||||
import { buildTimeRangeFilter } from '../utils/build_events_query';
|
||||
import { getQueryFilter } from '../utils/get_query_filter';
|
||||
|
||||
export interface BuildEqlSearchRequestParams {
|
||||
query: string;
|
||||
from: string;
|
||||
to: string;
|
||||
size: number;
|
||||
filters: RuleFilterArray | undefined;
|
||||
primaryTimestamp: TimestampOverride;
|
||||
secondaryTimestamp: TimestampOverride | undefined;
|
||||
exceptionFilter: Filter | undefined;
|
||||
}
|
||||
|
||||
export const buildEsqlSearchRequest = ({
|
||||
query,
|
||||
from,
|
||||
to,
|
||||
filters,
|
||||
primaryTimestamp,
|
||||
secondaryTimestamp,
|
||||
exceptionFilter,
|
||||
size,
|
||||
}: BuildEqlSearchRequestParams) => {
|
||||
const esFilter = getQueryFilter({
|
||||
query: '',
|
||||
language: 'esql',
|
||||
filters: filters || [],
|
||||
index: undefined,
|
||||
exceptionFilter,
|
||||
});
|
||||
|
||||
const rangeFilter = buildTimeRangeFilter({
|
||||
to,
|
||||
from,
|
||||
primaryTimestamp,
|
||||
secondaryTimestamp,
|
||||
});
|
||||
|
||||
const requestFilter: estypes.QueryDslQueryContainer[] = [rangeFilter, esFilter];
|
||||
|
||||
return {
|
||||
// we limit size of the response to maxAlertNumber + 1
|
||||
// ES|QL currently does not support pagination and returns 10,000 results
|
||||
query: `${query} | limit ${size + 1}`,
|
||||
filter: {
|
||||
bool: {
|
||||
filter: requestFilter,
|
||||
},
|
||||
},
|
||||
};
|
||||
};
|
|
@ -0,0 +1,53 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { validateNonExact } from '@kbn/securitysolution-io-ts-utils';
|
||||
import { ESQL_RULE_TYPE_ID } from '@kbn/securitysolution-rules';
|
||||
|
||||
import { SERVER_APP_ID } from '../../../../../common/constants';
|
||||
import type { EsqlRuleParams } from '../../rule_schema';
|
||||
import { esqlRuleParams } from '../../rule_schema';
|
||||
import { esqlExecutor } from './esql';
|
||||
import type { CreateRuleOptions, SecurityAlertType } from '../types';
|
||||
|
||||
export const createEsqlAlertType = (
|
||||
createOptions: CreateRuleOptions
|
||||
): SecurityAlertType<EsqlRuleParams, {}, {}, 'default'> => {
|
||||
const { version } = createOptions;
|
||||
return {
|
||||
id: ESQL_RULE_TYPE_ID,
|
||||
name: 'ES|QL Rule',
|
||||
validate: {
|
||||
params: {
|
||||
validate: (object: unknown) => {
|
||||
const [validated, errors] = validateNonExact(object, esqlRuleParams);
|
||||
if (errors != null) {
|
||||
throw new Error(errors);
|
||||
}
|
||||
if (validated == null) {
|
||||
throw new Error('Validation of rule params failed');
|
||||
}
|
||||
return validated;
|
||||
},
|
||||
},
|
||||
},
|
||||
actionGroups: [
|
||||
{
|
||||
id: 'default',
|
||||
name: 'Default',
|
||||
},
|
||||
],
|
||||
defaultActionGroupId: 'default',
|
||||
actionVariables: {
|
||||
context: [{ name: 'server', description: 'the server' }],
|
||||
},
|
||||
minimumLicenseRequired: 'basic',
|
||||
isExportable: false,
|
||||
producer: SERVER_APP_ID,
|
||||
executor: (params) => esqlExecutor({ ...params, version }),
|
||||
};
|
||||
};
|
|
@ -0,0 +1,166 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { performance } from 'perf_hooks';
|
||||
import type {
|
||||
AlertInstanceContext,
|
||||
AlertInstanceState,
|
||||
RuleExecutorServices,
|
||||
} from '@kbn/alerting-plugin/server';
|
||||
|
||||
import {
|
||||
computeIsESQLQueryAggregating,
|
||||
getIndexListFromEsqlQuery,
|
||||
} from '@kbn/securitysolution-utils';
|
||||
import { buildEsqlSearchRequest } from './build_esql_search_request';
|
||||
import { performEsqlRequest } from './esql_request';
|
||||
import { wrapEsqlAlerts } from './wrap_esql_alerts';
|
||||
import { createEnrichEventsFunction } from '../utils/enrichments';
|
||||
import { rowToDocument } from './utils';
|
||||
import { fetchSourceDocuments } from './fetch_source_documents';
|
||||
|
||||
import type { RunOpts } from '../types';
|
||||
|
||||
import {
|
||||
addToSearchAfterReturn,
|
||||
createSearchAfterReturnType,
|
||||
makeFloatString,
|
||||
getUnprocessedExceptionsWarnings,
|
||||
getMaxSignalsWarning,
|
||||
} from '../utils/utils';
|
||||
import type { EsqlRuleParams } from '../../rule_schema';
|
||||
import { withSecuritySpan } from '../../../../utils/with_security_span';
|
||||
|
||||
/**
|
||||
* ES|QL returns results as a single page. max size of 10,000
|
||||
* while we try increase size of the request to catch all events
|
||||
* we don't want to overload ES/Kibana with large responses
|
||||
*/
|
||||
const ESQL_PAGE_SIZE_CIRCUIT_BREAKER = 1000;
|
||||
|
||||
export const esqlExecutor = async ({
|
||||
runOpts: {
|
||||
completeRule,
|
||||
tuple,
|
||||
ruleExecutionLogger,
|
||||
bulkCreate,
|
||||
mergeStrategy,
|
||||
primaryTimestamp,
|
||||
secondaryTimestamp,
|
||||
exceptionFilter,
|
||||
unprocessedExceptions,
|
||||
alertTimestampOverride,
|
||||
publicBaseUrl,
|
||||
},
|
||||
services,
|
||||
state,
|
||||
spaceId,
|
||||
}: {
|
||||
runOpts: RunOpts<EsqlRuleParams>;
|
||||
services: RuleExecutorServices<AlertInstanceState, AlertInstanceContext, 'default'>;
|
||||
state: object;
|
||||
spaceId: string;
|
||||
version: string;
|
||||
}) => {
|
||||
const ruleParams = completeRule.ruleParams;
|
||||
|
||||
return withSecuritySpan('esqlExecutor', async () => {
|
||||
const result = createSearchAfterReturnType();
|
||||
let size = tuple.maxSignals;
|
||||
while (
|
||||
result.createdSignalsCount <= tuple.maxSignals &&
|
||||
size <= ESQL_PAGE_SIZE_CIRCUIT_BREAKER
|
||||
) {
|
||||
const esqlRequest = buildEsqlSearchRequest({
|
||||
query: ruleParams.query,
|
||||
from: tuple.from.toISOString(),
|
||||
to: tuple.to.toISOString(),
|
||||
size,
|
||||
filters: [],
|
||||
primaryTimestamp,
|
||||
secondaryTimestamp,
|
||||
exceptionFilter,
|
||||
});
|
||||
|
||||
ruleExecutionLogger.debug(`ES|QL query request: ${JSON.stringify(esqlRequest)}`);
|
||||
const exceptionsWarning = getUnprocessedExceptionsWarnings(unprocessedExceptions);
|
||||
if (exceptionsWarning) {
|
||||
result.warningMessages.push(exceptionsWarning);
|
||||
}
|
||||
|
||||
const esqlSignalSearchStart = performance.now();
|
||||
|
||||
const response = await performEsqlRequest({
|
||||
esClient: services.scopedClusterClient.asCurrentUser,
|
||||
requestParams: esqlRequest,
|
||||
});
|
||||
|
||||
const esqlSearchDuration = makeFloatString(performance.now() - esqlSignalSearchStart);
|
||||
result.searchAfterTimes = [esqlSearchDuration];
|
||||
|
||||
ruleExecutionLogger.debug(`ES|QL query request took: ${esqlSearchDuration}ms`);
|
||||
|
||||
const isRuleAggregating = computeIsESQLQueryAggregating(completeRule.ruleParams.query);
|
||||
|
||||
const results = response.values
|
||||
// slicing already processed results in previous iterations
|
||||
.slice(size - tuple.maxSignals)
|
||||
.map((row) => rowToDocument(response.columns, row));
|
||||
|
||||
const index = getIndexListFromEsqlQuery(completeRule.ruleParams.query);
|
||||
|
||||
const sourceDocuments = await fetchSourceDocuments({
|
||||
esClient: services.scopedClusterClient.asCurrentUser,
|
||||
results,
|
||||
index,
|
||||
isRuleAggregating,
|
||||
});
|
||||
|
||||
const wrappedAlerts = wrapEsqlAlerts({
|
||||
sourceDocuments,
|
||||
isRuleAggregating,
|
||||
results,
|
||||
spaceId,
|
||||
completeRule,
|
||||
mergeStrategy,
|
||||
alertTimestampOverride,
|
||||
ruleExecutionLogger,
|
||||
publicBaseUrl,
|
||||
tuple,
|
||||
});
|
||||
|
||||
const enrichAlerts = createEnrichEventsFunction({
|
||||
services,
|
||||
logger: ruleExecutionLogger,
|
||||
});
|
||||
const bulkCreateResult = await bulkCreate(
|
||||
wrappedAlerts,
|
||||
tuple.maxSignals - result.createdSignalsCount,
|
||||
enrichAlerts
|
||||
);
|
||||
|
||||
addToSearchAfterReturn({ current: result, next: bulkCreateResult });
|
||||
ruleExecutionLogger.debug(`Created ${bulkCreateResult.createdItemsCount} alerts`);
|
||||
|
||||
if (bulkCreateResult.alertsWereTruncated) {
|
||||
result.warningMessages.push(getMaxSignalsWarning());
|
||||
}
|
||||
|
||||
// no more results will be found
|
||||
if (response.values.length < size) {
|
||||
ruleExecutionLogger.debug(
|
||||
`End of search: Found ${response.values.length} results with page size ${size}`
|
||||
);
|
||||
break;
|
||||
}
|
||||
// ES|QL does not support pagination so we need to increase size of response to be able to catch all events
|
||||
size += tuple.maxSignals;
|
||||
}
|
||||
|
||||
return { ...result, state };
|
||||
});
|
||||
};
|
|
@ -0,0 +1,51 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { Logger, ElasticsearchClient } from '@kbn/core/server';
|
||||
import { getKbnServerError } from '@kbn/kibana-utils-plugin/server';
|
||||
|
||||
export interface EsqlResultColumn {
|
||||
name: string;
|
||||
type: 'date' | 'keyword';
|
||||
}
|
||||
|
||||
export type EsqlResultRow = Array<string | null>;
|
||||
|
||||
export interface EsqlTable {
|
||||
columns: EsqlResultColumn[];
|
||||
values: EsqlResultRow[];
|
||||
}
|
||||
|
||||
export const performEsqlRequest = async ({
|
||||
esClient,
|
||||
requestParams,
|
||||
}: {
|
||||
logger?: Logger;
|
||||
esClient: ElasticsearchClient;
|
||||
requestParams: Record<string, unknown>;
|
||||
}): Promise<EsqlTable> => {
|
||||
const search = async () => {
|
||||
try {
|
||||
const rawResponse = await esClient.transport.request<EsqlTable>({
|
||||
method: 'POST',
|
||||
path: '/_query',
|
||||
body: {
|
||||
...requestParams,
|
||||
},
|
||||
});
|
||||
return {
|
||||
rawResponse,
|
||||
isPartial: false,
|
||||
isRunning: false,
|
||||
};
|
||||
} catch (e) {
|
||||
throw getKbnServerError(e);
|
||||
}
|
||||
};
|
||||
|
||||
return (await search()).rawResponse;
|
||||
};
|
|
@ -0,0 +1,67 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { ElasticsearchClient } from '@kbn/core/server';
|
||||
import type * as estypes from '@elastic/elasticsearch/lib/api/types';
|
||||
|
||||
interface FetchSourceDocumentsArgs {
|
||||
isRuleAggregating: boolean;
|
||||
esClient: ElasticsearchClient;
|
||||
index: string[];
|
||||
results: Array<Record<string, string | null>>;
|
||||
}
|
||||
/**
|
||||
* fetches source documents by list of their ids
|
||||
* it used for a case when non-aggregating has _id property to enrich alert with source document,
|
||||
* if some of the properties missed from resulted query
|
||||
*/
|
||||
export const fetchSourceDocuments = async ({
|
||||
isRuleAggregating,
|
||||
results,
|
||||
esClient,
|
||||
index,
|
||||
}: FetchSourceDocumentsArgs): Promise<Record<string, { fields: estypes.SearchHit['fields'] }>> => {
|
||||
const ids = results.reduce<string[]>((acc, doc) => {
|
||||
if (doc._id) {
|
||||
acc.push(doc._id);
|
||||
}
|
||||
return acc;
|
||||
}, []);
|
||||
|
||||
// we will fetch source documents only for non-aggregating rules, since aggregating do not have _id
|
||||
if (ids.length === 0 || isRuleAggregating) {
|
||||
return {};
|
||||
}
|
||||
|
||||
const idsQuery = {
|
||||
query: {
|
||||
bool: {
|
||||
filter: {
|
||||
ids: { values: ids },
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const response = await esClient.search({
|
||||
index,
|
||||
body: {
|
||||
query: idsQuery.query,
|
||||
_source: false,
|
||||
fields: ['*'],
|
||||
},
|
||||
ignore_unavailable: true,
|
||||
});
|
||||
|
||||
return response.hits.hits.reduce<Record<string, { fields: estypes.SearchHit['fields'] }>>(
|
||||
(acc, hit) => {
|
||||
acc[hit._id] = { fields: hit.fields };
|
||||
return acc;
|
||||
},
|
||||
{}
|
||||
);
|
||||
};
|
|
@ -0,0 +1,8 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
export * from './row_to_document';
|
|
@ -0,0 +1,27 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import { rowToDocument } from './row_to_document';
|
||||
|
||||
describe('rowToDocument', () => {
|
||||
it('should convert row to document', () => {
|
||||
const columns = [
|
||||
{ name: '_id', type: 'keyword' as const },
|
||||
{ name: 'agent.name', type: 'keyword' as const },
|
||||
{ name: 'agent.version', type: 'keyword' as const },
|
||||
{ name: 'agent.type', type: 'keyword' as const },
|
||||
];
|
||||
|
||||
const row = ['abcd', null, '8.8.1', 'packetbeat'];
|
||||
expect(rowToDocument(columns, row)).toEqual({
|
||||
_id: 'abcd',
|
||||
'agent.name': null,
|
||||
'agent.version': '8.8.1',
|
||||
'agent.type': 'packetbeat',
|
||||
});
|
||||
});
|
||||
});
|
|
@ -0,0 +1,25 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import type { EsqlResultRow, EsqlResultColumn } from '../esql_request';
|
||||
|
||||
/**
|
||||
* transform ES|QL result row to JSON object
|
||||
* @param columns
|
||||
* @param row
|
||||
* @returns Record<string, string | null>
|
||||
*/
|
||||
export const rowToDocument = (
|
||||
columns: EsqlResultColumn[],
|
||||
row: EsqlResultRow
|
||||
): Record<string, string | null> => {
|
||||
return columns.reduce<Record<string, string | null>>((acc, column, i) => {
|
||||
acc[column.name] = row[i];
|
||||
|
||||
return acc;
|
||||
}, {});
|
||||
};
|
|
@ -0,0 +1,102 @@
|
|||
/*
|
||||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
|
||||
* or more contributor license agreements. Licensed under the Elastic License
|
||||
* 2.0; you may not use this file except in compliance with the Elastic License
|
||||
* 2.0.
|
||||
*/
|
||||
|
||||
import objectHash from 'object-hash';
|
||||
import type { Moment } from 'moment';
|
||||
import type * as estypes from '@elastic/elasticsearch/lib/api/types';
|
||||
|
||||
import type {
|
||||
BaseFieldsLatest,
|
||||
WrappedFieldsLatest,
|
||||
} from '../../../../../common/api/detection_engine/model/alerts';
|
||||
import type { ConfigType } from '../../../../config';
|
||||
import type { CompleteRule, EsqlRuleParams } from '../../rule_schema';
|
||||
import { buildReasonMessageForNewTermsAlert } from '../utils/reason_formatters';
|
||||
import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring';
|
||||
import { buildBulkBody } from '../factories/utils/build_bulk_body';
|
||||
import type { SignalSource } from '../types';
|
||||
|
||||
export const wrapEsqlAlerts = ({
|
||||
results,
|
||||
spaceId,
|
||||
completeRule,
|
||||
mergeStrategy,
|
||||
alertTimestampOverride,
|
||||
ruleExecutionLogger,
|
||||
publicBaseUrl,
|
||||
tuple,
|
||||
sourceDocuments,
|
||||
isRuleAggregating,
|
||||
}: {
|
||||
isRuleAggregating: boolean;
|
||||
sourceDocuments: Record<string, { fields: estypes.SearchHit['fields'] }>;
|
||||
results: Array<Record<string, string | null>>;
|
||||
spaceId: string | null | undefined;
|
||||
completeRule: CompleteRule<EsqlRuleParams>;
|
||||
mergeStrategy: ConfigType['alertMergeStrategy'];
|
||||
alertTimestampOverride: Date | undefined;
|
||||
ruleExecutionLogger: IRuleExecutionLogForExecutors;
|
||||
publicBaseUrl: string | undefined;
|
||||
tuple: {
|
||||
to: Moment;
|
||||
from: Moment;
|
||||
maxSignals: number;
|
||||
};
|
||||
}): Array<WrappedFieldsLatest<BaseFieldsLatest>> => {
|
||||
const wrapped = results.map<WrappedFieldsLatest<BaseFieldsLatest>>((document, i) => {
|
||||
const ruleRunId = tuple.from.toISOString() + tuple.to.toISOString();
|
||||
|
||||
// for aggregating rules when metadata _id is present, generate alert based on ES document event id
|
||||
const id =
|
||||
!isRuleAggregating && document._id
|
||||
? objectHash([
|
||||
document._id,
|
||||
document._version,
|
||||
document._index,
|
||||
`${spaceId}:${completeRule.alertId}`,
|
||||
])
|
||||
: objectHash([
|
||||
ruleRunId,
|
||||
completeRule.ruleParams.query,
|
||||
`${spaceId}:${completeRule.alertId}`,
|
||||
i,
|
||||
]);
|
||||
|
||||
// metadata fields need to be excluded from source, otherwise alerts creation fails
|
||||
const { _id, _version, _index, ...source } = document;
|
||||
|
||||
const baseAlert: BaseFieldsLatest = buildBulkBody(
|
||||
spaceId,
|
||||
completeRule,
|
||||
{
|
||||
_source: source as SignalSource,
|
||||
fields: _id ? sourceDocuments[_id]?.fields : undefined,
|
||||
_id: _id ?? '',
|
||||
_index: _index ?? '',
|
||||
},
|
||||
mergeStrategy,
|
||||
[],
|
||||
true,
|
||||
buildReasonMessageForNewTermsAlert,
|
||||
[],
|
||||
alertTimestampOverride,
|
||||
ruleExecutionLogger,
|
||||
id,
|
||||
publicBaseUrl
|
||||
);
|
||||
|
||||
return {
|
||||
_id: id,
|
||||
_index: _index ?? '',
|
||||
_source: {
|
||||
...baseAlert,
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
return wrapped;
|
||||
};
|
|
@ -120,7 +120,6 @@ describe('stripNonEcsFields', () => {
|
|||
});
|
||||
|
||||
expect(result).toEqual({
|
||||
agent: [],
|
||||
message: 'test message',
|
||||
});
|
||||
expect(removed).toEqual([
|
||||
|
@ -142,7 +141,7 @@ describe('stripNonEcsFields', () => {
|
|||
});
|
||||
|
||||
expect(result).toEqual({
|
||||
agent: { type: 'filebeat', name: [] },
|
||||
agent: { type: 'filebeat' },
|
||||
message: 'test message',
|
||||
});
|
||||
expect(removed).toEqual([
|
||||
|
@ -177,6 +176,42 @@ describe('stripNonEcsFields', () => {
|
|||
]);
|
||||
});
|
||||
|
||||
it('should strip conflicting fields that use dot notation and is an array', () => {
|
||||
const { result, removed } = stripNonEcsFields({
|
||||
'agent.name.text': ['1'],
|
||||
message: 'test message',
|
||||
});
|
||||
|
||||
expect(result).toEqual({
|
||||
message: 'test message',
|
||||
});
|
||||
|
||||
expect(removed).toEqual([
|
||||
{
|
||||
key: 'agent.name.text',
|
||||
value: '1',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('should strip conflicting fields that use dot notation and is an empty array', () => {
|
||||
const { result, removed } = stripNonEcsFields({
|
||||
'agent.name.text': [],
|
||||
message: 'test message',
|
||||
});
|
||||
|
||||
expect(result).toEqual({
|
||||
message: 'test message',
|
||||
});
|
||||
|
||||
expect(removed).toEqual([
|
||||
{
|
||||
key: 'agent.name.text',
|
||||
value: [],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('should not strip valid ECS fields that use dot notation', () => {
|
||||
const { result, removed } = stripNonEcsFields({
|
||||
'agent.name': 'some name',
|
||||
|
@ -289,7 +324,6 @@ describe('stripNonEcsFields', () => {
|
|||
|
||||
expect(result).toEqual({
|
||||
threat: {
|
||||
enrichments: [],
|
||||
'indicator.port': 443,
|
||||
},
|
||||
});
|
||||
|
|
|
@ -180,7 +180,7 @@ export const stripNonEcsFields = (doc: SourceFieldRecord): StripNonEcsFieldsRetu
|
|||
) => {
|
||||
const fullPath = [parentPath, documentKey].filter(Boolean).join('.');
|
||||
// if document array, traverse through each item w/o changing documentKey, parent, parentPath
|
||||
if (isArray(document)) {
|
||||
if (isArray(document) && document.length > 0) {
|
||||
document.slice().forEach((value) => {
|
||||
traverseAndDeleteInObj(value, documentKey, parent, parentPath);
|
||||
});
|
||||
|
@ -194,6 +194,9 @@ export const stripNonEcsFields = (doc: SourceFieldRecord): StripNonEcsFieldsRetu
|
|||
if (isArray(documentReference)) {
|
||||
const indexToDelete = documentReference.findIndex((item) => item === document);
|
||||
documentReference.splice(indexToDelete, 1);
|
||||
if (documentReference.length === 0) {
|
||||
delete parent[documentKey];
|
||||
}
|
||||
} else {
|
||||
delete parent[documentKey];
|
||||
}
|
||||
|
|
|
@ -6,6 +6,7 @@
|
|||
*/
|
||||
|
||||
export { createEqlAlertType } from './eql/create_eql_alert_type';
|
||||
export { createEsqlAlertType } from './esql/create_esql_alert_type';
|
||||
export { createIndicatorMatchAlertType } from './indicator_match/create_indicator_match_alert_type';
|
||||
export { createMlAlertType } from './ml/create_ml_alert_type';
|
||||
export { createQueryAlertType } from './query/create_query_alert_type';
|
||||
|
|
|
@ -10,7 +10,7 @@ import type { RuleParamsAndRefs } from '@kbn/alerting-plugin/server';
|
|||
|
||||
import type { RuleParams } from '../../rule_schema';
|
||||
|
||||
import { isMachineLearningParams } from '../utils/utils';
|
||||
import { isMachineLearningParams, isEsqlParams } from '../utils/utils';
|
||||
|
||||
import { extractExceptionsList } from './extract_exceptions_list';
|
||||
import { extractDataView } from './extract_data_view';
|
||||
|
@ -51,7 +51,7 @@ export const extractReferences = <TParams extends RuleParams>({
|
|||
|
||||
// if statement is needed here because dataViewId is not on the base rule params
|
||||
// much like how the index property is not on the base rule params either
|
||||
if (!isMachineLearningParams(params)) {
|
||||
if (!isMachineLearningParams(params) && !isEsqlParams(params)) {
|
||||
returnReferences = [
|
||||
...returnReferences,
|
||||
...extractDataView({
|
||||
|
|
|
@ -132,6 +132,9 @@ export const getFilter = async ({
|
|||
case 'eql': {
|
||||
throw new BadRequestError('Unsupported Rule of type "eql" supplied to getFilter');
|
||||
}
|
||||
case 'esql': {
|
||||
throw new BadRequestError('Unsupported Rule of type "esql" supplied to getFilter');
|
||||
}
|
||||
default: {
|
||||
return assertUnreachable(type);
|
||||
}
|
||||
|
|
|
@ -20,18 +20,27 @@ export const getQueryFilter = ({
|
|||
filters,
|
||||
index,
|
||||
exceptionFilter,
|
||||
fields = [],
|
||||
}: {
|
||||
query: RuleQuery;
|
||||
language: Language;
|
||||
filters: unknown;
|
||||
index: IndexPatternArray;
|
||||
exceptionFilter: Filter | undefined;
|
||||
fields?: DataViewFieldBase[];
|
||||
}): ESBoolQuery => {
|
||||
fields,
|
||||
}:
|
||||
| {
|
||||
query: RuleQuery;
|
||||
language: Language;
|
||||
filters: unknown;
|
||||
index: IndexPatternArray;
|
||||
exceptionFilter: Filter | undefined;
|
||||
fields?: DataViewFieldBase[];
|
||||
}
|
||||
| {
|
||||
index: undefined;
|
||||
query: RuleQuery;
|
||||
language: 'esql';
|
||||
filters: unknown;
|
||||
exceptionFilter: Filter | undefined;
|
||||
fields?: DataViewFieldBase[];
|
||||
}): ESBoolQuery => {
|
||||
const indexPattern: DataViewBase = {
|
||||
fields,
|
||||
title: index.join(),
|
||||
fields: fields ?? [],
|
||||
title: (index ?? []).join(),
|
||||
};
|
||||
|
||||
const config: EsQueryConfig = {
|
||||
|
|
|
@ -16,7 +16,7 @@ export interface BuildReasonMessageArgs {
|
|||
}
|
||||
|
||||
export interface BuildReasonMessageUtilArgs extends BuildReasonMessageArgs {
|
||||
type?: 'eql' | 'ml' | 'query' | 'threatMatch' | 'threshold' | 'new_terms';
|
||||
type?: 'eql' | 'ml' | 'query' | 'threatMatch' | 'threshold' | 'new_terms' | 'esql';
|
||||
}
|
||||
|
||||
export type BuildReasonMessage = (args: BuildReasonMessageArgs) => string;
|
||||
|
@ -123,6 +123,9 @@ created {alertSeverity} alert {alertName}.`,
|
|||
export const buildReasonMessageForEqlAlert = (args: BuildReasonMessageArgs) =>
|
||||
buildReasonMessageUtil({ ...args, type: 'eql' });
|
||||
|
||||
export const buildReasonMessageForEsqlAlert = (args: BuildReasonMessageArgs) =>
|
||||
buildReasonMessageUtil({ ...args, type: 'esql' });
|
||||
|
||||
export const buildReasonMessageForMlAlert = (args: BuildReasonMessageArgs) =>
|
||||
buildReasonMessageUtil({ ...args, type: 'ml' });
|
||||
|
||||
|
|
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue