[Security Solution][Detection Engine] adds ES|QL rule type to Security Detections rules (#165450)

## Summary

- related [epic](https://github.com/elastic/security-team/issues/6196)
- introduces new ES|QL rule type in Technical Preview Stage

- historical POC architecture
[document](https://docs.google.com/document/d/1hcKzNrDEIrmoWwWoqas1YZ-bd8Kk5NRjJNSUaCvSntM/edit#heading=h.gheuu8zcz481)(internal
link). Some of the information there can be outdated, but might be
useful for historical context of some tech decision. In future, detailed
technical documentation will be added

### UI

ES|QL query component introduced in rule edit/creation form

Rule name override supports values returned from ES|QL query
As agreed on Adv. correlation WG, we don't introduce similar possibility
for risk score/severity override at this point

<details>
<summary>How it looks like in UI</summary>
<img width="2082" alt="Screenshot 2023-09-21 at 11 52 59"
src="14c94e36-ca90-496d-a7a5-4a31899d25b6">
<img width="2079" alt="Screenshot 2023-09-21 at 11 53 14"
src="9abd53ec-a0f4-4481-8b1f-4ecccdc5feae">
<img width="2072" alt="Screenshot 2023-09-21 at 12 14 17"
src="58e4f9eb-c15f-4849-bba0-bc1b92e8c945">


</details>


### Context

We introduced concept of Aggregating and Non-aggregating rules for
ES|QL. It depends on, whether STATS..BY command used in query

**Aggregating rule** - is a rule that uses
[stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html)
grouping commands. So, its result can not be matched to a particular
document in ES. This can lead to possibly duplicated alerts, since we
are using document `id` to deduplicate alerts. We are going to introduce
suppression for all rule types in future, that would help to mitigate
this case
```
FROM logs*
| STATS count = COUNT(host.name) BY host.name
| SORT host.name
```

**Non-aggregating rule** - is a rule that does not use
[stats…by](https://esql.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/esql-stats-by.html)
grouping commands. Each row in result can be tracked to a source
document in ES. For this type of rule operator \`[metadata _id, _index,
_version]\` is required to be used after defining index source. This
would allow deduplicate alerts and link them with the source document.

```
FROM logs* [metadata _id, _index, _version]
| WHERE event.id == "test"
| LIMIT 10
```

### Serverless Feature Flag

ES|QL won't be available for Serverless as for 8.11 release, so it will
be hidden by Security experimental feature flag `esqlRulesDisabled`. All
UI changes will be hidden (it's mostly Form creation) and rule type
won't be registered, which prevents rule to be created, returned in
search if it exists or execute.

### Test envs
- Serverless qa, [admin link to
project](https://admin.qa.cld.elstc.co/projects/security/ef79684f92d64f27b69e1b04de86eb1a),
disabled there
- internal
[link](https://elastic.slack.com/archives/C03E8TR26HE/p1693848029955229)
to test env for Stateful


### Rule schema changes

introduces value `esql` to `type` property
introduces value `esql` to `language` property

### Tests coverage
- cypress tests (as per 27/09/2023 added cypress tests for rule
creation/edit/details,bulk_edit))
- functional tests for rule execution(exceptions, overrides, preview and
actual rule execution)
  - functional tests for bulk_edit

 #### Flaky test runner
- [cypress esql
tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3233#_),
non failed of added


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Vitalii Dmyterko 2023-09-30 09:45:34 +01:00 committed by GitHub
parent d35fa69138
commit b03b2fd477
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
125 changed files with 3889 additions and 112 deletions

1
.github/CODEOWNERS vendored
View file

@ -1274,6 +1274,7 @@ x-pack/plugins/cloud_integrations/cloud_full_story/server/config.ts @elastic/kib
/x-pack/plugins/security_solution/common/risk_engine @elastic/security-detection-engine
/x-pack/plugins/security_solution/public/common/components/sourcerer @elastic/security-detection-engine
/x-pack/plugins/security_solution/public/detection_engine/rule_creation @elastic/security-detection-engine
/x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui @elastic/security-detection-engine
/x-pack/plugins/security_solution/public/detection_engine/rule_exceptions @elastic/security-detection-engine
/x-pack/plugins/security_solution/public/detections/containers/detection_engine/lists @elastic/security-detection-engine

View file

@ -36,6 +36,7 @@ xpack.fleet.internal.registry.spec.max: '3.0'
# Serverless security specific options
xpack.securitySolution.enableExperimental:
- discoverInTimeline
- esqlRulesDisabled
xpack.ml.ad.enabled: true
xpack.ml.dfa.enabled: true

View file

@ -843,6 +843,9 @@ export const getDocLinks = ({ kibanaBranch }: GetDocLinkOptions): DocLinks => {
synthetics: {
featureRoles: `${ELASTIC_WEBSITE_URL}guide/en/observability/${DOC_LINK_VERSION}/synthetics-feature-roles.html`,
},
esql: {
statsBy: `${ELASTICSEARCH_DOCS}esql-stats-by.html`,
},
telemetry: {
settings: `${KIBANA_DOCS}telemetry-settings-kbn.html`,
},

View file

@ -601,6 +601,9 @@ export interface DocLinks {
readonly synthetics: {
readonly featureRoles: string;
};
readonly esql: {
readonly statsBy: string;
};
readonly telemetry: {
readonly settings: string;
};

View file

@ -8,7 +8,7 @@
import * as t from 'io-ts';
export const language = t.keyof({ eql: null, kuery: null, lucene: null });
export const language = t.keyof({ eql: null, kuery: null, lucene: null, esql: null });
export type Language = t.TypeOf<typeof language>;
export const languageOrUndefined = t.union([language, t.undefined]);

View file

@ -16,6 +16,7 @@ export const type = t.keyof({
threshold: null,
threat_match: null,
new_terms: null,
esql: null,
});
export type Type = t.TypeOf<typeof type>;

View file

@ -16,6 +16,7 @@ export const SIGNALS_ID = `siem.signals` as const;
*/
const RULE_TYPE_PREFIX = `siem` as const;
export const EQL_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.eqlRule` as const;
export const ESQL_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.esqlRule` as const;
export const INDICATOR_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.indicatorRule` as const;
export const ML_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.mlRule` as const;
export const QUERY_RULE_TYPE_ID = `${RULE_TYPE_PREFIX}.queryRule` as const;

View file

@ -8,6 +8,7 @@
import {
EQL_RULE_TYPE_ID,
ESQL_RULE_TYPE_ID,
INDICATOR_RULE_TYPE_ID,
ML_RULE_TYPE_ID,
NEW_TERMS_RULE_TYPE_ID,
@ -21,6 +22,7 @@ import {
*/
export const ruleTypeMappings = {
eql: EQL_RULE_TYPE_ID,
esql: ESQL_RULE_TYPE_ID,
machine_learning: ML_RULE_TYPE_ID,
query: QUERY_RULE_TYPE_ID,
saved_query: SAVED_QUERY_RULE_TYPE_ID,

View file

@ -9,3 +9,4 @@
export * from './src/add_remove_id_to_item';
export * from './src/transform_data_to_ndjson';
export * from './src/path_validations';
export * from './src/esql';

View file

@ -0,0 +1,65 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import { computeIsESQLQueryAggregating } from './compute_if_esql_query_aggregating';
describe('computeIsESQLQueryAggregating', () => {
describe('Aggregating query', () => {
it('should detect aggregating with STATS only', () => {
expect(computeIsESQLQueryAggregating('from packetbeat* | stats count(agent.name)')).toBe(
true
);
});
it('should detect aggregating with STATS..BY', () => {
expect(
computeIsESQLQueryAggregating('from packetbeat* | stats count(agent.name) by agent.name')
).toBe(true);
});
it('should detect aggregating with multiple spaces', () => {
expect(
computeIsESQLQueryAggregating(
'from packetbeat* | stats count(agent.name) by agent.name'
)
).toBe(true);
});
it('should detect aggregating case agnostic', () => {
expect(
computeIsESQLQueryAggregating('from packetbeat* | STATS count(agent.name) BY agent.name')
).toBe(true);
expect(computeIsESQLQueryAggregating('from packetbeat* | STATS count(agent.name)')).toBe(
true
);
});
it('should detect aggregating for multiple aggregation', () => {
expect(
computeIsESQLQueryAggregating(
'from packetbeat* | stats count(agent.name) by agent.name | stats distinct=count_distinct(agent.name)'
)
).toBe(true);
});
});
describe('Non-aggregating query', () => {
it('should detect non-aggregating', () => {
expect(computeIsESQLQueryAggregating('from packetbeat* | keep agent.*')).toBe(false);
});
it('should detect non-aggregating if word stats is part of variable name', () => {
expect(
computeIsESQLQueryAggregating('from packetbeat* | keep agent.some_stats | limit 10')
).toBe(false);
});
it('should detect non-aggregating if word stats is part of a path', () => {
expect(computeIsESQLQueryAggregating('from packetbeat* | keep agent.stats | limit 10')).toBe(
false
);
});
});
});

View file

@ -0,0 +1,16 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
/**
* compute if esqlQuery is aggregating/grouping, i.e. using STATS...BY command
* @param esqlQuery
* @returns boolean
*/
export const computeIsESQLQueryAggregating = (esqlQuery: string): boolean => {
return /\|\s+stats\s/i.test(esqlQuery);
};

View file

@ -0,0 +1,37 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import { getIndexListFromEsqlQuery } from './get_index_list_from_esql_query';
import { getIndexPatternFromESQLQuery } from '@kbn/es-query';
jest.mock('@kbn/es-query', () => {
return {
getIndexPatternFromESQLQuery: jest.fn(),
};
});
const getIndexPatternFromESQLQueryMock = getIndexPatternFromESQLQuery as jest.Mock;
describe('getIndexListFromEsqlQuery', () => {
it('should return empty array if index string is empty', () => {
getIndexPatternFromESQLQueryMock.mockReturnValue('');
expect(getIndexListFromEsqlQuery('')).toEqual([]);
});
it('should return single item array if one index present', () => {
getIndexPatternFromESQLQueryMock.mockReturnValue('test-1');
expect(getIndexListFromEsqlQuery('From test-1')).toEqual(['test-1']);
});
it('should return array if index string has multiple indices', () => {
getIndexPatternFromESQLQueryMock.mockReturnValue('test-1,test-2');
expect(getIndexListFromEsqlQuery('From test-1,test-2')).toEqual(['test-1', 'test-2']);
});
it('should trim spaces in index names', () => {
getIndexPatternFromESQLQueryMock.mockReturnValue('test-1 , test-2 ');
expect(getIndexListFromEsqlQuery('From test-1, test-2 ')).toEqual(['test-1', 'test-2']);
});
});

View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import { getIndexPatternFromESQLQuery } from '@kbn/es-query';
/**
* parses ES|QL query and returns array of indices
*/
export const getIndexListFromEsqlQuery = (query: string | undefined): string[] => {
const indexString = getIndexPatternFromESQLQuery(query);
return getIndexListFromIndexString(indexString);
};
/**
* transforms sting of indices, separated by commas to array
* index*, index2* => [index*, index2*]
*/
export const getIndexListFromIndexString = (indexString: string | undefined): string[] => {
if (!indexString) {
return [];
}
return indexString.split(',').map((index) => index.trim());
};

View file

@ -0,0 +1,10 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
export * from './compute_if_esql_query_aggregating';
export * from './get_index_list_from_esql_query';

View file

@ -11,7 +11,8 @@
"**/*.ts"
],
"kbn_references": [
"@kbn/i18n"
"@kbn/i18n",
"@kbn/es-query"
],
"exclude": [
"target/**/*",

View file

@ -10,6 +10,7 @@ import { i18n } from '@kbn/i18n';
import { DEFAULT_APP_CATEGORIES } from '@kbn/core-application-common';
import {
EQL_RULE_TYPE_ID,
ESQL_RULE_TYPE_ID,
INDICATOR_RULE_TYPE_ID,
ML_RULE_TYPE_ID,
NEW_TERMS_RULE_TYPE_ID,
@ -29,6 +30,7 @@ import type { SecurityFeatureParams } from './types';
const SECURITY_RULE_TYPES = [
LEGACY_NOTIFICATIONS_ID,
ESQL_RULE_TYPE_ID,
EQL_RULE_TYPE_ID,
INDICATOR_RULE_TYPE_ID,
ML_RULE_TYPE_ID,

View file

@ -11,6 +11,7 @@ import type {
MachineLearningRuleUpdateProps,
QueryRuleCreateProps,
QueryRuleUpdateProps,
EsqlRuleCreateProps,
SavedQueryRuleCreateProps,
ThreatMatchRuleCreateProps,
ThresholdRuleCreateProps,
@ -152,6 +153,23 @@ export const getCreateNewTermsRulesSchemaMock = (
history_window_start: 'now-7d',
});
export const getCreateEsqlRulesSchemaMock = (
ruleId = 'rule-1',
enabled = false
): EsqlRuleCreateProps => ({
description: 'Detecting root and admin users',
enabled,
name: 'Query with a rule id',
query: 'from auditbeat-* | where user.name=="root" or user.name=="admin"',
severity: 'high',
type: 'esql',
risk_score: 55,
language: 'esql',
rule_id: ruleId,
interval: '5m',
from: 'now-6m',
});
export const getUpdateRulesSchemaMock = (
id = '04128c15-0d1b-4716-a4c5-46997ac7f3bd'
): QueryRuleUpdateProps => ({

View file

@ -20,6 +20,7 @@ import {
getCreateThresholdRulesSchemaMock,
getCreateRulesSchemaMockWithDataView,
getCreateMachineLearningRulesSchemaMock,
getCreateEsqlRulesSchemaMock,
} from './rule_request_schema.mock';
import { buildResponseRuleSchema } from './build_rule_schemas';
@ -1206,6 +1207,38 @@ describe('rules schema', () => {
});
});
describe('esql rule type', () => {
it('should validate correct payload', () => {
const payload = getCreateEsqlRulesSchemaMock();
const decoded = RuleCreateProps.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual([]);
expect(message.schema).toEqual(payload);
});
it('should not validate index property', () => {
const payload = { ...getCreateEsqlRulesSchemaMock(), index: ['test*'] };
const decoded = RuleCreateProps.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "index,["test*"]"']);
});
it('should not validate data_view_id property', () => {
const payload = { ...getCreateEsqlRulesSchemaMock(), data_view_id: 'test' };
const decoded = RuleCreateProps.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "data_view_id"']);
});
it('should not validate filters property', () => {
const payload = { ...getCreateEsqlRulesSchemaMock(), filters: [] };
const decoded = RuleCreateProps.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "filters,[]"']);
});
});
describe('data_view_id', () => {
test('validates when "data_view_id" and index are defined', () => {
const payload = { ...getCreateRulesSchemaMockWithDataView(), index: ['auditbeat-*'] };

View file

@ -8,6 +8,7 @@
import { DEFAULT_INDICATOR_SOURCE_PATH } from '../../../../constants';
import type {
EqlRule,
EsqlRule,
MachineLearningRule,
QueryRule,
SavedQueryRule,
@ -103,6 +104,15 @@ export const getRulesMlSchemaMock = (anchorDate: string = ANCHOR_DATE): MachineL
};
};
export const getEsqlRuleSchemaMock = (anchorDate: string = ANCHOR_DATE): EsqlRule => {
return {
...getResponseBaseParams(anchorDate),
query: 'from auditbeat* | limit 10',
type: 'esql',
language: 'esql',
};
};
export const getThreatMatchingSchemaMock = (anchorDate: string = ANCHOR_DATE): ThreatMatchRule => {
return {
...getResponseBaseParams(anchorDate),

View file

@ -16,6 +16,7 @@ import {
getSavedQuerySchemaMock,
getThreatMatchingSchemaMock,
getRulesEqlSchemaMock,
getEsqlRuleSchemaMock,
} from './rule_response_schema.mock';
describe('Rule response schema', () => {
@ -167,6 +168,41 @@ describe('Rule response schema', () => {
});
});
describe('esql rule type', () => {
test('it should NOT validate a type of "esql" with "index" defined', () => {
const payload = { ...getEsqlRuleSchemaMock(), index: ['logs-*'] };
const decoded = RuleResponse.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "index,["logs-*"]"']);
expect(message.schema).toEqual({});
});
test('it should NOT validate a type of "esql" with "filters" defined', () => {
const payload = { ...getEsqlRuleSchemaMock(), filters: [] };
const decoded = RuleResponse.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "filters,[]"']);
expect(message.schema).toEqual({});
});
test('it should NOT validate a type of "esql" with a "saved_id" dependent', () => {
const payload = { ...getEsqlRuleSchemaMock(), saved_id: 'id' };
const decoded = RuleResponse.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "saved_id"']);
expect(message.schema).toEqual({});
});
});
describe('data_view_id', () => {
test('it should validate a type of "query" with "data_view_id" defined', () => {
const payload = { ...getRulesSchemaMock(), data_view_id: 'logs-*' };
@ -231,6 +267,17 @@ describe('Rule response schema', () => {
expect(getPaths(left(message.errors))).toEqual(['invalid keys "data_view_id"']);
expect(message.schema).toEqual({});
});
test('it should NOT validate a type of "esql" with "data_view_id" defined', () => {
const payload = { ...getEsqlRuleSchemaMock(), data_view_id: 'logs-*' };
const decoded = RuleResponse.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
expect(getPaths(left(message.errors))).toEqual(['invalid keys "data_view_id"']);
expect(message.schema).toEqual({});
});
});
describe('investigation_fields', () => {

View file

@ -213,6 +213,7 @@ export enum QueryLanguage {
'kuery' = 'kuery',
'lucene' = 'lucene',
'eql' = 'eql',
'esql' = 'esql',
}
export type KqlQueryLanguage = t.TypeOf<typeof KqlQueryLanguage>;
@ -253,6 +254,37 @@ export const EqlRulePatchProps = t.intersection([SharedPatchProps, eqlSchema.pat
export type EqlPatchParams = t.TypeOf<typeof EqlPatchParams>;
export const EqlPatchParams = eqlSchema.patch;
// -------------------------------------------------------------------------------------------------
// ES|QL rule schema
export type EsqlQueryLanguage = t.TypeOf<typeof EsqlQueryLanguage>;
export const EsqlQueryLanguage = t.literal('esql');
const esqlSchema = buildRuleSchemas({
required: {
type: t.literal('esql'),
language: EsqlQueryLanguage,
query: RuleQuery,
},
optional: {},
defaultable: {},
});
export type EsqlRule = t.TypeOf<typeof EsqlRule>;
export const EsqlRule = t.intersection([SharedResponseProps, esqlSchema.response]);
export type EsqlRuleCreateProps = t.TypeOf<typeof EsqlRuleCreateProps>;
export const EsqlRuleCreateProps = t.intersection([SharedCreateProps, esqlSchema.create]);
export type EsqlRuleUpdateProps = t.TypeOf<typeof EsqlRuleUpdateProps>;
export const EsqlRuleUpdateProps = t.intersection([SharedUpdateProps, esqlSchema.create]);
export type EsqlRulePatchProps = t.TypeOf<typeof EsqlRulePatchProps>;
export const EsqlRulePatchProps = t.intersection([SharedPatchProps, esqlSchema.patch]);
export type EsqlPatchParams = t.TypeOf<typeof EsqlPatchParams>;
export const EsqlPatchParams = esqlSchema.patch;
// -------------------------------------------------------------------------------------------------
// Indicator Match rule schema
@ -500,6 +532,7 @@ export const NewTermsPatchParams = newTermsSchema.patch;
export type TypeSpecificCreateProps = t.TypeOf<typeof TypeSpecificCreateProps>;
export const TypeSpecificCreateProps = t.union([
eqlSchema.create,
esqlSchema.create,
threatMatchSchema.create,
querySchema.create,
savedQuerySchema.create,
@ -511,6 +544,7 @@ export const TypeSpecificCreateProps = t.union([
export type TypeSpecificPatchProps = t.TypeOf<typeof TypeSpecificPatchProps>;
export const TypeSpecificPatchProps = t.union([
eqlSchema.patch,
esqlSchema.patch,
threatMatchSchema.patch,
querySchema.patch,
savedQuerySchema.patch,
@ -522,6 +556,7 @@ export const TypeSpecificPatchProps = t.union([
export type TypeSpecificResponse = t.TypeOf<typeof TypeSpecificResponse>;
export const TypeSpecificResponse = t.union([
eqlSchema.response,
esqlSchema.response,
threatMatchSchema.response,
querySchema.response,
savedQuerySchema.response,

View file

@ -87,6 +87,14 @@ export const RuleEqlQuery = t.exact(
})
);
export type RuleEsqlQuery = t.TypeOf<typeof RuleEsqlQuery>;
export const RuleEsqlQuery = t.exact(
t.type({
query: RuleQuery,
language: t.literal('esql'),
})
);
// -------------------------------------------------------------------------------------------------
// Rule schedule

View file

@ -53,6 +53,7 @@ import {
import {
BuildingBlockObject,
RuleEqlQuery,
RuleEsqlQuery,
InlineKqlQuery,
RuleKqlQuery,
RuleDataSource,
@ -147,6 +148,17 @@ export const DiffableEqlFields = buildSchema({
},
});
export type DiffableEsqlFields = t.TypeOf<typeof DiffableEsqlFields>;
export const DiffableEsqlFields = buildSchema({
required: {
type: t.literal('esql'),
esql_query: RuleEsqlQuery, // NOTE: new field
},
// this is a new type of rule, no prebuilt rules created yet.
// new properties might be added here during further rule type development
optional: {},
});
export type DiffableThreatMatchFields = t.TypeOf<typeof DiffableThreatMatchFields>;
export const DiffableThreatMatchFields = buildSchema({
required: {
@ -232,6 +244,7 @@ export const DiffableRule = t.intersection([
DiffableCustomQueryFields,
DiffableSavedQueryFields,
DiffableEqlFields,
DiffableEsqlFields,
DiffableThreatMatchFields,
DiffableThresholdFields,
DiffableMachineLearningFields,

View file

@ -10,6 +10,7 @@ import type {
DiffableCommonFields,
DiffableCustomQueryFields,
DiffableEqlFields,
DiffableEsqlFields,
DiffableMachineLearningFields,
DiffableNewTermsFields,
DiffableSavedQueryFields,
@ -24,6 +25,7 @@ export type CommonFieldsDiff = FieldsDiff<DiffableCommonFields>;
export type CustomQueryFieldsDiff = FieldsDiff<DiffableCustomQueryFields>;
export type SavedQueryFieldsDiff = FieldsDiff<DiffableSavedQueryFields>;
export type EqlFieldsDiff = FieldsDiff<DiffableEqlFields>;
export type EsqlFieldsDiff = FieldsDiff<DiffableEsqlFields>;
export type ThreatMatchFieldsDiff = FieldsDiff<DiffableThreatMatchFields>;
export type ThresholdFieldsDiff = FieldsDiff<DiffableThresholdFields>;
export type MachineLearningFieldsDiff = FieldsDiff<DiffableMachineLearningFields>;
@ -45,6 +47,7 @@ export type RuleFieldsDiff = CommonFieldsDiff &
| CustomQueryFieldsDiff
| SavedQueryFieldsDiff
| EqlFieldsDiff
| EsqlFieldsDiff
| ThreatMatchFieldsDiff
| ThresholdFieldsDiff
| MachineLearningFieldsDiff

View file

@ -444,6 +444,7 @@ export enum BulkActionsDryRunErrCode {
IMMUTABLE = 'IMMUTABLE',
MACHINE_LEARNING_AUTH = 'MACHINE_LEARNING_AUTH',
MACHINE_LEARNING_INDEX_PATTERN = 'MACHINE_LEARNING_INDEX_PATTERN',
ESQL_INDEX_PATTERN = 'ESQL_INDEX_PATTERN',
}
export const RISKY_HOSTS_DOC_LINK =

View file

@ -12,6 +12,7 @@ import {
normalizeMachineLearningJobIds,
normalizeThresholdField,
isMlRule,
isEsqlRule,
} from './utils';
import { hasLargeValueList } from '@kbn/securitysolution-list-utils';
@ -135,6 +136,16 @@ describe('isMlRule', () => {
});
});
describe('isEsqlRule', () => {
test('it returns true if a ES|QL rule', () => {
expect(isEsqlRule('esql')).toEqual(true);
});
test('it returns false if not a ES|QL rule', () => {
expect(isEsqlRule('query')).toEqual(false);
});
});
describe('#hasEqlSequenceQuery', () => {
describe('when a non-sequence query is passed', () => {
const query = 'process where process.name == "regsvr32.exe"';

View file

@ -46,6 +46,7 @@ export const isThreatMatchRule = (ruleType: Type | undefined): boolean =>
ruleType === 'threat_match';
export const isMlRule = (ruleType: Type | undefined): boolean => ruleType === 'machine_learning';
export const isNewTermsRule = (ruleType: Type | undefined): boolean => ruleType === 'new_terms';
export const isEsqlRule = (ruleType: Type | undefined): boolean => ruleType === 'esql';
export const normalizeThresholdField = (
thresholdField: string | string[] | null | undefined

View file

@ -110,6 +110,11 @@ export const allowedExperimentalValues = Object.freeze({
* */
discoverInTimeline: false,
/**
* disables ES|QL rules
*/
esqlRulesDisabled: false,
/**
* Enables Protection Updates tab in the Endpoint Policy Details page
*/

View file

@ -47,6 +47,8 @@
"files",
"controls",
"dataViewEditor",
"savedObjectsManagement",
"expressions",
"stackConnectors",
"discover",
"notifications",

View file

@ -0,0 +1,30 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { Query, AggregateQuery } from '@kbn/es-query';
/**
* converts AggregateQuery type to Query
* it needed because unified search bar emits 2 types of queries: Query and AggregateQuery
* on security side we deal with one type only (Query), so we converge it to this type only
*/
export const convertToQueryType = (query: Query | AggregateQuery): Query => {
if ('esql' in query) {
return {
query: query.esql,
language: 'esql',
};
}
if ('sql' in query) {
return {
query: query.sql,
language: 'sql',
};
}
return query;
};

View file

@ -8,15 +8,15 @@
import React, { memo, useMemo, useCallback, useState, useEffect } from 'react';
import deepEqual from 'fast-deep-equal';
import type { DataViewBase, Filter, Query, TimeRange } from '@kbn/es-query';
import type { DataViewBase, Filter, Query, AggregateQuery, TimeRange } from '@kbn/es-query';
import type { FilterManager, SavedQuery, SavedQueryTimeFilter } from '@kbn/data-plugin/public';
import { TimeHistory } from '@kbn/data-plugin/public';
import type { DataView } from '@kbn/data-views-plugin/public';
import type { SearchBarProps } from '@kbn/unified-search-plugin/public';
import { SearchBar } from '@kbn/unified-search-plugin/public';
import { Storage } from '@kbn/kibana-utils-plugin/public';
import { useKibana } from '../../lib/kibana';
import { convertToQueryType } from './convert_to_query_type';
export interface QueryBarComponentProps {
dataTestSubj?: string;
@ -64,18 +64,21 @@ export const QueryBar = memo<QueryBarComponentProps>(
const { data } = useKibana().services;
const [dataView, setDataView] = useState<DataView>();
const onQuerySubmit = useCallback(
(payload: { dateRange: TimeRange; query?: Query }) => {
(payload: { dateRange: TimeRange; query?: Query | AggregateQuery }) => {
if (payload.query != null && !deepEqual(payload.query, filterQuery)) {
onSubmitQuery(payload.query);
const payloadQuery = convertToQueryType(payload.query);
onSubmitQuery(payloadQuery);
}
},
[filterQuery, onSubmitQuery]
);
const onQueryChange = useCallback(
(payload: { dateRange: TimeRange; query?: Query }) => {
(payload: { dateRange: TimeRange; query?: Query | AggregateQuery }) => {
if (onChangedQuery && payload.query != null && !deepEqual(payload.query, filterQuery)) {
onChangedQuery(payload.query);
const payloadQuery = convertToQueryType(payload.query);
onChangedQuery(payloadQuery);
}
},
[filterQuery, onChangedQuery]
@ -109,11 +112,19 @@ export const QueryBar = memo<QueryBarComponentProps>(
[filterManager]
);
const isEsql = filterQuery?.language === 'esql';
const query = useMemo(() => {
if (isEsql && typeof filterQuery.query === 'string') {
return { esql: filterQuery.query };
}
return filterQuery;
}, [filterQuery, isEsql]);
useEffect(() => {
let dv: DataView;
if (isDataView(indexPattern)) {
setDataView(indexPattern);
} else {
} else if (!isEsql) {
const createDataView = async () => {
dv = await data.dataViews.create({ title: indexPattern.title });
setDataView(dv);
@ -125,7 +136,7 @@ export const QueryBar = memo<QueryBarComponentProps>(
data.dataViews.clearInstanceCache(dv?.id);
}
};
}, [data.dataViews, indexPattern]);
}, [data.dataViews, indexPattern, isEsql]);
const timeHistory = useMemo(() => new TimeHistory(new Storage(localStorage)), []);
const arrDataView = useMemo(() => (dataView != null ? [dataView] : []), [dataView]);
@ -138,7 +149,7 @@ export const QueryBar = memo<QueryBarComponentProps>(
indexPatterns={arrDataView}
isLoading={isLoading}
isRefreshPaused={isRefreshPaused}
query={filterQuery}
query={query}
onClearSavedQuery={onClearSavedQuery}
onFiltersUpdated={onFiltersUpdated}
onQueryChange={onQueryChange}

View file

@ -9,7 +9,7 @@ import type { CoreStart } from '@kbn/core/public';
import type { StartPlugins } from '../../../types';
type GlobalServices = Pick<CoreStart, 'application' | 'http' | 'uiSettings' | 'notifications'> &
Pick<StartPlugins, 'data' | 'unifiedSearch'>;
Pick<StartPlugins, 'data' | 'unifiedSearch' | 'expressions'>;
export class KibanaServices {
private static kibanaBranch?: string;
@ -27,12 +27,21 @@ export class KibanaServices {
prebuiltRulesPackageVersion,
uiSettings,
notifications,
expressions,
}: GlobalServices & {
kibanaBranch: string;
kibanaVersion: string;
prebuiltRulesPackageVersion?: string;
}) {
this.services = { application, data, http, uiSettings, unifiedSearch, notifications };
this.services = {
application,
data,
http,
uiSettings,
unifiedSearch,
notifications,
expressions,
};
this.kibanaBranch = kibanaBranch;
this.kibanaVersion = kibanaVersion;
this.prebuiltRulesPackageVersion = prebuiltRulesPackageVersion;

View file

@ -0,0 +1,83 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React, { useCallback } from 'react';
import type { EuiComboBoxOptionOption } from '@elastic/eui';
import { EuiFormRow, EuiComboBox } from '@elastic/eui';
import type { FieldHook } from '@kbn/es-ui-shared-plugin/static/forms/hook_form_lib';
import { useEsqlFieldOptions } from './use_esql_fields_options';
const AS_PLAIN_TEXT = { asPlainText: true };
const COMPONENT_WIDTH = 500;
interface AutocompleteFieldProps {
dataTestSubj: string;
field: FieldHook;
idAria: string;
isDisabled: boolean;
fieldType: 'string';
placeholder?: string;
esqlQuery: string | undefined;
}
/**
* autocomplete form component that works with ES|QL query
* it receives query as one of the parameters, fetches available fields and convert them to
* options, that populate autocomplete component
*/
export const EsqlAutocomplete: React.FC<AutocompleteFieldProps> = ({
dataTestSubj,
field,
idAria,
isDisabled,
fieldType,
placeholder,
esqlQuery,
}): JSX.Element => {
const handleValuesChange = useCallback(
([newOption]: EuiComboBoxOptionOption[]): void => {
field.setValue(newOption?.label ?? '');
},
[field]
);
const { options, isLoading } = useEsqlFieldOptions(esqlQuery, fieldType);
const value = field?.value;
const selectedOptions = typeof value === 'string' && value ? [{ label: value }] : [];
const isInvalid =
typeof value === 'string' && value ? !options.some((option) => option.label === value) : false;
return (
<EuiFormRow
data-test-subj={dataTestSubj}
describedByIds={idAria ? [idAria] : undefined}
fullWidth
helpText={field.helpText}
label={field.label}
labelAppend={field.labelAppend}
>
<EuiComboBox
placeholder={placeholder ?? ''}
options={options}
selectedOptions={selectedOptions}
onChange={handleValuesChange}
isLoading={isLoading}
isDisabled={isDisabled || isLoading}
isClearable={false}
singleSelection={AS_PLAIN_TEXT}
data-test-subj="esqlAutocompleteComboBox"
style={{ width: `${COMPONENT_WIDTH}px` }}
fullWidth
isInvalid={isInvalid}
/>
</EuiFormRow>
);
};
EsqlAutocomplete.displayName = 'EsqlAutocomplete';

View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { EsqlAutocomplete } from './esql_autocomplete';

View file

@ -0,0 +1,59 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { esqlToOptions } from './use_esql_fields_options';
describe('esqlToOptions', () => {
it('should return empty array if data is undefined', () => {
expect(esqlToOptions(undefined)).toEqual([]);
});
it('should return empty array if data is null', () => {
expect(esqlToOptions(null)).toEqual([]);
});
it('should return empty array if data has error', () => {
expect(esqlToOptions({ error: Error })).toEqual([]);
});
it('should transform all columns if fieldTYpe is not passed', () => {
expect(
esqlToOptions({
type: 'datatable',
rows: [],
columns: [
{ name: '@timestamp', id: '@timestamp', meta: { type: 'date' } },
{ name: 'agent.build.original', id: 'agent.build.original', meta: { type: 'string' } },
{ name: 'amqp.app-id', id: 'amqp.app-id', meta: { type: 'string' } },
{ name: 'amqp.auto-delete', id: 'amqp.auto-delete', meta: { type: 'number' } },
{ name: 'amqp.class-id', id: 'amqp.class-id', meta: { type: 'boolean' } },
],
})
).toEqual([
{ label: '@timestamp' },
{ label: 'agent.build.original' },
{ label: 'amqp.app-id' },
{ label: 'amqp.auto-delete' },
{ label: 'amqp.class-id' },
]);
});
it('should transform only columns of exact fieldType', () => {
expect(
esqlToOptions(
{
type: 'datatable',
rows: [],
columns: [
{ name: '@timestamp', id: '@timestamp', meta: { type: 'date' } },
{ name: 'agent.build.original', id: 'agent.build.original', meta: { type: 'string' } },
{ name: 'amqp.app-id', id: 'amqp.app-id', meta: { type: 'string' } },
{ name: 'amqp.auto-delete', id: 'amqp.auto-delete', meta: { type: 'number' } },
{ name: 'amqp.class-id', id: 'amqp.class-id', meta: { type: 'boolean' } },
],
},
'string'
)
).toEqual([{ label: 'agent.build.original' }, { label: 'amqp.app-id' }]);
});
});

View file

@ -0,0 +1,64 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { useMemo } from 'react';
import type { EuiComboBoxOptionOption } from '@elastic/eui';
import type { Datatable, ExpressionsStart } from '@kbn/expressions-plugin/public';
import { useQuery } from '@tanstack/react-query';
import { useKibana } from '@kbn/kibana-react-plugin/public';
import { getEsqlQueryConfig } from '../../logic/get_esql_query_config';
import type { FieldType } from '../../logic/esql_validator';
export const esqlToOptions = (
data: { error: unknown } | Datatable | undefined | null,
fieldType?: FieldType
) => {
if (data && 'error' in data) {
return [];
}
const options = (data?.columns ?? []).reduce<Array<{ label: string }>>((acc, { id, meta }) => {
// if fieldType absent, we do not filter columns by type
if (!fieldType || fieldType === meta.type) {
acc.push({ label: id });
}
return acc;
}, []);
return options;
};
type UseEsqlFieldOptions = (
esqlQuery: string | undefined,
fieldType: FieldType
) => {
isLoading: boolean;
options: Array<EuiComboBoxOptionOption<string>>;
};
/**
* fetches ES|QL fields and convert them to Combobox options
*/
export const useEsqlFieldOptions: UseEsqlFieldOptions = (esqlQuery, fieldType) => {
const kibana = useKibana<{ expressions: ExpressionsStart }>();
const { expressions } = kibana.services;
const queryConfig = getEsqlQueryConfig({ esqlQuery, expressions });
const { data, isLoading } = useQuery(queryConfig);
const options = useMemo(() => {
return esqlToOptions(data, fieldType);
}, [data, fieldType]);
return {
options,
isLoading,
};
};

View file

@ -0,0 +1,49 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React from 'react';
import { EuiPopover, EuiText, EuiButtonIcon } from '@elastic/eui';
import { Markdown } from '@kbn/kibana-react-plugin/public';
import * as i18n from './translations';
import { useBoolState } from '../../../../common/hooks/use_bool_state';
import { useKibana } from '../../../../common/lib/kibana';
const POPOVER_WIDTH = 640;
/**
* Icon and popover that gives hint to users how to get started with ES|QL rules
*/
const EsqlInfoIconComponent = () => {
const { docLinks } = useKibana().services;
const [isPopoverOpen, , closePopover, togglePopover] = useBoolState();
const button = (
<EuiButtonIcon iconType="iInCircle" onClick={togglePopover} aria-label={i18n.ARIA_LABEL} />
);
return (
<EuiPopover button={button} isOpen={isPopoverOpen} closePopover={closePopover}>
<EuiText style={{ width: POPOVER_WIDTH }} size="s">
<Markdown
markdown={i18n.getTooltipContent(
docLinks.links.esql.statsBy,
// Docs team will provide actual link to a new page before release
// For now, it's just a mock
docLinks.links.esql.statsBy
)}
/>
</EuiText>
</EuiPopover>
);
};
export const EsqlInfoIcon = React.memo(EsqlInfoIconComponent);
EsqlInfoIcon.displayName = 'EsqlInfoIcon';

View file

@ -0,0 +1,51 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { i18n } from '@kbn/i18n';
export const ARIA_LABEL = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlInfoAriaLabel',
{
defaultMessage: `Open help popover`,
}
);
export const getTooltipContent = (statsByLink: string, startUsingEsqlLink: string) =>
i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlInfoTooltipContent',
{
defaultMessage: `
### Aggregating rule
Is a rule that uses {statsByLink} grouping commands. So, its result can not be matched with a particular document in ES.
\`\`\`
FROM logs*
| STATS count = COUNT(host.name) BY host.name
| SORT host.name
\`\`\`
### Non-aggregating rule
Is a rule that does not use {statsByLink} grouping commands. Hence, each row in result can be tracked to a source document in ES. For this type of rule,
please use operator \`[metadata _id, _index, _version]\` after defining index source. This would allow deduplicate alerts and link them with the source document.
Example
\`\`\`
FROM logs* [metadata _id, _index, _version]
| WHERE event.id == "test"
| LIMIT 10
\`\`\`
Please, ensure, metadata properties \`id\`, \`_index\`, \`_version\` are carried over through pipe operators.
`,
values: {
statsByLink: `[STATS..BY](${statsByLink})`,
// Docs team will provide actual link to a new page before release
// startUsingEsqlLink: `[WIP: Get started using ES|QL rules](${startUsingEsqlLink})`,
},
}
);

View file

@ -0,0 +1,33 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { computeHasMetadataOperator } from './esql_validator';
describe('computeHasMetadataOperator', () => {
it('should be false if query does not have operator', () => {
expect(computeHasMetadataOperator('from test*')).toBe(false);
expect(computeHasMetadataOperator('from test* [metadata]')).toBe(false);
expect(computeHasMetadataOperator('from test* [metadata id]')).toBe(false);
expect(computeHasMetadataOperator('from metadata*')).toBe(false);
expect(computeHasMetadataOperator('from test* | keep metadata')).toBe(false);
expect(computeHasMetadataOperator('from test* | eval x="[metadata _id]"')).toBe(false);
});
it('should be true if query has operator', () => {
expect(computeHasMetadataOperator('from test* [metadata _id]')).toBe(true);
expect(computeHasMetadataOperator('from test* [metadata _id, _index]')).toBe(true);
expect(computeHasMetadataOperator('from test* [metadata _index, _id]')).toBe(true);
expect(computeHasMetadataOperator('from test* [ metadata _id ]')).toBe(true);
expect(computeHasMetadataOperator('from test* [ metadata _id] ')).toBe(true);
expect(computeHasMetadataOperator('from test* [ metadata _id] | limit 10')).toBe(true);
expect(
computeHasMetadataOperator(`from packetbeat* [metadata
_id ]
| limit 100`)
).toBe(true);
});
});

View file

@ -0,0 +1,94 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { isEmpty } from 'lodash';
import { computeIsESQLQueryAggregating } from '@kbn/securitysolution-utils';
import { KibanaServices } from '../../../common/lib/kibana';
import { securitySolutionQueryClient } from '../../../common/containers/query_client/query_client_provider';
import type { ValidationError, ValidationFunc } from '../../../shared_imports';
import { isEsqlRule } from '../../../../common/detection_engine/utils';
import type { DefineStepRule } from '../../../detections/pages/detection_engine/rules/types';
import type { FieldValueQueryBar } from '../../../detections/components/rules/query_bar';
import * as i18n from './translations';
import { getEsqlQueryConfig } from './get_esql_query_config';
export type FieldType = 'string';
export enum ERROR_CODES {
INVALID_ESQL = 'ERR_INVALID_ESQL',
ERR_MISSING_ID_FIELD_FROM_RESULT = 'ERR_MISSING_ID_FIELD_FROM_RESULT',
}
const constructValidationError = (error: Error) => {
return {
code: ERROR_CODES.INVALID_ESQL,
message: error?.message
? i18n.esqlValidationErrorMessage(error.message)
: i18n.ESQL_VALIDATION_UNKNOWN_ERROR,
error,
};
};
/**
* checks whether query has [metadata _id] operator
*/
export const computeHasMetadataOperator = (esqlQuery: string) => {
return /(?<!\|[\s\S.]*)\[\s*metadata[\s\S.]*_id[\s\S.]*\]/i.test(esqlQuery);
};
/**
* form validator for ES|QL queryBar
*/
export const esqlValidator = async (
...args: Parameters<ValidationFunc>
): Promise<ValidationError<ERROR_CODES> | void | undefined> => {
const [{ value, formData }] = args;
const { query: queryValue } = value as FieldValueQueryBar;
const query = queryValue.query as string;
const { ruleType } = formData as DefineStepRule;
const needsValidation = isEsqlRule(ruleType) && !isEmpty(query);
if (!needsValidation) {
return;
}
try {
const services = KibanaServices.get();
const isEsqlQueryAggregating = computeIsESQLQueryAggregating(query);
// non-aggregating query which does not have [metadata], is not a valid one
if (!isEsqlQueryAggregating && !computeHasMetadataOperator(query)) {
return {
code: ERROR_CODES.ERR_MISSING_ID_FIELD_FROM_RESULT,
message: i18n.ESQL_VALIDATION_MISSING_ID_IN_QUERY_ERROR,
};
}
const data = await securitySolutionQueryClient.fetchQuery(
getEsqlQueryConfig({ esqlQuery: query, expressions: services.expressions })
);
if (data && 'error' in data) {
return constructValidationError(data.error);
}
// check whether _id field is present in response
const isIdFieldPresent = (data?.columns ?? []).find(({ id }) => '_id' === id);
// for non-aggregating query, we want to disable queries w/o _id property returned in response
if (!isEsqlQueryAggregating && !isIdFieldPresent) {
return {
code: ERROR_CODES.ERR_MISSING_ID_FIELD_FROM_RESULT,
message: i18n.ESQL_VALIDATION_MISSING_ID_IN_QUERY_ERROR,
};
}
} catch (error) {
return constructValidationError(error);
}
};

View file

@ -0,0 +1,38 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { fetchFieldsFromESQL } from '@kbn/text-based-editor';
import type { ExpressionsStart } from '@kbn/expressions-plugin/public';
/**
* react-query configuration to be used to fetch ES|QL fields
* it sets limit in query to 0, so we don't fetch unnecessary results, only fields
*/
export const getEsqlQueryConfig = ({
esqlQuery,
expressions,
}: {
esqlQuery: string | undefined;
expressions: ExpressionsStart;
}) => {
const emptyResultsEsqlQuery = `${esqlQuery} | limit 0`;
return {
queryKey: [(esqlQuery ?? '').trim()],
queryFn: async () => {
if (!esqlQuery) {
return null;
}
try {
const res = await fetchFieldsFromESQL({ esql: emptyResultsEsqlQuery }, expressions);
return res;
} catch (e) {
return { error: e };
}
},
staleTime: 60 * 1000,
};
};

View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { i18n } from '@kbn/i18n';
export const ESQL_VALIDATION_UNKNOWN_ERROR = i18n.translate(
'xpack.securitySolution.detectionEngine.esqlValidation.unknownError',
{
defaultMessage: 'Unknown error while validating ES|QL',
}
);
export const esqlValidationErrorMessage = (message: string) =>
i18n.translate('xpack.securitySolution.detectionEngine.esqlValidation.errorMessage', {
values: { message },
defaultMessage: 'Error validating ES|QL: "{message}"',
});
export const ESQL_VALIDATION_MISSING_ID_IN_QUERY_ERROR = i18n.translate(
'xpack.securitySolution.detectionEngine.esqlValidation.missingIdInQueryError',
{
defaultMessage: `For non-aggregating rules(that don't use STATS..BY function), please write query that returns _id field from [metadata _id, _version, _index] operator`,
}
);

View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { useEsqlIndex } from './use_esql_index';
export { useEsqlQueryForAboutStep } from './use_esql_query_for_about_step';

View file

@ -0,0 +1,38 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { renderHook } from '@testing-library/react-hooks';
import { useEsqlIndex } from './use_esql_index';
const validEsqlQuery = 'from auditbeat* [metadata _id, _index, _version]';
describe('useEsqlIndex', () => {
it('should return empty array if isQueryReadEnabled is undefined', () => {
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'esql', undefined));
expect(result.current).toEqual([]);
});
it('should return empty array if isQueryReadEnabled is false', () => {
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'esql', false));
expect(result.current).toEqual([]);
});
it('should return empty array if rule type is not esql', () => {
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'query', true));
expect(result.current).toEqual([]);
});
it('should return empty array if query is empty', () => {
const { result } = renderHook(() => useEsqlIndex('', 'esql', true));
expect(result.current).toEqual([]);
});
it('should return parsed index array from a valid query', () => {
const { result } = renderHook(() => useEsqlIndex(validEsqlQuery, 'esql', true));
expect(result.current).toEqual(['auditbeat*']);
});
});

View file

@ -0,0 +1,38 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { useMemo } from 'react';
import { getIndexPatternFromESQLQuery } from '@kbn/es-query';
import { getIndexListFromIndexString } from '@kbn/securitysolution-utils';
import type { Query } from '@kbn/es-query';
import type { Type } from '@kbn/securitysolution-io-ts-alerting-types';
import { isEsqlRule } from '../../../../common/detection_engine/utils';
/**
* parses ES|QL query and returns memoized array of indices
* @param query - ES|QL query to retrieve index from
* @param ruleType - rule type value
* @param isQueryReadEnabled - if not enabled, return empty array. Useful if we know form or query is not valid and we don't want to retrieve index
* @returns
*/
export const useEsqlIndex = (
query: Query['query'],
ruleType: Type,
isQueryReadEnabled: boolean | undefined
) => {
const indexString = useMemo(() => {
if (!isQueryReadEnabled) {
return '';
}
const esqlQuery = typeof query === 'string' && isEsqlRule(ruleType) ? query : undefined;
return getIndexPatternFromESQLQuery(esqlQuery);
}, [query, isQueryReadEnabled, ruleType]);
const index = useMemo(() => getIndexListFromIndexString(indexString), [indexString]);
return index;
};

View file

@ -0,0 +1,38 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { useMemo } from 'react';
import { RuleStep } from '../../../detections/pages/detection_engine/rules/types';
import { isEsqlRule } from '../../../../common/detection_engine/utils';
import type { DefineStepRule } from '../../../detections/pages/detection_engine/rules/types';
interface UseEsqlQueryForAboutStepArgs {
defineStepData: DefineStepRule;
activeStep: RuleStep;
}
/**
* If about step not active, return query as undefined to prevent unnecessary re-renders
* ES|QL query can change frequently when user types it, so we don't want it to trigger about form step re-render, when * this from step not active
* When it is active, query would not change, as it can be edit only in define form step
*/
export const useEsqlQueryForAboutStep = ({
defineStepData,
activeStep,
}: UseEsqlQueryForAboutStepArgs) => {
const esqlQueryForAboutStep = useMemo(() => {
if (activeStep !== RuleStep.aboutRule) {
return undefined;
}
return typeof defineStepData.queryBar.query.query === 'string' &&
isEsqlRule(defineStepData.ruleType)
? defineStepData.queryBar.query.query
: undefined;
}, [defineStepData.queryBar.query.query, defineStepData.ruleType, activeStep]);
return esqlQueryForAboutStep;
};

View file

@ -7,7 +7,7 @@
/* eslint-disable complexity */
import { has, isEmpty } from 'lodash/fp';
import { has, isEmpty, get } from 'lodash/fp';
import type { Unit } from '@kbn/datemath';
import moment from 'moment';
import deepmerge from 'deepmerge';
@ -151,6 +151,20 @@ type NewTermsRuleFields<T> = Omit<
| 'threatMapping'
| 'eqlOptions'
>;
type EsqlRuleFields<T> = Omit<
T,
| 'anomalyThreshold'
| 'machineLearningJobId'
| 'threshold'
| 'threatIndex'
| 'threatQueryBar'
| 'threatMapping'
| 'eqlOptions'
| 'index'
| 'newTermsFields'
| 'historyWindowSize'
| 'dataViewId'
>;
const isMlFields = <T>(
fields:
@ -160,6 +174,7 @@ const isMlFields = <T>(
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| NewTermsRuleFields<T>
| EsqlRuleFields<T>
): fields is MlRuleFields<T> => has('anomalyThreshold', fields);
const isThresholdFields = <T>(
@ -170,6 +185,7 @@ const isThresholdFields = <T>(
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| NewTermsRuleFields<T>
| EsqlRuleFields<T>
): fields is ThresholdRuleFields<T> => has('threshold', fields);
const isThreatMatchFields = <T>(
@ -180,6 +196,7 @@ const isThreatMatchFields = <T>(
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| NewTermsRuleFields<T>
| EsqlRuleFields<T>
): fields is ThreatMatchRuleFields<T> => has('threatIndex', fields);
const isNewTermsFields = <T>(
@ -190,6 +207,7 @@ const isNewTermsFields = <T>(
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| NewTermsRuleFields<T>
| EsqlRuleFields<T>
): fields is NewTermsRuleFields<T> => has('newTermsFields', fields);
const isEqlFields = <T>(
@ -200,8 +218,20 @@ const isEqlFields = <T>(
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| NewTermsRuleFields<T>
| EsqlRuleFields<T>
): fields is EqlQueryRuleFields<T> => has('eqlOptions', fields);
const isEsqlFields = <T>(
fields:
| QueryRuleFields<T>
| EqlQueryRuleFields<T>
| MlRuleFields<T>
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| NewTermsRuleFields<T>
| EsqlRuleFields<T>
): fields is EsqlRuleFields<T> => get('queryBar.query.language', fields) === 'esql';
export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
fields: T,
type: Type
@ -211,6 +241,7 @@ export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
| MlRuleFields<T>
| ThresholdRuleFields<T>
| ThreatMatchRuleFields<T>
| EsqlRuleFields<T>
| NewTermsRuleFields<T> => {
switch (type) {
case 'machine_learning':
@ -279,6 +310,7 @@ export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
...eqlRuleFields
} = fields;
return eqlRuleFields;
case 'new_terms':
const {
anomalyThreshold: ___a,
@ -291,6 +323,23 @@ export const filterRuleFieldsForType = <T extends Partial<RuleFields>>(
...newTermsRuleFields
} = fields;
return newTermsRuleFields;
case 'esql':
const {
anomalyThreshold: _esql_a,
machineLearningJobId: _esql_m,
threshold: _esql_t,
threatIndex: _esql_removedThreatIndex,
threatQueryBar: _esql_removedThreatQueryBar,
threatMapping: _esql_removedThreatMapping,
newTermsFields: _esql_removedNewTermsFields,
historyWindowSize: _esql_removedHistoryWindowSize,
eqlOptions: _esql__eqlOptions,
index: _esql_index,
dataViewId: _esql_dataViewId,
...esqlRuleFields
} = fields;
return esqlRuleFields;
}
assertUnreachable(type);
};
@ -420,6 +469,11 @@ export const formatDefineStepData = (defineStepData: DefineStepRule): DefineStep
new_terms_fields: ruleFields.newTermsFields,
history_window_start: `now-${ruleFields.historyWindowSize}`,
}
: isEsqlFields(ruleFields) && !('index' in ruleFields)
? {
language: ruleFields.queryBar?.query?.language,
query: ruleFields.queryBar?.query?.query as string,
}
: {
...(ruleFields.groupByFields.length > 0
? {
@ -454,7 +508,7 @@ export const formatDefineStepData = (defineStepData: DefineStepRule): DefineStep
return {
...baseFields,
...typeFields,
data_view_id: ruleFields.dataViewId,
...('dataViewId' in ruleFields ? { data_view_id: ruleFields.dataViewId } : {}),
};
};

View file

@ -20,8 +20,13 @@ import React, { memo, useCallback, useRef, useState, useMemo, useEffect } from '
import styled from 'styled-components';
import type { DataViewListItem } from '@kbn/data-views-plugin/common';
import { useAppToasts } from '../../../../common/hooks/use_app_toasts';
import { isMlRule, isThreatMatchRule } from '../../../../../common/detection_engine/utils';
import {
isMlRule,
isThreatMatchRule,
isEsqlRule,
} from '../../../../../common/detection_engine/utils';
import { useCreateRule } from '../../../rule_management/logic';
import type { RuleCreateProps } from '../../../../../common/api/detection_engine/model/rule_schema';
import { useListsConfig } from '../../../../detections/containers/detection_engine/lists/use_lists_config';
@ -62,6 +67,7 @@ import {
import type { DefineStepRule } from '../../../../detections/pages/detection_engine/rules/types';
import { RuleStep } from '../../../../detections/pages/detection_engine/rules/types';
import { formatRule } from './helpers';
import { useEsqlIndex, useEsqlQueryForAboutStep } from '../../hooks';
import * as i18n from './translations';
import { SecurityPageName } from '../../../../app/types';
import {
@ -190,6 +196,11 @@ const CreateRulePageComponent: React.FC = () => {
[defineStepData.ruleType]
);
const isEsqlRuleValue = useMemo(
() => isEsqlRule(defineStepData.ruleType),
[defineStepData.ruleType]
);
const [openSteps, setOpenSteps] = useState({
[RuleStep.defineRule]: false,
[RuleStep.aboutRule]: false,
@ -206,11 +217,22 @@ const CreateRulePageComponent: React.FC = () => {
const [isQueryBarValid, setIsQueryBarValid] = useState(false);
const [isThreatQueryBarValid, setIsThreatQueryBarValid] = useState(false);
const esqlQueryForAboutStep = useEsqlQueryForAboutStep({ defineStepData, activeStep });
const esqlIndex = useEsqlIndex(
defineStepData.queryBar.query.query,
ruleType,
defineStepForm.isValid
);
const memoizedIndex = useMemo(
() => (isEsqlRuleValue ? esqlIndex : defineStepData.index),
[defineStepData.index, esqlIndex, isEsqlRuleValue]
);
const isPreviewDisabled = getIsRulePreviewDisabled({
ruleType,
isQueryBarValid,
isThreatQueryBarValid,
index: defineStepData.index,
index: memoizedIndex,
dataViewId: defineStepData.dataViewId,
dataSourceType: defineStepData.dataSourceType,
threatIndex: defineStepData.threatIndex,
@ -250,7 +272,7 @@ const CreateRulePageComponent: React.FC = () => {
}, [dataViews]);
const { indexPattern, isIndexPatternLoading, browserFields } = useRuleIndexPattern({
dataSourceType: defineStepData.dataSourceType,
index: defineStepData.index,
index: memoizedIndex,
dataViewId: defineStepData.dataViewId,
});
@ -494,7 +516,7 @@ const CreateRulePageComponent: React.FC = () => {
setIsQueryBarValid={setIsQueryBarValid}
setIsThreatQueryBarValid={setIsThreatQueryBarValid}
ruleType={defineStepData.ruleType}
index={defineStepData.index}
index={memoizedIndex}
threatIndex={defineStepData.threatIndex}
groupByFields={defineStepData.groupByFields}
dataSourceType={defineStepData.dataSourceType}
@ -518,7 +540,7 @@ const CreateRulePageComponent: React.FC = () => {
defineRuleNextStep,
defineStepData.dataSourceType,
defineStepData.groupByFields,
defineStepData.index,
memoizedIndex,
defineStepData.queryBar.saved_id,
defineStepData.queryBar.title,
defineStepData.ruleType,
@ -575,12 +597,13 @@ const CreateRulePageComponent: React.FC = () => {
<StepAboutRule
ruleType={defineStepData.ruleType}
machineLearningJobId={defineStepData.machineLearningJobId}
index={defineStepData.index}
index={memoizedIndex}
dataViewId={defineStepData.dataViewId}
timestampOverride={aboutStepData.timestampOverride}
isLoading={isCreateRuleLoading || loading}
isActive={activeStep === RuleStep.aboutRule}
form={aboutStepForm}
esqlQuery={esqlQueryForAboutStep}
/>
<NextStep
@ -598,12 +621,13 @@ const CreateRulePageComponent: React.FC = () => {
aboutStepForm,
activeStep,
defineStepData.dataViewId,
defineStepData.index,
memoizedIndex,
defineStepData.machineLearningJobId,
defineStepData.ruleType,
isCreateRuleLoading,
loading,
memoAboutStepReadOnly,
esqlQueryForAboutStep,
]
);
const memoAboutStepExtraAction = useMemo(

View file

@ -22,6 +22,8 @@ import React, { memo, useCallback, useEffect, useMemo, useRef, useState } from '
import { useParams } from 'react-router-dom';
import type { DataViewListItem } from '@kbn/data-views-plugin/common';
import { isEsqlRule } from '../../../../../common/detection_engine/utils';
import { RulePreview } from '../../../../detections/components/rules/rule_preview';
import { getIsRulePreviewDisabled } from '../../../../detections/components/rules/rule_preview/helpers';
import type { RuleUpdateProps } from '../../../../../common/api/detection_engine/model/rule_schema';
@ -64,6 +66,7 @@ import { useStartTransaction } from '../../../../common/lib/apm/use_start_transa
import { SINGLE_RULE_ACTIONS } from '../../../../common/lib/apm/user_actions';
import { useGetSavedQuery } from '../../../../detections/pages/detection_engine/rules/use_get_saved_query';
import { useRuleForms, useRuleIndexPattern } from '../form';
import { useEsqlIndex, useEsqlQueryForAboutStep } from '../../hooks';
import { CustomHeaderPageMemo } from '..';
const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
@ -144,11 +147,24 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
actionsStepDefault: ruleActionsData,
});
const esqlQueryForAboutStep = useEsqlQueryForAboutStep({ defineStepData, activeStep });
const esqlIndex = useEsqlIndex(
defineStepData.queryBar.query.query,
defineStepData.ruleType,
// allow to compute index from query only when query is valid or user switched to another tab
// to prevent multiple data view initiations with partly typed index names
defineStepForm.isValid || activeStep !== RuleStep.defineRule
);
const memoizedIndex = useMemo(
() => (isEsqlRule(defineStepData.ruleType) ? esqlIndex : defineStepData.index),
[defineStepData.index, esqlIndex, defineStepData.ruleType]
);
const isPreviewDisabled = getIsRulePreviewDisabled({
ruleType: defineStepData.ruleType,
isQueryBarValid,
isThreatQueryBarValid,
index: defineStepData.index,
index: memoizedIndex,
dataViewId: defineStepData.dataViewId,
dataSourceType: defineStepData.dataSourceType,
threatIndex: defineStepData.threatIndex,
@ -198,7 +214,7 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
const { indexPattern, isIndexPatternLoading, browserFields } = useRuleIndexPattern({
dataSourceType: defineStepData.dataSourceType,
index: defineStepData.index,
index: memoizedIndex,
dataViewId: defineStepData.dataViewId,
});
@ -236,7 +252,7 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
setIsQueryBarValid={setIsQueryBarValid}
setIsThreatQueryBarValid={setIsThreatQueryBarValid}
ruleType={defineStepData.ruleType}
index={defineStepData.index}
index={memoizedIndex}
threatIndex={defineStepData.threatIndex}
groupByFields={defineStepData.groupByFields}
dataSourceType={defineStepData.dataSourceType}
@ -270,10 +286,11 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
isActive={activeStep === RuleStep.aboutRule}
ruleType={defineStepData.ruleType}
machineLearningJobId={defineStepData.machineLearningJobId}
index={defineStepData.index}
index={memoizedIndex}
dataViewId={defineStepData.dataViewId}
timestampOverride={aboutStepData.timestampOverride}
form={aboutStepForm}
esqlQuery={esqlQueryForAboutStep}
key="aboutStep"
/>
)}
@ -365,6 +382,8 @@ const EditRulePageComponent: FC<{ rule: Rule }> = ({ rule }) => {
actionsStepData,
actionMessageParams,
actionsStepForm,
memoizedIndex,
esqlQueryForAboutStep,
]
);

View file

@ -7,6 +7,7 @@
import { useEffect, useState, useMemo, useCallback } from 'react';
import type { DataViewBase } from '@kbn/es-query';
import { getIndexListFromEsqlQuery } from '@kbn/securitysolution-utils';
import type { FieldSpec, DataViewSpec } from '@kbn/data-views-plugin/common';
@ -39,6 +40,10 @@ export const useFetchIndexPatterns = (rules: Rule[] | null): ReturnUseFetchExcep
() => rules != null && isSingleRule && rules[0].type === 'machine_learning',
[isSingleRule, rules]
);
const isEsqlRule = useMemo(
() => rules != null && isSingleRule && rules[0].type === 'esql',
[isSingleRule, rules]
);
useEffect(() => {
const fetchAndSetActiveSpace = async () => {
@ -59,13 +64,14 @@ export const useFetchIndexPatterns = (rules: Rule[] | null): ReturnUseFetchExcep
[isSingleRule, rules, activeSpaceId]
);
const memoNonDataViewIndexPatterns = useMemo(
() =>
!memoDataViewId && rules != null && isSingleRule && rules[0].index != null
? rules[0].index
: [],
[memoDataViewId, isSingleRule, rules]
);
const memoNonDataViewIndexPatterns = useMemo(() => {
if (isEsqlRule) {
return getIndexListFromEsqlQuery(rules?.[0].query);
}
return !memoDataViewId && rules != null && isSingleRule && rules[0].index != null
? rules[0].index
: [];
}, [memoDataViewId, isSingleRule, rules, isEsqlRule]);
// Index pattern logic for ML
const memoMlJobIds = useMemo(

View file

@ -201,6 +201,8 @@ const getRuleTypeDescription = (ruleType: Type) => {
return descriptionStepI18n.THRESHOLD_TYPE_DESCRIPTION;
case 'eql':
return descriptionStepI18n.EQL_TYPE_DESCRIPTION;
case 'esql':
return descriptionStepI18n.ESQL_TYPE_DESCRIPTION;
case 'threat_match':
return descriptionStepI18n.THREAT_MATCH_TYPE_DESCRIPTION;
case 'new_terms':
@ -358,7 +360,7 @@ const prepareDefinitionSectionListItems = (
}
}
if ('filters' in rule && rule.filters && rule.filters.length > 0) {
if ('filters' in rule && 'data_view_id' in rule && rule.filters?.length) {
definitionSectionListItems.push({
title: savedQuery
? descriptionStepI18n.SAVED_QUERY_FILTERS_LABEL

View file

@ -6,10 +6,15 @@
*/
import { useMemo } from 'react';
import { getIndexListFromEsqlQuery } from '@kbn/securitysolution-utils';
import { useSecurityJobs } from '../../../common/components/ml_popover/hooks/use_security_jobs';
import { useGetInstalledJob } from '../../../common/components/ml/hooks/use_get_jobs';
export const useRuleIndices = (machineLearningJobId?: string[], defaultRuleIndices?: string[]) => {
export const useRuleIndices = (
machineLearningJobId?: string[],
defaultRuleIndices?: string[],
esqlQuery?: string
) => {
const memoMlJobIds = useMemo(() => machineLearningJobId ?? [], [machineLearningJobId]);
const { loading: mlSecurityJobLoading, jobs } = useSecurityJobs();
const memoSelectedMlJobs = useMemo(
@ -32,10 +37,12 @@ export const useRuleIndices = (machineLearningJobId?: string[], defaultRuleIndic
const memoRuleIndices = useMemo(() => {
if (memoMlIndices.length > 0) {
return memoMlIndices;
} else if (esqlQuery) {
return getIndexListFromEsqlQuery(esqlQuery);
} else {
return defaultRuleIndices ?? [];
}
}, [defaultRuleIndices, memoMlIndices]);
}, [defaultRuleIndices, esqlQuery, memoMlIndices]);
return {
mlJobLoading: mlSecurityJobLoading || mlInstalledJobLoading,

View file

@ -63,6 +63,10 @@ describe('Component BulkEditRuleErrorsList', () => {
BulkActionsDryRunErrCode.MACHINE_LEARNING_INDEX_PATTERN,
"2 custom machine learning rules (these rules don't have index patterns)",
],
[
BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN,
"2 custom ES|QL rules (these rules don't have index patterns)",
],
[
BulkActionsDryRunErrCode.MACHINE_LEARNING_AUTH,
"2 machine learning rules can't be edited (test failure)",

View file

@ -56,6 +56,16 @@ const BulkEditRuleErrorItem = ({
/>
</li>
);
case BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN:
return (
<li key={message}>
<FormattedMessage
id="xpack.securitySolution.detectionEngine.rules.allRules.bulkActions.esqlRulesIndexEditDescription"
defaultMessage="{rulesCount, plural, =1 {# custom ES|QL rule} other {# custom ES|QL rules}} (these rules don't have index patterns)"
values={{ rulesCount }}
/>
</li>
);
default:
return (
<li key={message}>

View file

@ -63,6 +63,16 @@ describe('prepareSearchParams', () => {
excludeRuleTypes: ['machine_learning'],
},
],
[
BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN,
{
filter: '',
tags: [],
showCustomRules: false,
showElasticRules: false,
excludeRuleTypes: ['esql'],
},
],
[
BulkActionsDryRunErrCode.IMMUTABLE,
{

View file

@ -49,6 +49,12 @@ export const prepareSearchParams = ({
excludeRuleTypes: [...(modifiedFilterOptions.excludeRuleTypes ?? []), 'machine_learning'],
};
break;
case BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN:
modifiedFilterOptions = {
...modifiedFilterOptions,
excludeRuleTypes: [...(modifiedFilterOptions.excludeRuleTypes ?? []), 'esql'],
};
break;
}
});

View file

@ -6,11 +6,11 @@
*/
import React from 'react';
import { EuiIcon, EuiToolTip, EuiBetaBadge } from '@elastic/eui';
import { EuiIcon, EuiToolTip } from '@elastic/eui';
import type { LicenseService } from '../../../../../common/license';
import { minimumLicenseForSuppression } from '../../../../../common/api/detection_engine/model/rule_schema';
import { TechnicalPreviewBadge } from '../technical_preview_badge';
import * as i18n from './translations';
interface TechnicalPreviewBadgeProps {
@ -18,14 +18,12 @@ interface TechnicalPreviewBadgeProps {
license: LicenseService;
}
export const TechnicalPreviewBadge = ({ label, license }: TechnicalPreviewBadgeProps) => (
export const AlertSuppressionTechnicalPreviewBadge = ({
label,
license,
}: TechnicalPreviewBadgeProps) => (
<>
{label}
<EuiBetaBadge
label={i18n.ALERT_SUPPRESSION_TECHNICAL_PREVIEW}
style={{ verticalAlign: 'middle', marginLeft: '8px' }}
size="s"
/>
<TechnicalPreviewBadge label={label} />
{!license.isAtLeast(minimumLicenseForSuppression) && (
<EuiToolTip position="top" content={i18n.ALERT_SUPPRESSION_INSUFFICIENT_LICENSE}>
<EuiIcon type={'warning'} size="l" color="#BD271E" style={{ marginLeft: '8px' }} />

View file

@ -46,7 +46,8 @@ import type {
import { GroupByOptions } from '../../../pages/detection_engine/rules/types';
import { defaultToEmptyTag } from '../../../../common/components/empty_value';
import { ThreatEuiFlexGroup } from './threat_description';
import { TechnicalPreviewBadge } from './technical_preview_badge';
import { AlertSuppressionTechnicalPreviewBadge } from './alert_suppression_technical_preview_badge';
import { TechnicalPreviewBadge } from '../technical_preview_badge';
import type { LicenseService } from '../../../../../common/license';
import { AlertSuppressionMissingFieldsStrategy } from '../../../../../common/api/detection_engine/model/rule_schema';
const NoteDescriptionContainer = styled(EuiFlexItem)`
@ -449,6 +450,14 @@ export const buildRuleTypeDescription = (label: string, ruleType: Type): ListIte
},
];
}
case 'esql': {
return [
{
title: label,
description: <TechnicalPreviewBadge label={i18n.ESQL_TYPE_DESCRIPTION} />,
},
];
}
default:
return assertUnreachable(ruleType);
}
@ -571,7 +580,7 @@ export const buildAlertSuppressionDescription = (
</EuiFlexGroup>
);
const title = <TechnicalPreviewBadge label={label} license={license} />;
const title = <AlertSuppressionTechnicalPreviewBadge label={label} license={license} />;
return [
{
title,
@ -591,7 +600,7 @@ export const buildAlertSuppressionWindowDescription = (
? `${value.value}${value.unit}`
: i18n.ALERT_SUPPRESSION_PER_RULE_EXECUTION;
const title = <TechnicalPreviewBadge label={label} license={license} />;
const title = <AlertSuppressionTechnicalPreviewBadge label={label} license={license} />;
return [
{
title,
@ -614,7 +623,7 @@ export const buildAlertSuppressionMissingFieldsDescription = (
? i18n.ALERT_SUPPRESSION_SUPPRESS_ON_MISSING_FIELDS
: i18n.ALERT_SUPPRESSION_DO_NOT_SUPPRESS_ON_MISSING_FIELDS;
const title = <TechnicalPreviewBadge label={label} license={license} />;
const title = <AlertSuppressionTechnicalPreviewBadge label={label} license={license} />;
return [
{
title,

View file

@ -91,6 +91,13 @@ export const NEW_TERMS_TYPE_DESCRIPTION = i18n.translate(
}
);
export const ESQL_TYPE_DESCRIPTION = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.esqlRuleTypeDescription',
{
defaultMessage: 'ES|QL',
}
);
export const THRESHOLD_RESULTS_ALL = i18n.translate(
'xpack.securitySolution.detectionEngine.ruleDescription.thresholdResultsAllDescription',
{
@ -134,13 +141,6 @@ export const ALERT_SUPPRESSION_INSUFFICIENT_LICENSE = i18n.translate(
}
);
export const ALERT_SUPPRESSION_TECHNICAL_PREVIEW = i18n.translate(
'xpack.securitySolution.detectionEngine.ruleDescription.alertSuppressionTechnicalPreview',
{
defaultMessage: 'Technical Preview',
}
);
export const ALERT_SUPPRESSION_PER_RULE_EXECUTION = i18n.translate(
'xpack.securitySolution.detectionEngine.ruleDescription.alertSuppressionPerRuleExecution',
{

View file

@ -113,6 +113,39 @@ const isNewTermsPreviewDisabled = (newTermsFields: string[]): boolean => {
return newTermsFields.length === 0 || newTermsFields.length > MAX_NUMBER_OF_NEW_TERMS_FIELDS;
};
const isEsqlPreviewDisabled = ({
isQueryBarValid,
queryBar,
}: {
queryBar: FieldValueQueryBar;
isQueryBarValid: boolean;
}): boolean => {
return !isQueryBarValid || isEmpty(queryBar.query.query);
};
const isThreatMatchPreviewDisabled = ({
isThreatQueryBarValid,
threatIndex,
threatMapping,
}: {
threatIndex: string[];
threatMapping: ThreatMapping;
isThreatQueryBarValid: boolean;
}): boolean => {
if (!isThreatQueryBarValid || !threatIndex.length || !threatMapping) {
return true;
} else if (
!threatMapping.length ||
!threatMapping[0].entries?.length ||
!threatMapping[0].entries[0].field ||
!threatMapping[0].entries[0].value
) {
return true;
}
return false;
};
export const getIsRulePreviewDisabled = ({
ruleType,
isQueryBarValid,
@ -138,6 +171,9 @@ export const getIsRulePreviewDisabled = ({
queryBar: FieldValueQueryBar;
newTermsFields: string[];
}) => {
if (ruleType === 'esql') {
return isEsqlPreviewDisabled({ isQueryBarValid, queryBar });
}
if (
!isQueryBarValid ||
(dataSourceType === DataSourceType.DataView && !dataViewId) ||
@ -146,14 +182,11 @@ export const getIsRulePreviewDisabled = ({
return true;
}
if (ruleType === 'threat_match') {
if (!isThreatQueryBarValid || !threatIndex.length || !threatMapping) return true;
if (
!threatMapping.length ||
!threatMapping[0].entries?.length ||
!threatMapping[0].entries[0].field ||
!threatMapping[0].entries[0].value
)
return true;
return isThreatMatchPreviewDisabled({
threatIndex,
threatMapping,
isThreatQueryBarValid,
});
}
if (ruleType === 'machine_learning') {
return machineLearningJobId.length === 0;

View file

@ -10,8 +10,15 @@ import { mount, shallow } from 'enzyme';
import { SelectRuleType } from '.';
import { TestProviders, useFormFieldMock } from '../../../../common/mock';
import { useIsExperimentalFeatureEnabled } from '../../../../common/hooks/use_experimental_features';
jest.mock('../../../../common/lib/kibana');
jest.mock('../../../../common/hooks/use_experimental_features', () => ({
useIsExperimentalFeatureEnabled: jest.fn().mockReturnValue(false),
}));
const useIsExperimentalFeatureEnabledMock = useIsExperimentalFeatureEnabled as jest.Mock;
describe('SelectRuleType', () => {
it('renders correctly', () => {
const Component = () => {
@ -51,6 +58,7 @@ describe('SelectRuleType', () => {
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeTruthy();
});
it('renders only the card selected when in update mode of "eql"', () => {
@ -71,6 +79,7 @@ describe('SelectRuleType', () => {
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
});
it('renders only the card selected when in update mode of "machine_learning', () => {
@ -91,6 +100,7 @@ describe('SelectRuleType', () => {
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
});
it('renders only the card selected when in update mode of "query', () => {
@ -111,6 +121,7 @@ describe('SelectRuleType', () => {
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
});
it('renders only the card selected when in update mode of "threshold"', () => {
@ -131,6 +142,7 @@ describe('SelectRuleType', () => {
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
});
it('renders only the card selected when in update mode of "threat_match', () => {
@ -151,6 +163,47 @@ describe('SelectRuleType', () => {
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
});
it('renders only the card selected when in update mode of "esql"', () => {
const field = useFormFieldMock<unknown>({ value: 'esql' });
const wrapper = mount(
<TestProviders>
<SelectRuleType
describedByIds={[]}
field={field}
isUpdateView={true}
hasValidLicense={true}
isMlAdmin={true}
/>
</TestProviders>
);
expect(wrapper.find('[data-test-subj="customRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="machineLearningRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="thresholdRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="eqlRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="threatMatchRuleType"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeTruthy();
});
it('should not render "esql" rule type if its feature disabled', () => {
useIsExperimentalFeatureEnabledMock.mockReturnValue(false);
const Component = () => {
const field = useFormFieldMock();
return (
<SelectRuleType
field={field}
describedByIds={[]}
isUpdateView={false}
hasValidLicense={true}
isMlAdmin={true}
/>
);
};
const wrapper = shallow(<Component />);
expect(wrapper.find('[data-test-subj="esqlRuleType"]').exists()).toBeFalsy();
});
});

View file

@ -16,10 +16,13 @@ import {
isQueryRule,
isThreatMatchRule,
isNewTermsRule,
isEsqlRule,
} from '../../../../../common/detection_engine/utils';
import type { FieldHook } from '../../../../shared_imports';
import * as i18n from './translations';
import { MlCardDescription } from './ml_card_description';
import { TechnicalPreviewBadge } from '../technical_preview_badge';
import { useIsExperimentalFeatureEnabled } from '../../../../common/hooks/use_experimental_features';
interface SelectRuleTypeProps {
describedByIds: string[];
@ -44,6 +47,9 @@ export const SelectRuleType: React.FC<SelectRuleTypeProps> = memo(
const setThreshold = useCallback(() => setType('threshold'), [setType]);
const setThreatMatch = useCallback(() => setType('threat_match'), [setType]);
const setNewTerms = useCallback(() => setType('new_terms'), [setType]);
const setEsql = useCallback(() => setType('esql'), [setType]);
const isEsqlFeatureEnabled = !useIsExperimentalFeatureEnabled('esqlRulesDisabled');
const eqlSelectableConfig = useMemo(
() => ({
@ -94,6 +100,14 @@ export const SelectRuleType: React.FC<SelectRuleTypeProps> = memo(
[ruleType, setNewTerms]
);
const esqlSelectableConfig = useMemo(
() => ({
onClick: setEsql,
isSelected: isEsqlRule(ruleType),
}),
[ruleType, setEsql]
);
return (
<EuiFormRow
fullWidth
@ -181,6 +195,19 @@ export const SelectRuleType: React.FC<SelectRuleTypeProps> = memo(
/>
</EuiFlexItem>
)}
{isEsqlFeatureEnabled && (!isUpdateView || esqlSelectableConfig.isSelected) && (
<EuiFlexItem>
<EuiCard
data-test-subj="esqlRuleType"
title={<TechnicalPreviewBadge label={i18n.ESQL_TYPE_TITLE} />}
titleSize="xs"
description={i18n.ESQL_TYPE_DESCRIPTION}
icon={<EuiIcon type="logoElasticsearch" size="l" />}
selectable={esqlSelectableConfig}
layout="horizontal"
/>
</EuiFlexItem>
)}
</EuiFlexGrid>
</EuiFormRow>
);

View file

@ -22,6 +22,20 @@ export const EQL_TYPE_DESCRIPTION = i18n.translate(
}
);
export const ESQL_TYPE_TITLE = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.ruleTypeField.esqlTypeTitle',
{
defaultMessage: 'ES|QL',
}
);
export const ESQL_TYPE_DESCRIPTION = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.ruleTypeField.esqlTypeDescription',
{
defaultMessage: 'Use The Elasticsearch Query Language (ES|QL) to search or aggregate events',
}
);
export const QUERY_TYPE_TITLE = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.ruleTypeField.queryTypeTitle',
{

View file

@ -12,7 +12,8 @@ import styled from 'styled-components';
import type { DataViewBase } from '@kbn/es-query';
import type { Severity, Type } from '@kbn/securitysolution-io-ts-alerting-types';
import { isThreatMatchRule } from '../../../../../common/detection_engine/utils';
import { isThreatMatchRule, isEsqlRule } from '../../../../../common/detection_engine/utils';
import type { RuleStepProps, AboutStepRule } from '../../../pages/detection_engine/rules/types';
import { AddItem } from '../add_item_form';
import { StepRuleDescription } from '../description_step';
@ -33,6 +34,7 @@ import { useFetchIndex } from '../../../../common/containers/source';
import { DEFAULT_INDICATOR_SOURCE_PATH } from '../../../../../common/constants';
import { useKibana } from '../../../../common/lib/kibana';
import { useRuleIndices } from '../../../../detection_engine/rule_management/logic/use_rule_indices';
import { EsqlAutocomplete } from '../../../../detection_engine/rule_creation/components/esql_autocomplete';
import { MultiSelectFieldsAutocomplete } from '../multi_select_fields';
const CommonUseField = getUseField({ component: Field });
@ -44,7 +46,7 @@ interface StepAboutRuleProps extends RuleStepProps {
dataViewId: string | undefined;
timestampOverride: string;
form: FormHook<AboutStepRule>;
esqlQuery?: string | undefined;
// TODO: https://github.com/elastic/kibana/issues/161456
// The About step page contains EuiRange component which does not work properly within memoized parents.
// EUI team suggested not to memoize EuiRange/EuiDualRange: https://github.com/elastic/eui/issues/6846
@ -83,10 +85,12 @@ const StepAboutRuleComponent: FC<StepAboutRuleProps> = ({
isUpdateView = false,
isLoading,
form,
esqlQuery,
}) => {
const { data } = useKibana().services;
const isThreatMatchRuleValue = useMemo(() => isThreatMatchRule(ruleType), [ruleType]);
const isEsqlRuleValue = useMemo(() => isEsqlRule(ruleType), [ruleType]);
const { ruleIndices } = useRuleIndices(machineLearningJobId, index);
@ -327,18 +331,33 @@ const StepAboutRuleComponent: FC<StepAboutRuleProps> = ({
</>
)}
<EuiSpacer size="l" />
<UseField
path="ruleNameOverride"
component={AutocompleteField}
componentProps={{
dataTestSubj: 'detectionEngineStepAboutRuleRuleNameOverride',
fieldType: 'string',
idAria: 'detectionEngineStepAboutRuleRuleNameOverride',
indices: indexPattern,
isDisabled: isLoading || indexPatternLoading,
placeholder: '',
}}
/>
{isEsqlRuleValue ? (
<UseField
path="ruleNameOverride"
component={EsqlAutocomplete}
componentProps={{
dataTestSubj: 'detectionEngineStepAboutRuleRuleNameOverrideForEsqlRuleType',
idAria: 'detectionEngineStepAboutRuleRuleNameOverrideForEsqlRuleType',
esqlQuery,
fieldType: 'string',
isDisabled: isLoading,
}}
/>
) : (
<UseField
path="ruleNameOverride"
component={AutocompleteField}
componentProps={{
dataTestSubj: 'detectionEngineStepAboutRuleRuleNameOverride',
fieldType: 'string',
idAria: 'detectionEngineStepAboutRuleRuleNameOverride',
indices: indexPattern,
isDisabled: isLoading || indexPatternLoading,
placeholder: '',
}}
/>
)}
<EuiSpacer size="l" />
<UseField
path="timestampOverride"

View file

@ -18,7 +18,7 @@ import {
EuiRadioGroup,
} from '@elastic/eui';
import type { FC } from 'react';
import React, { memo, useCallback, useState, useEffect, useMemo } from 'react';
import React, { memo, useCallback, useState, useEffect, useMemo, useRef } from 'react';
import styled from 'styled-components';
import { i18n as i18nCore } from '@kbn/i18n';
@ -54,6 +54,7 @@ import { PickTimeline } from '../pick_timeline';
import { StepContentWrapper } from '../step_content_wrapper';
import { ThresholdInput } from '../threshold_input';
import { SuppressionInfoIcon } from '../suppression_info_icon';
import { EsqlInfoIcon } from '../../../../detection_engine/rule_creation/components/esql_info_icon';
import { Field, Form, getUseField, UseField, UseMultiFields } from '../../../../shared_imports';
import type { FormHook } from '../../../../shared_imports';
import { schema } from './schema';
@ -65,6 +66,7 @@ import {
isThreatMatchRule,
isThresholdRule,
isQueryRule,
isEsqlRule,
} from '../../../../../common/detection_engine/utils';
import { EqlQueryBar } from '../eql_query_bar';
import { DataViewSelector } from '../data_view_selector';
@ -174,6 +176,8 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
const [threatIndexModified, setThreatIndexModified] = useState(false);
const license = useLicense();
const esqlQueryRef = useRef<DefineStepRule['queryBar'] | undefined>(undefined);
const { getFields, reset, setFieldValue } = form;
const setRuleTypeCallback = useSetFieldValueWithCallback({
@ -312,6 +316,43 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
}
}, [ruleType, previousRuleType, getFields]);
/**
* ensures when user switches between rule types, written ES|QL query is not getting lost
* additional work is required in this code area, as currently switching to EQL will result in query lose
* https://github.com/elastic/kibana/issues/166933
*/
useEffect(() => {
const { queryBar: currentQuery } = getFields();
if (currentQuery == null) {
return;
}
const currentQueryValue = currentQuery.value as DefineStepRule['queryBar'];
// sets ES|QL query to a default value or earlier added one, when switching to ES|QL rule type
if (isEsqlRule(ruleType)) {
if (previousRuleType && !isEsqlRule(previousRuleType)) {
currentQuery.reset({
defaultValue: esqlQueryRef.current ?? defaultCustomQuery.forEsqlRules,
});
}
// otherwise reset it to default values of other rule types
} else if (isEsqlRule(previousRuleType)) {
// sets ES|QL query value to reference, so it can be used when user switch back from one rule type to another
if (currentQueryValue?.query?.language === 'esql') {
esqlQueryRef.current = currentQueryValue;
}
const defaultValue = isThreatMatchRule(ruleType)
? defaultCustomQuery.forThreatMatchRules
: defaultCustomQuery.forNormalRules;
currentQuery.reset({
defaultValue,
});
}
}, [ruleType, previousRuleType, getFields]);
// if saved query failed to load:
// - reset shouldLoadFormDynamically to false, as non existent query cannot be used for loading and execution
const handleSavedQueryError = useCallback(() => {
@ -570,6 +611,45 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
handleResetIndices,
]);
const queryBarProps = useMemo(
() =>
({
idAria: 'detectionEngineStepDefineRuleQueryBar',
indexPattern,
isDisabled: isLoading,
isLoading,
dataTestSubj: 'detectionEngineStepDefineRuleQueryBar',
onValidityChange: setIsQueryBarValid,
} as QueryBarDefineRuleProps),
[indexPattern, isLoading, setIsQueryBarValid]
);
const esqlQueryBarConfig = useMemo(
() => ({
...schema.queryBar,
label: i18n.ESQL_QUERY,
labelAppend: <EsqlInfoIcon />,
}),
[]
);
const EsqlQueryBarMemo = useMemo(
() => (
<UseField
key="QueryBarDefineRule"
path="queryBar"
config={esqlQueryBarConfig}
component={QueryBarDefineRule}
componentProps={{
...queryBarProps,
dataTestSubj: 'detectionEngineStepDefineRuleEsqlQueryBar',
idAria: 'detectionEngineStepDefineRuleEsqlQueryBar',
}}
/>
),
[queryBarProps, esqlQueryBarConfig]
);
const QueryBarMemo = useMemo(
() => (
<UseField
@ -688,8 +768,10 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
/>
<RuleTypeEuiFormRow $isVisible={!isMlRule(ruleType)} fullWidth>
<>
<EuiSpacer size="s" />
{DataSource}
<StyledVisibleContainer isVisible={!isEsqlRule(ruleType)}>
<EuiSpacer size="s" />
{DataSource}
</StyledVisibleContainer>
<EuiSpacer size="s" />
{isEqlRule(ruleType) ? (
<UseField
@ -715,6 +797,8 @@ const StepDefineRuleComponent: FC<StepDefineRuleProps> = ({
label: i18n.EQL_QUERY_BAR_LABEL,
}}
/>
) : isEsqlRule(ruleType) ? (
EsqlQueryBarMemo
) : (
QueryBarMemo
)}

View file

@ -17,7 +17,7 @@ import {
customValidators,
} from '../../../../common/components/threat_match/helpers';
import {
isEqlRule,
isEsqlRule,
isNewTermsRule,
isQueryRule,
isThreatMatchRule,
@ -31,15 +31,16 @@ import { FIELD_TYPES, fieldValidators } from '../../../../shared_imports';
import type { DefineStepRule } from '../../../pages/detection_engine/rules/types';
import { DataSourceType } from '../../../pages/detection_engine/rules/types';
import { debounceAsync, eqlValidator } from '../eql_query_bar/validators';
import { esqlValidator } from '../../../../detection_engine/rule_creation/logic/esql_validator';
import {
CUSTOM_QUERY_REQUIRED,
EQL_QUERY_REQUIRED,
INVALID_CUSTOM_QUERY,
INDEX_HELPER_TEXT,
THREAT_MATCH_INDEX_HELPER_TEXT,
THREAT_MATCH_REQUIRED,
THREAT_MATCH_EMPTIES,
} from './translations';
import { getQueryRequiredMessage } from './utils';
export const schema: FormSchema<DefineStepRule> = {
index: {
@ -60,7 +61,9 @@ export const schema: FormSchema<DefineStepRule> = {
): ReturnType<ValidationFunc<{}, ERROR_CODE>> | undefined => {
const [{ formData }] = args;
const skipValidation =
isMlRule(formData.ruleType) || formData.dataSourceType !== DataSourceType.IndexPatterns;
isMlRule(formData.ruleType) ||
isEsqlRule(formData.ruleType) ||
formData.dataSourceType !== DataSourceType.IndexPatterns;
if (skipValidation) {
return;
@ -129,6 +132,7 @@ export const schema: FormSchema<DefineStepRule> = {
},
eqlOptions: {},
queryBar: {
fieldsToValidateOnChange: ['queryBar'],
validations: [
{
validator: (
@ -150,7 +154,7 @@ export const schema: FormSchema<DefineStepRule> = {
// https://github.com/elastic/kibana/issues/159060
return undefined;
}
const message = isEqlRule(formData.ruleType) ? EQL_QUERY_REQUIRED : CUSTOM_QUERY_REQUIRED;
const message = getQueryRequiredMessage(formData.ruleType);
return { code: 'ERR_FIELD_MISSING', path, message };
},
},
@ -181,6 +185,9 @@ export const schema: FormSchema<DefineStepRule> = {
{
validator: debounceAsync(eqlValidator, 300),
},
{
validator: debounceAsync(esqlValidator, 300),
},
],
},
ruleType: {

View file

@ -21,6 +21,13 @@ export const EQL_QUERY_REQUIRED = i18n.translate(
}
);
export const ESQL_QUERY_REQUIRED = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlQueryFieldRequiredError',
{
defaultMessage: 'An ES|QL query is required.',
}
);
export const INVALID_CUSTOM_QUERY = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.customQueryFieldInvalidError',
{
@ -172,6 +179,13 @@ export const ALERT_SUPPRESSION_MISSING_FIELDS_DO_NOT_SUPPRESS_OPTION = i18n.tran
}
);
export const ESQL_QUERY = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.esqlQueryLabel',
{
defaultMessage: 'ES|QL query',
}
);
export const GROUP_BY_FIELD_LICENSE_WARNING = i18n.translate(
'xpack.securitySolution.detectionEngine.createRule.stepDefineRule.groupBy.licenseWarning',
{

View file

@ -5,8 +5,14 @@
* 2.0.
*/
import type { Type } from '@kbn/securitysolution-io-ts-alerting-types';
import type { BrowserField } from '../../../../common/containers/source';
import { CUSTOM_QUERY_REQUIRED, EQL_QUERY_REQUIRED, ESQL_QUERY_REQUIRED } from './translations';
import { isEqlRule, isEsqlRule } from '../../../../../common/detection_engine/utils';
/**
* Filters out fields, that are not supported in terms aggregation.
* Terms aggregation supports limited number of types:
@ -19,3 +25,18 @@ export const getTermsAggregationFields = (fields: BrowserField[]): BrowserField[
return fields.filter((field) => field.aggregatable === true && allowedTypesSet.has(field.type));
};
/**
* return query is required message depends on a rule type
*/
export const getQueryRequiredMessage = (ruleType: Type) => {
if (isEsqlRule(ruleType)) {
return ESQL_QUERY_REQUIRED;
}
if (isEqlRule(ruleType)) {
return EQL_QUERY_REQUIRED;
}
return CUSTOM_QUERY_REQUIRED;
};

View file

@ -0,0 +1,26 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React from 'react';
import { EuiBetaBadge } from '@elastic/eui';
import * as i18n from './translations';
interface TechnicalPreviewBadgeProps {
label: string;
}
export const TechnicalPreviewBadge = ({ label }: TechnicalPreviewBadgeProps) => (
<>
{label}
<EuiBetaBadge
label={i18n.TECHNICAL_PREVIEW}
style={{ verticalAlign: 'middle', marginLeft: '8px' }}
size="s"
/>
</>
);

View file

@ -0,0 +1,15 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { i18n } from '@kbn/i18n';
export const TECHNICAL_PREVIEW = i18n.translate(
'xpack.securitySolution.detectionEngine.ruleDescription.TechnicalPreviewLabel',
{
defaultMessage: 'Technical Preview',
}
);

View file

@ -359,6 +359,7 @@ const commonRuleParamsKeys = [
'version',
];
const queryRuleParams = ['index', 'filters', 'language', 'query', 'saved_id', 'response_actions'];
const esqlRuleParams = ['filters', 'language', 'query', 'response_actions'];
const machineLearningRuleParams = ['anomaly_threshold', 'machine_learning_job_id'];
const thresholdRuleParams = ['threshold', ...queryRuleParams];
@ -379,6 +380,8 @@ const getRuleSpecificRuleParamKeys = (ruleType: Type) => {
return machineLearningRuleParams;
case 'threshold':
return thresholdRuleParams;
case 'esql':
return esqlRuleParams;
case 'new_terms':
case 'threat_match':
case 'query':

View file

@ -132,4 +132,9 @@ const threatQueryBarDefaultValue: DefineStepRule['queryBar'] = {
export const defaultCustomQuery = {
forNormalRules: stepDefineDefaultValue.queryBar,
forThreatMatchRules: threatQueryBarDefaultValue,
forEsqlRules: {
query: { query: '', language: 'esql' },
filters: [],
saved_id: null,
},
};

View file

@ -49,6 +49,7 @@ import type { CloudExperimentsPluginStart } from '@kbn/cloud-experiments-plugin/
import type { GuidedOnboardingPluginStart } from '@kbn/guided-onboarding-plugin/public';
import type { DataViewsServicePublic } from '@kbn/data-views-plugin/public';
import type { ContentManagementPublicStart } from '@kbn/content-management-plugin/public';
import type { ExpressionsStart } from '@kbn/expressions-plugin/public';
import type { RouteProps } from 'react-router-dom';
import type { DiscoverStart } from '@kbn/discover-plugin/public';
@ -132,6 +133,7 @@ export interface StartPlugins {
fieldFormats: FieldFormatsStartCommon;
discover: DiscoverStart;
navigation: NavigationPublicPluginStart;
expressions: ExpressionsStart;
dataViewEditor: DataViewEditorStart;
savedSearch: SavedSearchPublicPluginStart;
}

View file

@ -14,6 +14,7 @@ import type {
DiffableCommonFields,
DiffableCustomQueryFields,
DiffableEqlFields,
DiffableEsqlFields,
DiffableMachineLearningFields,
DiffableNewTermsFields,
DiffableRule,
@ -23,6 +24,7 @@ import type {
CommonFieldsDiff,
CustomQueryFieldsDiff,
EqlFieldsDiff,
EsqlFieldsDiff,
MachineLearningFieldsDiff,
NewTermsFieldsDiff,
RuleFieldsDiff,
@ -142,6 +144,16 @@ export const calculateRuleFieldsDiff = (
...calculateNewTermsFieldsDiff({ base_version, current_version, target_version }),
};
}
case 'esql': {
if (hasBaseVersion) {
invariant(base_version.type === 'esql', BASE_TYPE_ERROR);
}
invariant(target_version.type === 'esql', TARGET_TYPE_ERROR);
return {
...commonFieldsDiff,
...calculateEsqlFieldsDiff({ base_version, current_version, target_version }),
};
}
default: {
return assertUnreachable(current_version, 'Unhandled rule type');
}
@ -226,6 +238,17 @@ const eqlFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableEqlFields> = {
tiebreaker_field: simpleDiffAlgorithm,
};
const calculateEsqlFieldsDiff = (
ruleVersions: ThreeVersionsOf<DiffableEsqlFields>
): EsqlFieldsDiff => {
return calculateFieldsDiffFor(ruleVersions, esqlFieldsDiffAlgorithms);
};
const esqlFieldsDiffAlgorithms: FieldsDiffAlgorithmsFor<DiffableEsqlFields> = {
type: simpleDiffAlgorithm,
esql_query: simpleDiffAlgorithm,
};
const calculateThreatMatchFieldsDiff = (
ruleVersions: ThreeVersionsOf<DiffableThreatMatchFields>
): ThreatMatchFieldsDiff => {

View file

@ -10,6 +10,8 @@ import { assertUnreachable } from '../../../../../../../common/utility_types';
import type {
EqlRule,
EqlRuleCreateProps,
EsqlRule,
EsqlRuleCreateProps,
MachineLearningRule,
MachineLearningRuleCreateProps,
NewTermsRule,
@ -29,6 +31,7 @@ import type {
DiffableCommonFields,
DiffableCustomQueryFields,
DiffableEqlFields,
DiffableEsqlFields,
DiffableMachineLearningFields,
DiffableNewTermsFields,
DiffableRule,
@ -40,6 +43,7 @@ import { extractBuildingBlockObject } from './extract_building_block_object';
import {
extractInlineKqlQuery,
extractRuleEqlQuery,
extractRuleEsqlQuery,
extractRuleKqlQuery,
} from './extract_rule_data_query';
import { extractRuleDataSource } from './extract_rule_data_source';
@ -91,6 +95,11 @@ export const convertRuleToDiffable = (rule: RuleResponse | PrebuiltRuleAsset): D
...commonFields,
...extractDiffableNewTermsFieldsFromRuleObject(rule),
};
case 'esql':
return {
...commonFields,
...extractDiffableEsqlFieldsFromRuleObject(rule),
};
default:
return assertUnreachable(rule, 'Unhandled rule type');
}
@ -176,6 +185,15 @@ const extractDiffableEqlFieldsFromRuleObject = (
};
};
const extractDiffableEsqlFieldsFromRuleObject = (
rule: EsqlRule | EsqlRuleCreateProps
): DiffableEsqlFields => {
return {
type: rule.type,
esql_query: extractRuleEsqlQuery(rule.query, rule.language),
};
};
const extractDiffableThreatMatchFieldsFromRuleObject = (
rule: ThreatMatchRule | ThreatMatchRuleCreateProps
): DiffableThreatMatchFields => {

View file

@ -7,6 +7,7 @@
import type {
EqlQueryLanguage,
EsqlQueryLanguage,
KqlQueryLanguage,
RuleFilterArray,
RuleQuery,
@ -14,6 +15,7 @@ import type {
import type {
InlineKqlQuery,
RuleEqlQuery,
RuleEsqlQuery,
RuleKqlQuery,
} from '../../../../../../../common/api/detection_engine/prebuilt_rules';
import { KqlQueryType } from '../../../../../../../common/api/detection_engine/prebuilt_rules';
@ -58,3 +60,13 @@ export const extractRuleEqlQuery = (
filters: filters ?? [],
};
};
export const extractRuleEsqlQuery = (
query: RuleQuery,
language: EsqlQueryLanguage
): RuleEsqlQuery => {
return {
query,
language,
};
};

View file

@ -376,6 +376,41 @@ describe('ruleParamsModifier', () => {
"Index patterns can't be overwritten. Machine learning rule doesn't have index patterns property"
);
});
test('should throw error on adding index pattern if rule is of ES|QL type', () => {
expect(() =>
ruleParamsModifier({ type: 'esql' } as RuleAlertType['params'], [
{
type: BulkActionEditType.add_index_patterns,
value: ['my-index-*'],
},
])
).toThrow("Index patterns can't be added. ES|QL rule doesn't have index patterns property");
});
test('should throw error on deleting index pattern if rule is of ES|QL type', () => {
expect(() =>
ruleParamsModifier({ type: 'esql' } as RuleAlertType['params'], [
{
type: BulkActionEditType.delete_index_patterns,
value: ['my-index-*'],
},
])
).toThrow("Index patterns can't be deleted. ES|QL rule doesn't have index patterns property");
});
test('should throw error on overwriting index pattern if rule is of ES|QL type', () => {
expect(() =>
ruleParamsModifier({ type: 'esql' } as RuleAlertType['params'], [
{
type: BulkActionEditType.set_index_patterns,
value: ['my-index-*'],
},
])
).toThrow(
"Index patterns can't be overwritten. ES|QL rule doesn't have index patterns property"
);
});
});
describe('timeline', () => {

View file

@ -85,6 +85,10 @@ const applyBulkActionEditToRuleParams = (
ruleParams.type !== 'machine_learning',
"Index patterns can't be added. Machine learning rule doesn't have index patterns property"
);
invariant(
ruleParams.type !== 'esql',
"Index patterns can't be added. ES|QL rule doesn't have index patterns property"
);
if (shouldSkipIndexPatternsBulkAction(ruleParams.index, ruleParams.dataViewId, action)) {
isActionSkipped = true;
@ -103,6 +107,10 @@ const applyBulkActionEditToRuleParams = (
ruleParams.type !== 'machine_learning',
"Index patterns can't be deleted. Machine learning rule doesn't have index patterns property"
);
invariant(
ruleParams.type !== 'esql',
"Index patterns can't be deleted. ES|QL rule doesn't have index patterns property"
);
if (
!action.overwrite_data_views &&
@ -126,6 +134,10 @@ const applyBulkActionEditToRuleParams = (
ruleParams.type !== 'machine_learning',
"Index patterns can't be overwritten. Machine learning rule doesn't have index patterns property"
);
invariant(
ruleParams.type !== 'esql',
"Index patterns can't be overwritten. ES|QL rule doesn't have index patterns property"
);
if (shouldSkipIndexPatternsBulkAction(ruleParams.index, ruleParams.dataViewId, action)) {
isActionSkipped = true;

View file

@ -8,6 +8,7 @@
import type { Type as RuleType } from '@kbn/securitysolution-io-ts-alerting-types';
import { invariant } from '../../../../../../common/utils/invariant';
import { isMlRule } from '../../../../../../common/machine_learning/helpers';
import { isEsqlRule } from '../../../../../../common/detection_engine/utils';
import { BulkActionsDryRunErrCode } from '../../../../../../common/constants';
import type { BulkActionEditPayload } from '../../../../../../common/api/detection_engine/rule_management/bulk_actions/bulk_actions_route';
import { BulkActionEditType } from '../../../../../../common/api/detection_engine/rule_management/bulk_actions/bulk_actions_route';
@ -129,4 +130,15 @@ export const dryRunValidateBulkEditRule = async ({
),
BulkActionsDryRunErrCode.MACHINE_LEARNING_INDEX_PATTERN
);
// if rule is es|ql, index pattern action can't be applied to it
await throwDryRunError(
() =>
invariant(
!isEsqlRule(rule.params.type) ||
!edit.some((action) => isIndexPatternsBulkEditAction(action.type)),
"ES|QL rule doesn't have index patterns"
),
BulkActionsDryRunErrCode.ESQL_INDEX_PATTERN
);
};

View file

@ -7,6 +7,7 @@
import {
EQL_RULE_TYPE_ID,
ESQL_RULE_TYPE_ID,
INDICATOR_RULE_TYPE_ID,
ML_RULE_TYPE_ID,
NEW_TERMS_RULE_TYPE_ID,
@ -18,6 +19,7 @@ import {
import { enrichFilterWithRuleTypeMapping } from './enrich_filter_with_rule_type_mappings';
const allAlertTypeIds = `alert.attributes.alertTypeId: ${EQL_RULE_TYPE_ID}
OR alert.attributes.alertTypeId: ${ESQL_RULE_TYPE_ID}
OR alert.attributes.alertTypeId: ${ML_RULE_TYPE_ID}
OR alert.attributes.alertTypeId: ${QUERY_RULE_TYPE_ID}
OR alert.attributes.alertTypeId: ${SAVED_QUERY_RULE_TYPE_ID}

View file

@ -29,6 +29,7 @@ import type {
} from '../../../../../common/api/detection_engine/model/rule_schema';
import {
EqlPatchParams,
EsqlPatchParams,
MachineLearningPatchParams,
NewTermsPatchParams,
QueryPatchParams,
@ -59,6 +60,8 @@ import type {
BaseRuleParams,
EqlRuleParams,
EqlSpecificRuleParams,
EsqlRuleParams,
EsqlSpecificRuleParams,
ThreatRuleParams,
ThreatSpecificRuleParams,
QueryRuleParams,
@ -107,6 +110,13 @@ export const typeSpecificSnakeToCamel = (
tiebreakerField: params.tiebreaker_field,
};
}
case 'esql': {
return {
type: params.type,
language: params.language,
query: params.query,
};
}
case 'threat_match': {
return {
type: params.type,
@ -206,6 +216,17 @@ const patchEqlParams = (
};
};
const patchEsqlParams = (
params: EsqlPatchParams,
existingRule: EsqlRuleParams
): EsqlSpecificRuleParams => {
return {
type: existingRule.type,
language: params.language ?? existingRule.language,
query: params.query ?? existingRule.query,
};
};
const patchThreatMatchParams = (
params: ThreatMatchPatchParams,
existingRule: ThreatRuleParams
@ -341,6 +362,13 @@ export const patchTypeSpecificSnakeToCamel = (
}
return patchEqlParams(validated, existingRule);
}
case 'esql': {
const [validated, error] = validateNonExact(params, EsqlPatchParams);
if (validated == null) {
throw parseValidationError(error);
}
return patchEsqlParams(validated, existingRule);
}
case 'threat_match': {
const [validated, error] = validateNonExact(params, ThreatMatchPatchParams);
if (validated == null) {
@ -526,6 +554,13 @@ export const typeSpecificCamelToSnake = (params: TypeSpecificRuleParams): TypeSp
tiebreaker_field: params.tiebreakerField,
};
}
case 'esql': {
return {
type: params.type,
language: params.language,
query: params.query,
};
}
case 'threat_match': {
return {
type: params.type,

View file

@ -53,6 +53,7 @@ import type {
} from '../../../rule_types/types';
import {
createEqlAlertType,
createEsqlAlertType,
createIndicatorMatchAlertType,
createMlAlertType,
createQueryAlertType,
@ -419,6 +420,27 @@ export const previewRulesRoute = async (
}
);
break;
case 'esql':
if (config.experimentalFeatures.esqlRulesDisabled) {
throw Error('ES|QL rule type is not supported');
}
const esqlAlertType = previewRuleTypeWrapper(createEsqlAlertType(ruleOptions));
await runExecutors(
esqlAlertType.executor,
esqlAlertType.id,
esqlAlertType.name,
previewRuleParams,
() => true,
{
create: alertInstanceFactoryStub,
alertLimit: {
getValue: () => 1000,
setLimitReached: () => {},
},
done: () => ({ getRecoveredAlerts: () => [] }),
}
);
break;
case 'machine_learning':
const mlAlertType = previewRuleTypeWrapper(createMlAlertType(ruleOptions));
await runExecutors(

View file

@ -28,6 +28,7 @@ import {
import {
SIGNALS_ID,
EQL_RULE_TYPE_ID,
ESQL_RULE_TYPE_ID,
INDICATOR_RULE_TYPE_ID,
ML_RULE_TYPE_ID,
QUERY_RULE_TYPE_ID,
@ -141,6 +142,15 @@ export const eqlRuleParams = t.intersection([baseRuleParams, eqlSpecificRulePara
export type EqlSpecificRuleParams = t.TypeOf<typeof eqlSpecificRuleParams>;
export type EqlRuleParams = t.TypeOf<typeof eqlRuleParams>;
const esqlSpecificRuleParams = t.type({
type: t.literal('esql'),
language: t.literal('esql'),
query: RuleQuery,
});
export const esqlRuleParams = t.intersection([baseRuleParams, esqlSpecificRuleParams]);
export type EsqlSpecificRuleParams = t.TypeOf<typeof esqlSpecificRuleParams>;
export type EsqlRuleParams = t.TypeOf<typeof esqlRuleParams>;
const threatSpecificRuleParams = t.type({
type: t.literal('threat_match'),
language: nonEqlLanguages,
@ -244,6 +254,7 @@ export type NewTermsRuleParams = t.TypeOf<typeof newTermsRuleParams>;
export const typeSpecificRuleParams = t.union([
eqlSpecificRuleParams,
esqlSpecificRuleParams,
threatSpecificRuleParams,
querySpecificRuleParams,
savedQuerySpecificRuleParams,
@ -265,6 +276,7 @@ export interface CompleteRule<T extends RuleParams> {
export const allRuleTypes = t.union([
t.literal(SIGNALS_ID),
t.literal(EQL_RULE_TYPE_ID),
t.literal(ESQL_RULE_TYPE_ID),
t.literal(INDICATOR_RULE_TYPE_ID),
t.literal(ML_RULE_TYPE_ID),
t.literal(QUERY_RULE_TYPE_ID),

View file

@ -16,6 +16,7 @@ import { buildExceptionFilter } from '@kbn/lists-plugin/server/services/exceptio
import { technicalRuleFieldMap } from '@kbn/rule-registry-plugin/common/assets/field_maps/technical_rule_field_map';
import type { FieldMap } from '@kbn/alerts-as-data-utils';
import { parseScheduleDates } from '@kbn/securitysolution-io-ts-utils';
import { getIndexListFromEsqlQuery } from '@kbn/securitysolution-utils';
import type { FormatAlert } from '@kbn/alerting-plugin/server/types';
import {
checkPrivilegesFromEsClient,
@ -24,6 +25,7 @@ import {
hasReadIndexPrivileges,
hasTimestampFields,
isMachineLearningParams,
isEsqlParams,
} from './utils/utils';
import { DEFAULT_MAX_SIGNALS, DEFAULT_SEARCH_AFTER_PAGE_SIZE } from '../../../../common/constants';
import type { CreateSecurityRuleTypeWrapper } from './types';
@ -209,13 +211,16 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper =
/**
* Data Views Logic
* Use of data views is supported for all rules other than ML.
* Use of data views is supported for all rules other than ML and Esql.
* Rules can define both a data view and index pattern, but on execution:
* - Data view is used if it is defined
* - Rule exits early if data view defined is not found (ie: it's been deleted)
* - If no data view defined, falls to using existing index logic
* Esql rules has index in query, which can be retrieved
*/
if (!isMachineLearningParams(params)) {
if (isEsqlParams(params)) {
inputIndex = getIndexListFromEsqlQuery(params.query);
} else if (!isMachineLearningParams(params)) {
try {
const { index, runtimeMappings: dataViewRuntimeMappings } = await getInputIndex({
index: params.index,
@ -322,7 +327,7 @@ export const createSecurityRuleTypeWrapper: CreateSecurityRuleTypeWrapper =
});
}
if (!isMachineLearningParams(params)) {
if (!isMachineLearningParams(params) && !isEsqlParams(params)) {
inputIndexFields = await getFieldsForWildcard({
index: inputIndex,
dataViews: services.dataViews,

View file

@ -0,0 +1,65 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import type { Filter } from '@kbn/es-query';
import type {
RuleFilterArray,
TimestampOverride,
} from '../../../../../common/api/detection_engine/model/rule_schema';
import { buildTimeRangeFilter } from '../utils/build_events_query';
import { getQueryFilter } from '../utils/get_query_filter';
export interface BuildEqlSearchRequestParams {
query: string;
from: string;
to: string;
size: number;
filters: RuleFilterArray | undefined;
primaryTimestamp: TimestampOverride;
secondaryTimestamp: TimestampOverride | undefined;
exceptionFilter: Filter | undefined;
}
export const buildEsqlSearchRequest = ({
query,
from,
to,
filters,
primaryTimestamp,
secondaryTimestamp,
exceptionFilter,
size,
}: BuildEqlSearchRequestParams) => {
const esFilter = getQueryFilter({
query: '',
language: 'esql',
filters: filters || [],
index: undefined,
exceptionFilter,
});
const rangeFilter = buildTimeRangeFilter({
to,
from,
primaryTimestamp,
secondaryTimestamp,
});
const requestFilter: estypes.QueryDslQueryContainer[] = [rangeFilter, esFilter];
return {
// we limit size of the response to maxAlertNumber + 1
// ES|QL currently does not support pagination and returns 10,000 results
query: `${query} | limit ${size + 1}`,
filter: {
bool: {
filter: requestFilter,
},
},
};
};

View file

@ -0,0 +1,53 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { validateNonExact } from '@kbn/securitysolution-io-ts-utils';
import { ESQL_RULE_TYPE_ID } from '@kbn/securitysolution-rules';
import { SERVER_APP_ID } from '../../../../../common/constants';
import type { EsqlRuleParams } from '../../rule_schema';
import { esqlRuleParams } from '../../rule_schema';
import { esqlExecutor } from './esql';
import type { CreateRuleOptions, SecurityAlertType } from '../types';
export const createEsqlAlertType = (
createOptions: CreateRuleOptions
): SecurityAlertType<EsqlRuleParams, {}, {}, 'default'> => {
const { version } = createOptions;
return {
id: ESQL_RULE_TYPE_ID,
name: 'ES|QL Rule',
validate: {
params: {
validate: (object: unknown) => {
const [validated, errors] = validateNonExact(object, esqlRuleParams);
if (errors != null) {
throw new Error(errors);
}
if (validated == null) {
throw new Error('Validation of rule params failed');
}
return validated;
},
},
},
actionGroups: [
{
id: 'default',
name: 'Default',
},
],
defaultActionGroupId: 'default',
actionVariables: {
context: [{ name: 'server', description: 'the server' }],
},
minimumLicenseRequired: 'basic',
isExportable: false,
producer: SERVER_APP_ID,
executor: (params) => esqlExecutor({ ...params, version }),
};
};

View file

@ -0,0 +1,166 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { performance } from 'perf_hooks';
import type {
AlertInstanceContext,
AlertInstanceState,
RuleExecutorServices,
} from '@kbn/alerting-plugin/server';
import {
computeIsESQLQueryAggregating,
getIndexListFromEsqlQuery,
} from '@kbn/securitysolution-utils';
import { buildEsqlSearchRequest } from './build_esql_search_request';
import { performEsqlRequest } from './esql_request';
import { wrapEsqlAlerts } from './wrap_esql_alerts';
import { createEnrichEventsFunction } from '../utils/enrichments';
import { rowToDocument } from './utils';
import { fetchSourceDocuments } from './fetch_source_documents';
import type { RunOpts } from '../types';
import {
addToSearchAfterReturn,
createSearchAfterReturnType,
makeFloatString,
getUnprocessedExceptionsWarnings,
getMaxSignalsWarning,
} from '../utils/utils';
import type { EsqlRuleParams } from '../../rule_schema';
import { withSecuritySpan } from '../../../../utils/with_security_span';
/**
* ES|QL returns results as a single page. max size of 10,000
* while we try increase size of the request to catch all events
* we don't want to overload ES/Kibana with large responses
*/
const ESQL_PAGE_SIZE_CIRCUIT_BREAKER = 1000;
export const esqlExecutor = async ({
runOpts: {
completeRule,
tuple,
ruleExecutionLogger,
bulkCreate,
mergeStrategy,
primaryTimestamp,
secondaryTimestamp,
exceptionFilter,
unprocessedExceptions,
alertTimestampOverride,
publicBaseUrl,
},
services,
state,
spaceId,
}: {
runOpts: RunOpts<EsqlRuleParams>;
services: RuleExecutorServices<AlertInstanceState, AlertInstanceContext, 'default'>;
state: object;
spaceId: string;
version: string;
}) => {
const ruleParams = completeRule.ruleParams;
return withSecuritySpan('esqlExecutor', async () => {
const result = createSearchAfterReturnType();
let size = tuple.maxSignals;
while (
result.createdSignalsCount <= tuple.maxSignals &&
size <= ESQL_PAGE_SIZE_CIRCUIT_BREAKER
) {
const esqlRequest = buildEsqlSearchRequest({
query: ruleParams.query,
from: tuple.from.toISOString(),
to: tuple.to.toISOString(),
size,
filters: [],
primaryTimestamp,
secondaryTimestamp,
exceptionFilter,
});
ruleExecutionLogger.debug(`ES|QL query request: ${JSON.stringify(esqlRequest)}`);
const exceptionsWarning = getUnprocessedExceptionsWarnings(unprocessedExceptions);
if (exceptionsWarning) {
result.warningMessages.push(exceptionsWarning);
}
const esqlSignalSearchStart = performance.now();
const response = await performEsqlRequest({
esClient: services.scopedClusterClient.asCurrentUser,
requestParams: esqlRequest,
});
const esqlSearchDuration = makeFloatString(performance.now() - esqlSignalSearchStart);
result.searchAfterTimes = [esqlSearchDuration];
ruleExecutionLogger.debug(`ES|QL query request took: ${esqlSearchDuration}ms`);
const isRuleAggregating = computeIsESQLQueryAggregating(completeRule.ruleParams.query);
const results = response.values
// slicing already processed results in previous iterations
.slice(size - tuple.maxSignals)
.map((row) => rowToDocument(response.columns, row));
const index = getIndexListFromEsqlQuery(completeRule.ruleParams.query);
const sourceDocuments = await fetchSourceDocuments({
esClient: services.scopedClusterClient.asCurrentUser,
results,
index,
isRuleAggregating,
});
const wrappedAlerts = wrapEsqlAlerts({
sourceDocuments,
isRuleAggregating,
results,
spaceId,
completeRule,
mergeStrategy,
alertTimestampOverride,
ruleExecutionLogger,
publicBaseUrl,
tuple,
});
const enrichAlerts = createEnrichEventsFunction({
services,
logger: ruleExecutionLogger,
});
const bulkCreateResult = await bulkCreate(
wrappedAlerts,
tuple.maxSignals - result.createdSignalsCount,
enrichAlerts
);
addToSearchAfterReturn({ current: result, next: bulkCreateResult });
ruleExecutionLogger.debug(`Created ${bulkCreateResult.createdItemsCount} alerts`);
if (bulkCreateResult.alertsWereTruncated) {
result.warningMessages.push(getMaxSignalsWarning());
}
// no more results will be found
if (response.values.length < size) {
ruleExecutionLogger.debug(
`End of search: Found ${response.values.length} results with page size ${size}`
);
break;
}
// ES|QL does not support pagination so we need to increase size of response to be able to catch all events
size += tuple.maxSignals;
}
return { ...result, state };
});
};

View file

@ -0,0 +1,51 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { Logger, ElasticsearchClient } from '@kbn/core/server';
import { getKbnServerError } from '@kbn/kibana-utils-plugin/server';
export interface EsqlResultColumn {
name: string;
type: 'date' | 'keyword';
}
export type EsqlResultRow = Array<string | null>;
export interface EsqlTable {
columns: EsqlResultColumn[];
values: EsqlResultRow[];
}
export const performEsqlRequest = async ({
esClient,
requestParams,
}: {
logger?: Logger;
esClient: ElasticsearchClient;
requestParams: Record<string, unknown>;
}): Promise<EsqlTable> => {
const search = async () => {
try {
const rawResponse = await esClient.transport.request<EsqlTable>({
method: 'POST',
path: '/_query',
body: {
...requestParams,
},
});
return {
rawResponse,
isPartial: false,
isRunning: false,
};
} catch (e) {
throw getKbnServerError(e);
}
};
return (await search()).rawResponse;
};

View file

@ -0,0 +1,67 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { ElasticsearchClient } from '@kbn/core/server';
import type * as estypes from '@elastic/elasticsearch/lib/api/types';
interface FetchSourceDocumentsArgs {
isRuleAggregating: boolean;
esClient: ElasticsearchClient;
index: string[];
results: Array<Record<string, string | null>>;
}
/**
* fetches source documents by list of their ids
* it used for a case when non-aggregating has _id property to enrich alert with source document,
* if some of the properties missed from resulted query
*/
export const fetchSourceDocuments = async ({
isRuleAggregating,
results,
esClient,
index,
}: FetchSourceDocumentsArgs): Promise<Record<string, { fields: estypes.SearchHit['fields'] }>> => {
const ids = results.reduce<string[]>((acc, doc) => {
if (doc._id) {
acc.push(doc._id);
}
return acc;
}, []);
// we will fetch source documents only for non-aggregating rules, since aggregating do not have _id
if (ids.length === 0 || isRuleAggregating) {
return {};
}
const idsQuery = {
query: {
bool: {
filter: {
ids: { values: ids },
},
},
},
};
const response = await esClient.search({
index,
body: {
query: idsQuery.query,
_source: false,
fields: ['*'],
},
ignore_unavailable: true,
});
return response.hits.hits.reduce<Record<string, { fields: estypes.SearchHit['fields'] }>>(
(acc, hit) => {
acc[hit._id] = { fields: hit.fields };
return acc;
},
{}
);
};

View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './row_to_document';

View file

@ -0,0 +1,27 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { rowToDocument } from './row_to_document';
describe('rowToDocument', () => {
it('should convert row to document', () => {
const columns = [
{ name: '_id', type: 'keyword' as const },
{ name: 'agent.name', type: 'keyword' as const },
{ name: 'agent.version', type: 'keyword' as const },
{ name: 'agent.type', type: 'keyword' as const },
];
const row = ['abcd', null, '8.8.1', 'packetbeat'];
expect(rowToDocument(columns, row)).toEqual({
_id: 'abcd',
'agent.name': null,
'agent.version': '8.8.1',
'agent.type': 'packetbeat',
});
});
});

View file

@ -0,0 +1,25 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { EsqlResultRow, EsqlResultColumn } from '../esql_request';
/**
* transform ES|QL result row to JSON object
* @param columns
* @param row
* @returns Record<string, string | null>
*/
export const rowToDocument = (
columns: EsqlResultColumn[],
row: EsqlResultRow
): Record<string, string | null> => {
return columns.reduce<Record<string, string | null>>((acc, column, i) => {
acc[column.name] = row[i];
return acc;
}, {});
};

View file

@ -0,0 +1,102 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import objectHash from 'object-hash';
import type { Moment } from 'moment';
import type * as estypes from '@elastic/elasticsearch/lib/api/types';
import type {
BaseFieldsLatest,
WrappedFieldsLatest,
} from '../../../../../common/api/detection_engine/model/alerts';
import type { ConfigType } from '../../../../config';
import type { CompleteRule, EsqlRuleParams } from '../../rule_schema';
import { buildReasonMessageForNewTermsAlert } from '../utils/reason_formatters';
import type { IRuleExecutionLogForExecutors } from '../../rule_monitoring';
import { buildBulkBody } from '../factories/utils/build_bulk_body';
import type { SignalSource } from '../types';
export const wrapEsqlAlerts = ({
results,
spaceId,
completeRule,
mergeStrategy,
alertTimestampOverride,
ruleExecutionLogger,
publicBaseUrl,
tuple,
sourceDocuments,
isRuleAggregating,
}: {
isRuleAggregating: boolean;
sourceDocuments: Record<string, { fields: estypes.SearchHit['fields'] }>;
results: Array<Record<string, string | null>>;
spaceId: string | null | undefined;
completeRule: CompleteRule<EsqlRuleParams>;
mergeStrategy: ConfigType['alertMergeStrategy'];
alertTimestampOverride: Date | undefined;
ruleExecutionLogger: IRuleExecutionLogForExecutors;
publicBaseUrl: string | undefined;
tuple: {
to: Moment;
from: Moment;
maxSignals: number;
};
}): Array<WrappedFieldsLatest<BaseFieldsLatest>> => {
const wrapped = results.map<WrappedFieldsLatest<BaseFieldsLatest>>((document, i) => {
const ruleRunId = tuple.from.toISOString() + tuple.to.toISOString();
// for aggregating rules when metadata _id is present, generate alert based on ES document event id
const id =
!isRuleAggregating && document._id
? objectHash([
document._id,
document._version,
document._index,
`${spaceId}:${completeRule.alertId}`,
])
: objectHash([
ruleRunId,
completeRule.ruleParams.query,
`${spaceId}:${completeRule.alertId}`,
i,
]);
// metadata fields need to be excluded from source, otherwise alerts creation fails
const { _id, _version, _index, ...source } = document;
const baseAlert: BaseFieldsLatest = buildBulkBody(
spaceId,
completeRule,
{
_source: source as SignalSource,
fields: _id ? sourceDocuments[_id]?.fields : undefined,
_id: _id ?? '',
_index: _index ?? '',
},
mergeStrategy,
[],
true,
buildReasonMessageForNewTermsAlert,
[],
alertTimestampOverride,
ruleExecutionLogger,
id,
publicBaseUrl
);
return {
_id: id,
_index: _index ?? '',
_source: {
...baseAlert,
},
};
});
return wrapped;
};

View file

@ -120,7 +120,6 @@ describe('stripNonEcsFields', () => {
});
expect(result).toEqual({
agent: [],
message: 'test message',
});
expect(removed).toEqual([
@ -142,7 +141,7 @@ describe('stripNonEcsFields', () => {
});
expect(result).toEqual({
agent: { type: 'filebeat', name: [] },
agent: { type: 'filebeat' },
message: 'test message',
});
expect(removed).toEqual([
@ -177,6 +176,42 @@ describe('stripNonEcsFields', () => {
]);
});
it('should strip conflicting fields that use dot notation and is an array', () => {
const { result, removed } = stripNonEcsFields({
'agent.name.text': ['1'],
message: 'test message',
});
expect(result).toEqual({
message: 'test message',
});
expect(removed).toEqual([
{
key: 'agent.name.text',
value: '1',
},
]);
});
it('should strip conflicting fields that use dot notation and is an empty array', () => {
const { result, removed } = stripNonEcsFields({
'agent.name.text': [],
message: 'test message',
});
expect(result).toEqual({
message: 'test message',
});
expect(removed).toEqual([
{
key: 'agent.name.text',
value: [],
},
]);
});
it('should not strip valid ECS fields that use dot notation', () => {
const { result, removed } = stripNonEcsFields({
'agent.name': 'some name',
@ -289,7 +324,6 @@ describe('stripNonEcsFields', () => {
expect(result).toEqual({
threat: {
enrichments: [],
'indicator.port': 443,
},
});

View file

@ -180,7 +180,7 @@ export const stripNonEcsFields = (doc: SourceFieldRecord): StripNonEcsFieldsRetu
) => {
const fullPath = [parentPath, documentKey].filter(Boolean).join('.');
// if document array, traverse through each item w/o changing documentKey, parent, parentPath
if (isArray(document)) {
if (isArray(document) && document.length > 0) {
document.slice().forEach((value) => {
traverseAndDeleteInObj(value, documentKey, parent, parentPath);
});
@ -194,6 +194,9 @@ export const stripNonEcsFields = (doc: SourceFieldRecord): StripNonEcsFieldsRetu
if (isArray(documentReference)) {
const indexToDelete = documentReference.findIndex((item) => item === document);
documentReference.splice(indexToDelete, 1);
if (documentReference.length === 0) {
delete parent[documentKey];
}
} else {
delete parent[documentKey];
}

View file

@ -6,6 +6,7 @@
*/
export { createEqlAlertType } from './eql/create_eql_alert_type';
export { createEsqlAlertType } from './esql/create_esql_alert_type';
export { createIndicatorMatchAlertType } from './indicator_match/create_indicator_match_alert_type';
export { createMlAlertType } from './ml/create_ml_alert_type';
export { createQueryAlertType } from './query/create_query_alert_type';

View file

@ -10,7 +10,7 @@ import type { RuleParamsAndRefs } from '@kbn/alerting-plugin/server';
import type { RuleParams } from '../../rule_schema';
import { isMachineLearningParams } from '../utils/utils';
import { isMachineLearningParams, isEsqlParams } from '../utils/utils';
import { extractExceptionsList } from './extract_exceptions_list';
import { extractDataView } from './extract_data_view';
@ -51,7 +51,7 @@ export const extractReferences = <TParams extends RuleParams>({
// if statement is needed here because dataViewId is not on the base rule params
// much like how the index property is not on the base rule params either
if (!isMachineLearningParams(params)) {
if (!isMachineLearningParams(params) && !isEsqlParams(params)) {
returnReferences = [
...returnReferences,
...extractDataView({

View file

@ -132,6 +132,9 @@ export const getFilter = async ({
case 'eql': {
throw new BadRequestError('Unsupported Rule of type "eql" supplied to getFilter');
}
case 'esql': {
throw new BadRequestError('Unsupported Rule of type "esql" supplied to getFilter');
}
default: {
return assertUnreachable(type);
}

View file

@ -20,18 +20,27 @@ export const getQueryFilter = ({
filters,
index,
exceptionFilter,
fields = [],
}: {
query: RuleQuery;
language: Language;
filters: unknown;
index: IndexPatternArray;
exceptionFilter: Filter | undefined;
fields?: DataViewFieldBase[];
}): ESBoolQuery => {
fields,
}:
| {
query: RuleQuery;
language: Language;
filters: unknown;
index: IndexPatternArray;
exceptionFilter: Filter | undefined;
fields?: DataViewFieldBase[];
}
| {
index: undefined;
query: RuleQuery;
language: 'esql';
filters: unknown;
exceptionFilter: Filter | undefined;
fields?: DataViewFieldBase[];
}): ESBoolQuery => {
const indexPattern: DataViewBase = {
fields,
title: index.join(),
fields: fields ?? [],
title: (index ?? []).join(),
};
const config: EsQueryConfig = {

View file

@ -16,7 +16,7 @@ export interface BuildReasonMessageArgs {
}
export interface BuildReasonMessageUtilArgs extends BuildReasonMessageArgs {
type?: 'eql' | 'ml' | 'query' | 'threatMatch' | 'threshold' | 'new_terms';
type?: 'eql' | 'ml' | 'query' | 'threatMatch' | 'threshold' | 'new_terms' | 'esql';
}
export type BuildReasonMessage = (args: BuildReasonMessageArgs) => string;
@ -123,6 +123,9 @@ created {alertSeverity} alert {alertName}.`,
export const buildReasonMessageForEqlAlert = (args: BuildReasonMessageArgs) =>
buildReasonMessageUtil({ ...args, type: 'eql' });
export const buildReasonMessageForEsqlAlert = (args: BuildReasonMessageArgs) =>
buildReasonMessageUtil({ ...args, type: 'esql' });
export const buildReasonMessageForMlAlert = (args: BuildReasonMessageArgs) =>
buildReasonMessageUtil({ ...args, type: 'ml' });

Some files were not shown because too many files have changed in this diff Show more