github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 17:28:26 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Response Ops][Alerting] Adding group by options to ES query rule type (#144689)

Browse source 
Resolves https://github.com/elastic/kibana/issues/89481

## Summary

Adds group by options to the ES query rule type, both DSL and KQL
options. This is the same limited group by options that are offered in
the index threshold rule type so I used the same UI components and rule
parameter names. I moved some aggregation building code to `common` so
they could be reused. All existing ES query rules are migrated to be
`count over all` rules.

## To Verify

* Create the following types of rules and verify they work as expected.
Verify for both DSL query and KQL query
* `count over all` rule - this should run the same as before, where it
counts the number of documents that matches the query and applies the
threshold condition to that value. `{{context.hits}}` is all the
documents that match the query if the threshold condition is met.
* `<metric> over all` rule - this calculates the specific aggregation
metric and applies the threshold condition to the aggregated metric (for
example, `avg event.duration`). `{{context.hits}}` is all the documents
that match the query if the threshold condition is met.
* `count over top N terms` - this will apply a term aggregation to the
query and matches the threshold condition to each term bucket (for
example, `count over top 10 event.action` will apply the threshold
condition to the count of documents within each `event.action` bucket).
`{{context.hits}}` is the result of the top hits aggregation within each
term bucket if the threshold condition is met for that bucket.
* `<metric> over top N terms` - this will apply a term aggregation and a
metric sub-aggregation to the query and matches the threshold condition
to the metric value within each term bucket (for example, `avg
event.duration over top 10 event.action` will apply the threshold
condition to the average value of `event.duration` within each
`event.action` bucket). `{{context.hits}}` is the result of the top hits
aggregation within each term bucket if the threshold condition is met
for that bucket.
* Verify the migration by creating a DSL and KQL query in an older
version of Kibana and then upgrading to this PR. The rules should still
continue running successfully.


### Checklist

- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
This commit is contained in:
Ying Mao 2022-12-15 11:03:58 -05:00 • committed by GitHub
parent e709523410
commit fdf4dea9bd
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
74 changed files with 5236 additions and 691 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
docs/user/alerting/images/rule-types-es-query-conditions.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 344 KiB

After

Width:  |  Height:  |  Size: 157 KiB

Before After
Before After

23
docs/user/alerting/rule-types/es-query.asciidoc
View file

@ -10,8 +10,7 @@ threshold condition is met.
=== Create the rule
Fill in the <<defining-rules-general-details, rule details>>, then select
*{es} query*.
*{es} query*. {es} query rule can be defined using KQL/Lucene or Query DSL.
[float]
=== Define the conditions
@ -19,26 +18,26 @@ Fill in the <<defining-rules-general-details, rule details>>, then select
Define properties to detect the condition.
[role="screenshot"]
image::user/alerting/images/rule-types-es-query-conditions.png[Six clauses define the condition to detect]
image::user/alerting/images/rule-types-es-query-conditions.png[Eight clauses define the condition to detect]
Index:: Specifies an *index or data view* and a *time field* that is used for
the *time window*.
Size:: Specifies the number of documents to pass to the configured actions when
the threshold condition is met.
{es} query:: Specifies the ES DSL query. The number of documents that
match this query is evaluated against the threshold condition. Only the `query`, `fields`, `_source` and `runtime_mappings`
fields are used, other DSL fields are not considered.
{es} query:: Specifies the ES DSL query. Only the `query`, `fields`, `_source` and `runtime_mappings` fields are used, other DSL fields are not considered.
When:: Specifies how the value to be compared to the threshold is calculated. The value is calculated by aggregating a numeric field within the *time window*. The aggregation options are: `count`, `average`, `sum`, `min`, and `max`. When using `count` the document count is used and an aggregation field is not necessary.
Over or Grouped Over:: Specifies whether the aggregation is applied over all documents or split into groups using a grouping field. If grouping is used, an <<alerting-concepts-alerts,alert>> will be created for each group when it meets the condition. To limit the number of alerts on high cardinality fields, you must specify the number of groups to check against the threshold. Only the *top* groups are checked.
Threshold:: Defines a threshold value and a comparison operator (`is above`,
`is above or equals`, `is below`, `is below or equals`, or `is between`). The
number of documents that match the specified query is compared to this
threshold.
`is above or equals`, `is below`, `is below or equals`, or `is between`). The value
calculated by the aggregation is compared to this threshold.
Time window:: Defines how far back to search for documents, using the
*time field* set in the *index* clause. Generally this value should be set to a
value higher than the *check every* value in the
<<defining-rules-general-details, general rule details>>, to avoid gaps in
detection.
Size:: Specifies the number of documents to pass to the configured actions when
the threshold condition is met.
Exclude matches from previous run:: Turn on to avoid alert duplication by
excluding documents that have already been detected by the previous rule run.
excluding documents that have already been detected by the previous rule run. This
option is not available when a grouping field is specified.
[float]
=== Add action variables

2
src/core/server/integration_tests/saved_objects/migrations/check_registered_types.test.ts
View file

@ -57,7 +57,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
Object {
"action": "7858e6d5a9f231bf23f6f2e57328eb0095b26735",
"action_task_params": "bbd38cbfd74bf6713586fe078e3fa92db2234299",
"alert": "d95e8ef645ae9f797b93a9a64d8ab9d35d484064",
"alert": "c29c5e28a6f1d075e528a9273a1a07b080625565",
"api_key_pending_invalidation": "9b4bc1235337da9a87ef05a1d1f4858b2a3b77c6",
"apm-indices": "ceb0870f3a74e2ffc3a1cd3a3c73af76baca0999",
"apm-server-schema": "2bfd2998d3873872e1366458ce553def85418f91",

39
x-pack/plugins/alerting/server/saved_objects/migrations/8.7/index.ts Normal file
View file

@ -0,0 +1,39 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { SavedObjectUnsanitizedDoc } from '@kbn/core-saved-objects-server';
import { EncryptedSavedObjectsPluginSetup } from '@kbn/encrypted-saved-objects-plugin/server';
import { createEsoMigration, isEsQueryRuleType, pipeMigrations } from '../utils';
import { RawRule } from '../../../types';
function addGroupByToEsQueryRule(
doc: SavedObjectUnsanitizedDoc<RawRule>
): SavedObjectUnsanitizedDoc<RawRule> {
// Adding another check in for isEsQueryRuleType in case we add more migrations
if (isEsQueryRuleType(doc)) {
return {
...doc,
attributes: {
...doc.attributes,
params: {
...doc.attributes.params,
aggType: 'count',
groupBy: 'all',
},
},
};
}
return doc;
}
export const getMigrations870 = (encryptedSavedObjects: EncryptedSavedObjectsPluginSetup) =>
createEsoMigration(
encryptedSavedObjects,
(doc): doc is SavedObjectUnsanitizedDoc<RawRule> => isEsQueryRuleType(doc),
pipeMigrations(addGroupByToEsQueryRule)
);

35
x-pack/plugins/alerting/server/saved_objects/migrations/index.test.ts
View file

@ -2508,6 +2508,41 @@ describe('successful migrations', () => {
});
});
describe('8.7.0', () => {
test('migrates es_query rule params and adds group by fields', () => {
const migration870 = getMigrations(encryptedSavedObjectsSetup, {}, isPreconfigured)['8.7.0'];
const rule = getMockData(
{
params: { esQuery: '{ "query": "test-query" }', searchType: 'esQuery' },
alertTypeId: '.es-query',
},
true
);
const migratedAlert870 = migration870(rule, migrationContext);
expect(migratedAlert870.attributes.params).toEqual({
esQuery: '{ "query": "test-query" }',
searchType: 'esQuery',
aggType: 'count',
groupBy: 'all',
});
});
test('does not migrate rule params if rule is not es query', () => {
const migration870 = getMigrations(encryptedSavedObjectsSetup, {}, isPreconfigured)['8.7.0'];
const rule = getMockData(
{
params: { foo: true },
alertTypeId: '.not-es-query',
},
true
);
const migratedAlert870 = migration870(rule, migrationContext);
expect(migratedAlert870.attributes.params).toEqual({ foo: true });
});
});
describe('Metrics Inventory Threshold rule', () => {
test('Migrates incorrect action group spelling', () => {
const migration800 = getMigrations(encryptedSavedObjectsSetup, {}, isPreconfigured)['8.0.0'];

2
x-pack/plugins/alerting/server/saved_objects/migrations/index.ts
View file

@ -29,6 +29,7 @@ import { getMigrations830 } from './8.3';
import { getMigrations841 } from './8.4';
import { getMigrations850 } from './8.5';
import { getMigrations860 } from './8.6';
import { getMigrations870 } from './8.7';
import { AlertLogMeta, AlertMigration } from './types';
import { MINIMUM_SS_MIGRATION_VERSION } from './constants';
import { createEsoMigration, isEsQueryRuleType, pipeMigrations } from './utils';
@ -77,6 +78,7 @@ export function getMigrations(
'8.4.1': executeMigrationWithErrorHandling(getMigrations841(encryptedSavedObjects), '8.4.1'),
'8.5.0': executeMigrationWithErrorHandling(getMigrations850(encryptedSavedObjects), '8.5.0'),
'8.6.0': executeMigrationWithErrorHandling(getMigrations860(encryptedSavedObjects), '8.6.0'),
'8.7.0': executeMigrationWithErrorHandling(getMigrations870(encryptedSavedObjects), '8.7.0'),
},
getSearchSourceMigrations(encryptedSavedObjects, searchSourceMigrations)
);

21
x-pack/plugins/stack_alerts/common/build_sorted_events_query.test.ts
View file

@ -57,9 +57,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},
@ -108,9 +105,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},
@ -160,9 +154,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},
@ -213,9 +204,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},
@ -271,9 +259,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},
@ -329,9 +314,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},
@ -380,9 +362,6 @@ describe('buildSortedEventsQuery', () => {
],
},
},
{
match_all: {},
},
],
},
},

7
x-pack/plugins/stack_alerts/common/build_sorted_events_query.ts
View file

@ -72,12 +72,7 @@ export const buildSortedEventsQuery = ({
docvalue_fields: docFields,
query: {
bool: {
filter: [
...filterWithTime,
{
match_all: {},
},
],
filter: [...filterWithTime],
},
},
...(aggs ? { aggs } : {}),

2
x-pack/plugins/stack_alerts/server/rule_types/lib/comparator.test.ts → x-pack/plugins/stack_alerts/common/comparator.test.ts
View file

@ -6,7 +6,7 @@
*/
import { getComparatorScript } from './comparator';
import { Comparator } from '../../../common/comparator_types';
import { Comparator } from './comparator_types';
describe('getComparatorScript', () => {
it('correctly returns script when comparator is LT', () => {

81
x-pack/plugins/stack_alerts/common/comparator.ts Normal file
View file

@ -0,0 +1,81 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { Comparator } from './comparator_types';
export type ComparatorFn = (value: number, threshold: number[]) => boolean;
const humanReadableComparators = new Map<Comparator, string>([
[Comparator.LT, 'less than'],
[Comparator.LT_OR_EQ, 'less than or equal to'],
[Comparator.GT_OR_EQ, 'greater than or equal to'],
[Comparator.GT, 'greater than'],
[Comparator.BETWEEN, 'between'],
[Comparator.NOT_BETWEEN, 'not between'],
]);
export const ComparatorFns = new Map<Comparator, ComparatorFn>([
[Comparator.LT, (value: number, threshold: number[]) => value < threshold[0]],
[Comparator.LT_OR_EQ, (value: number, threshold: number[]) => value <= threshold[0]],
[Comparator.GT_OR_EQ, (value: number, threshold: number[]) => value >= threshold[0]],
[Comparator.GT, (value: number, threshold: number[]) => value > threshold[0]],
[
Comparator.BETWEEN,
(value: number, threshold: number[]) => value >= threshold[0] && value <= threshold[1],
],
[
Comparator.NOT_BETWEEN,
(value: number, threshold: number[]) => value < threshold[0] || value > threshold[1],
],
]);
export const getComparatorScript = (
comparator: Comparator,
threshold: number[],
fieldName: string
) => {
if (threshold.length === 0) {
throw new Error('Threshold value required');
}
function getThresholdString(thresh: number) {
return Number.isInteger(thresh) ? `${thresh}L` : `${thresh}`;
}
switch (comparator) {
case Comparator.LT:
return `${fieldName} < ${getThresholdString(threshold[0])}`;
case Comparator.LT_OR_EQ:
return `${fieldName} <= ${getThresholdString(threshold[0])}`;
case Comparator.GT:
return `${fieldName} > ${getThresholdString(threshold[0])}`;
case Comparator.GT_OR_EQ:
return `${fieldName} >= ${getThresholdString(threshold[0])}`;
case Comparator.BETWEEN:
if (threshold.length < 2) {
throw new Error('Threshold values required');
}
return `${fieldName} >= ${getThresholdString(
threshold[0]
)} && ${fieldName} <= ${getThresholdString(threshold[1])}`;
case Comparator.NOT_BETWEEN:
if (threshold.length < 2) {
throw new Error('Threshold values required');
}
return `${fieldName} < ${getThresholdString(
threshold[0]
)} || ${fieldName} > ${getThresholdString(threshold[1])}`;
}
};
export const ComparatorFnNames = new Set(ComparatorFns.keys());
export function getHumanReadableComparator(comparator: Comparator) {
return humanReadableComparators.has(comparator)
? humanReadableComparators.get(comparator)
: comparator;
}

6
x-pack/plugins/stack_alerts/common/index.ts
View file

@ -5,4 +5,10 @@
* 2.0.
*/
export {
ComparatorFns,
getComparatorScript,
ComparatorFnNames,
getHumanReadableComparator,
} from './comparator';
export { STACK_ALERTS_FEATURE_ID } from './constants';

8
x-pack/plugins/stack_alerts/public/rule_types/es_query/constants.ts
View file

@ -18,6 +18,9 @@ export const DEFAULT_VALUES = {
TIME_WINDOW_SIZE: 5,
TIME_WINDOW_UNIT: 'm',
THRESHOLD: [1000],
AGGREGATION_TYPE: 'count',
TERM_SIZE: 5,
GROUP_BY: 'all',
EXCLUDE_PREVIOUS_HITS: true,
};
@ -27,6 +30,11 @@ export const COMMON_EXPRESSION_ERRORS = {
threshold1: new Array<string>(),
timeWindowSize: new Array<string>(),
size: new Array<string>(),
aggField: new Array<string>(),
aggType: new Array<string>(),
groupBy: new Array<string>(),
termSize: new Array<string>(),
termField: new Array<string>(),
};
export const SEARCH_SOURCE_ONLY_EXPRESSION_ERRORS = {

62
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/es_query_expression.test.tsx
View file

@ -114,6 +114,8 @@ const defaultEsQueryExpressionParams: EsQueryRuleParams<SearchType.esQuery> = {
timeWindowUnit: 's',
index: ['test-index'],
timeField: '@timestamp',
aggType: 'count',
groupBy: 'all',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
excludeHitsFromPreviousRun: true,
};
@ -204,7 +206,7 @@ describe('EsQueryRuleTypeExpression', () => {
expect(testQueryButton.prop('disabled')).toBe(true);
});
test('should show success message if Test Query is successful', async () => {
test('should show success message if ungrouped Test Query is successful', async () => {
const searchResponseMock$ = of<IKibanaSearchResponse>({
rawResponse: {
hits: {
@ -215,14 +217,12 @@ describe('EsQueryRuleTypeExpression', () => {
dataMock.search.search.mockImplementation(() => searchResponseMock$);
const wrapper = await setup(defaultEsQueryExpressionParams);
const testQueryButton = wrapper.find('button[data-test-subj="testQuery"]');
testQueryButton.simulate('click');
expect(dataMock.search.search).toHaveBeenCalled();
await act(async () => {
await nextTick();
wrapper.update();
});
expect(wrapper.find('[data-test-subj="testQuerySuccess"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="testQueryError"]').exists()).toBeFalsy();
expect(wrapper.find('EuiText[data-test-subj="testQuerySuccess"]').text()).toEqual(
@ -230,6 +230,62 @@ describe('EsQueryRuleTypeExpression', () => {
);
});
test('should show success message if grouped Test Query is successful', async () => {
const searchResponseMock$ = of<IKibanaSearchResponse>({
rawResponse: {
hits: {
total: 1234,
},
aggregations: {
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 103,
buckets: [
{
key: 'execute',
doc_count: 120,
},
{
key: 'execute-start',
doc_count: 120,
},
{
key: 'active-instance',
doc_count: 100,
},
{
key: 'execute-action',
doc_count: 100,
},
{
key: 'new-instance',
doc_count: 100,
},
],
},
},
},
});
dataMock.search.search.mockImplementation(() => searchResponseMock$);
const wrapper = await setup({
...defaultEsQueryExpressionParams,
termField: 'the-term',
termSize: 10,
});
const testQueryButton = wrapper.find('button[data-test-subj="testQuery"]');
testQueryButton.simulate('click');
expect(dataMock.search.search).toHaveBeenCalled();
await act(async () => {
await nextTick();
wrapper.update();
});
expect(wrapper.find('[data-test-subj="testQuerySuccess"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="testQueryError"]').exists()).toBeFalsy();
expect(wrapper.find('EuiText[data-test-subj="testQuerySuccess"]').text()).toEqual(
`Grouped query matched 5 groups in the last 15s.`
);
});
test('should show success message if Test Query is successful (with partial result)', async () => {
const partial = {
isRunning: true,

161
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/es_query_expression.tsx
View file

@ -16,13 +16,22 @@ import { XJson } from '@kbn/es-ui-shared-plugin/public';
import { CodeEditor } from '@kbn/kibana-react-plugin/public';
import { getFields, RuleTypeParamsExpressionProps } from '@kbn/triggers-actions-ui-plugin/public';
import { parseDuration } from '@kbn/alerting-plugin/common';
import {
FieldOption,
buildAggregation,
parseAggregationResults,
isGroupAggregation,
isCountAggregation,
BUCKET_SELECTOR_FIELD,
} from '@kbn/triggers-actions-ui-plugin/public/common';
import { Comparator } from '../../../../common/comparator_types';
import { getComparatorScript } from '../../../../common';
import { hasExpressionValidationErrors } from '../validation';
import { buildSortedEventsQuery } from '../../../../common/build_sorted_events_query';
import { EsQueryRuleParams, EsQueryRuleMetaData, SearchType } from '../types';
import { IndexSelectPopover } from '../../components/index_select_popover';
import { DEFAULT_VALUES } from '../constants';
import { RuleCommonExpressions } from '../rule_common_expressions';
import { totalHitsToNumber } from '../test_query_row';
import { useTriggerUiActionServices } from '../util';
const { useXJsonMode } = XJson;
@ -39,6 +48,11 @@ export const EsQueryExpression: React.FC<
threshold,
timeWindowSize,
timeWindowUnit,
aggType,
aggField,
groupBy,
termSize,
termField,
excludeHitsFromPreviousRun,
} = ruleParams;
@ -51,6 +65,9 @@ export const EsQueryExpression: React.FC<
thresholdComparator: thresholdComparator ?? DEFAULT_VALUES.THRESHOLD_COMPARATOR,
size: size ?? DEFAULT_VALUES.SIZE,
esQuery: esQuery ?? DEFAULT_VALUES.QUERY,
aggType: aggType ?? DEFAULT_VALUES.AGGREGATION_TYPE,
groupBy: groupBy ?? DEFAULT_VALUES.GROUP_BY,
termSize: termSize ?? DEFAULT_VALUES.TERM_SIZE,
searchType: SearchType.esQuery,
excludeHitsFromPreviousRun:
excludeHitsFromPreviousRun ?? DEFAULT_VALUES.EXCLUDE_PREVIOUS_HITS,
@ -71,15 +88,7 @@ export const EsQueryExpression: React.FC<
const services = useTriggerUiActionServices();
const { http, docLinks } = services;
const [esFields, setEsFields] = useState<
Array<{
name: string;
type: string;
normalizedType: string;
searchable: boolean;
aggregatable: boolean;
}>
>([]);
const [esFields, setEsFields] = useState<FieldOption[]>([]);
const { convertToJson, setXJson, xJson } = useXJsonMode(DEFAULT_VALUES.QUERY);
const setDefaultExpressionValues = async () => {
@ -87,7 +96,7 @@ export const EsQueryExpression: React.FC<
setXJson(esQuery ?? DEFAULT_VALUES.QUERY);
if (index && index.length > 0) {
await refreshEsFields();
await refreshEsFields(index);
}
};
@ -96,17 +105,21 @@ export const EsQueryExpression: React.FC<
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
const refreshEsFields = async () => {
if (index) {
const currentEsFields = await getFields(http, index);
setEsFields(currentEsFields);
}
const refreshEsFields = async (indices: string[]) => {
const currentEsFields = await getFields(http, indices);
setEsFields(currentEsFields);
};
const onTestQuery = useCallback(async () => {
const isGroupAgg = isGroupAggregation(termField);
const isCountAgg = isCountAggregation(aggType);
const window = `${timeWindowSize}${timeWindowUnit}`;
if (hasExpressionValidationErrors(currentRuleParams)) {
return { nrOfDocs: 0, timeWindow: window };
return {
testResults: { results: [], truncated: false },
isGrouped: isGroupAgg,
timeWindow: window,
};
}
const timeWindow = parseDuration(window);
const parsedQuery = JSON.parse(esQuery);
@ -122,13 +135,43 @@ export const EsQueryExpression: React.FC<
searchAfterSortId: undefined,
timeField: timeField ? timeField : '',
track_total_hits: true,
aggs: buildAggregation({
aggType,
aggField,
termField,
termSize,
condition: {
conditionScript: getComparatorScript(
(thresholdComparator ?? DEFAULT_VALUES.THRESHOLD_COMPARATOR) as Comparator,
threshold,
BUCKET_SELECTOR_FIELD
),
},
}),
}),
})
);
const hits = rawResponse.hits;
return { nrOfDocs: totalHitsToNumber(hits.total), timeWindow: window };
}, [timeWindowSize, timeWindowUnit, currentRuleParams, esQuery, data.search, index, timeField]);
return {
testResults: parseAggregationResults({ isCountAgg, isGroupAgg, esResult: rawResponse }),
isGrouped: isGroupAgg,
timeWindow: window,
};
}, [
timeWindowSize,
timeWindowUnit,
currentRuleParams,
esQuery,
data.search,
index,
timeField,
aggType,
aggField,
termField,
termSize,
threshold,
thresholdComparator,
]);
return (
<Fragment>
@ -155,7 +198,7 @@ export const EsQueryExpression: React.FC<
// reset expression fields if indices are deleted
if (indices.length === 0) {
setRuleProperty('params', {
...ruleParams,
timeField: ruleParams.timeField,
index: indices,
esQuery: DEFAULT_VALUES.QUERY,
size: DEFAULT_VALUES.SIZE,
@ -163,10 +206,14 @@ export const EsQueryExpression: React.FC<
timeWindowSize: DEFAULT_VALUES.TIME_WINDOW_SIZE,
timeWindowUnit: DEFAULT_VALUES.TIME_WINDOW_UNIT,
threshold: DEFAULT_VALUES.THRESHOLD,
timeField: '',
aggType: DEFAULT_VALUES.AGGREGATION_TYPE,
groupBy: DEFAULT_VALUES.GROUP_BY,
termSize: DEFAULT_VALUES.TERM_SIZE,
searchType: SearchType.esQuery,
excludeHitsFromPreviousRun: DEFAULT_VALUES.EXCLUDE_PREVIOUS_HITS,
});
} else {
await refreshEsFields();
await refreshEsFields(indices);
}
}}
onTimeFieldChange={(updatedTimeField: string) => setParam('timeField', updatedTimeField)}
@ -229,26 +276,62 @@ export const EsQueryExpression: React.FC<
timeWindowSize={timeWindowSize}
timeWindowUnit={timeWindowUnit}
size={size}
onChangeThreshold={(selectedThresholds) => setParam('threshold', selectedThresholds)}
onChangeThresholdComparator={(selectedThresholdComparator) =>
setParam('thresholdComparator', selectedThresholdComparator)
}
onChangeWindowSize={(selectedWindowSize: number | undefined) =>
setParam('timeWindowSize', selectedWindowSize)
}
onChangeWindowUnit={(selectedWindowUnit: string) =>
setParam('timeWindowUnit', selectedWindowUnit)
}
onChangeSizeValue={(updatedValue) => {
setParam('size', updatedValue);
}}
esFields={esFields}
aggType={aggType}
aggField={aggField}
groupBy={groupBy}
termSize={termSize}
termField={termField}
onChangeSelectedAggField={useCallback(
(selectedAggField?: string) => setParam('aggField', selectedAggField),
[setParam]
)}
onChangeSelectedAggType={useCallback(
(selectedAggType: string) => setParam('aggType', selectedAggType),
[setParam]
)}
onChangeSelectedGroupBy={useCallback(
(selectedGroupBy) => setParam('groupBy', selectedGroupBy),
[setParam]
)}
onChangeSelectedTermField={useCallback(
(selectedTermField) => setParam('termField', selectedTermField),
[setParam]
)}
onChangeSelectedTermSize={useCallback(
(selectedTermSize?: number) => setParam('termSize', selectedTermSize),
[setParam]
)}
onChangeThreshold={useCallback(
(selectedThresholds) => setParam('threshold', selectedThresholds),
[setParam]
)}
onChangeThresholdComparator={useCallback(
(selectedThresholdComparator) =>
setParam('thresholdComparator', selectedThresholdComparator),
[setParam]
)}
onChangeWindowSize={useCallback(
(selectedWindowSize: number | undefined) =>
setParam('timeWindowSize', selectedWindowSize),
[setParam]
)}
onChangeWindowUnit={useCallback(
(selectedWindowUnit: string) => setParam('timeWindowUnit', selectedWindowUnit),
[setParam]
)}
onChangeSizeValue={useCallback(
(updatedValue) => setParam('size', updatedValue),
[setParam]
)}
errors={errors}
hasValidationErrors={hasExpressionValidationErrors(currentRuleParams)}
onTestFetch={onTestQuery}
excludeHitsFromPreviousRun={excludeHitsFromPreviousRun}
onChangeExcludeHitsFromPreviousRun={(exclude) => {
setParam('excludeHitsFromPreviousRun', exclude);
}}
onChangeExcludeHitsFromPreviousRun={useCallback(
(exclude) => setParam('excludeHitsFromPreviousRun', exclude),
[setParam]
)}
/>
<EuiSpacer />

13
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/expression.test.tsx
View file

@ -13,7 +13,12 @@ import { httpServiceMock } from '@kbn/core/public/mocks';
import { dataPluginMock } from '@kbn/data-plugin/public/mocks';
import { dataViewPluginMocks } from '@kbn/data-views-plugin/public/mocks';
import { unifiedSearchPluginMock } from '@kbn/unified-search-plugin/public/mocks';
import { CommonRuleParams, EsQueryRuleMetaData, EsQueryRuleParams, SearchType } from '../types';
import {
CommonEsQueryRuleParams,
EsQueryRuleMetaData,
EsQueryRuleParams,
SearchType,
} from '../types';
import { EsQueryRuleTypeExpression } from './expression';
import { chartPluginMock } from '@kbn/charts-plugin/public/mocks';
import { Subject } from 'rxjs';
@ -54,6 +59,7 @@ const defaultEsQueryRuleParams: EsQueryRuleParams<SearchType.esQuery> = {
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
searchType: SearchType.esQuery,
excludeHitsFromPreviousRun: true,
aggType: 'count',
};
const defaultSearchSourceRuleParams: EsQueryRuleParams<SearchType.searchSource> = {
size: 100,
@ -66,6 +72,7 @@ const defaultSearchSourceRuleParams: EsQueryRuleParams<SearchType.searchSource>
searchType: SearchType.searchSource,
searchConfiguration: {},
excludeHitsFromPreviousRun: true,
aggType: 'count',
};
const dataViewPluginMock = dataViewPluginMocks.createStartContract();
@ -146,7 +153,7 @@ const Wrapper: React.FC<{
ruleParams: EsQueryRuleParams<SearchType.searchSource> | EsQueryRuleParams<SearchType.esQuery>;
metadata?: EsQueryRuleMetaData;
}> = ({ ruleParams, metadata }) => {
const [currentRuleParams, setCurrentRuleParams] = useState<CommonRuleParams>(ruleParams);
const [currentRuleParams, setCurrentRuleParams] = useState<CommonEsQueryRuleParams>(ruleParams);
const errors = {
index: [],
esQuery: [],
@ -168,7 +175,7 @@ const Wrapper: React.FC<{
}}
setRuleProperty={(name, params) => {
if (name === 'params') {
setCurrentRuleParams(params as CommonRuleParams);
setCurrentRuleParams(params as CommonEsQueryRuleParams);
}
}}
errors={errors}

2
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/query_form_type_chooser.tsx
View file

@ -106,7 +106,7 @@ export const QueryFormTypeChooser: React.FC<QueryFormTypeProps> = ({
/>
</h5>
</EuiTitle>
<EuiListGroup flush gutterSize="m" size="l" maxWidth={false}>
<EuiListGroup flush gutterSize="m" size="m" maxWidth={false}>
{FORM_TYPE_ITEMS.map((item) => (
<EuiListGroupItem
wrapText

342
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/search_source_expression.test.tsx
View file

@ -22,6 +22,7 @@ import { copyToClipboard, EuiLoadingSpinner } from '@elastic/eui';
import { KibanaContextProvider } from '@kbn/kibana-react-plugin/public';
import { indexPatternEditorPluginMock as dataViewEditorPluginMock } from '@kbn/data-view-editor-plugin/public/mocks';
import { ReactWrapper } from 'enzyme';
import { DataPlugin } from '@kbn/data-plugin/public';
jest.mock('@elastic/eui', () => {
const original = jest.requireActual('@elastic/eui');
@ -56,9 +57,10 @@ const defaultSearchSourceExpressionParams: EsQueryRuleParams<SearchType.searchSo
index: '90943e30-9a47-11e8-b64d-95841ca0b247',
},
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const mockSearchResult = new Subject();
const testResultComplete = {
rawResponse: {
hits: {
@ -67,6 +69,42 @@ const testResultComplete = {
},
};
const testResultGroupedComplete = {
rawResponse: {
hits: {
total: 1234,
},
aggregations: {
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 103,
buckets: [
{
key: 'execute',
doc_count: 120,
},
{
key: 'execute-start',
doc_count: 120,
},
{
key: 'active-instance',
doc_count: 100,
},
{
key: 'execute-action',
doc_count: 100,
},
{
key: 'new-instance',
doc_count: 100,
},
],
},
},
},
};
const testResultPartial = {
partial: true,
running: true,
@ -88,87 +126,6 @@ const searchSourceFieldsMock = {
},
};
const searchSourceMock = {
id: 'data_source6',
fields: searchSourceFieldsMock,
getField: (name: string) => {
return (searchSourceFieldsMock as Record<string, object>)[name] || '';
},
setField: jest.fn(),
createCopy: jest.fn(() => {
return searchSourceMock;
}),
setParent: jest.fn(() => {
return searchSourceMock;
}),
fetch$: jest.fn(() => {
return mockSearchResult;
}),
getSearchRequestBody: jest.fn(() => ({
fields: [
{
field: '@timestamp',
format: 'date_time',
},
{
field: 'timestamp',
format: 'date_time',
},
{
field: 'utc_time',
format: 'date_time',
},
],
script_fields: {},
stored_fields: ['*'],
runtime_mappings: {
hour_of_day: {
type: 'long',
script: {
source: "emit(doc['timestamp'].value.getHour());",
},
},
},
_source: {
excludes: [],
},
query: {
bool: {
must: [],
filter: [
{
bool: {
must_not: {
bool: {
should: [
{
match: {
response: '200',
},
},
],
minimum_should_match: 1,
},
},
},
},
{
range: {
timestamp: {
format: 'strict_date_optional_time',
gte: '2022-06-19T02:49:51.192Z',
lte: '2022-06-24T02:49:51.192Z',
},
},
},
],
should: [],
must_not: [],
},
},
})),
} as unknown as ISearchSource;
const savedQueryMock = {
id: 'test-id',
attributes: {
@ -182,61 +139,147 @@ const savedQueryMock = {
},
};
const dataMock = dataPluginMock.createStartContract();
(dataMock.search.searchSource.create as jest.Mock).mockImplementation(() =>
Promise.resolve(searchSourceMock)
);
(dataViewPluginMock.getIds as jest.Mock) = jest.fn().mockImplementation(() => Promise.resolve([]));
dataViewPluginMock.getDefaultDataView = jest.fn(() => Promise.resolve(null));
dataViewPluginMock.get = jest.fn();
(dataMock.query.savedQueries.getSavedQuery as jest.Mock).mockImplementation(() =>
Promise.resolve(savedQueryMock)
);
dataMock.query.savedQueries.findSavedQueries = jest.fn(() =>
Promise.resolve({ total: 0, queries: [] })
);
const setup = (alertParams: EsQueryRuleParams<SearchType.searchSource>) => {
const errors = {
size: [],
timeField: [],
timeWindowSize: [],
searchConfiguration: [],
};
const dataViewEditorMock = dataViewEditorPluginMock.createStartContract();
return mountWithIntl(
<KibanaContextProvider
services={{
dataViews: dataViewPluginMock,
data: dataMock,
uiSettings: uiSettingsMock,
dataViewEditor: dataViewEditorMock,
unifiedSearch: unifiedSearchMock,
}}
>
<SearchSourceExpression
ruleInterval="1m"
ruleThrottle="1m"
alertNotifyWhen="onThrottleInterval"
ruleParams={alertParams}
setRuleParams={() => {}}
setRuleProperty={() => {}}
errors={errors}
unifiedSearch={unifiedSearchMock}
data={dataMock}
dataViews={dataViewPluginMock}
defaultActionGroupId=""
actionGroups={[]}
charts={chartsStartMock}
metadata={{ adHocDataViewList: [] }}
onChangeMetaData={jest.fn()}
/>
</KibanaContextProvider>
);
};
describe('SearchSourceAlertTypeExpression', () => {
let dataMock: jest.Mocked<ReturnType<DataPlugin['start']>>;
let searchSourceMock: ISearchSource;
let mockSearchResult: Subject<unknown>;
beforeEach(() => {
mockSearchResult = new Subject();
searchSourceMock = {
id: 'data_source6',
fields: searchSourceFieldsMock,
getField: (name: string) => {
return (searchSourceFieldsMock as Record<string, object>)[name] || '';
},
setField: jest.fn(),
createCopy: jest.fn(() => {
return searchSourceMock;
}),
setParent: jest.fn(() => {
return searchSourceMock;
}),
fetch$: jest.fn(() => {
return mockSearchResult;
}),
getSearchRequestBody: jest.fn(() => ({
fields: [
{
field: '@timestamp',
format: 'date_time',
},
{
field: 'timestamp',
format: 'date_time',
},
{
field: 'utc_time',
format: 'date_time',
},
],
script_fields: {},
stored_fields: ['*'],
runtime_mappings: {
hour_of_day: {
type: 'long',
script: {
source: "emit(doc['timestamp'].value.getHour());",
},
},
},
_source: {
excludes: [],
},
query: {
bool: {
must: [],
filter: [
{
bool: {
must_not: {
bool: {
should: [
{
match: {
response: '200',
},
},
],
minimum_should_match: 1,
},
},
},
},
{
range: {
timestamp: {
format: 'strict_date_optional_time',
gte: '2022-06-19T02:49:51.192Z',
lte: '2022-06-24T02:49:51.192Z',
},
},
},
],
should: [],
must_not: [],
},
},
})),
} as unknown as ISearchSource;
dataMock = dataPluginMock.createStartContract();
(dataMock.search.searchSource.create as jest.Mock).mockImplementation(() =>
Promise.resolve(searchSourceMock)
);
(dataMock.query.savedQueries.getSavedQuery as jest.Mock).mockImplementation(() =>
Promise.resolve(savedQueryMock)
);
dataMock.query.savedQueries.findSavedQueries = jest.fn(() =>
Promise.resolve({ total: 0, queries: [] })
);
});
const setup = (ruleParams: EsQueryRuleParams<SearchType.searchSource>) => {
const errors = {
size: [],
timeField: [],
timeWindowSize: [],
searchConfiguration: [],
};
const dataViewEditorMock = dataViewEditorPluginMock.createStartContract();
return mountWithIntl(
<KibanaContextProvider
services={{
dataViews: dataViewPluginMock,
data: dataMock,
uiSettings: uiSettingsMock,
dataViewEditor: dataViewEditorMock,
unifiedSearch: unifiedSearchMock,
}}
>
<SearchSourceExpression
ruleInterval="1m"
ruleThrottle="1m"
alertNotifyWhen="onThrottleInterval"
ruleParams={ruleParams}
setRuleParams={() => {}}
setRuleProperty={() => {}}
errors={errors}
unifiedSearch={unifiedSearchMock}
data={dataMock}
dataViews={dataViewPluginMock}
defaultActionGroupId=""
actionGroups={[]}
charts={chartsStartMock}
metadata={{ adHocDataViewList: [] }}
onChangeMetaData={jest.fn()}
/>
</KibanaContextProvider>
);
};
test('should render correctly', async () => {
let wrapper = setup(defaultSearchSourceExpressionParams);
@ -264,7 +307,7 @@ describe('SearchSourceAlertTypeExpression', () => {
expect(testButton.prop('disabled')).toBeTruthy();
});
test('should show success message if Test Query is successful', async () => {
test('should show success message if ungrouped Test Query is successful', async () => {
let wrapper = setup(defaultSearchSourceExpressionParams);
await act(async () => {
await nextTick();
@ -293,6 +336,39 @@ describe('SearchSourceAlertTypeExpression', () => {
);
});
test('should show success message if grouped Test Query is successful', async () => {
let wrapper = setup({
...defaultSearchSourceExpressionParams,
termField: 'the-term',
termSize: 10,
});
await act(async () => {
await nextTick();
});
wrapper = await wrapper.update();
await act(async () => {
const testButton = findTestSubject(wrapper, 'testQuery');
expect(testButton.prop('disabled')).toBeFalsy();
testButton.simulate('click');
wrapper.update();
});
wrapper = await wrapper.update();
await act(async () => {
mockSearchResult.next(testResultPartial);
mockSearchResult.next(testResultGroupedComplete);
mockSearchResult.complete();
await nextTick();
wrapper.update();
});
expect(wrapper.find('[data-test-subj="testQuerySuccess"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="testQueryError"]').exists()).toBeFalsy();
expect(wrapper.find('EuiText[data-test-subj="testQuerySuccess"]').text()).toEqual(
`Grouped query matched 5 groups in the last 15s.`
);
});
it('should call copyToClipboard with the serialized query when the copy query button is clicked', async () => {
let wrapper = null as unknown as ReactWrapper;
await act(async () => {

10
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/search_source_expression.tsx
View file

@ -37,6 +37,11 @@ export const SearchSourceExpression = ({
size,
savedQueryId,
searchConfiguration,
aggType,
aggField,
groupBy,
termField,
termSize,
excludeHitsFromPreviousRun,
} = ruleParams;
const { data } = useTriggerUiActionServices();
@ -80,6 +85,11 @@ export const SearchSourceExpression = ({
threshold: threshold ?? DEFAULT_VALUES.THRESHOLD,
thresholdComparator: thresholdComparator ?? DEFAULT_VALUES.THRESHOLD_COMPARATOR,
size: size ?? DEFAULT_VALUES.SIZE,
aggType: aggType ?? DEFAULT_VALUES.AGGREGATION_TYPE,
aggField,
groupBy: groupBy ?? DEFAULT_VALUES.GROUP_BY,
termField,
termSize: termSize ?? DEFAULT_VALUES.TERM_SIZE,
excludeHitsFromPreviousRun:
excludeHitsFromPreviousRun ?? DEFAULT_VALUES.EXCLUDE_PREVIOUS_HITS,
});

129
x-pack/plugins/stack_alerts/public/rule_types/es_query/expression/search_source_expression_form.tsx
View file

@ -20,13 +20,22 @@ import {
type SavedQuery,
type ISearchSource,
} from '@kbn/data-plugin/public';
import {
BUCKET_SELECTOR_FIELD,
buildAggregation,
FieldOption,
isCountAggregation,
isGroupAggregation,
parseAggregationResults,
} from '@kbn/triggers-actions-ui-plugin/public/common';
import { getComparatorScript } from '../../../../common';
import { Comparator } from '../../../../common/comparator_types';
import { STACK_ALERTS_FEATURE_ID } from '../../../../common';
import { CommonRuleParams, EsQueryRuleMetaData, EsQueryRuleParams, SearchType } from '../types';
import { DEFAULT_VALUES } from '../constants';
import { DataViewSelectPopover } from '../../components/data_view_select_popover';
import { RuleCommonExpressions } from '../rule_common_expressions';
import { totalHitsToNumber } from '../test_query_row';
import { useTriggerUiActionServices } from '../util';
import { useTriggerUiActionServices, convertFieldSpecToFieldOption } from '../util';
import { hasExpressionValidationErrors } from '../validation';
const HIDDEN_FILTER_PANEL_OPTIONS: SearchBarProps['hiddenFilterPanelOptions'] = [
@ -34,30 +43,15 @@ const HIDDEN_FILTER_PANEL_OPTIONS: SearchBarProps['hiddenFilterPanelOptions'] =
'disableFilter',
];
interface LocalState {
interface LocalState extends CommonRuleParams {
index?: DataView;
filter: Filter[];
query: Query;
thresholdComparator: CommonRuleParams['thresholdComparator'];
threshold: CommonRuleParams['threshold'];
timeWindowSize: CommonRuleParams['timeWindowSize'];
timeWindowUnit: CommonRuleParams['timeWindowUnit'];
size: CommonRuleParams['size'];
excludeHitsFromPreviousRun: CommonRuleParams['excludeHitsFromPreviousRun'];
}
interface LocalStateAction {
type:
| SearchSourceParamsAction['type']
| (
| 'threshold'
| 'thresholdComparator'
| 'timeWindowSize'
| 'timeWindowUnit'
| 'size'
| 'excludeHitsFromPreviousRun'
);
payload: SearchSourceParamsAction['payload'] | (number[] | number | string | boolean);
type: SearchSourceParamsAction['type'] | keyof CommonRuleParams;
payload: SearchSourceParamsAction['payload'] | (number[] | number | string | boolean | undefined);
}
type LocalStateReducer = (prevState: LocalState, action: LocalStateAction) => LocalState;
@ -111,6 +105,11 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
thresholdComparator: ruleParams.thresholdComparator ?? DEFAULT_VALUES.THRESHOLD_COMPARATOR,
timeWindowSize: ruleParams.timeWindowSize ?? DEFAULT_VALUES.TIME_WINDOW_SIZE,
timeWindowUnit: ruleParams.timeWindowUnit ?? DEFAULT_VALUES.TIME_WINDOW_UNIT,
aggType: ruleParams.aggType ?? DEFAULT_VALUES.AGGREGATION_TYPE,
aggField: ruleParams.aggField,
groupBy: ruleParams.groupBy ?? DEFAULT_VALUES.GROUP_BY,
termSize: ruleParams.termSize ?? DEFAULT_VALUES.TERM_SIZE,
termField: ruleParams.termField,
size: ruleParams.size ?? DEFAULT_VALUES.SIZE,
excludeHitsFromPreviousRun:
ruleParams.excludeHitsFromPreviousRun ?? DEFAULT_VALUES.EXCLUDE_PREVIOUS_HITS,
@ -120,11 +119,15 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
const { index: dataView, query, filter: filters } = ruleConfiguration;
const dataViews = useMemo(() => (dataView ? [dataView] : []), [dataView]);
const onSelectDataView = useCallback(
(newDataView: DataView) => dispatch({ type: 'index', payload: newDataView }),
[]
const [esFields, setEsFields] = useState<FieldOption[]>(
dataView ? convertFieldSpecToFieldOption(dataView.fields.map((field) => field.toSpec())) : []
);
const onSelectDataView = useCallback((newDataView: DataView) => {
setEsFields(convertFieldSpecToFieldOption(newDataView.fields.map((field) => field.toSpec())));
dispatch({ type: 'index', payload: newDataView });
}, []);
const onUpdateFilters = useCallback((newFilters) => {
dispatch({ type: 'filter', payload: mapAndFlattenFilters(newFilters) });
}, []);
@ -180,6 +183,33 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
[]
);
const onChangeSelectedAggField = useCallback(
(selectedAggField?: string) => dispatch({ type: 'aggField', payload: selectedAggField }),
[]
);
const onChangeSelectedAggType = useCallback(
(selectedAggType: string) => dispatch({ type: 'aggType', payload: selectedAggType }),
[]
);
const onChangeSelectedGroupBy = useCallback(
(selectedGroupBy?: string) =>
selectedGroupBy && dispatch({ type: 'groupBy', payload: selectedGroupBy }),
[]
);
const onChangeSelectedTermField = useCallback(
(selectedTermField?: string) => dispatch({ type: 'termField', payload: selectedTermField }),
[]
);
const onChangeSelectedTermSize = useCallback(
(selectedTermSize?: number) =>
selectedTermSize && dispatch({ type: 'termSize', payload: selectedTermSize }),
[]
);
const onChangeSelectedThreshold = useCallback(
(selectedThresholds?: number[]) =>
selectedThresholds && dispatch({ type: 'threshold', payload: selectedThresholds }),
@ -208,8 +238,34 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
'filter',
timeFilter ? [timeFilter, ...ruleConfiguration.filter] : ruleConfiguration.filter
);
testSearchSource.setField(
'aggs',
buildAggregation({
aggType: ruleParams.aggType,
aggField: ruleParams.aggField,
termField: ruleParams.termField,
termSize: ruleParams.termSize,
condition: {
conditionScript: getComparatorScript(
(ruleParams.thresholdComparator ?? DEFAULT_VALUES.THRESHOLD_COMPARATOR) as Comparator,
ruleParams.threshold,
BUCKET_SELECTOR_FIELD
),
},
})
);
return testSearchSource;
}, [searchSource, timeWindow, ruleConfiguration]);
}, [
searchSource,
timeWindow,
ruleConfiguration,
ruleParams.aggType,
ruleParams.aggField,
ruleParams.termField,
ruleParams.termSize,
ruleParams.threshold,
ruleParams.thresholdComparator,
]);
const onCopyQuery = useCallback(() => {
const testSearchSource = createTestSearchSource();
@ -217,10 +273,16 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
}, [createTestSearchSource]);
const onTestFetch = useCallback(async () => {
const isGroupAgg = isGroupAggregation(ruleParams.termField);
const isCountAgg = isCountAggregation(ruleParams.aggType);
const testSearchSource = createTestSearchSource();
const { rawResponse } = await lastValueFrom(testSearchSource.fetch$());
return { nrOfDocs: totalHitsToNumber(rawResponse.hits.total), timeWindow };
}, [timeWindow, createTestSearchSource]);
return {
testResults: parseAggregationResults({ isCountAgg, isGroupAgg, esResult: rawResponse }),
isGrouped: isGroupAgg,
timeWindow,
};
}, [timeWindow, createTestSearchSource, ruleParams.aggType, ruleParams.termField]);
return (
<Fragment>
@ -232,16 +294,13 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
/>
</h5>
</EuiTitle>
<EuiSpacer size="s" />
<DataViewSelectPopover
dataView={dataView}
metadata={props.metadata}
onSelectDataView={onSelectDataView}
onChangeMetaData={props.onChangeMetaData}
/>
{Boolean(dataView?.id) && (
<>
<EuiSpacer size="s" />
@ -289,6 +348,17 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
timeWindowSize={ruleConfiguration.timeWindowSize}
timeWindowUnit={ruleConfiguration.timeWindowUnit}
size={ruleConfiguration.size}
esFields={esFields}
aggType={ruleConfiguration.aggType}
aggField={ruleConfiguration.aggField}
groupBy={ruleConfiguration.groupBy}
termSize={ruleConfiguration.termSize}
termField={ruleConfiguration.termField}
onChangeSelectedAggField={onChangeSelectedAggField}
onChangeSelectedAggType={onChangeSelectedAggType}
onChangeSelectedGroupBy={onChangeSelectedGroupBy}
onChangeSelectedTermField={onChangeSelectedTermField}
onChangeSelectedTermSize={onChangeSelectedTermSize}
onChangeThreshold={onChangeSelectedThreshold}
onChangeThresholdComparator={onChangeSelectedThresholdComparator}
onChangeWindowSize={onChangeWindowSize}
@ -301,7 +371,6 @@ export const SearchSourceExpressionForm = (props: SearchSourceExpressionFormProp
excludeHitsFromPreviousRun={ruleConfiguration.excludeHitsFromPreviousRun}
onChangeExcludeHitsFromPreviousRun={onChangeExcludeHitsFromPreviousRun}
/>
<EuiSpacer />
</Fragment>
);

274
x-pack/plugins/stack_alerts/public/rule_types/es_query/rule_common_expressions/rule_common_expressions.test.tsx Normal file
View file

@ -0,0 +1,274 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React from 'react';
import { mountWithIntl, nextTick } from '@kbn/test-jest-helpers';
import { act } from 'react-dom/test-utils';
import { RuleCommonExpressions } from './rule_common_expressions';
import {
builtInAggregationTypes,
builtInComparators,
getTimeUnitLabel,
TIME_UNITS,
} from '@kbn/triggers-actions-ui-plugin/public';
import { DEFAULT_VALUES } from '../constants';
import { CommonEsQueryRuleParams } from '../types';
const errors = {
index: new Array<string>(),
size: new Array<string>(),
timeField: new Array<string>(),
threshold0: new Array<string>(),
threshold1: new Array<string>(),
esQuery: new Array<string>(),
thresholdComparator: new Array<string>(),
timeWindowSize: new Array<string>(),
searchConfiguration: new Array<string>(),
searchType: new Array<string>(),
aggField: new Array<string>(),
aggType: new Array<string>(),
groupBy: new Array<string>(),
termSize: new Array<string>(),
termField: new Array<string>(),
};
describe('RuleCommonExpressions', () => {
const onChangeExcludeHitsFromPreviousRunFn = jest.fn();
function getCommonParams(overrides = {}) {
return {
thresholdComparator: '>',
threshold: [0],
timeWindowSize: 15,
timeWindowUnit: 's',
aggType: 'count',
size: 100,
...overrides,
} as unknown as CommonEsQueryRuleParams;
}
async function setup({
ruleParams,
hasValidationErrors = false,
excludeHitsFromPreviousRun = true,
}: {
ruleParams: CommonEsQueryRuleParams;
hasValidationErrors?: boolean;
excludeHitsFromPreviousRun?: boolean;
}) {
const wrapper = mountWithIntl(
<RuleCommonExpressions
esFields={[]}
thresholdComparator={ruleParams.thresholdComparator}
threshold={ruleParams.threshold}
timeWindowSize={ruleParams.timeWindowSize}
timeWindowUnit={ruleParams.timeWindowUnit}
size={ruleParams.size}
aggType={ruleParams.aggType}
aggField={ruleParams.aggField}
groupBy={ruleParams.groupBy}
termSize={ruleParams.termSize}
termField={ruleParams.termField}
onChangeSelectedAggField={() => {}}
onChangeSelectedAggType={() => {}}
onChangeSelectedGroupBy={() => {}}
onChangeSelectedTermField={() => {}}
onChangeSelectedTermSize={() => {}}
onChangeThreshold={() => {}}
onChangeThresholdComparator={() => {}}
onChangeWindowSize={() => {}}
onChangeWindowUnit={() => {}}
onChangeSizeValue={() => {}}
errors={errors}
hasValidationErrors={hasValidationErrors}
onTestFetch={async () => {
return {
testResults: { results: [], truncated: false },
isGrouped: false,
timeWindow: '1m',
};
}}
excludeHitsFromPreviousRun={excludeHitsFromPreviousRun}
onChangeExcludeHitsFromPreviousRun={onChangeExcludeHitsFromPreviousRunFn}
/>
);
await act(async () => {
await nextTick();
wrapper.update();
});
return wrapper;
}
test(`should render RuleCommonExpressions with expected components when aggType doesn't require field`, async () => {
const wrapper = await setup({ ruleParams: getCommonParams() });
expect(wrapper.find('[data-test-subj="thresholdHelpPopover"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="whenExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="aggTypeExpression"]').exists()).toBeFalsy();
expect(wrapper.find('[data-test-subj="groupByExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="thresholdExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="forLastExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="sizeValueExpression"]').exists()).toBeTruthy();
const excludeHitsButton = wrapper.find(
'[data-test-subj="excludeHitsFromPreviousRunExpression"]'
);
expect(excludeHitsButton.exists()).toBeTruthy();
expect(excludeHitsButton.first().prop('checked')).toBeTruthy();
expect(excludeHitsButton.first().prop('disabled')).toBe(false);
const testQueryButton = wrapper.find('EuiButton[data-test-subj="testQuery"]');
expect(testQueryButton.exists()).toBeTruthy();
expect(testQueryButton.prop('disabled')).toBe(false);
});
test(`should render RuleCommonExpressions with expected components when aggType does require field`, async () => {
const wrapper = await setup({ ruleParams: getCommonParams({ aggType: 'avg' }) });
expect(wrapper.find('[data-test-subj="thresholdHelpPopover"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="whenExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="aggTypeExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="groupByExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="thresholdExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="forLastExpression"]').exists()).toBeTruthy();
expect(wrapper.find('[data-test-subj="sizeValueExpression"]').exists()).toBeTruthy();
const excludeHitsButton = wrapper.find(
'[data-test-subj="excludeHitsFromPreviousRunExpression"]'
);
expect(excludeHitsButton.exists()).toBeTruthy();
expect(excludeHitsButton.first().prop('checked')).toBeTruthy();
const testQueryButton = wrapper.find('EuiButton[data-test-subj="testQuery"]');
expect(testQueryButton.exists()).toBeTruthy();
expect(testQueryButton.prop('disabled')).toBe(false);
});
test(`should set default rule common params when params are undefined`, async () => {
const wrapper = await setup({
ruleParams: getCommonParams({
aggType: undefined,
thresholdComparator: undefined,
timeWindowSize: undefined,
timeWindowUnit: undefined,
groupBy: undefined,
threshold: undefined,
}),
});
expect(wrapper.find('button[data-test-subj="whenExpression"]').text()).toEqual(
`when ${builtInAggregationTypes[DEFAULT_VALUES.AGGREGATION_TYPE].text}`
);
expect(wrapper.find('button[data-test-subj="groupByExpression"]').text()).toEqual(
`over ${DEFAULT_VALUES.GROUP_BY} documents `
);
expect(wrapper.find('[data-test-subj="aggTypeExpression"]').exists()).toBeFalsy();
expect(wrapper.find('button[data-test-subj="thresholdPopover"]').text()).toEqual(
`${builtInComparators[DEFAULT_VALUES.THRESHOLD_COMPARATOR].text} ${
DEFAULT_VALUES.THRESHOLD[0]
}`
);
expect(wrapper.find('button[data-test-subj="forLastExpression"]').text()).toEqual(
`for the last ${DEFAULT_VALUES.TIME_WINDOW_SIZE} ${getTimeUnitLabel(
DEFAULT_VALUES.TIME_WINDOW_UNIT as TIME_UNITS,
DEFAULT_VALUES.TIME_WINDOW_SIZE.toString()
)}`
);
});
test(`should use rule params when common params are defined`, async () => {
const aggType = 'avg';
const thresholdComparator = 'between';
const timeWindowSize = 987;
const timeWindowUnit = 's';
const threshold = [3, 1003];
const groupBy = 'top';
const termSize = '27';
const termField = 'host.name';
const wrapper = await setup({
ruleParams: getCommonParams({
aggType,
thresholdComparator,
timeWindowSize,
timeWindowUnit,
termSize,
termField,
groupBy,
threshold,
}),
});
expect(wrapper.find('button[data-test-subj="whenExpression"]').text()).toEqual(
`when ${builtInAggregationTypes[aggType].text}`
);
expect(wrapper.find('button[data-test-subj="groupByExpression"]').text()).toEqual(
`grouped over ${groupBy} ${termSize} '${termField}'`
);
expect(wrapper.find('button[data-test-subj="thresholdPopover"]').text()).toEqual(
`${builtInComparators[thresholdComparator].text} ${threshold[0]} AND ${threshold[1]}`
);
expect(wrapper.find('button[data-test-subj="forLastExpression"]').text()).toEqual(
`for the last ${timeWindowSize} ${getTimeUnitLabel(
timeWindowUnit as TIME_UNITS,
timeWindowSize.toString()
)}`
);
});
test(`should disable excludeHitsFromPreviousRuns when groupBy is not all`, async () => {
const aggType = 'avg';
const thresholdComparator = 'between';
const timeWindowSize = 987;
const timeWindowUnit = 's';
const threshold = [3, 1003];
const groupBy = 'top';
const termSize = '27';
const termField = 'host.name';
const wrapper = await setup({
ruleParams: getCommonParams({
aggType,
thresholdComparator,
timeWindowSize,
timeWindowUnit,
termSize,
termField,
groupBy,
threshold,
}),
});
expect(
wrapper
.find('[data-test-subj="excludeHitsFromPreviousRunExpression"]')
.first()
.prop('disabled')
).toBe(true);
});
test(`should render excludeHitsFromPreviousRuns as unchecked when excludeHitsFromPreviousRun is false`, async () => {
const wrapper = await setup({
ruleParams: getCommonParams(),
excludeHitsFromPreviousRun: false,
});
const excludeHitsButton = wrapper.find(
'[data-test-subj="excludeHitsFromPreviousRunExpression"]'
);
expect(excludeHitsButton.exists()).toBeTruthy();
expect(excludeHitsButton.first().prop('checked')).toBeFalsy();
wrapper
.find('input[data-test-subj="excludeHitsFromPreviousRunExpression"]')
.simulate('change', { target: { checked: true } });
expect(onChangeExcludeHitsFromPreviousRunFn).toHaveBeenCalledWith(true);
});
test(`should render test query button disabled if hasValidationErrors is true`, async () => {
const wrapper = await setup({ ruleParams: getCommonParams(), hasValidationErrors: true });
const testQueryButton = wrapper.find('EuiButton[data-test-subj="testQuery"]');
expect(testQueryButton.exists()).toBeTruthy();
expect(testQueryButton.prop('disabled')).toBe(true);
});
});

75
x-pack/plugins/stack_alerts/public/rule_types/es_query/rule_common_expressions/rule_common_expressions.tsx
View file

@ -5,7 +5,7 @@
* 2.0.
*/
import React from 'react';
import React, { useEffect, useState } from 'react';
import { FormattedMessage } from '@kbn/i18n-react';
import {
EuiCheckbox,
@ -18,25 +18,30 @@ import {
} from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import {
builtInAggregationTypes,
ForLastExpression,
GroupByExpression,
IErrorObject,
OfExpression,
ThresholdExpression,
ValueExpression,
WhenExpression,
} from '@kbn/triggers-actions-ui-plugin/public';
import { builtInGroupByTypes, FieldOption } from '@kbn/triggers-actions-ui-plugin/public/common';
import { CommonRuleParams } from '../types';
import { DEFAULT_VALUES } from '../constants';
import { TestQueryRow, TestQueryRowProps } from '../test_query_row';
import { QueryThresholdHelpPopover } from './threshold_help_popover';
export interface RuleCommonExpressionsProps {
thresholdComparator?: CommonRuleParams['thresholdComparator'];
threshold?: CommonRuleParams['threshold'];
timeWindowSize: CommonRuleParams['timeWindowSize'];
timeWindowUnit: CommonRuleParams['timeWindowUnit'];
size: CommonRuleParams['size'];
excludeHitsFromPreviousRun: CommonRuleParams['excludeHitsFromPreviousRun'];
export interface RuleCommonExpressionsProps extends CommonRuleParams {
esFields: FieldOption[];
errors: IErrorObject;
hasValidationErrors: boolean;
onChangeSelectedAggField: Parameters<typeof OfExpression>[0]['onChangeSelectedAggField'];
onChangeSelectedAggType: Parameters<typeof WhenExpression>[0]['onChangeSelectedAggType'];
onChangeSelectedGroupBy: Parameters<typeof GroupByExpression>[0]['onChangeSelectedGroupBy'];
onChangeSelectedTermField: Parameters<typeof GroupByExpression>[0]['onChangeSelectedTermField'];
onChangeSelectedTermSize: Parameters<typeof GroupByExpression>[0]['onChangeSelectedTermSize'];
onChangeThreshold: Parameters<typeof ThresholdExpression>[0]['onChangeSelectedThreshold'];
onChangeThresholdComparator: Parameters<
typeof ThresholdExpression
@ -50,13 +55,24 @@ export interface RuleCommonExpressionsProps {
}
export const RuleCommonExpressions: React.FC<RuleCommonExpressionsProps> = ({
esFields,
thresholdComparator,
threshold,
timeWindowSize,
timeWindowUnit,
aggType,
aggField,
groupBy,
termField,
termSize,
size,
errors,
hasValidationErrors,
onChangeSelectedAggField,
onChangeSelectedAggType,
onChangeSelectedGroupBy,
onChangeSelectedTermField,
onChangeSelectedTermSize,
onChangeThreshold,
onChangeThresholdComparator,
onChangeWindowSize,
@ -67,18 +83,54 @@ export const RuleCommonExpressions: React.FC<RuleCommonExpressionsProps> = ({
excludeHitsFromPreviousRun,
onChangeExcludeHitsFromPreviousRun,
}) => {
const [isExcludeHitsDisabled, setIsExcludeHitsDisabled] = useState<boolean>(false);
useEffect(() => {
if (groupBy) {
setIsExcludeHitsDisabled(groupBy !== builtInGroupByTypes.all.value);
}
}, [groupBy]);
return (
<>
<EuiTitle size="xs">
<h4>
<FormattedMessage
id="xpack.stackAlerts.esQuery.ui.conditionsPrompt"
defaultMessage="Set the threshold and time window"
defaultMessage="Set the group, threshold, and time window"
/>{' '}
<QueryThresholdHelpPopover />
</h4>
</EuiTitle>
<EuiSpacer size="s" />
<WhenExpression
display="fullWidth"
data-test-subj="whenExpression"
aggType={aggType ?? DEFAULT_VALUES.AGGREGATION_TYPE}
onChangeSelectedAggType={onChangeSelectedAggType}
/>
{aggType && builtInAggregationTypes[aggType].fieldRequired ? (
<OfExpression
aggField={aggField}
data-test-subj="aggTypeExpression"
fields={esFields}
aggType={aggType}
errors={errors}
display="fullWidth"
onChangeSelectedAggField={onChangeSelectedAggField}
/>
) : null}
<GroupByExpression
groupBy={groupBy || DEFAULT_VALUES.GROUP_BY}
data-test-subj="groupByExpression"
termField={termField}
termSize={termSize}
errors={errors}
fields={esFields}
display="fullWidth"
onChangeSelectedGroupBy={onChangeSelectedGroupBy}
onChangeSelectedTermField={onChangeSelectedTermField}
onChangeSelectedTermSize={onChangeSelectedTermSize}
/>
<ThresholdExpression
data-test-subj="thresholdExpression"
thresholdComparator={thresholdComparator ?? DEFAULT_VALUES.THRESHOLD_COMPARATOR}
@ -92,8 +144,8 @@ export const RuleCommonExpressions: React.FC<RuleCommonExpressionsProps> = ({
<ForLastExpression
data-test-subj="forLastExpression"
popupPosition="upLeft"
timeWindowSize={timeWindowSize}
timeWindowUnit={timeWindowUnit}
timeWindowSize={timeWindowSize ?? DEFAULT_VALUES.TIME_WINDOW_SIZE}
timeWindowUnit={timeWindowUnit ?? DEFAULT_VALUES.TIME_WINDOW_UNIT}
display="fullWidth"
errors={errors}
onChangeWindowSize={onChangeWindowSize}
@ -138,6 +190,7 @@ export const RuleCommonExpressions: React.FC<RuleCommonExpressionsProps> = ({
<EuiSpacer size="m" />
<EuiFormRow>
<EuiCheckbox
disabled={isExcludeHitsDisabled}
data-test-subj="excludeHitsFromPreviousRunExpression"
checked={excludeHitsFromPreviousRun}
id="excludeHitsFromPreviousRunExpressionId"

5
x-pack/plugins/stack_alerts/public/rule_types/es_query/rule_common_expressions/threshold_help_popover.tsx
View file

@ -46,7 +46,7 @@ export class QueryThresholdHelpPopover extends Component<{}, State> {
<p>
<FormattedMessage
id="xpack.stackAlerts.esQuery.ui.thresholdHelp.threshold"
defaultMessage="Each time the rule runs, it checks whether the number of documents that match your query meets this threshold."
defaultMessage="Each time the rule runs, it checks whether the number of documents that match your query meets this threshold. If there is a grouped over clause, the rule checks the condition for the specified number of top groups."
/>
</p>
<p>
@ -77,6 +77,7 @@ export class QueryThresholdHelpPopover extends Component<{}, State> {
return (
<EuiPopover
id="thresholdHelpPopover"
data-test-subj="thresholdHelpPopover"
anchorPosition="upLeft"
button={
<EuiButtonIcon
@ -95,7 +96,7 @@ export class QueryThresholdHelpPopover extends Component<{}, State> {
<EuiPopoverTitle>
<FormattedMessage
id="xpack.stackAlerts.esQuery.ui.thresholdHelp.title"
defaultMessage="Set the threshold and time window"
defaultMessage="Set the group, threshold and time window"
/>
</EuiPopoverTitle>
{this._renderContent()}

2
x-pack/plugins/stack_alerts/public/rule_types/es_query/test_query_row/index.ts
View file

@ -7,4 +7,4 @@
export { TestQueryRow } from './test_query_row';
export type { TestQueryRowProps } from './test_query_row';
export { useTestQuery, totalHitsToNumber } from './use_test_query';
export { useTestQuery } from './use_test_query';

10
x-pack/plugins/stack_alerts/public/rule_types/es_query/test_query_row/test_query_row.test.tsx
View file

@ -21,7 +21,15 @@ jest.mock('@elastic/eui', () => {
});
const COPIED_QUERY = 'COPIED QUERY';
const onFetch = () => Promise.resolve({ nrOfDocs: 42, timeWindow: '5m' });
const onFetch = () =>
Promise.resolve({
testResults: {
results: [{ group: 'all documents', hits: [], count: 42 }],
truncated: false,
},
isGrouped: false,
timeWindow: '5m',
});
const onCopyQuery = () => COPIED_QUERY;
describe('TestQueryRow', () => {

7
x-pack/plugins/stack_alerts/public/rule_types/es_query/test_query_row/test_query_row.tsx
View file

@ -16,10 +16,15 @@ import {
EuiToolTip,
} from '@elastic/eui';
import { FormattedMessage } from '@kbn/i18n-react';
import { ParsedAggregationResults } from '@kbn/triggers-actions-ui-plugin/common';
import { useTestQuery } from './use_test_query';
export interface TestQueryRowProps {
fetch: () => Promise<{ nrOfDocs: number; timeWindow: string }>;
fetch: () => Promise<{
testResults: ParsedAggregationResults;
isGrouped: boolean;
timeWindow: string;
}>;
copyQuery?: () => string;
hasValidationErrors: boolean;
}

39
x-pack/plugins/stack_alerts/public/rule_types/es_query/test_query_row/use_test_query.test.ts
View file

@ -10,9 +10,17 @@ import { act } from 'react-test-renderer';
import { useTestQuery } from './use_test_query';
describe('useTestQuery', () => {
test('returning a valid result', async () => {
test('returning a valid result for ungrouped result', async () => {
const { result } = renderHook(useTestQuery, {
initialProps: () => Promise.resolve({ nrOfDocs: 1, timeWindow: '1s' }),
initialProps: () =>
Promise.resolve({
testResults: {
results: [{ group: 'all documents', hits: [], count: 1 }],
truncated: false,
},
isGrouped: false,
timeWindow: '1s',
}),
});
await act(async () => {
await result.current.onTestQuery();
@ -22,6 +30,33 @@ describe('useTestQuery', () => {
expect(result.current.testQueryResult).toContain('1s');
expect(result.current.testQueryResult).toContain('1 document');
});
test('returning a valid result for grouped result', async () => {
const { result } = renderHook(useTestQuery, {
initialProps: () =>
Promise.resolve({
testResults: {
results: [
{ group: 'a', count: 1, value: 10, hits: [] },
{ group: 'b', count: 2, value: 20, hits: [] },
],
truncated: false,
},
isGrouped: true,
timeWindow: '1s',
}),
});
await act(async () => {
await result.current.onTestQuery();
});
expect(result.current.testQueryLoading).toBe(false);
expect(result.current.testQueryError).toBe(null);
expect(result.current.testQueryResult).toContain('1s');
expect(result.current.testQueryResult).toContain(
'Grouped query matched 2 groups in the last 1s.'
);
});
test('returning an error', async () => {
const errorMsg = 'How dare you writing such a query';
const { result } = renderHook(useTestQuery, {

48
x-pack/plugins/stack_alerts/public/rule_types/es_query/test_query_row/use_test_query.ts
View file

@ -7,7 +7,7 @@
import { useState, useCallback, useEffect } from 'react';
import { i18n } from '@kbn/i18n';
import * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import type { ParsedAggregationResults } from '@kbn/triggers-actions-ui-plugin/common';
interface TestQueryResponse {
result: string | null;
@ -25,7 +25,13 @@ const TEST_QUERY_INITIAL_RESPONSE: TestQueryResponse = {
* Hook used to test the data fetching execution by returning a number of found documents
* Or in error in case it's failing
*/
export function useTestQuery(fetch: () => Promise<{ nrOfDocs: number; timeWindow: string }>) {
export function useTestQuery(
fetch: () => Promise<{
testResults: ParsedAggregationResults;
isGrouped: boolean;
timeWindow: string;
}>
) {
const [testQueryResponse, setTestQueryResponse] = useState<TestQueryResponse>(
TEST_QUERY_INITIAL_RESPONSE
);
@ -43,16 +49,32 @@ export function useTestQuery(fetch: () => Promise<{ nrOfDocs: number; timeWindow
});
try {
const { nrOfDocs, timeWindow } = await fetch();
const { testResults, isGrouped, timeWindow } = await fetch();
setTestQueryResponse({
result: i18n.translate('xpack.stackAlerts.esQuery.ui.numQueryMatchesText', {
defaultMessage: 'Query matched {count} documents in the last {window}.',
values: { count: nrOfDocs, window: timeWindow },
}),
error: null,
isLoading: false,
});
if (isGrouped) {
setTestQueryResponse({
result: i18n.translate('xpack.stackAlerts.esQuery.ui.testQueryGroupedResponse', {
defaultMessage: 'Grouped query matched {groups} groups in the last {window}.',
values: {
groups: testResults.results.length,
window: timeWindow,
},
}),
error: null,
isLoading: false,
});
} else {
const ungroupedQueryResponse =
testResults.results.length > 0 ? testResults.results[0] : { count: 0 };
setTestQueryResponse({
result: i18n.translate('xpack.stackAlerts.esQuery.ui.numQueryMatchesText', {
defaultMessage: 'Query matched {count} documents in the last {window}.',
values: { count: ungroupedQueryResponse?.count ?? 0, window: timeWindow },
}),
error: null,
isLoading: false,
});
}
} catch (err) {
const message = err?.body?.attributes?.error?.root_cause[0]?.reason || err?.body?.message;
@ -74,7 +96,3 @@ export function useTestQuery(fetch: () => Promise<{ nrOfDocs: number; timeWindow
testQueryLoading: testQueryResponse.isLoading,
};
}
export function totalHitsToNumber(total: estypes.SearchHitsMetadata['total']): number {
return typeof total === 'number' ? total : total?.value ?? 0;
}

19
x-pack/plugins/stack_alerts/public/rule_types/es_query/types.ts
View file

@ -10,34 +10,35 @@ import { SerializedSearchSourceFields } from '@kbn/data-plugin/common';
import { EuiComboBoxOptionOption } from '@elastic/eui';
import type { DataView } from '@kbn/data-views-plugin/public';
export interface Comparator {
text: string;
value: string;
requiredValues: number;
}
export enum SearchType {
esQuery = 'esQuery',
searchSource = 'searchSource',
}
export interface CommonRuleParams extends RuleTypeParams {
export interface CommonRuleParams {
size: number;
thresholdComparator?: string;
threshold: number[];
timeWindowSize: number;
timeWindowUnit: string;
aggType: string;
aggField?: string;
groupBy?: string;
termSize?: number;
termField?: string;
excludeHitsFromPreviousRun: boolean;
}
export interface CommonEsQueryRuleParams extends RuleTypeParams, CommonRuleParams {}
export interface EsQueryRuleMetaData {
adHocDataViewList: DataView[];
isManagementPage?: boolean;
}
export type EsQueryRuleParams<T = SearchType> = T extends SearchType.searchSource
? CommonRuleParams & OnlySearchSourceRuleParams
: CommonRuleParams & OnlyEsQueryRuleParams;
? CommonEsQueryRuleParams & OnlySearchSourceRuleParams
: CommonEsQueryRuleParams & OnlyEsQueryRuleParams;
export interface OnlyEsQueryRuleParams {
esQuery: string;

125
x-pack/plugins/stack_alerts/public/rule_types/es_query/util.test.ts Normal file
View file

@ -0,0 +1,125 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { convertFieldSpecToFieldOption } from './util';
describe('convertFieldSpecToFieldOption', () => {
test('should correctly convert FieldSpec to FieldOption', () => {
expect(
convertFieldSpecToFieldOption([
{
count: 0,
name: '@timestamp',
type: 'date',
esTypes: ['date'],
scripted: false,
searchable: true,
aggregatable: true,
readFromDocValues: true,
shortDotsEnable: false,
isMapped: true,
},
{
count: 0,
name: 'ecs.version',
type: 'string',
esTypes: ['keyword'],
scripted: false,
searchable: true,
aggregatable: true,
readFromDocValues: true,
shortDotsEnable: false,
isMapped: true,
},
{
count: 0,
name: 'error.message',
type: 'string',
esTypes: ['text'],
scripted: false,
searchable: true,
aggregatable: false,
readFromDocValues: false,
shortDotsEnable: false,
isMapped: true,
},
{
count: 0,
name: 'event.duration',
type: 'number',
esTypes: ['long'],
scripted: false,
searchable: true,
aggregatable: true,
readFromDocValues: true,
shortDotsEnable: false,
isMapped: true,
},
{
count: 0,
name: 'event.risk_score',
type: 'number',
esTypes: ['float'],
scripted: false,
searchable: true,
aggregatable: true,
readFromDocValues: true,
shortDotsEnable: false,
isMapped: true,
},
{
count: 0,
name: 'user.name',
type: 'string',
esTypes: ['keyword'],
scripted: false,
searchable: false,
aggregatable: false,
readFromDocValues: true,
shortDotsEnable: false,
isMapped: false,
},
])
).toEqual([
{
name: '@timestamp',
type: 'date',
normalizedType: 'date',
aggregatable: true,
searchable: true,
},
{
name: 'ecs.version',
type: 'keyword',
normalizedType: 'keyword',
aggregatable: true,
searchable: true,
},
{
name: 'error.message',
type: 'text',
normalizedType: 'text',
aggregatable: false,
searchable: true,
},
{
name: 'event.duration',
type: 'long',
normalizedType: 'number',
aggregatable: true,
searchable: true,
},
{
name: 'event.risk_score',
type: 'float',
normalizedType: 'number',
aggregatable: true,
searchable: true,
},
]);
});
});

27
x-pack/plugins/stack_alerts/public/rule_types/es_query/util.ts
View file

@ -5,7 +5,9 @@
* 2.0.
*/
import { FieldSpec } from '@kbn/data-views-plugin/common';
import { useKibana } from '@kbn/triggers-actions-ui-plugin/public';
import { FieldOption } from '@kbn/triggers-actions-ui-plugin/public/common';
import { EsQueryRuleParams, SearchType } from './types';
export const isSearchSourceRule = (
@ -14,4 +16,29 @@ export const isSearchSourceRule = (
return ruleParams.searchType === 'searchSource';
};
export const convertFieldSpecToFieldOption = (fieldSpec: FieldSpec[]): FieldOption[] => {
return (fieldSpec ?? [])
.filter((spec: FieldSpec) => spec.isMapped)
.map((spec: FieldSpec) => {
const converted = {
name: spec.name,
searchable: spec.searchable,
aggregatable: spec.aggregatable,
type: spec.type,
normalizedType: spec.type,
};
if (spec.type === 'string') {
const esType = spec.esTypes && spec.esTypes.length > 0 ? spec.esTypes[0] : spec.type;
converted.type = esType;
converted.normalizedType = esType;
} else if (spec.type === 'number') {
const esType = spec.esTypes && spec.esTypes.length > 0 ? spec.esTypes[0] : spec.type;
converted.type = esType;
}
return converted;
});
};
export const useTriggerUiActionServices = () => useKibana().services;

75
x-pack/plugins/stack_alerts/public/rule_types/es_query/validation.test.ts
View file

@ -26,6 +26,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.index.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.index[0]).toBe('Index is required.');
@ -41,11 +43,66 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.timeField.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.timeField[0]).toBe('Time field is required.');
});
test('if aggField property is invalid should return proper error message', () => {
const initialParams: EsQueryRuleParams<SearchType.esQuery> = {
index: ['test'],
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
timeWindowSize: 1,
timeWindowUnit: 's',
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'avg',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.aggField.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.aggField[0]).toBe(
'Aggregation field is required.'
);
});
test('if termSize property is not set should return proper error message', () => {
const initialParams: EsQueryRuleParams<SearchType.esQuery> = {
index: ['test'],
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
timeWindowSize: 1,
timeWindowUnit: 's',
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'top',
};
expect(validateExpression(initialParams).errors.termSize.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.termSize[0]).toBe('Term size is required.');
});
test('if termField property is not set should return proper error message', () => {
const initialParams: EsQueryRuleParams<SearchType.esQuery> = {
index: ['test'],
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
timeWindowSize: 1,
timeWindowUnit: 's',
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'top',
termSize: 10,
};
expect(validateExpression(initialParams).errors.termField.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.termField[0]).toBe('Term field is required.');
});
test('if esQuery property is invalid JSON should return proper error message', () => {
const initialParams: EsQueryRuleParams<SearchType.esQuery> = {
index: ['test'],
@ -56,6 +113,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.esQuery.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.esQuery[0]).toBe('Query must be valid JSON.');
@ -71,6 +130,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.esQuery.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.esQuery[0]).toBe(`Query field is required.`);
@ -102,6 +163,8 @@ describe('expression params validation', () => {
thresholdComparator: '<',
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.threshold0.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.threshold0[0]).toBe('Threshold 0 is required.');
@ -118,6 +181,8 @@ describe('expression params validation', () => {
thresholdComparator: 'between',
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.threshold1.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.threshold1[0]).toBe('Threshold 1 is required.');
@ -134,6 +199,8 @@ describe('expression params validation', () => {
thresholdComparator: 'between',
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.threshold1.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.threshold1[0]).toBe(
@ -151,6 +218,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.size.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.size[0]).toBe(
@ -168,6 +237,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.size.length).toBe(0);
});
@ -182,6 +253,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.size.length).toBeGreaterThan(0);
expect(validateExpression(initialParams).errors.size[0]).toBe(
@ -199,6 +272,8 @@ describe('expression params validation', () => {
threshold: [0],
timeField: '@timestamp',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
expect(validateExpression(initialParams).errors.size.length).toBe(0);
expect(hasExpressionValidationErrors(initialParams)).toBe(false);

53
x-pack/plugins/stack_alerts/public/rule_types/es_query/validation.ts
View file

@ -7,7 +7,12 @@
import { defaultsDeep, isNil } from 'lodash';
import { i18n } from '@kbn/i18n';
import { ValidationResult, builtInComparators } from '@kbn/triggers-actions-ui-plugin/public';
import {
ValidationResult,
builtInComparators,
builtInAggregationTypes,
builtInGroupByTypes,
} from '@kbn/triggers-actions-ui-plugin/public';
import { EsQueryRuleParams, SearchType } from './types';
import { isSearchSourceRule } from './util';
import {
@ -17,7 +22,17 @@ import {
} from './constants';
const validateCommonParams = (ruleParams: EsQueryRuleParams) => {
const { size, threshold, timeWindowSize, thresholdComparator } = ruleParams;
const {
size,
threshold,
timeWindowSize,
thresholdComparator,
aggType,
aggField,
groupBy,
termSize,
termField,
} = ruleParams;
const errors: typeof COMMON_EXPRESSION_ERRORS = defaultsDeep({}, COMMON_EXPRESSION_ERRORS);
if (!('index' in ruleParams) && !ruleParams.searchType) {
@ -30,6 +45,40 @@ const validateCommonParams = (ruleParams: EsQueryRuleParams) => {
return errors;
}
if (aggType && builtInAggregationTypes[aggType].fieldRequired && !aggField) {
errors.aggField.push(
i18n.translate('xpack.stackAlerts.esQuery.ui.validation.error.requiredAggFieldText', {
defaultMessage: 'Aggregation field is required.',
})
);
}
if (
groupBy &&
builtInGroupByTypes[groupBy] &&
builtInGroupByTypes[groupBy].sizeRequired &&
!termSize
) {
errors.termSize.push(
i18n.translate('xpack.stackAlerts.esQuery.ui.validation.error.requiredTermSizedText', {
defaultMessage: 'Term size is required.',
})
);
}
if (
groupBy &&
builtInGroupByTypes[groupBy].validNormalizedTypes &&
builtInGroupByTypes[groupBy].validNormalizedTypes.length > 0 &&
!termField
) {
errors.termField.push(
i18n.translate('xpack.stackAlerts.esQuery.ui.validation.error.requiredTermFieldText', {
defaultMessage: 'Term field is required.',
})
);
}
if (!threshold || threshold.length === 0 || threshold[0] === undefined) {
errors.threshold0.push(
i18n.translate('xpack.stackAlerts.esQuery.ui.validation.error.requiredThreshold0Text', {

172
x-pack/plugins/stack_alerts/server/rule_types/es_query/action_context.test.ts
View file

@ -5,10 +5,15 @@
* 2.0.
*/
import { EsQueryRuleActionContext, addMessages } from './action_context';
import {
EsQueryRuleActionContext,
addMessages,
getContextConditionsDescription,
} from './action_context';
import { EsQueryRuleParams, EsQueryRuleParamsSchema } from './rule_type_params';
import { Comparator } from '../../../common/comparator_types';
describe('ActionContext', () => {
describe('addMessages', () => {
it('generates expected properties', async () => {
const params = EsQueryRuleParamsSchema.validate({
index: ['[index]'],
@ -20,6 +25,8 @@ describe('ActionContext', () => {
thresholdComparator: '>',
threshold: [4],
searchType: 'esQuery',
aggType: 'count',
groupBy: 'all',
}) as EsQueryRuleParams;
const base: EsQueryRuleActionContext = {
date: '2020-01-01T00:00:00.000Z',
@ -28,7 +35,7 @@ describe('ActionContext', () => {
hits: [],
link: 'link-mock',
};
const context = addMessages('[rule-name]', base, params);
const context = addMessages({ ruleName: '[rule-name]', baseContext: base, params });
expect(context.title).toMatchInlineSnapshot(`"rule '[rule-name]' matched query"`);
expect(context.message).toEqual(
`rule '[rule-name]' is active:
@ -51,6 +58,8 @@ describe('ActionContext', () => {
thresholdComparator: '>',
threshold: [4],
searchType: 'esQuery',
aggType: 'count',
groupBy: 'all',
}) as EsQueryRuleParams;
const base: EsQueryRuleActionContext = {
date: '2020-01-01T00:00:00.000Z',
@ -59,7 +68,12 @@ describe('ActionContext', () => {
hits: [],
link: 'link-mock',
};
const context = addMessages('[rule-name]', base, params, true);
const context = addMessages({
ruleName: '[rule-name]',
baseContext: base,
params,
isRecovered: true,
});
expect(context.title).toMatchInlineSnapshot(`"rule '[rule-name]' recovered"`);
expect(context.message).toEqual(
`rule '[rule-name]' is recovered:
@ -82,6 +96,8 @@ describe('ActionContext', () => {
thresholdComparator: 'between',
threshold: [4, 5],
searchType: 'esQuery',
aggType: 'count',
groupBy: 'all',
}) as EsQueryRuleParams;
const base: EsQueryRuleActionContext = {
date: '2020-01-01T00:00:00.000Z',
@ -90,7 +106,7 @@ describe('ActionContext', () => {
hits: [],
link: 'link-mock',
};
const context = addMessages('[rule-name]', base, params);
const context = addMessages({ ruleName: '[rule-name]', baseContext: base, params });
expect(context.title).toMatchInlineSnapshot(`"rule '[rule-name]' matched query"`);
expect(context.message).toEqual(
`rule '[rule-name]' is active:
@ -98,7 +114,153 @@ describe('ActionContext', () => {
- Value: 4
- Conditions Met: count between 4 and 5 over 5m
- Timestamp: 2020-01-01T00:00:00.000Z
- Link: link-mock`
);
});
it('generates expected properties when group is specified', async () => {
const params = EsQueryRuleParamsSchema.validate({
index: ['[index]'],
timeField: '[timeField]',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
timeWindowSize: 5,
timeWindowUnit: 'm',
thresholdComparator: '>',
threshold: [4],
searchType: 'esQuery',
aggType: 'count',
groupBy: 'top',
termField: 'host.name',
termSize: 5,
}) as EsQueryRuleParams;
const base: EsQueryRuleActionContext = {
date: '2020-01-01T00:00:00.000Z',
value: 42,
conditions: `count for group "host-1" not greater than 4`,
hits: [],
link: 'link-mock',
};
const context = addMessages({
ruleName: '[rule-name]',
baseContext: base,
params,
group: 'host-1',
});
expect(context.title).toMatchInlineSnapshot(
`"rule '[rule-name]' matched query for group host-1"`
);
expect(context.message).toEqual(
`rule '[rule-name]' is active:
- Value: 42
- Conditions Met: count for group "host-1" not greater than 4 over 5m
- Timestamp: 2020-01-01T00:00:00.000Z
- Link: link-mock`
);
});
});
describe('getContextConditionsDescription', () => {
it('should return conditions correctly', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
aggType: 'count',
});
expect(result).toBe(`Number of matching documents is greater than 10`);
});
it('should return conditions correctly when isRecovered is true', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
aggType: 'count',
isRecovered: true,
});
expect(result).toBe(`Number of matching documents is NOT greater than 10`);
});
it('should return conditions correctly when multiple thresholds provided', () => {
const result = getContextConditionsDescription({
comparator: Comparator.BETWEEN,
threshold: [10, 20],
aggType: 'count',
isRecovered: true,
});
expect(result).toBe(`Number of matching documents is NOT between 10 and 20`);
});
it('should return conditions correctly when group is specified', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
aggType: 'count',
group: 'host-1',
});
expect(result).toBe(`Number of matching documents for group "host-1" is greater than 10`);
});
it('should return conditions correctly when group is specified and isRecovered is true', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
aggType: 'count',
isRecovered: true,
group: 'host-1',
});
expect(result).toBe(`Number of matching documents for group "host-1" is NOT greater than 10`);
});
it('should return conditions correctly when aggType is not count', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
aggType: 'min',
aggField: 'numericField',
});
expect(result).toBe(
`Number of matching documents where min of numericField is greater than 10`
);
});
it('should return conditions correctly when aggType is not count and isRecovered is true', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
aggType: 'min',
aggField: 'numericField',
isRecovered: true,
});
expect(result).toBe(
`Number of matching documents where min of numericField is NOT greater than 10`
);
});
it('should return conditions correctly when group is specified and aggType is not count', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
group: 'host-1',
aggType: 'max',
aggField: 'numericField',
});
expect(result).toBe(
`Number of matching documents for group "host-1" where max of numericField is greater than 10`
);
});
it('should return conditions correctly when group is specified, aggType is not count and isRecovered is true', () => {
const result = getContextConditionsDescription({
comparator: Comparator.GT,
threshold: [10],
isRecovered: true,
group: 'host-1',
aggType: 'max',
aggField: 'numericField',
});
expect(result).toBe(
`Number of matching documents for group "host-1" where max of numericField is NOT greater than 10`
);
});
});

54
x-pack/plugins/stack_alerts/server/rule_types/es_query/action_context.ts
View file

@ -9,6 +9,8 @@ import { i18n } from '@kbn/i18n';
import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import { AlertInstanceContext } from '@kbn/alerting-plugin/server';
import { EsQueryRuleParams } from './rule_type_params';
import { Comparator } from '../../../common/comparator_types';
import { getHumanReadableComparator } from '../../../common';
// rule type context provided to actions
export interface ActionContext extends EsQueryRuleActionContext {
@ -32,17 +34,25 @@ export interface EsQueryRuleActionContext extends AlertInstanceContext {
link: string;
}
export function addMessages(
ruleName: string,
baseContext: EsQueryRuleActionContext,
params: EsQueryRuleParams,
isRecovered: boolean = false
): ActionContext {
interface AddMessagesOpts {
ruleName: string;
baseContext: EsQueryRuleActionContext;
params: EsQueryRuleParams;
group?: string;
isRecovered?: boolean;
}
export function addMessages({
ruleName,
baseContext,
params,
group,
isRecovered = false,
}: AddMessagesOpts): ActionContext {
const title = i18n.translate('xpack.stackAlerts.esQuery.alertTypeContextSubjectTitle', {
defaultMessage: `rule '{name}' {verb}`,
values: {
name: ruleName,
verb: isRecovered ? 'recovered' : 'matched query',
verb: isRecovered ? 'recovered' : `matched query${group ? ` for group ${group}` : ''}`,
},
});
@ -67,3 +77,33 @@ export function addMessages(
return { ...baseContext, title, message };
}
interface GetContextConditionsDescriptionOpts {
comparator: Comparator;
threshold: number[];
aggType: string;
aggField?: string;
isRecovered?: boolean;
group?: string;
}
export function getContextConditionsDescription({
comparator,
threshold,
aggType,
aggField,
isRecovered = false,
group,
}: GetContextConditionsDescriptionOpts) {
return i18n.translate('xpack.stackAlerts.esQuery.alertTypeContextConditionsDescription', {
defaultMessage:
'Number of matching documents{groupCondition}{aggCondition} is {negation}{thresholdComparator} {threshold}',
values: {
aggCondition: aggType === 'count' ? '' : ` where ${aggType} of ${aggField}`,
groupCondition: group ? ` for group "${group}"` : '',
thresholdComparator: getHumanReadableComparator(comparator),
threshold: threshold.join(' and '),
negation: isRecovered ? 'NOT ' : '',
},
});
}

511
x-pack/plugins/stack_alerts/server/rule_types/es_query/executor.test.ts
View file

@ -5,28 +5,506 @@
* 2.0.
*/
import {
getSearchParams,
getValidTimefieldSort,
tryToParseAsDate,
getContextConditionsDescription,
} from './executor';
import { OnlyEsQueryRuleParams } from './types';
import { of } from 'rxjs';
import { CoreSetup } from '@kbn/core/server';
import { executor, getSearchParams, getValidTimefieldSort, tryToParseAsDate } from './executor';
import { ExecutorOptions, OnlyEsQueryRuleParams } from './types';
import { Comparator } from '../../../common/comparator_types';
import { elasticsearchServiceMock } from '@kbn/core-elasticsearch-server-mocks';
import { loggerMock } from '@kbn/logging-mocks';
import { createSearchSourceMock } from '@kbn/data-plugin/common/search/search_source/mocks';
import { ISearchStartSearchSource } from '@kbn/data-plugin/common';
import { EsQueryRuleParams } from './rule_type_params';
import { FetchEsQueryOpts } from './lib/fetch_es_query';
import { FetchSearchSourceQueryOpts } from './lib/fetch_search_source_query';
const logger = loggerMock.create();
const scopedClusterClientMock = elasticsearchServiceMock.createScopedClusterClient();
const createSearchSourceClientMock = () => {
const searchSourceMock = createSearchSourceMock();
searchSourceMock.fetch$ = jest.fn().mockImplementation(() => of({ rawResponse: { took: 5 } }));
return {
searchSourceMock,
searchSourceClientMock: {
create: jest.fn().mockReturnValue(searchSourceMock),
createEmpty: jest.fn().mockReturnValue(searchSourceMock),
} as unknown as ISearchStartSearchSource,
};
};
const { searchSourceClientMock } = createSearchSourceClientMock();
const mockFetchEsQuery = jest.fn();
jest.mock('./lib/fetch_es_query', () => ({
fetchEsQuery: (...args: [FetchEsQueryOpts]) => mockFetchEsQuery(...args),
}));
const mockFetchSearchSourceQuery = jest.fn();
jest.mock('./lib/fetch_search_source_query', () => ({
fetchSearchSourceQuery: (...args: [FetchSearchSourceQueryOpts]) =>
mockFetchSearchSourceQuery(...args),
}));
const scheduleActions = jest.fn();
const replaceState = jest.fn(() => ({ scheduleActions }));
const mockCreateAlert = jest.fn(() => ({ replaceState }));
const mockGetRecoveredAlerts = jest.fn().mockReturnValue([]);
const mockSetLimitReached = jest.fn();
const mockNow = jest.getRealSystemTime();
describe('es_query executor', () => {
beforeAll(() => {
jest.useFakeTimers();
jest.setSystemTime(mockNow);
});
beforeEach(() => {
jest.clearAllMocks();
});
afterAll(() => {
jest.useRealTimers();
});
const defaultProps = {
size: 3,
timeWindowSize: 5,
timeWindowUnit: 'm',
threshold: [],
thresholdComparator: '>=',
thresholdComparator: '>=' as Comparator,
esQuery: '{ "query": "test-query" }',
index: ['test-index'],
timeField: '',
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
describe('executor', () => {
const services = {
scopedClusterClient: scopedClusterClientMock,
savedObjectsClient: {
get: () => ({ attributes: { consumer: 'alerts' } }),
},
searchSourceClient: searchSourceClientMock,
alertFactory: {
create: mockCreateAlert,
alertLimit: {
getValue: jest.fn().mockReturnValue(1000),
setLimitReached: mockSetLimitReached,
},
done: () => ({
getRecoveredAlerts: mockGetRecoveredAlerts,
}),
},
alertWithLifecycle: jest.fn(),
logger,
shouldWriteAlerts: () => true,
};
const coreMock = {
http: { basePath: { publicBaseUrl: 'https://localhost:5601' } },
} as CoreSetup;
const defaultExecutorOptions = {
params: defaultProps,
services,
rule: { id: 'test-rule-id', name: 'test-rule-name' },
state: { latestTimestamp: undefined },
spaceId: 'default',
logger,
} as unknown as ExecutorOptions<EsQueryRuleParams>;
it('should throw error for invalid comparator', async () => {
await expect(() =>
executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: { ...defaultProps, thresholdComparator: '?' },
})
).rejects.toThrowErrorMatchingInlineSnapshot(`"invalid thresholdComparator specified: ?"`);
});
it('should call fetchEsQuery if searchType is esQuery', async () => {
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'all documents',
count: 491,
hits: [],
},
],
truncated: false,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, defaultExecutorOptions);
expect(mockFetchEsQuery).toHaveBeenCalledWith({
ruleId: 'test-rule-id',
name: 'test-rule-name',
alertLimit: 1000,
params: defaultProps,
timestamp: undefined,
services: {
scopedClusterClient: scopedClusterClientMock,
logger,
},
});
expect(mockFetchSearchSourceQuery).not.toHaveBeenCalled();
});
it('should call fetchSearchSourceQuery if searchType is searchSource', async () => {
mockFetchSearchSourceQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'all documents',
count: 491,
hits: [],
},
],
truncated: false,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
params: { ...defaultProps, searchConfiguration: {}, searchType: 'searchSource' },
});
expect(mockFetchSearchSourceQuery).toHaveBeenCalledWith({
ruleId: 'test-rule-id',
alertLimit: 1000,
params: { ...defaultProps, searchConfiguration: {}, searchType: 'searchSource' },
latestTimestamp: undefined,
services: {
searchSourceClient: searchSourceClientMock,
logger,
},
});
expect(mockFetchEsQuery).not.toHaveBeenCalled();
});
it('should not create alert if compare function returns false for ungrouped alert', async () => {
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'all documents',
count: 491,
hits: [],
},
],
truncated: false,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: { ...defaultProps, threshold: [500], thresholdComparator: '>=' as Comparator },
});
expect(mockCreateAlert).not.toHaveBeenCalled();
expect(mockSetLimitReached).toHaveBeenCalledTimes(1);
expect(mockSetLimitReached).toHaveBeenCalledWith(false);
});
it('should create alert if compare function returns true for ungrouped alert', async () => {
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'all documents',
count: 491,
hits: [],
},
],
truncated: false,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: { ...defaultProps, threshold: [200], thresholdComparator: '>=' as Comparator },
});
expect(mockCreateAlert).toHaveBeenCalledTimes(1);
expect(mockCreateAlert).toHaveBeenNthCalledWith(1, 'query matched');
expect(scheduleActions).toHaveBeenCalledTimes(1);
expect(scheduleActions).toHaveBeenNthCalledWith(1, 'query matched', {
conditions: 'Number of matching documents is greater than or equal to 200',
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is active:
- Value: 491
- Conditions Met: Number of matching documents is greater than or equal to 200 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' matched query",
value: 491,
});
expect(mockSetLimitReached).toHaveBeenCalledTimes(1);
expect(mockSetLimitReached).toHaveBeenCalledWith(false);
});
it('should create as many alerts as number of results in parsedResults for grouped alert', async () => {
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'host-1',
count: 291,
hits: [],
},
{
group: 'host-2',
count: 477,
hits: [],
},
{
group: 'host-3',
count: 999,
hits: [],
},
],
truncated: false,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: {
...defaultProps,
threshold: [200],
thresholdComparator: '>=' as Comparator,
groupBy: 'top',
termSize: 10,
termField: 'host.name',
},
});
expect(mockCreateAlert).toHaveBeenCalledTimes(3);
expect(mockCreateAlert).toHaveBeenNthCalledWith(1, 'host-1');
expect(mockCreateAlert).toHaveBeenNthCalledWith(2, 'host-2');
expect(mockCreateAlert).toHaveBeenNthCalledWith(3, 'host-3');
expect(scheduleActions).toHaveBeenCalledTimes(3);
expect(scheduleActions).toHaveBeenNthCalledWith(1, 'query matched', {
conditions:
'Number of matching documents for group "host-1" is greater than or equal to 200',
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is active:
- Value: 291
- Conditions Met: Number of matching documents for group "host-1" is greater than or equal to 200 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' matched query for group host-1",
value: 291,
});
expect(scheduleActions).toHaveBeenNthCalledWith(2, 'query matched', {
conditions:
'Number of matching documents for group "host-2" is greater than or equal to 200',
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is active:
- Value: 477
- Conditions Met: Number of matching documents for group "host-2" is greater than or equal to 200 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' matched query for group host-2",
value: 477,
});
expect(scheduleActions).toHaveBeenNthCalledWith(3, 'query matched', {
conditions:
'Number of matching documents for group "host-3" is greater than or equal to 200',
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is active:
- Value: 999
- Conditions Met: Number of matching documents for group "host-3" is greater than or equal to 200 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' matched query for group host-3",
value: 999,
});
expect(mockSetLimitReached).toHaveBeenCalledTimes(1);
expect(mockSetLimitReached).toHaveBeenCalledWith(false);
});
it('should set limit as reached if results are truncated', async () => {
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'host-1',
count: 291,
hits: [],
},
{
group: 'host-2',
count: 477,
hits: [],
},
{
group: 'host-3',
count: 999,
hits: [],
},
],
truncated: true,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: {
...defaultProps,
threshold: [200],
thresholdComparator: '>=' as Comparator,
groupBy: 'top',
termSize: 10,
termField: 'host.name',
},
});
expect(mockCreateAlert).toHaveBeenCalledTimes(3);
expect(mockCreateAlert).toHaveBeenNthCalledWith(1, 'host-1');
expect(mockCreateAlert).toHaveBeenNthCalledWith(2, 'host-2');
expect(mockCreateAlert).toHaveBeenNthCalledWith(3, 'host-3');
expect(scheduleActions).toHaveBeenCalledTimes(3);
expect(mockSetLimitReached).toHaveBeenCalledTimes(1);
expect(mockSetLimitReached).toHaveBeenCalledWith(true);
});
it('should correctly handle recovered alerts for ungrouped alert', async () => {
const mockSetContext = jest.fn();
mockGetRecoveredAlerts.mockReturnValueOnce([
{
getId: () => 'query matched',
setContext: mockSetContext,
},
]);
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: {
results: [
{
group: 'all documents',
count: 491,
hits: [],
},
],
truncated: false,
},
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: { ...defaultProps, threshold: [500], thresholdComparator: '>=' as Comparator },
});
expect(mockCreateAlert).not.toHaveBeenCalled();
expect(mockSetContext).toHaveBeenCalledTimes(1);
expect(mockSetContext).toHaveBeenNthCalledWith(1, {
conditions: 'Number of matching documents is NOT greater than or equal to 500',
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is recovered:
- Value: 0
- Conditions Met: Number of matching documents is NOT greater than or equal to 500 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' recovered",
value: 0,
});
expect(mockSetLimitReached).toHaveBeenCalledTimes(1);
expect(mockSetLimitReached).toHaveBeenCalledWith(false);
});
it('should correctly handle recovered alerts for grouped alerts', async () => {
const mockSetContext = jest.fn();
mockGetRecoveredAlerts.mockReturnValueOnce([
{
getId: () => 'host-1',
setContext: mockSetContext,
},
{
getId: () => 'host-2',
setContext: mockSetContext,
},
]);
mockFetchEsQuery.mockResolvedValueOnce({
parsedResults: { results: [], truncated: false },
dateStart: new Date().toISOString(),
dateEnd: new Date().toISOString(),
});
await executor(coreMock, {
...defaultExecutorOptions,
// @ts-expect-error
params: {
...defaultProps,
threshold: [200],
thresholdComparator: '>=' as Comparator,
groupBy: 'top',
termSize: 10,
termField: 'host.name',
},
});
expect(mockCreateAlert).not.toHaveBeenCalled();
expect(mockSetContext).toHaveBeenCalledTimes(2);
expect(mockSetContext).toHaveBeenNthCalledWith(1, {
conditions: `Number of matching documents for group "host-1" is NOT greater than or equal to 200`,
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is recovered:
- Value: 0
- Conditions Met: Number of matching documents for group "host-1" is NOT greater than or equal to 200 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' recovered",
value: 0,
});
expect(mockSetContext).toHaveBeenNthCalledWith(2, {
conditions: `Number of matching documents for group "host-2" is NOT greater than or equal to 200`,
date: new Date(mockNow).toISOString(),
hits: [],
link: 'https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id',
message: `rule 'test-rule-name' is recovered:
- Value: 0
- Conditions Met: Number of matching documents for group "host-2" is NOT greater than or equal to 200 over 5m
- Timestamp: ${new Date(mockNow).toISOString()}
- Link: https://localhost:5601/app/management/insightsAndAlerting/triggersActions/rule/test-rule-id`,
title: "rule 'test-rule-name' recovered",
value: 0,
});
expect(mockSetLimitReached).toHaveBeenCalledTimes(1);
expect(mockSetLimitReached).toHaveBeenCalledWith(false);
});
});
describe('tryToParseAsDate', () => {
it.each<[string | number]>([['2019-01-01T00:00:00.000Z'], [1546300800000]])(
'should parse as date correctly',
@ -85,21 +563,4 @@ describe('es_query executor', () => {
).toThrow('invalid format for windowSize: "5r"');
});
});
describe('getContextConditionsDescription', () => {
it('should return conditions correctly', () => {
const result = getContextConditionsDescription(Comparator.GT, [10]);
expect(result).toBe(`Number of matching documents is greater than 10`);
});
it('should return conditions correctly when isRecovered is true', () => {
const result = getContextConditionsDescription(Comparator.GT, [10], true);
expect(result).toBe(`Number of matching documents is NOT greater than 10`);
});
it('should return conditions correctly when multiple thresholds provided', () => {
const result = getContextConditionsDescription(Comparator.BETWEEN, [10, 20], true);
expect(result).toBe(`Number of matching documents is NOT between 10 and 20`);
});
});
});

179
x-pack/plugins/stack_alerts/server/rule_types/es_query/executor.ts
View file

@ -8,14 +8,18 @@ import { sha256 } from 'js-sha256';
import { i18n } from '@kbn/i18n';
import { CoreSetup } from '@kbn/core/server';
import { parseDuration } from '@kbn/alerting-plugin/server';
import { addMessages, EsQueryRuleActionContext } from './action_context';
import { ComparatorFns, getHumanReadableComparator } from '../lib';
import { isGroupAggregation, UngroupedGroupId } from '@kbn/triggers-actions-ui-plugin/common';
import { ComparatorFns } from '../../../common';
import {
addMessages,
EsQueryRuleActionContext,
getContextConditionsDescription,
} from './action_context';
import { ExecutorOptions, OnlyEsQueryRuleParams, OnlySearchSourceRuleParams } from './types';
import { ActionGroupId, ConditionMetAlertInstanceId } from './constants';
import { fetchEsQuery } from './lib/fetch_es_query';
import { EsQueryRuleParams } from './rule_type_params';
import { fetchSearchSourceQuery } from './lib/fetch_search_source_query';
import { Comparator } from '../../../common/comparator_types';
import { isEsQueryRule } from './util';
export async function executor(core: CoreSetup, options: ExecutorOptions<EsQueryRuleParams>) {
@ -31,36 +35,43 @@ export async function executor(core: CoreSetup, options: ExecutorOptions<EsQuery
const { alertFactory, scopedClusterClient, searchSourceClient } = services;
const currentTimestamp = new Date().toISOString();
const publicBaseUrl = core.http.basePath.publicBaseUrl ?? '';
const alertLimit = alertFactory.alertLimit.getValue();
const compareFn = ComparatorFns.get(params.thresholdComparator);
if (compareFn == null) {
throw new Error(getInvalidComparatorError(params.thresholdComparator));
}
let latestTimestamp: string | undefined = tryToParseAsDate(state.latestTimestamp);
// During each rule execution, we run the configured query, get a hit count
// (hits.total) and retrieve up to params.size hits. We
// evaluate the threshold condition using the value of hits.total. If the threshold
// condition is met, the hits are counted toward the query match and we update
// the rule state with the timestamp of the latest hit. In the next execution
// of the rule, the latestTimestamp will be used to gate the query in order to
const isGroupAgg = isGroupAggregation(params.termField);
// For ungrouped queries, we run the configured query during each rule run, get a hit count
// and retrieve up to params.size hits. We evaluate the threshold condition using the
// value of the hit count. If the threshold condition is met, the hits are counted
// toward the query match and we update the rule state with the timestamp of the latest hit.
// In the next run of the rule, the latestTimestamp will be used to gate the query in order to
// avoid counting a document multiple times.
const { numMatches, searchResult, dateStart, dateEnd } = esQueryRule
? await fetchEsQuery(ruleId, name, params as OnlyEsQueryRuleParams, latestTimestamp, {
scopedClusterClient,
logger,
// latestTimestamp will be ignored if set for grouped queries
let latestTimestamp: string | undefined = tryToParseAsDate(state.latestTimestamp);
const { parsedResults, dateStart, dateEnd } = esQueryRule
? await fetchEsQuery({
ruleId,
name,
alertLimit,
params: params as OnlyEsQueryRuleParams,
timestamp: latestTimestamp,
services: {
scopedClusterClient,
logger,
},
})
: await fetchSearchSourceQuery(ruleId, params as OnlySearchSourceRuleParams, latestTimestamp, {
searchSourceClient,
logger,
: await fetchSearchSourceQuery({
ruleId,
alertLimit,
params: params as OnlySearchSourceRuleParams,
latestTimestamp,
services: {
searchSourceClient,
logger,
},
});
// apply the rule condition
const conditionMet = compareFn(numMatches, params.threshold);
const base = publicBaseUrl;
const spacePrefix = spaceId !== 'default' ? `/s/${spaceId}` : '';
const link = esQueryRule
@ -68,56 +79,89 @@ export async function executor(core: CoreSetup, options: ExecutorOptions<EsQuery
: `${base}${spacePrefix}/app/discover#/viewAlert/${ruleId}?from=${dateStart}&to=${dateEnd}&checksum=${getChecksum(
params as OnlyEsQueryRuleParams
)}`;
const baseContext: Omit<EsQueryRuleActionContext, 'conditions'> = {
title: name,
date: currentTimestamp,
value: numMatches,
hits: searchResult.hits.hits,
link,
};
const unmetGroupValues: Record<string, number> = {};
for (const result of parsedResults.results) {
const alertId = result.group;
const value = result.value ?? result.count;
if (conditionMet) {
// group aggregations use the bucket selector agg to compare conditions
// within the ES query, so only 'met' results are returned, therefore we don't need
// to use the compareFn
const met = isGroupAgg ? true : compareFn(value, params.threshold);
if (!met) {
unmetGroupValues[alertId] = value;
continue;
}
const baseContext: Omit<EsQueryRuleActionContext, 'conditions'> = {
title: name,
date: currentTimestamp,
value,
hits: result.hits,
link,
};
const baseActiveContext: EsQueryRuleActionContext = {
...baseContext,
conditions: getContextConditionsDescription(params.thresholdComparator, params.threshold),
conditions: getContextConditionsDescription({
comparator: params.thresholdComparator,
threshold: params.threshold,
aggType: params.aggType,
aggField: params.aggField,
...(isGroupAgg ? { group: alertId } : {}),
}),
} as EsQueryRuleActionContext;
const actionContext = addMessages(name, baseActiveContext, params);
const alertInstance = alertFactory.create(ConditionMetAlertInstanceId);
alertInstance
const actionContext = addMessages({
ruleName: name,
baseContext: baseActiveContext,
params,
...(isGroupAgg ? { group: alertId } : {}),
});
const alert = alertFactory.create(
alertId === UngroupedGroupId && !isGroupAgg ? ConditionMetAlertInstanceId : alertId
);
alert
// store the params we would need to recreate the query that led to this alert instance
.replaceState({ latestTimestamp, dateStart, dateEnd })
.scheduleActions(ActionGroupId, actionContext);
// update the timestamp based on the current search results
const firstValidTimefieldSort = getValidTimefieldSort(
searchResult.hits.hits.find((hit) => getValidTimefieldSort(hit.sort))?.sort
);
if (firstValidTimefieldSort) {
latestTimestamp = firstValidTimefieldSort;
if (!isGroupAgg) {
// update the timestamp based on the current search results
const firstValidTimefieldSort = getValidTimefieldSort(
result.hits.find((hit) => getValidTimefieldSort(hit.sort))?.sort
);
if (firstValidTimefieldSort) {
latestTimestamp = firstValidTimefieldSort;
}
}
// we only create one alert if the condition is met, so we would only ever
// reach the alert limit if the limit is less than 1
alertFactory.alertLimit.setLimitReached(alertLimit < 1);
} else {
alertFactory.alertLimit.setLimitReached(false);
}
alertFactory.alertLimit.setLimitReached(parsedResults.truncated);
const { getRecoveredAlerts } = alertFactory.done();
for (const alert of getRecoveredAlerts()) {
for (const recoveredAlert of getRecoveredAlerts()) {
const alertId = recoveredAlert.getId();
const baseRecoveryContext: EsQueryRuleActionContext = {
...baseContext,
conditions: getContextConditionsDescription(
params.thresholdComparator,
params.threshold,
true
),
title: name,
date: currentTimestamp,
value: unmetGroupValues[alertId] ?? 0,
hits: [],
link,
conditions: getContextConditionsDescription({
comparator: params.thresholdComparator,
threshold: params.threshold,
isRecovered: true,
aggType: params.aggType,
aggField: params.aggField,
...(isGroupAgg ? { group: alertId } : {}),
}),
} as EsQueryRuleActionContext;
const recoveryContext = addMessages(name, baseRecoveryContext, params, true);
alert.setContext(recoveryContext);
const recoveryContext = addMessages({
ruleName: name,
baseContext: baseRecoveryContext,
params,
isRecovered: true,
...(isGroupAgg ? { group: alertId } : {}),
});
recoveredAlert.setContext(recoveryContext);
}
return { latestTimestamp };
}
@ -198,18 +242,3 @@ export function getInvalidComparatorError(comparator: string) {
},
});
}
export function getContextConditionsDescription(
comparator: Comparator,
threshold: number[],
isRecovered: boolean = false
) {
return i18n.translate('xpack.stackAlerts.esQuery.alertTypeContextConditionsDescription', {
defaultMessage: 'Number of matching documents is {negation}{thresholdComparator} {threshold}',
values: {
thresholdComparator: getHumanReadableComparator(comparator),
threshold: threshold.join(' and '),
negation: isRecovered ? 'NOT ' : '',
},
});
}

357
x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_es_query.test.ts Normal file
View file

@ -0,0 +1,357 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { OnlyEsQueryRuleParams } from '../types';
import { Comparator } from '../../../../common/comparator_types';
import { fetchEsQuery } from './fetch_es_query';
import { elasticsearchServiceMock } from '@kbn/core-elasticsearch-server-mocks';
import { loggerMock } from '@kbn/logging-mocks';
import { getSearchParams } from './get_search_params';
jest.mock('@kbn/triggers-actions-ui-plugin/common', () => {
const actual = jest.requireActual('@kbn/triggers-actions-ui-plugin/common');
return {
...actual,
parseAggregationResults: jest.fn(),
};
});
const mockNow = jest.getRealSystemTime();
const defaultParams: OnlyEsQueryRuleParams = {
index: ['test-index'],
size: 100,
timeField: '@timestamp',
timeWindowSize: 5,
timeWindowUnit: 'm',
thresholdComparator: Comparator.LT,
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
};
const logger = loggerMock.create();
const scopedClusterClientMock = elasticsearchServiceMock.createScopedClusterClient();
describe('fetchEsQuery', () => {
beforeAll(() => {
jest.useFakeTimers();
jest.setSystemTime(mockNow);
});
beforeEach(() => {
jest.clearAllMocks();
});
afterAll(() => {
jest.useRealTimers();
});
const services = {
scopedClusterClient: scopedClusterClientMock,
logger,
};
it('should add time filter if timestamp if defined and excludeHitsFromPreviousRun is true', async () => {
const params = defaultParams;
const { dateStart, dateEnd } = getSearchParams(params);
await fetchEsQuery({
ruleId: 'abc',
name: 'test-rule',
params,
timestamp: '2020-02-09T23:15:41.941Z',
services,
});
expect(scopedClusterClientMock.asCurrentUser.search).toHaveBeenCalledWith(
{
allow_no_indices: true,
body: {
aggs: {},
docvalue_fields: [
{
field: '@timestamp',
format: 'strict_date_optional_time',
},
],
query: {
bool: {
filter: [
{
bool: {
filter: [
{
match_all: {},
},
{
bool: {
must_not: [
{
bool: {
filter: [
{
range: {
'@timestamp': {
format: 'strict_date_optional_time',
lte: '2020-02-09T23:15:41.941Z',
},
},
},
],
},
},
],
},
},
],
},
},
{
bool: {
filter: [
{
range: {
'@timestamp': {
format: 'strict_date_optional_time',
gte: dateStart,
lte: dateEnd,
},
},
},
],
},
},
],
},
},
sort: [
{
'@timestamp': {
order: 'desc',
},
},
],
},
ignore_unavailable: true,
index: ['test-index'],
size: 100,
track_total_hits: true,
},
{ meta: true }
);
});
it('should not add time filter if timestamp is undefined', async () => {
const params = defaultParams;
const { dateStart, dateEnd } = getSearchParams(params);
await fetchEsQuery({
ruleId: 'abc',
name: 'test-rule',
params,
timestamp: undefined,
services,
});
expect(scopedClusterClientMock.asCurrentUser.search).toHaveBeenCalledWith(
{
allow_no_indices: true,
body: {
aggs: {},
docvalue_fields: [
{
field: '@timestamp',
format: 'strict_date_optional_time',
},
],
query: {
bool: {
filter: [
{
match_all: {},
},
{
bool: {
filter: [
{
range: {
'@timestamp': {
format: 'strict_date_optional_time',
gte: dateStart,
lte: dateEnd,
},
},
},
],
},
},
],
},
},
sort: [
{
'@timestamp': {
order: 'desc',
},
},
],
},
ignore_unavailable: true,
index: ['test-index'],
size: 100,
track_total_hits: true,
},
{ meta: true }
);
});
it('should not add time filter if excludeHitsFromPreviousRun is false', async () => {
const params = { ...defaultParams, excludeHitsFromPreviousRun: false };
const { dateStart, dateEnd } = getSearchParams(params);
await fetchEsQuery({
ruleId: 'abc',
name: 'test-rule',
params,
timestamp: '2020-02-09T23:15:41.941Z',
services,
});
expect(scopedClusterClientMock.asCurrentUser.search).toHaveBeenCalledWith(
{
allow_no_indices: true,
body: {
aggs: {},
docvalue_fields: [
{
field: '@timestamp',
format: 'strict_date_optional_time',
},
],
query: {
bool: {
filter: [
{
match_all: {},
},
{
bool: {
filter: [
{
range: {
'@timestamp': {
format: 'strict_date_optional_time',
gte: dateStart,
lte: dateEnd,
},
},
},
],
},
},
],
},
},
sort: [
{
'@timestamp': {
order: 'desc',
},
},
],
},
ignore_unavailable: true,
index: ['test-index'],
size: 100,
track_total_hits: true,
},
{ meta: true }
);
});
it('should set size: 0 and top hits size to size parameter if grouping alerts', async () => {
const params = { ...defaultParams, groupBy: 'top', termField: 'host.name', termSize: 10 };
const { dateStart, dateEnd } = getSearchParams(params);
await fetchEsQuery({
ruleId: 'abc',
name: 'test-rule',
params,
timestamp: undefined,
services,
});
expect(scopedClusterClientMock.asCurrentUser.search).toHaveBeenCalledWith(
{
allow_no_indices: true,
body: {
aggs: {
groupAgg: {
aggs: {
conditionSelector: {
bucket_selector: {
buckets_path: {
compareValue: '_count',
},
script: 'params.compareValue < 0L',
},
},
topHitsAgg: {
top_hits: {
size: 100,
},
},
},
terms: {
field: 'host.name',
size: 10,
},
},
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
},
docvalue_fields: [
{
field: '@timestamp',
format: 'strict_date_optional_time',
},
],
query: {
bool: {
filter: [
{
match_all: {},
},
{
bool: {
filter: [
{
range: {
'@timestamp': {
format: 'strict_date_optional_time',
gte: dateStart,
lte: dateEnd,
},
},
},
],
},
},
],
},
},
sort: [
{
'@timestamp': {
order: 'desc',
},
},
],
},
ignore_unavailable: true,
index: ['test-index'],
size: 0,
track_total_hits: true,
},
{ meta: true }
);
});
});

67
x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_es_query.ts
View file

@ -5,27 +5,46 @@
* 2.0.
*/
import { IScopedClusterClient, Logger } from '@kbn/core/server';
import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import {
BUCKET_SELECTOR_FIELD,
buildAggregation,
isCountAggregation,
parseAggregationResults,
} from '@kbn/triggers-actions-ui-plugin/common';
import { isGroupAggregation } from '@kbn/triggers-actions-ui-plugin/common';
import { getComparatorScript } from '../../../../common';
import { OnlyEsQueryRuleParams } from '../types';
import { buildSortedEventsQuery } from '../../../../common/build_sorted_events_query';
import { ES_QUERY_ID } from '../constants';
import { getSearchParams } from './get_search_params';
/**
* Fetching matching documents for a given rule from elasticsearch by a given index and query
*/
export async function fetchEsQuery(
ruleId: string,
name: string,
params: OnlyEsQueryRuleParams,
timestamp: string | undefined,
export interface FetchEsQueryOpts {
ruleId: string;
name: string;
params: OnlyEsQueryRuleParams;
timestamp: string | undefined;
services: {
scopedClusterClient: IScopedClusterClient;
logger: Logger;
}
) {
};
alertLimit?: number;
}
/**
* Fetching matching documents for a given rule from elasticsearch by a given index and query
*/
export async function fetchEsQuery({
ruleId,
name,
params,
timestamp,
services,
alertLimit,
}: FetchEsQueryOpts) {
const { scopedClusterClient, logger } = services;
const esClient = scopedClusterClient.asCurrentUser;
const isGroupAgg = isGroupAggregation(params.termField);
const isCountAgg = isCountAggregation(params.aggType);
const {
// eslint-disable-next-line @typescript-eslint/naming-convention
parsedQuery: { query, fields, runtime_mappings, _source },
@ -69,7 +88,7 @@ export async function fetchEsQuery(
from: dateStart,
to: dateEnd,
filter,
size: params.size,
size: isGroupAgg ? 0 : params.size,
sortOrder: 'desc',
searchAfterSortId: undefined,
timeField: params.timeField,
@ -77,6 +96,21 @@ export async function fetchEsQuery(
fields,
runtime_mappings,
_source,
aggs: buildAggregation({
aggType: params.aggType,
aggField: params.aggField,
termField: params.termField,
termSize: params.termSize,
condition: {
resultLimit: alertLimit,
conditionScript: getComparatorScript(
params.thresholdComparator,
params.threshold,
BUCKET_SELECTOR_FIELD
),
},
...(isGroupAgg ? { topHitsSize: params.size } : {}),
}),
});
logger.debug(
@ -88,9 +122,14 @@ export async function fetchEsQuery(
logger.debug(
` es query rule ${ES_QUERY_ID}:${ruleId} "${name}" result - ${JSON.stringify(searchResult)}`
);
return {
numMatches: (searchResult.hits.total as estypes.SearchTotalHits).value,
searchResult,
parsedResults: parseAggregationResults({
isCountAgg,
isGroupAgg,
esResult: searchResult,
resultLimit: alertLimit,
}),
dateStart,
dateEnd,
};

80
x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.test.ts
View file

@ -38,6 +38,8 @@ const defaultParams: OnlySearchSourceRuleParams = {
searchConfiguration: {},
searchType: 'searchSource',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
timeField: 'time',
};
@ -66,6 +68,7 @@ describe('fetchSearchSourceQuery', () => {
undefined
);
const searchRequest = searchSource.getSearchRequestBody();
expect(searchRequest.size).toMatchInlineSnapshot(`100`);
expect(searchRequest.query).toMatchInlineSnapshot(`
Object {
"bool": Object {
@ -86,6 +89,7 @@ describe('fetchSearchSourceQuery', () => {
},
}
`);
expect(searchRequest.aggs).toMatchInlineSnapshot(`Object {}`);
expect(dateStart).toMatch('2020-02-09T23:10:41.941Z');
expect(dateEnd).toMatch('2020-02-09T23:15:41.941Z');
});
@ -101,6 +105,7 @@ describe('fetchSearchSourceQuery', () => {
'2020-02-09T23:12:41.941Z'
);
const searchRequest = searchSource.getSearchRequestBody();
expect(searchRequest.size).toMatchInlineSnapshot(`100`);
expect(searchRequest.query).toMatchInlineSnapshot(`
Object {
"bool": Object {
@ -128,6 +133,7 @@ describe('fetchSearchSourceQuery', () => {
},
}
`);
expect(searchRequest.aggs).toMatchInlineSnapshot(`Object {}`);
});
it('with latest timestamp in before the given time range ', async () => {
@ -141,6 +147,7 @@ describe('fetchSearchSourceQuery', () => {
'2020-01-09T22:12:41.941Z'
);
const searchRequest = searchSource.getSearchRequestBody();
expect(searchRequest.size).toMatchInlineSnapshot(`100`);
expect(searchRequest.query).toMatchInlineSnapshot(`
Object {
"bool": Object {
@ -161,6 +168,7 @@ describe('fetchSearchSourceQuery', () => {
},
}
`);
expect(searchRequest.aggs).toMatchInlineSnapshot(`Object {}`);
});
it('does not add time range if excludeHitsFromPreviousRun is false', async () => {
@ -174,6 +182,7 @@ describe('fetchSearchSourceQuery', () => {
'2020-02-09T23:12:41.941Z'
);
const searchRequest = searchSource.getSearchRequestBody();
expect(searchRequest.size).toMatchInlineSnapshot(`100`);
expect(searchRequest.query).toMatchInlineSnapshot(`
Object {
"bool": Object {
@ -194,6 +203,77 @@ describe('fetchSearchSourceQuery', () => {
},
}
`);
expect(searchRequest.aggs).toMatchInlineSnapshot(`Object {}`);
});
it('should set size: 0 and top hits size to size parameter if grouping alerts', async () => {
const params = {
...defaultParams,
excludeHitsFromPreviousRun: false,
groupBy: 'top',
termField: 'host.name',
termSize: 10,
};
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { searchSource } = updateSearchSource(
searchSourceInstance,
params,
'2020-02-09T23:12:41.941Z'
);
const searchRequest = searchSource.getSearchRequestBody();
expect(searchRequest.size).toMatchInlineSnapshot(`0`);
expect(searchRequest.query).toMatchInlineSnapshot(`
Object {
"bool": Object {
"filter": Array [
Object {
"range": Object {
"time": Object {
"format": "strict_date_optional_time",
"gte": "2020-02-09T23:10:41.941Z",
"lte": "2020-02-09T23:15:41.941Z",
},
},
},
],
"must": Array [],
"must_not": Array [],
"should": Array [],
},
}
`);
expect(searchRequest.aggs).toMatchInlineSnapshot(`
Object {
"groupAgg": Object {
"aggs": Object {
"conditionSelector": Object {
"bucket_selector": Object {
"buckets_path": Object {
"compareValue": "_count",
},
"script": "params.compareValue < 0L",
},
},
"topHitsAgg": Object {
"top_hits": Object {
"size": 100,
},
},
},
"terms": Object {
"field": "host.name",
"size": 10,
},
},
"groupAggCount": Object {
"stats_bucket": Object {
"buckets_path": "groupAgg._count",
},
},
}
`);
});
});
});

64
x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.ts
View file

@ -12,25 +12,45 @@ import {
ISearchStartSearchSource,
SortDirection,
} from '@kbn/data-plugin/common';
import {
BUCKET_SELECTOR_FIELD,
buildAggregation,
isCountAggregation,
parseAggregationResults,
} from '@kbn/triggers-actions-ui-plugin/common';
import { isGroupAggregation } from '@kbn/triggers-actions-ui-plugin/common';
import { OnlySearchSourceRuleParams } from '../types';
import { getComparatorScript } from '../../../../common';
export async function fetchSearchSourceQuery(
ruleId: string,
params: OnlySearchSourceRuleParams,
latestTimestamp: string | undefined,
export interface FetchSearchSourceQueryOpts {
ruleId: string;
params: OnlySearchSourceRuleParams;
latestTimestamp: string | undefined;
services: {
logger: Logger;
searchSourceClient: ISearchStartSearchSource;
}
) {
logger: Logger;
};
alertLimit?: number;
}
export async function fetchSearchSourceQuery({
ruleId,
params,
latestTimestamp,
services,
alertLimit,
}: FetchSearchSourceQueryOpts) {
const { logger, searchSourceClient } = services;
const isGroupAgg = isGroupAggregation(params.termField);
const isCountAgg = isCountAggregation(params.aggType);
const initialSearchSource = await searchSourceClient.create(params.searchConfiguration);
const { searchSource, dateStart, dateEnd } = updateSearchSource(
initialSearchSource,
params,
latestTimestamp
latestTimestamp,
alertLimit
);
logger.debug(
@ -42,8 +62,7 @@ export async function fetchSearchSourceQuery(
const searchResult = await searchSource.fetch();
return {
numMatches: Number(searchResult.hits.total),
searchResult,
parsedResults: parseAggregationResults({ isCountAgg, isGroupAgg, esResult: searchResult }),
dateStart,
dateEnd,
};
@ -52,15 +71,18 @@ export async function fetchSearchSourceQuery(
export function updateSearchSource(
searchSource: ISearchSource,
params: OnlySearchSourceRuleParams,
latestTimestamp: string | undefined
latestTimestamp: string | undefined,
alertLimit?: number
) {
const isGroupAgg = isGroupAggregation(params.termField);
const index = searchSource.getField('index')!;
const timeFieldName = params.timeField || index.timeFieldName;
if (!timeFieldName) {
throw new Error('Invalid data view without timeFieldName.');
}
searchSource.setField('size', params.size);
searchSource.setField('size', isGroupAgg ? 0 : params.size);
const timerangeFilter = getTime(index, {
from: `now-${params.timeWindowSize}${params.timeWindowUnit}`,
@ -83,6 +105,24 @@ export function updateSearchSource(
const searchSourceChild = searchSource.createChild();
searchSourceChild.setField('filter', filters as Filter[]);
searchSourceChild.setField('sort', [{ [timeFieldName]: SortDirection.desc }]);
searchSourceChild.setField(
'aggs',
buildAggregation({
aggType: params.aggType,
aggField: params.aggField,
termField: params.termField,
termSize: params.termSize,
condition: {
resultLimit: alertLimit,
conditionScript: getComparatorScript(
params.thresholdComparator,
params.threshold,
BUCKET_SELECTOR_FIELD
),
},
...(isGroupAgg ? { topHitsSize: params.size } : {}),
})
);
return {
searchSource: searchSourceChild,
dateStart,

24
x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.test.ts
View file

@ -110,6 +110,8 @@ describe('ruleType', () => {
thresholdComparator: Comparator.LT,
threshold: [0],
searchType: 'esQuery',
aggType: 'count',
groupBy: 'all',
};
expect(ruleType.validate?.params?.validate(params)).toBeTruthy();
@ -129,6 +131,8 @@ describe('ruleType', () => {
thresholdComparator: Comparator.BETWEEN,
threshold: [0],
searchType: 'esQuery',
aggType: 'count',
groupBy: 'all',
};
expect(() => paramsSchema.validate(params)).toThrowErrorMatchingInlineSnapshot(
@ -144,10 +148,12 @@ describe('ruleType', () => {
size: 100,
timeWindowSize: 5,
timeWindowUnit: 'm',
thresholdComparator: Comparator.BETWEEN,
thresholdComparator: Comparator.GT,
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -179,6 +185,8 @@ describe('ruleType', () => {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -226,6 +234,8 @@ describe('ruleType', () => {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -276,6 +286,8 @@ describe('ruleType', () => {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -320,6 +332,8 @@ describe('ruleType', () => {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -393,6 +407,8 @@ describe('ruleType', () => {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -440,6 +456,8 @@ describe('ruleType', () => {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
const ruleServices: RuleExecutorServicesMock = alertsMock.createRuleExecutorServices();
@ -510,6 +528,8 @@ describe('ruleType', () => {
searchConfiguration: {},
searchType: 'searchSource',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
afterAll(() => {
@ -530,6 +550,8 @@ describe('ruleType', () => {
threshold: [0],
esQuery: '',
searchType: 'searchSource',
aggType: 'count',
groupBy: 'all',
};
expect(() => paramsSchema.validate(params)).toThrowErrorMatchingInlineSnapshot(

69
x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.test.ts
View file

@ -6,6 +6,7 @@
*/
import { TypeOf } from '@kbn/config-schema';
import { MAX_GROUPS } from '@kbn/triggers-actions-ui-plugin/server';
import type { Writable } from '@kbn/utility-types';
import { Comparator } from '../../../common/comparator_types';
import {
@ -25,9 +26,11 @@ const DefaultParams: Writable<Partial<EsQueryRuleParams>> = {
threshold: [0],
searchType: 'esQuery',
excludeHitsFromPreviousRun: true,
aggType: 'count',
groupBy: 'all',
};
describe('alertType Params validate()', () => {
describe('ruleType Params validate()', () => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
let params: any;
beforeEach(() => {
@ -129,6 +132,70 @@ describe('alertType Params validate()', () => {
);
});
it('fails for invalid aggType', async () => {
params.aggType = 42;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[aggType]: expected value of type [string] but got [number]"`
);
params.aggType = '-not-a-valid-aggType-';
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[aggType]: invalid aggType: \\"-not-a-valid-aggType-\\""`
);
});
it('fails for invalid aggField', async () => {
params.aggField = 42;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[aggField]: expected value of type [string] but got [number]"`
);
params.aggField = '';
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[aggField]: value has length [0] but it must have a minimum length of [1]."`
);
});
it('fails for invalid termField', async () => {
params.groupBy = 'top';
params.termField = 42;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[termField]: expected value of type [string] but got [number]"`
);
params.termField = '';
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[termField]: value has length [0] but it must have a minimum length of [1]."`
);
});
it('fails for invalid termSize', async () => {
params.groupBy = 'top';
params.termField = 'fee';
params.termSize = 'foo';
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[termSize]: expected value of type [number] but got [string]"`
);
params.termSize = MAX_GROUPS + 1;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[termSize]: must be less than or equal to 1000"`
);
params.termSize = 0;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[termSize]: Value must be equal to or greater than [1]."`
);
});
it('fails for invalid aggType/aggField', async () => {
params.aggType = 'avg';
delete params.aggField;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(
`"[aggField]: must have a value when [aggType] is \\"avg\\""`
);
});
it('fails for invalid timeWindowSize', async () => {
delete params.timeWindowSize;
expect(onValidate()).toThrowErrorMatchingInlineSnapshot(

62
x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type_params.ts
View file

@ -7,11 +7,16 @@
import { i18n } from '@kbn/i18n';
import { schema, TypeOf } from '@kbn/config-schema';
import { validateTimeWindowUnits } from '@kbn/triggers-actions-ui-plugin/server';
import {
validateTimeWindowUnits,
validateAggType,
validateGroupBy,
MAX_GROUPS,
} from '@kbn/triggers-actions-ui-plugin/server';
import { RuleTypeState } from '@kbn/alerting-plugin/server';
import { SerializedSearchSourceFields } from '@kbn/data-plugin/common';
import { ComparatorFnNames } from '../../../common';
import { Comparator } from '../../../common/comparator_types';
import { ComparatorFnNames } from '../lib';
import { getComparatorSchemaType } from '../lib/comparator';
export const ES_QUERY_MAX_HITS_PER_EXECUTION = 10000;
@ -35,6 +40,16 @@ const EsQueryRuleParamsSchemaProperties = {
timeWindowUnit: schema.string({ validate: validateTimeWindowUnits }),
threshold: schema.arrayOf(schema.number(), { minSize: 1, maxSize: 2 }),
thresholdComparator: getComparatorSchemaType(validateComparator),
// aggregation type
aggType: schema.string({ validate: validateAggType }),
// aggregation field
aggField: schema.maybe(schema.string({ minLength: 1 })),
// how to group
groupBy: schema.string({ validate: validateGroupBy }),
// field to group on (for groupBy: top)
termField: schema.maybe(schema.string({ minLength: 1 })),
// limit on number of groups returned
termSize: schema.maybe(schema.number({ min: 1 })),
searchType: schema.oneOf([schema.literal('searchSource'), schema.literal('esQuery')], {
defaultValue: 'esQuery',
}),
@ -74,7 +89,17 @@ const betweenComparators = new Set(['between', 'notBetween']);
// using direct type not allowed, circular reference, so body is typed to any
function validateParams(anyParams: unknown): string | undefined {
const { esQuery, thresholdComparator, threshold, searchType } = anyParams as EsQueryRuleParams;
const {
esQuery,
thresholdComparator,
threshold,
searchType,
aggType,
aggField,
groupBy,
termField,
termSize,
} = anyParams as EsQueryRuleParams;
if (betweenComparators.has(thresholdComparator) && threshold.length === 1) {
return i18n.translate('xpack.stackAlerts.esQuery.invalidThreshold2ErrorMessage', {
@ -86,6 +111,37 @@ function validateParams(anyParams: unknown): string | undefined {
});
}
if (aggType !== 'count' && !aggField) {
return i18n.translate('xpack.stackAlerts.esQuery.aggTypeRequiredErrorMessage', {
defaultMessage: '[aggField]: must have a value when [aggType] is "{aggType}"',
values: {
aggType,
},
});
}
// check grouping
if (groupBy === 'top') {
if (termField == null) {
return i18n.translate('xpack.stackAlerts.esQuery.termFieldRequiredErrorMessage', {
defaultMessage: '[termField]: termField required when [groupBy] is top',
});
}
if (termSize == null) {
return i18n.translate('xpack.stackAlerts.esQuery.termSizeRequiredErrorMessage', {
defaultMessage: '[termSize]: termSize required when [groupBy] is top',
});
}
if (termSize > MAX_GROUPS) {
return i18n.translate('xpack.stackAlerts.esQuery.invalidTermSizeMaximumErrorMessage', {
defaultMessage: '[termSize]: must be less than or equal to {maxGroups}',
values: {
maxGroups: MAX_GROUPS,
},
});
}
}
if (searchType === 'searchSource') {
return;
}

12
x-pack/plugins/stack_alerts/server/rule_types/index_threshold/rule_type.ts
View file

@ -11,12 +11,16 @@ import {
TimeSeriesQuery,
TIME_SERIES_BUCKET_SELECTOR_FIELD,
} from '@kbn/triggers-actions-ui-plugin/server';
import { isGroupAggregation } from '@kbn/triggers-actions-ui-plugin/common';
import { RuleType, RuleExecutorOptions, StackAlertsStartDeps } from '../../types';
import { Params, ParamsSchema } from './rule_type_params';
import { ActionContext, BaseActionContext, addMessages } from './action_context';
import { STACK_ALERTS_FEATURE_ID } from '../../../common';
import { ComparatorFns, getHumanReadableComparator } from '../lib';
import { getComparatorScript } from '../lib/comparator';
import {
ComparatorFns,
getComparatorScript,
getHumanReadableComparator,
STACK_ALERTS_FEATURE_ID,
} from '../../../common';
export const ID = '.index-threshold';
export const ActionGroupId = 'threshold met';
@ -192,7 +196,7 @@ export function getRuleType(
});
logger.debug(`rule ${ID}:${ruleId} "${name}" query result: ${JSON.stringify(result)}`);
const isGroupAgg = !!queryParams.termField;
const isGroupAgg = isGroupAggregation(queryParams.termField);
const unmetGroupValues: Record<string, number> = {};
const agg = params.aggField ? `${params.aggType}(${params.aggField})` : `${params.aggType}`;

2
x-pack/plugins/stack_alerts/server/rule_types/index_threshold/rule_type_params.ts
View file

@ -11,7 +11,7 @@ import {
CoreQueryParamsSchemaProperties,
validateCoreQueryBody,
} from '@kbn/triggers-actions-ui-plugin/server';
import { ComparatorFnNames } from '../lib';
import { ComparatorFnNames } from '../../../common';
import { Comparator } from '../../../common/comparator_types';
import { getComparatorSchemaType } from '../lib/comparator';

74
x-pack/plugins/stack_alerts/server/rule_types/lib/comparator.ts
View file

@ -4,75 +4,9 @@
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { schema } from '@kbn/config-schema';
import { Comparator } from '../../../common/comparator_types';
export type ComparatorFn = (value: number, threshold: number[]) => boolean;
const humanReadableComparators = new Map<Comparator, string>([
[Comparator.LT, 'less than'],
[Comparator.LT_OR_EQ, 'less than or equal to'],
[Comparator.GT_OR_EQ, 'greater than or equal to'],
[Comparator.GT, 'greater than'],
[Comparator.BETWEEN, 'between'],
[Comparator.NOT_BETWEEN, 'not between'],
]);
export const ComparatorFns = new Map<Comparator, ComparatorFn>([
[Comparator.LT, (value: number, threshold: number[]) => value < threshold[0]],
[Comparator.LT_OR_EQ, (value: number, threshold: number[]) => value <= threshold[0]],
[Comparator.GT_OR_EQ, (value: number, threshold: number[]) => value >= threshold[0]],
[Comparator.GT, (value: number, threshold: number[]) => value > threshold[0]],
[
Comparator.BETWEEN,
(value: number, threshold: number[]) => value >= threshold[0] && value <= threshold[1],
],
[
Comparator.NOT_BETWEEN,
(value: number, threshold: number[]) => value < threshold[0] || value > threshold[1],
],
]);
export const getComparatorScript = (
comparator: Comparator,
threshold: number[],
fieldName: string
) => {
if (threshold.length === 0) {
throw new Error('Threshold value required');
}
function getThresholdString(thresh: number) {
return Number.isInteger(thresh) ? `${thresh}L` : `${thresh}`;
}
switch (comparator) {
case Comparator.LT:
return `${fieldName} < ${getThresholdString(threshold[0])}`;
case Comparator.LT_OR_EQ:
return `${fieldName} <= ${getThresholdString(threshold[0])}`;
case Comparator.GT:
return `${fieldName} > ${getThresholdString(threshold[0])}`;
case Comparator.GT_OR_EQ:
return `${fieldName} >= ${getThresholdString(threshold[0])}`;
case Comparator.BETWEEN:
if (threshold.length < 2) {
throw new Error('Threshold values required');
}
return `${fieldName} >= ${getThresholdString(
threshold[0]
)} && ${fieldName} <= ${getThresholdString(threshold[1])}`;
case Comparator.NOT_BETWEEN:
if (threshold.length < 2) {
throw new Error('Threshold values required');
}
return `${fieldName} < ${getThresholdString(
threshold[0]
)} || ${fieldName} > ${getThresholdString(threshold[1])}`;
}
};
export const getComparatorSchemaType = (validate: (comparator: Comparator) => string | void) =>
schema.oneOf(
[
@ -85,11 +19,3 @@ export const getComparatorSchemaType = (validate: (comparator: Comparator) => st
],
{ validate }
);
export const ComparatorFnNames = new Set(ComparatorFns.keys());
export function getHumanReadableComparator(comparator: Comparator) {
return humanReadableComparators.has(comparator)
? humanReadableComparators.get(comparator)
: comparator;
}

8
x-pack/plugins/stack_alerts/server/rule_types/lib/index.ts
View file

@ -1,8 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { ComparatorFns, ComparatorFnNames, getHumanReadableComparator } from './comparator';

1
x-pack/plugins/translations/translations/fr-FR.json
View file

@ -32308,7 +32308,6 @@
"xpack.spaces.spacesTitle": "Espaces",
"xpack.spaces.uiApi.errorBoundaryToastMessage": "Rechargez la page pour continuer.",
"xpack.spaces.uiApi.errorBoundaryToastTitle": "Impossible de charger la ressource Kibana",
"xpack.stackAlerts.esQuery.alertTypeContextConditionsDescription": "Le nombre de documents correspondants est {negation} {thresholdComparator} {threshold}",
"xpack.stackAlerts.esQuery.alertTypeContextMessageDescription": "La règle {name} est {verb} :\n\n- Valeur : {value}\n- Conditions remplies : {conditions} sur {window}\n- Horodatage : {date}\n- Lien : {link}",
"xpack.stackAlerts.esQuery.alertTypeContextSubjectTitle": "règle '{name}' : {verb}",
"xpack.stackAlerts.esQuery.invalidComparatorErrorMessage": "thresholdComparator spécifié non valide : {comparator}",

1
x-pack/plugins/translations/translations/ja-JP.json
View file

@ -32278,7 +32278,6 @@
"xpack.spaces.spacesTitle": "スペース",
"xpack.spaces.uiApi.errorBoundaryToastMessage": "続行するにはページを再読み込みしてください。",
"xpack.spaces.uiApi.errorBoundaryToastTitle": "Kibanaアセットを読み込めませんでした",
"xpack.stackAlerts.esQuery.alertTypeContextConditionsDescription": "一致するドキュメント数は{negation}{thresholdComparator} {threshold}です",
"xpack.stackAlerts.esQuery.alertTypeContextMessageDescription": "ルール'{name}'は{verb}です。\n\n- 値：{value}\n- 条件が満たされました：{window} の {conditions}\n- タイムスタンプ：{date}\n- リンク：{link}",
"xpack.stackAlerts.esQuery.alertTypeContextSubjectTitle": "ルール'{name}' {verb}",
"xpack.stackAlerts.esQuery.invalidComparatorErrorMessage": "無効な thresholdComparator が指定されました：{comparator}",

1
x-pack/plugins/translations/translations/zh-CN.json
View file

@ -32312,7 +32312,6 @@
"xpack.spaces.spacesTitle": "工作区",
"xpack.spaces.uiApi.errorBoundaryToastMessage": "重新加载页面以继续。",
"xpack.spaces.uiApi.errorBoundaryToastTitle": "无法加载 Kibana 资产",
"xpack.stackAlerts.esQuery.alertTypeContextConditionsDescription": "匹配文档的数目为 {negation}{thresholdComparator} {threshold}",
"xpack.stackAlerts.esQuery.alertTypeContextMessageDescription": "规则“{name}”为 {verb}：\n\n- 值：{value}\n- 满足的条件：{conditions} 超过 {window}\n- 时间戳：{date}\n- 链接：{link}",
"xpack.stackAlerts.esQuery.alertTypeContextSubjectTitle": "规则“{name}”{verb}",
"xpack.stackAlerts.esQuery.invalidComparatorErrorMessage": "指定的 thresholdComparator 无效：{comparator}",

3
x-pack/plugins/triggers_actions_ui/common/data/index.ts
View file

@ -5,6 +5,9 @@
* 2.0.
*/
export * from './lib/date_range_info';
export * from './lib/build_agg';
export * from './lib/parse_aggregation_results';
export interface TimeSeriesResult {
results: TimeSeriesResultRow[];
truncated: boolean;

861
x-pack/plugins/triggers_actions_ui/common/data/lib/build_agg.test.ts Normal file
View file

@ -0,0 +1,861 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { buildAggregation } from './build_agg';
describe('buildAgg', () => {
describe('count over all (aggType = count and termField is undefined)', () => {
it('should create correct aggregation when condition params are undefined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'count',
aggField: undefined,
termField: undefined,
termSize: undefined,
})
).toEqual({
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
},
});
});
it('should create correct aggregation when condition params are undefined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'count',
aggField: undefined,
termField: undefined,
termSize: undefined,
})
).toEqual({});
});
it('should create correct aggregation when condition params are defined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'count',
aggField: undefined,
termField: undefined,
termSize: undefined,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'count',
aggField: undefined,
termField: undefined,
termSize: undefined,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({});
});
it('should not add top hits aggregation even if topHitsSize is specified', async () => {
expect(
buildAggregation({
aggType: 'count',
aggField: undefined,
termField: undefined,
termSize: undefined,
topHitsSize: 10,
})
).toEqual({});
});
});
describe('count over top N termField (aggType = count and termField is specified)', () => {
it('should create correct aggregation when condition params are undefined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'count',
aggField: undefined,
termField: 'the-term',
termSize: 10,
})
).toEqual({
groupAgg: {
terms: {
field: 'the-term',
size: 10,
},
aggs: {
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
},
},
},
});
});
it('should create correct aggregation when condition params are undefined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'count',
aggField: undefined,
termField: 'the-term',
termSize: 10,
})
).toEqual({
groupAgg: {
terms: {
field: 'the-term',
size: 10,
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'count',
aggField: undefined,
termField: 'the-term',
termSize: 10,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
groupAgg: {
terms: {
field: 'the-term',
size: 10,
},
aggs: {
conditionSelector: {
bucket_selector: {
buckets_path: {
compareValue: '_count',
},
script: `params.compareValue > 1`,
},
},
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
},
},
},
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'count',
aggField: undefined,
termField: 'the-term',
termSize: 10,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
groupAgg: {
terms: {
field: 'the-term',
size: 10,
},
aggs: {
conditionSelector: {
bucket_selector: {
buckets_path: {
compareValue: '_count',
},
script: `params.compareValue > 1`,
},
},
},
},
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
});
});
it('should add top hits aggregation if topHitsSize is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'count',
aggField: undefined,
termField: 'the-term',
termSize: 10,
topHitsSize: 15,
})
).toEqual({
groupAgg: {
terms: {
field: 'the-term',
size: 10,
},
aggs: {
topHitsAgg: {
top_hits: {
size: 15,
},
},
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
},
},
},
});
});
});
describe('aggregate metric over all (aggType != count and termField is undefined)', () => {
it('should create correct aggregation when condition params are undefined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'avg',
aggField: 'avg-field',
termField: undefined,
termSize: undefined,
})
).toEqual({
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
sortValueAgg: {
avg: {
field: 'avg-field',
},
},
});
});
it('should create correct aggregation when condition params are undefined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'avg',
aggField: 'avg-field',
termField: undefined,
termSize: undefined,
})
).toEqual({
metricAgg: {
avg: {
field: 'avg-field',
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'avg',
aggField: 'avg-field',
termField: undefined,
termSize: undefined,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
sortValueAgg: {
avg: {
field: 'avg-field',
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'avg',
aggField: 'avg-field',
termField: undefined,
termSize: undefined,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
metricAgg: {
avg: {
field: 'avg-field',
},
},
});
});
it('should not add top hits aggregation even if topHitsSize is specified', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'avg',
aggField: 'avg-field',
termField: undefined,
termSize: undefined,
})
).toEqual({
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
sortValueAgg: {
avg: {
field: 'avg-field',
},
},
});
});
});
describe('aggregate metric over top N termField (aggType != count and termField is specified)', () => {
it('should create correct aggregation when condition params are undefined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'avg',
aggField: 'avg-field',
termField: 'the-field',
termSize: 20,
})
).toEqual({
groupAgg: {
terms: {
field: 'the-field',
order: {
sortValueAgg: 'desc',
},
size: 20,
},
aggs: {
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
sortValueAgg: {
avg: {
field: 'avg-field',
},
},
},
},
});
});
it('should create correct aggregation when condition params are undefined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'avg',
aggField: 'avg-field',
termField: 'the-field',
termSize: 20,
})
).toEqual({
groupAgg: {
terms: {
field: 'the-field',
order: {
metricAgg: 'desc',
},
size: 20,
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is defined', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'avg',
aggField: 'avg-field',
termField: 'the-field',
termSize: 20,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
groupAgg: {
terms: {
field: 'the-field',
order: {
sortValueAgg: 'desc',
},
size: 20,
},
aggs: {
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
conditionSelector: {
bucket_selector: {
buckets_path: {
compareValue: 'sortValueAgg',
},
script: 'params.compareValue > 1',
},
},
sortValueAgg: {
avg: {
field: 'avg-field',
},
},
},
},
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
});
});
it('should create correct aggregation when condition params are defined and timeSeries is undefined', async () => {
expect(
buildAggregation({
aggType: 'avg',
aggField: 'avg-field',
termField: 'the-field',
termSize: 20,
condition: {
resultLimit: 1000,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
groupAgg: {
terms: {
field: 'the-field',
order: {
metricAgg: 'desc',
},
size: 20,
},
aggs: {
conditionSelector: {
bucket_selector: {
buckets_path: {
compareValue: 'metricAgg',
},
script: 'params.compareValue > 1',
},
},
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
});
});
it('should add topHitsAgg if topHitsSize is defined', () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'avg',
aggField: 'avg-field',
termField: 'the-field',
termSize: 20,
topHitsSize: 15,
})
).toEqual({
groupAgg: {
terms: {
field: 'the-field',
order: {
sortValueAgg: 'desc',
},
size: 20,
},
aggs: {
topHitsAgg: {
top_hits: {
size: 15,
},
},
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
aggs: {
metricAgg: {
avg: {
field: 'avg-field',
},
},
},
},
sortValueAgg: {
avg: {
field: 'avg-field',
},
},
},
},
});
});
});
it('should correctly apply the resultLimit if specified', async () => {
expect(
buildAggregation({
timeSeries: {
timeField: 'time-field',
timeWindowSize: 5,
timeWindowUnit: 'm',
dateStart: '2021-04-22T15:19:31Z',
dateEnd: '2021-04-22T15:20:31Z',
interval: '1m',
},
aggType: 'count',
aggField: undefined,
termField: 'the-term',
termSize: 100,
condition: {
resultLimit: 5,
conditionScript: `params.compareValue > 1`,
},
})
).toEqual({
groupAgg: {
terms: {
field: 'the-term',
size: 6,
},
aggs: {
conditionSelector: {
bucket_selector: {
buckets_path: {
compareValue: '_count',
},
script: `params.compareValue > 1`,
},
},
dateAgg: {
date_range: {
field: 'time-field',
format: 'strict_date_time',
ranges: [
{
from: '2021-04-22T15:14:31.000Z',
to: '2021-04-22T15:19:31.000Z',
},
{
from: '2021-04-22T15:15:31.000Z',
to: '2021-04-22T15:20:31.000Z',
},
],
},
},
},
},
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
});
});
});

187
x-pack/plugins/triggers_actions_ui/common/data/lib/build_agg.ts Normal file
View file

@ -0,0 +1,187 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { AggregationsAggregationContainer } from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import { DateRangeInfo, getDateRangeInfo } from './date_range_info';
export interface BuildAggregationOpts {
timeSeries?: {
timeField: string;
dateStart?: string;
dateEnd?: string;
interval?: string;
timeWindowSize: number;
timeWindowUnit: string;
};
aggType: string;
aggField?: string;
termSize?: number;
termField?: string;
topHitsSize?: number;
condition?: {
resultLimit?: number;
conditionScript: string;
};
}
export const BUCKET_SELECTOR_PATH_NAME = 'compareValue';
export const BUCKET_SELECTOR_FIELD = `params.${BUCKET_SELECTOR_PATH_NAME}`;
export const DEFAULT_GROUPS = 100;
export const isCountAggregation = (aggType: string) => aggType === 'count';
export const isGroupAggregation = (termField?: string) => !!termField;
export const buildAggregation = ({
timeSeries,
aggType,
aggField,
termField,
termSize,
condition,
topHitsSize,
}: BuildAggregationOpts): Record<string, AggregationsAggregationContainer> => {
const aggContainer = {
aggs: {},
};
const isCountAgg = isCountAggregation(aggType);
const isGroupAgg = isGroupAggregation(termField);
const isDateAgg = !!timeSeries;
const includeConditionInQuery = !!condition;
let dateRangeInfo: DateRangeInfo | null = null;
if (isDateAgg) {
const { timeWindowSize, timeWindowUnit, dateStart, dateEnd, interval } = timeSeries;
const window = `${timeWindowSize}${timeWindowUnit}`;
dateRangeInfo = getDateRangeInfo({ dateStart, dateEnd, window, interval });
}
// Cap the maximum number of terms returned to the resultLimit if defined
// Use resultLimit + 1 because we're using the bucket selector aggregation
// to apply the threshold condition to the ES query. We don't seem to be
// able to get the true cardinality from the bucket selector (i.e., get
// the number of buckets that matched the selector condition without actually
// retrieving the bucket data). By using resultLimit + 1, we can count the number
// of buckets returned and if the value is greater than resultLimit, we know that
// there is additional alert data that we're not returning.
let terms = termSize || DEFAULT_GROUPS;
terms =
includeConditionInQuery && condition.resultLimit
? terms > condition.resultLimit
? condition.resultLimit + 1
: terms
: terms;
let aggParent: any = aggContainer;
const getAggName = () => (isDateAgg ? 'sortValueAgg' : 'metricAgg');
// first, add a group aggregation, if requested
if (isGroupAgg) {
aggParent.aggs = {
groupAgg: {
terms: {
field: termField,
size: terms,
},
},
...(includeConditionInQuery
? {
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
}
: {}),
};
// if not count add an order
if (!isCountAgg) {
const sortOrder = aggType === 'min' ? 'asc' : 'desc';
aggParent.aggs.groupAgg.terms!.order = {
[getAggName()]: sortOrder,
};
} else if (includeConditionInQuery) {
aggParent.aggs.groupAgg.aggs = {
conditionSelector: {
bucket_selector: {
buckets_path: {
[BUCKET_SELECTOR_PATH_NAME]: '_count',
},
script: condition.conditionScript,
},
},
};
}
aggParent = aggParent.aggs.groupAgg;
}
// next, add the time window aggregation
if (isDateAgg) {
aggParent.aggs = {
...aggParent.aggs,
dateAgg: {
date_range: {
field: timeSeries.timeField,
format: 'strict_date_time',
ranges: dateRangeInfo!.dateRanges,
},
},
};
}
if (isGroupAgg && topHitsSize) {
aggParent.aggs = {
...aggParent.aggs,
topHitsAgg: {
top_hits: {
size: topHitsSize,
},
},
};
}
// if not count, add a sorted value agg
if (!isCountAgg) {
aggParent.aggs = {
...aggParent.aggs,
[getAggName()]: {
[aggType]: {
field: aggField,
},
},
};
if (isGroupAgg && includeConditionInQuery) {
aggParent.aggs.conditionSelector = {
bucket_selector: {
buckets_path: {
[BUCKET_SELECTOR_PATH_NAME]: getAggName(),
},
script: condition.conditionScript,
},
};
}
}
if (timeSeries && dateRangeInfo) {
aggParent = aggParent.aggs.dateAgg;
// finally, the metric aggregation, if requested
if (!isCountAgg) {
aggParent.aggs = {
metricAgg: {
[aggType]: {
field: aggField,
},
},
};
}
}
return aggContainer.aggs;
};

0
x-pack/plugins/triggers_actions_ui/server/data/lib/date_range_info.test.ts → x-pack/plugins/triggers_actions_ui/common/data/lib/date_range_info.test.ts
View file

5
x-pack/plugins/triggers_actions_ui/server/data/lib/date_range_info.ts → x-pack/plugins/triggers_actions_ui/common/data/lib/date_range_info.ts
View file

@ -5,10 +5,11 @@
* 2.0.
*/
import { parseDuration } from '@kbn/alerting-plugin/common';
import { i18n } from '@kbn/i18n';
import { times } from 'lodash';
import { parseDuration } from '@kbn/alerting-plugin/server';
import { MAX_INTERVALS } from '..';
export const MAX_INTERVALS = 1000;
// dates as numbers are epoch millis
// dates as strings are ISO

617
x-pack/plugins/triggers_actions_ui/common/data/lib/parse_aggregation_results.test.ts Normal file
View file

@ -0,0 +1,617 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { parseAggregationResults } from './parse_aggregation_results';
const sampleHit = {
_index: '.kibana-event-log-8.6.0-000001',
_id: 'RSPAXYQB0WpSSRUF7Vm1',
_score: null,
_source: {
'@timestamp': '2022-11-09T18:57:14.918Z',
event: {
provider: 'alerting',
action: 'execute-start',
kind: 'alert',
category: ['stackAlerts'],
start: '2022-11-09T18:57:14.918Z',
},
kibana: {
alert: {
rule: {
rule_type_id: '.es-query',
consumer: 'alerts',
execution: {
uuid: '52d62e0b-d66e-4739-911f-e3aa96eaa99a',
},
},
},
saved_objects: [
{
rel: 'primary',
type: 'alert',
id: 'ca8d9770-605f-11ed-8444-a37ecf51689e',
type_id: '.es-query',
},
],
space_ids: ['default'],
task: {
scheduled: '2022-11-09T18:57:13.614Z',
schedule_delay: 1304000000,
},
server_uuid: '5b2de169-2785-441b-ae8c-186a1936b17d',
version: '8.6.0',
},
rule: {
id: 'ca8d9770-605f-11ed-8444-a37ecf51689e',
license: 'basic',
category: '.es-query',
ruleset: 'stackAlerts',
},
message: 'rule execution start: "ca8d9770-605f-11ed-8444-a37ecf51689e"',
ecs: {
version: '1.8.0',
},
},
fields: {
'@timestamp': ['2022-11-09T18:57:14.918Z'],
},
sort: [1668020234918],
};
describe('parseAggregationResults', () => {
it('correctly parses results for count over all', () => {
expect(
parseAggregationResults({
isCountAgg: true,
isGroupAgg: false,
esResult: {
took: 238,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: {
total: 491,
max_score: null,
hits: [sampleHit],
},
},
})
).toEqual({
results: [
{
group: 'all documents',
count: 491,
hits: [sampleHit],
},
],
truncated: false,
});
});
it('correctly parses results for count over top N termField', () => {
expect(
parseAggregationResults({
isCountAgg: true,
isGroupAgg: true,
esResult: {
took: 233,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: { total: 643, max_score: null, hits: [] },
aggregations: {
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 103,
buckets: [
{
key: 'execute',
doc_count: 120,
},
{
key: 'execute-start',
doc_count: 120,
},
{
key: 'active-instance',
doc_count: 100,
},
{
key: 'execute-action',
doc_count: 100,
},
{
key: 'new-instance',
doc_count: 100,
},
],
},
},
},
})
).toEqual({
results: [
{
group: 'execute',
count: 120,
hits: [],
},
{
group: 'execute-start',
count: 120,
hits: [],
},
{
group: 'active-instance',
count: 100,
hits: [],
},
{
group: 'execute-action',
count: 100,
hits: [],
},
{
group: 'new-instance',
count: 100,
hits: [],
},
],
truncated: false,
});
});
it('correctly parses results for count over top N termField with topHits', () => {
expect(
parseAggregationResults({
isCountAgg: true,
isGroupAgg: true,
esResult: {
took: 233,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: { total: 643, max_score: null, hits: [] },
aggregations: {
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 103,
buckets: [
{
key: 'execute',
doc_count: 120,
topHitsAgg: {
hits: {
total: {
value: 120,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
},
{
key: 'execute-start',
doc_count: 120,
topHitsAgg: {
hits: {
total: {
value: 120,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
},
{
key: 'active-instance',
doc_count: 100,
topHitsAgg: {
hits: {
total: {
value: 100,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
},
{
key: 'execute-action',
doc_count: 100,
topHitsAgg: {
hits: {
total: {
value: 100,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
},
{
key: 'new-instance',
doc_count: 100,
topHitsAgg: {
hits: {
total: {
value: 100,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
},
],
},
},
},
})
).toEqual({
results: [
{
group: 'execute',
count: 120,
hits: [sampleHit],
},
{
group: 'execute-start',
count: 120,
hits: [sampleHit],
},
{
group: 'active-instance',
count: 100,
hits: [sampleHit],
},
{
group: 'execute-action',
count: 100,
hits: [sampleHit],
},
{
group: 'new-instance',
count: 100,
hits: [sampleHit],
},
],
truncated: false,
});
});
it('correctly parses results for aggregate metric over all', () => {
expect(
parseAggregationResults({
isCountAgg: false,
isGroupAgg: false,
esResult: {
took: 238,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: { total: 643, max_score: null, hits: [sampleHit] },
aggregations: {
metricAgg: {
value: 3578195238.095238,
},
},
},
})
).toEqual({
results: [
{
group: 'all documents',
hits: [sampleHit],
count: 643,
value: 3578195238.095238,
},
],
truncated: false,
});
});
it('correctly parses results for aggregate metric over top N termField', () => {
expect(
parseAggregationResults({
isCountAgg: false,
isGroupAgg: true,
esResult: {
took: 238,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: { total: 643, max_score: null, hits: [] },
aggregations: {
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 240,
buckets: [
{
key: 'execute-action',
doc_count: 120,
metricAgg: {
value: null,
},
},
{
key: 'execute-start',
doc_count: 139,
metricAgg: {
value: null,
},
},
{
key: 'starting',
doc_count: 1,
metricAgg: {
value: null,
},
},
{
key: 'recovered-instance',
doc_count: 120,
metricAgg: {
value: 12837500000,
},
},
{
key: 'execute',
doc_count: 139,
metricAgg: {
value: 137647482.0143885,
},
},
],
},
},
},
})
).toEqual({
results: [
{
group: 'execute-action',
count: 120,
hits: [],
value: null,
},
{
group: 'execute-start',
count: 139,
hits: [],
value: null,
},
{
group: 'starting',
count: 1,
hits: [],
value: null,
},
{
group: 'recovered-instance',
count: 120,
hits: [],
value: 12837500000,
},
{
group: 'execute',
count: 139,
hits: [],
value: 137647482.0143885,
},
],
truncated: false,
});
});
it('correctly parses results for aggregate metric over top N termField with topHits', () => {
expect(
parseAggregationResults({
isCountAgg: false,
isGroupAgg: true,
esResult: {
took: 238,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: { total: 643, max_score: null, hits: [] },
aggregations: {
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 240,
buckets: [
{
key: 'execute-action',
doc_count: 120,
topHitsAgg: {
hits: {
total: {
value: 120,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
metricAgg: {
value: null,
},
},
{
key: 'execute-start',
doc_count: 139,
topHitsAgg: {
hits: {
total: {
value: 139,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
metricAgg: {
value: null,
},
},
{
key: 'starting',
doc_count: 1,
topHitsAgg: {
hits: {
total: {
value: 1,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
metricAgg: {
value: null,
},
},
{
key: 'recovered-instance',
doc_count: 120,
topHitsAgg: {
hits: {
total: {
value: 120,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
metricAgg: {
value: 12837500000,
},
},
{
key: 'execute',
doc_count: 139,
topHitsAgg: {
hits: {
total: {
value: 139,
relation: 'eq',
},
max_score: 0,
hits: [sampleHit],
},
},
metricAgg: {
value: 137647482.0143885,
},
},
],
},
},
},
})
).toEqual({
results: [
{
group: 'execute-action',
count: 120,
hits: [sampleHit],
value: null,
},
{
group: 'execute-start',
count: 139,
hits: [sampleHit],
value: null,
},
{
group: 'starting',
count: 1,
hits: [sampleHit],
value: null,
},
{
group: 'recovered-instance',
count: 120,
hits: [sampleHit],
value: 12837500000,
},
{
group: 'execute',
count: 139,
hits: [sampleHit],
value: 137647482.0143885,
},
],
truncated: false,
});
});
it('correctly returns truncated status when resultLimit is reached', () => {
expect(
parseAggregationResults({
isCountAgg: true,
isGroupAgg: true,
esResult: {
took: 233,
timed_out: false,
_shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
hits: { total: 643, max_score: null, hits: [] },
aggregations: {
groupAggCount: {
count: 5,
},
groupAgg: {
doc_count_error_upper_bound: 0,
sum_other_doc_count: 103,
buckets: [
{
key: 'execute',
doc_count: 120,
},
{
key: 'execute-start',
doc_count: 120,
},
{
key: 'active-instance',
doc_count: 100,
},
{
key: 'execute-action',
doc_count: 100,
},
{
key: 'new-instance',
doc_count: 100,
},
],
},
},
},
resultLimit: 3,
})
).toEqual({
results: [
{
group: 'execute',
count: 120,
hits: [],
},
{
group: 'execute-start',
count: 120,
hits: [],
},
{
group: 'active-instance',
count: 100,
hits: [],
},
],
truncated: true,
});
});
});

94
x-pack/plugins/triggers_actions_ui/common/data/lib/parse_aggregation_results.ts Normal file
View file

@ -0,0 +1,94 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import {
SearchResponse,
SearchHit,
SearchHitsMetadata,
AggregationsSingleMetricAggregateBase,
} from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
export const UngroupedGroupId = 'all documents';
export interface ParsedAggregationGroup {
group: string;
count: number;
hits: Array<SearchHit<unknown>>;
value?: number;
}
export interface ParsedAggregationResults {
results: ParsedAggregationGroup[];
truncated: boolean;
}
interface ParseAggregationResultsOpts {
isCountAgg: boolean;
isGroupAgg: boolean;
esResult: SearchResponse<unknown>;
resultLimit?: number;
}
export const parseAggregationResults = ({
isCountAgg,
isGroupAgg,
esResult,
resultLimit,
}: ParseAggregationResultsOpts): ParsedAggregationResults => {
const aggregations = esResult?.aggregations || {};
// add a fake 'all documents' group aggregation, if a group aggregation wasn't used
if (!isGroupAgg) {
aggregations.groupAgg = {
buckets: [
{
key: UngroupedGroupId,
doc_count: totalHitsToNumber(esResult.hits.total),
topHitsAgg: {
hits: {
hits: esResult.hits.hits ?? [],
},
},
...(!isCountAgg
? {
metricAgg: {
value:
(aggregations.metricAgg as AggregationsSingleMetricAggregateBase)?.value ?? 0,
},
}
: {}),
},
],
};
}
// @ts-expect-error specify aggregations type explicitly
const groupBuckets = aggregations.groupAgg?.buckets || [];
// @ts-expect-error specify aggregations type explicitly
const numGroupsTotal = aggregations.groupAggCount?.count ?? 0;
const results: ParsedAggregationResults = {
results: [],
truncated: resultLimit ? numGroupsTotal > resultLimit : false,
};
for (const groupBucket of groupBuckets) {
if (resultLimit && results.results.length === resultLimit) break;
const groupName: string = `${groupBucket?.key}`;
const groupResult: any = {
group: groupName,
count: groupBucket?.doc_count,
hits: groupBucket?.topHitsAgg?.hits?.hits ?? [],
...(!isCountAgg ? { value: groupBucket?.metricAgg?.value } : {}),
};
results.results.push(groupResult);
}
return results;
};
function totalHitsToNumber(total: SearchHitsMetadata['total']): number {
return typeof total === 'number' ? total : total?.value ?? 0;
}

46
x-pack/plugins/triggers_actions_ui/public/common/expression_items/group_by_over.test.tsx
View file

@ -7,6 +7,8 @@
import * as React from 'react';
import { shallow } from 'enzyme';
import { mountWithIntl, nextTick } from '@kbn/test-jest-helpers';
import { act } from 'react-dom/test-utils';
import { GroupByExpression } from './group_by_over';
import { FormattedMessage } from '@kbn/i18n-react';
@ -18,7 +20,7 @@ describe('group by expression', () => {
const wrapper = shallow(
<GroupByExpression
errors={{ termSize: [], termField: [] }}
fields={[{}]}
fields={[]}
groupBy={'all'}
onChangeSelectedGroupBy={onChangeSelectedGroupBy}
onChangeSelectedTermField={onChangeSelectedTermField}
@ -53,7 +55,15 @@ describe('group by expression', () => {
const wrapper = shallow(
<GroupByExpression
errors={{ termSize: [], termField: [] }}
fields={[{ normalizedType: 'number', name: 'test', text: 'test text' }]}
fields={[
{
normalizedType: 'number',
name: 'test',
type: 'long',
searchable: true,
aggregatable: true,
},
]}
groupBy={'top'}
onChangeSelectedGroupBy={onChangeSelectedGroupBy}
onChangeSelectedTermField={onChangeSelectedTermField}
@ -90,7 +100,7 @@ describe('group by expression', () => {
const wrapper = shallow(
<GroupByExpression
errors={{ termSize: [], termField: [] }}
fields={[{}]}
fields={[]}
groupBy={'all'}
onChangeSelectedGroupBy={onChangeSelectedGroupBy}
onChangeSelectedTermField={onChangeSelectedTermField}
@ -108,4 +118,34 @@ describe('group by expression', () => {
)
).toBeTruthy();
});
it('clears selected agg field if fields does not contain current selection', async () => {
const onChangeSelectedTermField = jest.fn();
const wrapper = mountWithIntl(
<GroupByExpression
errors={{ termSize: [], termField: [] }}
fields={[
{
normalizedType: 'number',
name: 'test',
type: 'long',
searchable: true,
aggregatable: true,
},
]}
termField="notavailable"
groupBy={'all'}
onChangeSelectedGroupBy={() => {}}
onChangeSelectedTermSize={() => {}}
onChangeSelectedTermField={onChangeSelectedTermField}
/>
);
await act(async () => {
await nextTick();
wrapper.update();
});
expect(onChangeSelectedTermField).toHaveBeenCalledWith('');
});
});

56
x-pack/plugins/triggers_actions_ui/public/common/expression_items/group_by_over.tsx
View file

@ -5,8 +5,9 @@
* 2.0.
*/
import React, { useState } from 'react';
import React, { useEffect, useState } from 'react';
import { FormattedMessage } from '@kbn/i18n-react';
import { css } from '@emotion/react';
import { i18n } from '@kbn/i18n';
import {
EuiExpression,
@ -18,17 +19,21 @@ import {
EuiFieldNumber,
} from '@elastic/eui';
import { builtInGroupByTypes } from '../constants';
import { GroupByType } from '../types';
import { FieldOption, GroupByType } from '../types';
import { ClosablePopoverTitle } from './components';
import { IErrorObject } from '../../types';
interface GroupByOverFieldOption {
text: string;
value: string;
}
export interface GroupByExpressionProps {
groupBy: string;
errors: IErrorObject;
onChangeSelectedTermSize: (selectedTermSize?: number) => void;
onChangeSelectedTermField: (selectedTermField?: string) => void;
onChangeSelectedGroupBy: (selectedGroupBy?: string) => void;
fields: Record<string, any>;
fields: FieldOption[];
termSize?: number;
termField?: string;
customGroupByTypes?: {
@ -67,7 +72,7 @@ export const GroupByExpression = ({
const [groupByPopoverOpen, setGroupByPopoverOpen] = useState(false);
const MIN_TERM_SIZE = 1;
const MAX_TERM_SIZE = 1000;
const firstFieldOption = {
const firstFieldOption: GroupByOverFieldOption = {
text: i18n.translate(
'xpack.triggersActionsUI.common.expressionItems.groupByType.timeFieldOptionLabel',
{
@ -77,6 +82,31 @@ export const GroupByExpression = ({
value: '',
};
const availableFieldOptions: GroupByOverFieldOption[] = fields.reduce(
(options: GroupByOverFieldOption[], field: FieldOption) => {
if (groupByTypes[groupBy].validNormalizedTypes.includes(field.normalizedType)) {
options.push({
text: field.name,
value: field.name,
});
}
return options;
},
[firstFieldOption]
);
useEffect(() => {
// if current field set doesn't contain selected field, clear selection
if (
termField &&
termField.length > 0 &&
fields.length > 0 &&
!fields.find((field: FieldOption) => field.name === termField)
) {
onChangeSelectedTermField('');
}
}, [termField, fields, onChangeSelectedTermField]);
return (
<EuiPopover
button={
@ -155,6 +185,9 @@ export const GroupByExpression = ({
<EuiFlexItem grow={false}>
<EuiFormRow isInvalid={errors.termSize.length > 0} error={errors.termSize}>
<EuiFieldNumber
css={css`
min-width: 50px;
`}
isInvalid={errors.termSize.length > 0}
value={termSize || ''}
onChange={(e) => {
@ -179,20 +212,7 @@ export const GroupByExpression = ({
onChange={(e) => {
onChangeSelectedTermField(e.target.value);
}}
options={fields.reduce(
(options: any, field: { name: string; normalizedType: string }) => {
if (
groupByTypes[groupBy].validNormalizedTypes.includes(field.normalizedType)
) {
options.push({
text: field.name,
value: field.name,
});
}
return options;
},
[firstFieldOption]
)}
options={availableFieldOptions}
onBlur={() => {
if (termField === undefined) {
onChangeSelectedTermField('');

30
x-pack/plugins/triggers_actions_ui/public/common/expression_items/of.test.tsx
View file

@ -7,6 +7,8 @@
import * as React from 'react';
import { shallow } from 'enzyme';
import { mountWithIntl, nextTick } from '@kbn/test-jest-helpers';
import { act } from 'react-dom/test-utils';
import { OfExpression } from './of';
import { FormattedMessage } from '@kbn/i18n-react';
@ -152,4 +154,32 @@ describe('of expression', () => {
'Helptext test message'
);
});
it('clears selected agg field if fields does not contain current selection', async () => {
const onChangeSelectedAggField = jest.fn();
const wrapper = mountWithIntl(
<OfExpression
aggType="count"
errors={{ aggField: [] }}
fields={[
{
normalizedType: 'number',
name: 'test',
type: 'long',
searchable: true,
aggregatable: true,
},
]}
aggField="notavailable"
onChangeSelectedAggField={onChangeSelectedAggField}
/>
);
await act(async () => {
await nextTick();
wrapper.update();
});
expect(onChangeSelectedAggField).toHaveBeenCalledWith(undefined);
});
});

41
x-pack/plugins/triggers_actions_ui/public/common/expression_items/of.tsx
View file

@ -5,7 +5,7 @@
* 2.0.
*/
import React, { useState } from 'react';
import React, { useEffect, useState } from 'react';
import { i18n } from '@kbn/i18n';
import { FormattedMessage } from '@kbn/i18n-react';
import {
@ -17,16 +17,19 @@ import {
EuiComboBox,
} from '@elastic/eui';
import { builtInAggregationTypes } from '../constants';
import { AggregationType } from '../types';
import { AggregationType, FieldOption } from '../types';
import { IErrorObject } from '../../types';
import { ClosablePopoverTitle } from './components';
import './of.scss';
interface OfFieldOption {
label: string;
}
export interface OfExpressionProps {
aggType: string;
aggField?: string;
errors: IErrorObject;
onChangeSelectedAggField: (selectedAggType?: string) => void;
onChangeSelectedAggField: (selectedAggField?: string) => void;
fields: Record<string, any>;
customAggTypesOptions?: {
[key: string]: AggregationType;
@ -71,14 +74,28 @@ export const OfExpression = ({
};
const aggregationTypes = customAggTypesOptions ?? builtInAggregationTypes;
const availablefieldsOptions = fields.reduce((esFieldOptions: any[], field: any) => {
if (aggregationTypes[aggType].validNormalizedTypes.includes(field.normalizedType)) {
esFieldOptions.push({
label: field.name,
});
const availableFieldOptions: OfFieldOption[] = fields.reduce(
(esFieldOptions: OfFieldOption[], field: FieldOption) => {
if (aggregationTypes[aggType].validNormalizedTypes.includes(field.normalizedType)) {
esFieldOptions.push({
label: field.name,
});
}
return esFieldOptions;
},
[]
);
useEffect(() => {
// if current field set doesn't contain selected field, clear selection
if (
aggField &&
fields.length > 0 &&
!fields.find((field: FieldOption) => field.name === aggField)
) {
onChangeSelectedAggField(undefined);
}
return esFieldOptions;
}, []);
}, [aggField, fields, onChangeSelectedAggField]);
return (
<EuiPopover
@ -133,8 +150,8 @@ export const OfExpression = ({
data-test-subj="availablefieldsOptionsComboBox"
isInvalid={errors.aggField.length > 0 && aggField !== undefined}
placeholder={firstFieldOption.text}
options={availablefieldsOptions}
noSuggestions={!availablefieldsOptions.length}
options={availableFieldOptions}
noSuggestions={!availableFieldOptions.length}
selectedOptions={aggField ? [{ label: aggField }] : []}
onChange={(selectedOptions) => {
onChangeSelectedAggField(

10
x-pack/plugins/triggers_actions_ui/public/common/index.ts
View file

@ -25,4 +25,12 @@ export { connectorDeprecatedMessage, deprecatedMessage } from './connectors_sele
export type { IOption } from './index_controls';
export { getFields, getIndexOptions, firstFieldOption } from './index_controls';
export { getTimeFieldOptions, useKibana } from './lib';
export type { Comparator, AggregationType, GroupByType, RuleStatus } from './types';
export type { Comparator, AggregationType, GroupByType, RuleStatus, FieldOption } from './types';
export {
BUCKET_SELECTOR_FIELD,
buildAggregation,
isCountAggregation,
isGroupAggregation,
parseAggregationResults,
} from '../../common';
export type { ParsedAggregationGroup } from '../../common';

11
x-pack/plugins/triggers_actions_ui/public/common/lib/data_apis.ts
View file

@ -7,6 +7,7 @@
import { HttpSetup } from '@kbn/core/public';
import { DataViewsContract, DataView } from '@kbn/data-views-plugin/public';
import { FieldOption } from '../types';
const DATA_API_ROOT = '/internal/triggers_actions_ui/data';
@ -47,15 +48,7 @@ export async function getESIndexFields({
}: {
indexes: string[];
http: HttpSetup;
}): Promise<
Array<{
name: string;
type: string;
normalizedType: string;
searchable: boolean;
aggregatable: boolean;
}>
> {
}): Promise<FieldOption[]> {
const { fields } = await http.post<{ fields: ReturnType<typeof getESIndexFields> }>(
`${DATA_API_ROOT}/_fields`,
{ body: JSON.stringify({ indexPatterns: indexes }) }

8
x-pack/plugins/triggers_actions_ui/public/common/types.ts
View file

@ -25,4 +25,12 @@ export interface GroupByType {
validNormalizedTypes: string[];
}
export interface FieldOption {
name: string;
type: string;
normalizedType: string;
searchable: boolean;
aggregatable: boolean;
}
export type { RuleStatus } from '../types';

3
x-pack/plugins/triggers_actions_ui/server/data/index.ts
View file

@ -15,10 +15,11 @@ export {
CoreQueryParamsSchemaProperties,
validateCoreQueryBody,
validateTimeWindowUnits,
validateAggType,
validateGroupBy,
} from './lib';
// future enhancement: make these configurable?
export const MAX_INTERVALS = 1000;
export const MAX_GROUPS = 1000;
export const DEFAULT_GROUPS = 100;

2
x-pack/plugins/triggers_actions_ui/server/data/lib/core_query_types.ts
View file

@ -95,7 +95,7 @@ export function validateCoreQueryBody(anyParams: unknown): string | undefined {
const AggTypes = new Set(['count', 'avg', 'min', 'max', 'sum']);
function validateAggType(aggType: string): string | undefined {
export function validateAggType(aggType: string): string | undefined {
if (AggTypes.has(aggType)) {
return;
}

2
x-pack/plugins/triggers_actions_ui/server/data/lib/index.ts
View file

@ -12,4 +12,6 @@ export {
CoreQueryParamsSchemaProperties,
validateCoreQueryBody,
validateTimeWindowUnits,
validateAggType,
validateGroupBy,
} from './core_query_types';

134
x-pack/plugins/triggers_actions_ui/server/data/lib/time_series_query.ts
View file

@ -10,8 +10,12 @@ import { Logger } from '@kbn/core/server';
import type { ElasticsearchClient } from '@kbn/core/server';
import { getEsErrorMessage } from '@kbn/alerting-plugin/server';
import { toElasticsearchQuery, fromKueryExpression } from '@kbn/es-query';
import { DEFAULT_GROUPS } from '..';
import { getDateRangeInfo } from './date_range_info';
import {
buildAggregation,
getDateRangeInfo,
isCountAggregation,
isGroupAggregation,
} from '../../../common';
import {
TimeSeriesQuery,
@ -48,6 +52,7 @@ export async function timeSeriesQuery(
const window = `${timeWindowSize}${timeWindowUnit}`;
const dateRangeInfo = getDateRangeInfo({ dateStart, dateEnd, window, interval });
const { aggType, aggField, termField, termSize } = queryParams;
// core query
// Constructing a typesafe ES query in JS is problematic, use any escapehatch for now
@ -72,123 +77,32 @@ export async function timeSeriesQuery(
],
},
},
// aggs: {...}, filled in below
aggs: buildAggregation({
timeSeries: {
timeField,
timeWindowSize,
timeWindowUnit,
dateStart,
dateEnd,
interval,
},
aggType,
aggField,
termField,
termSize,
condition: conditionParams,
}),
},
ignore_unavailable: true,
allow_no_indices: true,
};
// add the aggregations
const { aggType, aggField, termField, termSize } = queryParams;
const isCountAgg = aggType === 'count';
const isGroupAgg = !!termField;
const isCountAgg = isCountAggregation(aggType);
const isGroupAgg = isGroupAggregation(termField);
const includeConditionInQuery = !!conditionParams;
// Cap the maximum number of terms returned to the resultLimit if defined
// Use resultLimit + 1 because we're using the bucket selector aggregation
// to apply the threshold condition to the ES query. We don't seem to be
// able to get the true cardinality from the bucket selector (i.e., get
// the number of buckets that matched the selector condition without actually
// retrieving the bucket data). By using resultLimit + 1, we can count the number
// of buckets returned and if the value is greater than resultLimit, we know that
// there is additional alert data that we're not returning.
let terms = termSize || DEFAULT_GROUPS;
terms = includeConditionInQuery
? terms > conditionParams.resultLimit
? conditionParams.resultLimit + 1
: terms
: terms;
let aggParent = esQuery.body;
// first, add a group aggregation, if requested
if (isGroupAgg) {
aggParent.aggs = {
groupAgg: {
terms: {
field: termField,
size: terms,
},
},
...(includeConditionInQuery
? {
groupAggCount: {
stats_bucket: {
buckets_path: 'groupAgg._count',
},
},
}
: {}),
};
// if not count add an order
if (!isCountAgg) {
const sortOrder = aggType === 'min' ? 'asc' : 'desc';
aggParent.aggs.groupAgg.terms.order = {
sortValueAgg: sortOrder,
};
} else if (includeConditionInQuery) {
aggParent.aggs.groupAgg.aggs = {
conditionSelector: {
bucket_selector: {
buckets_path: {
[TIME_SERIES_BUCKET_SELECTOR_PATH_NAME]: '_count',
},
script: conditionParams.conditionScript,
},
},
};
}
aggParent = aggParent.aggs.groupAgg;
}
// next, add the time window aggregation
aggParent.aggs = {
...aggParent.aggs,
dateAgg: {
date_range: {
field: timeField,
format: 'strict_date_time',
ranges: dateRangeInfo.dateRanges,
},
},
};
// if not count, add a sorted value agg
if (!isCountAgg) {
aggParent.aggs.sortValueAgg = {
[aggType]: {
field: aggField,
},
};
if (isGroupAgg && includeConditionInQuery) {
aggParent.aggs.conditionSelector = {
bucket_selector: {
buckets_path: {
[TIME_SERIES_BUCKET_SELECTOR_PATH_NAME]: 'sortValueAgg',
},
script: conditionParams.conditionScript,
},
};
}
}
aggParent = aggParent.aggs.dateAgg;
// finally, the metric aggregation, if requested
if (!isCountAgg) {
aggParent.aggs = {
metricAgg: {
[aggType]: {
field: aggField,
},
},
};
}
const logPrefix = 'indexThreshold timeSeriesQuery: callCluster';
logger.debug(`${logPrefix} call: ${JSON.stringify(esQuery)}`);
let esResult: estypes.SearchResponse<unknown>;

6
x-pack/plugins/triggers_actions_ui/server/data/lib/time_series_types.ts
View file

@ -12,12 +12,12 @@ import { i18n } from '@kbn/i18n';
import { schema, TypeOf } from '@kbn/config-schema';
import { parseDuration } from '@kbn/alerting-plugin/server';
import { MAX_INTERVALS } from '..';
import { CoreQueryParamsSchemaProperties, validateCoreQueryBody } from './core_query_types';
import {
getTooManyIntervalsErrorMessage,
MAX_INTERVALS,
getDateStartAfterDateEndErrorMessage,
} from './date_range_info';
getTooManyIntervalsErrorMessage,
} from '../../../common/data';
export type { TimeSeriesResult, TimeSeriesResultRow, MetricResult } from '../../../common/data';

3
x-pack/plugins/triggers_actions_ui/server/index.ts
View file

@ -14,7 +14,8 @@ export {
CoreQueryParamsSchemaProperties,
validateCoreQueryBody,
validateTimeWindowUnits,
MAX_INTERVALS,
validateAggType,
validateGroupBy,
MAX_GROUPS,
DEFAULT_GROUPS,
TIME_SERIES_BUCKET_SELECTOR_FIELD,

22
x-pack/test/alerting_api_integration/spaces_only/tests/alerting/builtin_alert_types/es_query/common.ts
View file

@ -13,7 +13,7 @@ import {
getUrlPrefix,
ObjectRemover,
} from '../../../../../common/lib';
import { createEsDocuments } from '../lib/create_test_data';
import { createEsDocuments, createEsDocumentsWithGroups } from '../lib/create_test_data';
export const RULE_TYPE_ID = '.es-query';
export const CONNECTOR_TYPE_ID = '.index';
@ -63,6 +63,8 @@ export interface CreateRuleParams {
searchType?: 'searchSource';
notifyWhen?: string;
indexName?: string;
aggType?: string;
groupBy?: string;
}
export function getRuleServices(getService: FtrProviderContext['getService']) {
@ -89,6 +91,23 @@ export function getRuleServices(getService: FtrProviderContext['getService']) {
);
}
async function createGroupedEsDocumentsInGroups(
groups: number,
endDate: string,
indexTool: ESTestIndexTool = esTestIndexTool,
indexName: string = ES_TEST_INDEX_NAME
) {
await createEsDocumentsWithGroups({
es,
esTestIndexTool: indexTool,
endDate,
intervals: RULE_INTERVALS_TO_WRITE,
intervalMillis: RULE_INTERVAL_MILLIS,
groups,
indexName,
});
}
async function waitForDocs(count: number): Promise<any[]> {
return await esTestIndexToolOutput.waitForDocs(
ES_TEST_INDEX_SOURCE,
@ -104,6 +123,7 @@ export function getRuleServices(getService: FtrProviderContext['getService']) {
esTestIndexToolOutput,
esTestIndexToolDataStream,
createEsDocumentsInGroups,
createGroupedEsDocumentsInGroups,
waitForDocs,
};
}

2
x-pack/test/alerting_api_integration/spaces_only/tests/alerting/builtin_alert_types/es_query/query_dsl_only.ts
View file

@ -332,6 +332,8 @@ export default function ruleTests({ getService }: FtrProviderContext) {
thresholdComparator: params.thresholdComparator,
threshold: params.threshold,
searchType: params.searchType,
aggType: params.aggType || 'count',
groupBy: params.groupBy || 'all',
...ruleParams,
},
})

313
x-pack/test/alerting_api_integration/spaces_only/tests/alerting/builtin_alert_types/es_query/rule.ts
View file

@ -35,6 +35,7 @@ export default function ruleTests({ getService }: FtrProviderContext) {
esTestIndexToolOutput,
esTestIndexToolDataStream,
createEsDocumentsInGroups,
createGroupedEsDocumentsInGroups,
} = getRuleServices(getService);
describe('rule', async () => {
@ -126,7 +127,7 @@ export default function ruleTests({ getService }: FtrProviderContext) {
},
] as const,
].forEach(([searchType, initData]) =>
it(`runs correctly: threshold on hit count < > for ${searchType} search type`, async () => {
it(`runs correctly: threshold on ungrouped hit count < > for ${searchType} search type`, async () => {
// write documents from now to the future end date in groups
await createEsDocumentsInGroups(ES_GROUPS_TO_WRITE, endDate);
await initData();
@ -155,6 +156,306 @@ export default function ruleTests({ getService }: FtrProviderContext) {
})
);
[
[
'esQuery',
async () => {
await createRule({
name: 'never fire',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
thresholdComparator: '<',
threshold: [0],
aggType: 'avg',
aggField: 'testedValue',
});
await createRule({
name: 'always fire',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
thresholdComparator: '>',
threshold: [-1],
aggType: 'avg',
aggField: 'testedValue',
});
},
] as const,
[
'searchSource',
async () => {
const esTestDataView = await indexPatterns.create(
{ title: ES_TEST_INDEX_NAME, timeFieldName: 'date' },
{ override: true },
getUrlPrefix(Spaces.space1.id)
);
await createRule({
name: 'never fire',
size: 100,
thresholdComparator: '<',
threshold: [0],
searchType: 'searchSource',
searchConfiguration: {
query: {
query: '',
language: 'kuery',
},
index: esTestDataView.id,
filter: [],
},
aggType: 'avg',
aggField: 'testedValue',
});
await createRule({
name: 'always fire',
size: 100,
thresholdComparator: '>',
threshold: [-1],
searchType: 'searchSource',
searchConfiguration: {
query: {
query: '',
language: 'kuery',
},
index: esTestDataView.id,
filter: [],
},
aggType: 'avg',
aggField: 'testedValue',
});
},
] as const,
].forEach(([searchType, initData]) =>
it(`runs correctly: threshold on ungrouped agg metric < > for ${searchType} search type`, async () => {
// write documents from now to the future end date in groups
await createEsDocumentsInGroups(ES_GROUPS_TO_WRITE, endDate);
await initData();
const docs = await waitForDocs(2);
for (let i = 0; i < docs.length; i++) {
const doc = docs[i];
const { previousTimestamp, hits } = doc._source;
const { name, title, message } = doc._source.params;
expect(name).to.be('always fire');
expect(title).to.be(`rule 'always fire' matched query`);
const messagePattern =
/rule 'always fire' is active:\n\n- Value: \d+.?\d*\n- Conditions Met: Number of matching documents where avg of testedValue is greater than -1 over 20s\n- Timestamp: \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z/;
expect(message).to.match(messagePattern);
expect(hits).not.to.be.empty();
// during the first execution, the latestTimestamp value should be empty
// since this rule always fires, the latestTimestamp value should be updated each execution
if (!i) {
expect(previousTimestamp).to.be.empty();
} else {
expect(previousTimestamp).not.to.be.empty();
}
}
})
);
[
[
'esQuery',
async () => {
await createRule({
name: 'never fire',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
thresholdComparator: '<',
threshold: [0],
groupBy: 'top',
termField: 'group',
termSize: 2,
});
await createRule({
name: 'always fire',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
thresholdComparator: '>',
threshold: [-1],
groupBy: 'top',
termField: 'group',
termSize: 2,
});
},
] as const,
[
'searchSource',
async () => {
const esTestDataView = await indexPatterns.create(
{ title: ES_TEST_INDEX_NAME, timeFieldName: 'date' },
{ override: true },
getUrlPrefix(Spaces.space1.id)
);
await createRule({
name: 'never fire',
size: 100,
thresholdComparator: '<',
threshold: [0],
searchType: 'searchSource',
searchConfiguration: {
query: {
query: '',
language: 'kuery',
},
index: esTestDataView.id,
filter: [],
},
groupBy: 'top',
termField: 'group',
termSize: 2,
});
await createRule({
name: 'always fire',
size: 100,
thresholdComparator: '>',
threshold: [-1],
searchType: 'searchSource',
searchConfiguration: {
query: {
query: '',
language: 'kuery',
},
index: esTestDataView.id,
filter: [],
},
groupBy: 'top',
termField: 'group',
termSize: 2,
});
},
] as const,
].forEach(([searchType, initData]) =>
it(`runs correctly: threshold on grouped hit count < > for ${searchType} search type`, async () => {
// write documents from now to the future end date in groups
await createGroupedEsDocumentsInGroups(ES_GROUPS_TO_WRITE, endDate);
await initData();
const docs = await waitForDocs(2);
for (let i = 0; i < docs.length; i++) {
const doc = docs[i];
const { previousTimestamp, hits } = doc._source;
const { name, title, message } = doc._source.params;
expect(name).to.be('always fire');
const titlePattern = /rule 'always fire' matched query for group group-\d/;
expect(title).to.match(titlePattern);
const messagePattern =
/rule 'always fire' is active:\n\n- Value: \d+\n- Conditions Met: Number of matching documents for group \"group-\d\" is greater than -1 over 20s\n- Timestamp: \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z/;
expect(message).to.match(messagePattern);
expect(hits).not.to.be.empty();
expect(previousTimestamp).to.be.empty();
}
})
);
[
[
'esQuery',
async () => {
await createRule({
name: 'never fire',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
thresholdComparator: '<',
threshold: [0],
groupBy: 'top',
termField: 'group',
termSize: 2,
aggType: 'avg',
aggField: 'testedValue',
});
await createRule({
name: 'always fire',
esQuery: `{\n \"query\":{\n \"match_all\" : {}\n }\n}`,
size: 100,
thresholdComparator: '>',
threshold: [-1],
groupBy: 'top',
termField: 'group',
termSize: 2,
aggType: 'avg',
aggField: 'testedValue',
});
},
] as const,
[
'searchSource',
async () => {
const esTestDataView = await indexPatterns.create(
{ title: ES_TEST_INDEX_NAME, timeFieldName: 'date' },
{ override: true },
getUrlPrefix(Spaces.space1.id)
);
await createRule({
name: 'never fire',
size: 100,
thresholdComparator: '<',
threshold: [0],
searchType: 'searchSource',
searchConfiguration: {
query: {
query: '',
language: 'kuery',
},
index: esTestDataView.id,
filter: [],
},
groupBy: 'top',
termField: 'group',
termSize: 2,
aggType: 'avg',
aggField: 'testedValue',
});
await createRule({
name: 'always fire',
size: 100,
thresholdComparator: '>',
threshold: [-1],
searchType: 'searchSource',
searchConfiguration: {
query: {
query: '',
language: 'kuery',
},
index: esTestDataView.id,
filter: [],
},
groupBy: 'top',
termField: 'group',
termSize: 2,
aggType: 'avg',
aggField: 'testedValue',
});
},
] as const,
].forEach(([searchType, initData]) =>
it(`runs correctly: threshold on grouped agg metric < > for ${searchType} search type`, async () => {
// write documents from now to the future end date in groups
await createGroupedEsDocumentsInGroups(ES_GROUPS_TO_WRITE, endDate);
await initData();
const docs = await waitForDocs(2);
for (let i = 0; i < docs.length; i++) {
const doc = docs[i];
const { previousTimestamp, hits } = doc._source;
const { name, title, message } = doc._source.params;
expect(name).to.be('always fire');
const titlePattern = /rule 'always fire' matched query for group group-\d/;
expect(title).to.match(titlePattern);
const messagePattern =
/rule 'always fire' is active:\n\n- Value: \d+.?\d*\n- Conditions Met: Number of matching documents for group \"group-\d\" where avg of testedValue is greater than -1 over 20s\n- Timestamp: \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z/;
expect(message).to.match(messagePattern);
expect(hits).not.to.be.empty();
expect(previousTimestamp).to.be.empty();
}
})
);
[
[
'esQuery',
@ -697,6 +998,11 @@ export default function ruleTests({ getService }: FtrProviderContext) {
notifyWhen?: string;
indexName?: string;
excludeHitsFromPreviousRun?: boolean;
aggType?: string;
aggField?: string;
groupBy?: string;
termField?: string;
termSize?: number;
}
async function createRule(params: CreateRuleParams): Promise<string> {
@ -772,6 +1078,11 @@ export default function ruleTests({ getService }: FtrProviderContext) {
thresholdComparator: params.thresholdComparator,
threshold: params.threshold,
searchType: params.searchType,
aggType: params.aggType ?? 'count',
groupBy: params.groupBy ?? 'all',
aggField: params.aggField,
termField: params.termField,
termSize: params.termSize,
...(params.excludeHitsFromPreviousRun !== undefined && {
excludeHitsFromPreviousRun: params.excludeHitsFromPreviousRun,
}),

29
x-pack/test/alerting_api_integration/spaces_only/tests/alerting/migrations.ts
View file

@ -484,7 +484,7 @@ export default function createGetTests({ getService }: FtrProviderContext) {
);
});
it('8.5.0 doesnt reformat ES Query rules that dot have a runetime field on them', async () => {
it('8.5.0 doesnt reformat ES Query rules that dot have a runtime field on them', async () => {
const response = await es.get<{
alert: {
params: {
@ -581,5 +581,32 @@ export default function createGetTests({ getService }: FtrProviderContext) {
expect(alert?.lastRun?.warning).to.eql('warning reason');
expect(alert?.lastRun?.outcomeMsg).to.eql('warning message');
});
it('8.7.0 adds aggType and groupBy to ES query rules', async () => {
const response = await es.search<RawRule>(
{
index: '.kibana',
body: {
query: {
bool: {
must: [
{
term: {
'alert.alertTypeId': '.es-query',
},
},
],
},
},
},
},
{ meta: true }
);
expect(response.statusCode).to.eql(200);
response.body.hits.hits.forEach((hit) => {
expect((hit?._source?.alert as RawRule)?.params?.aggType).to.eql('count');
expect((hit?._source?.alert as RawRule)?.params?.groupBy).to.eql('all');
});
});
});
}