github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 09:19:04 -04:00
Code Issues Projects Releases Packages Wiki Activity

[search source] ES Query rule loads fewer fields on query execution (#183694)

Browse source 
## Summary

tldr; ES Query alert execution creates less field_caps traffic, date
fields being accessed in alert message via `fields.*` might not render.

--

This PR reduces the number of fields loaded via field caps to the
minimum required to run a query, rather than the full field list. It
adds a `createLazy` method to the Search Source Service which internally
loads fields via a DataViewLazy object and then adds them to a DataView
object. This is to minimize changes and ship code quickly - SearchSource
objects expose the DataView object they use and kibana apps may use
this. It will take time to migrate away from this since the DataView
object is used both internally and referenced externally. A key element
of this code is the ability to extract a field list from a query so a
limited (rather than complete) set of fields can be loaded.*

One side effect of loading fewer fields is that date fields available
via `fields.*` in the alert message may no longer work. Previously, all
fields were loaded including all date fields. Now, date fields are only
loaded if they're part of the query. This has been determined to be a
small corner case and an acceptable tradeoff.

Only the ES Query rule is using this new method of loading fields. While
further work is needed before wider adoption, this should prevent
significant data transfer savings via a reduction in field_caps usage.

Depends upon https://github.com/elastic/kibana/pull/183573

---

\* We don't need to load all fields to create a query, rather we need to
load all the fields where some attribute will change the output of a
query. Sometimes the translation from KQL to DSL is the same no matter
the field type (or any other attribute) and sometimes the translation is
dependent field type and other attributes. Generally speaking, we need
the latter.

There are additional complexities - we need to know which fields are
dates (and date nanos) when their values are displayed so their values
can be made uniform. In some circumstances we need to load a set of
fields due to source field exclusion - its not supported in ES so Kibana
submits a list of individual field names.

Finally, there are times where we solve a simpler problem rather than
the problem definition. Its easier to get a list of all fields
referenced in a KQL statement instead of only getting the subset we
need. A couple of extra fields is unlikely to result in performance
degradation.

---

Places where the field list is inspected -
```
packages/kbn-es-query/src/es_query/filter_matches_index.ts
packages/kbn-es-query/src/es_query/from_nested_filter.ts
packages/kbn-es-query/src/es_query/migrate_filter.ts
packages/kbn-es-query/src/kuery/functions/exists.ts
packages/kbn-es-query/src/kuery/functions/is.ts
packages/kbn-es-query/src/kuery/functions/utils/get_fields.ts
```

This looks like its worth closer examination since it looks at the
length of the field list -
https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/kuery/functions/is.ts#L110

Next steps -
- [x] Discuss above usage and make sure all cases are covered in this PR
- [x] Add statement to PR on lack of date formatting
- [x] Add test to verify reduction of fields requested

---------

Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Lukas Olson <lukas@elastic.co>
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
This commit is contained in:
Matthew Kime 2024-06-02 11:48:51 -05:00 • committed by GitHub
parent f1c854b9db
commit 28bef6540b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
17 changed files with 196 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

1
packages/kbn-generate-csv/src/generate_csv.test.ts
View file

@ -73,6 +73,7 @@ describe('CsvGenerator', () => {
const mockSearchSourceService: jest.Mocked<ISearchStartSearchSource> = {
create: jest.fn().mockReturnValue(searchSourceMock),
createLazy: jest.fn().mockReturnValue(searchSourceMock),
createEmpty: jest.fn().mockReturnValue(searchSourceMock),
telemetry: jest.fn(),
inject: jest.fn(),

65
src/plugins/data/common/search/search_source/create_search_source.test.ts
View file

@ -8,7 +8,7 @@
import { createSearchSource as createSearchSourceFactory } from './create_search_source';
import { SearchSourceDependencies } from './search_source';
import type { DataView, DataViewsContract } from '@kbn/data-views-plugin/common';
import type { DataView, DataViewsContract, DataViewLazy } from '@kbn/data-views-plugin/common';
import type { Filter } from '@kbn/es-query';
describe('createSearchSource', () => {
@ -24,6 +24,10 @@ describe('createSearchSource', () => {
search: jest.fn(),
onResponse: (req, res) => res,
scriptedFieldsEnabled: true,
dataViews: {
getMetaFields: jest.fn(),
getShortDotsEnable: jest.fn(),
} as unknown as DataViewsContract,
};
indexPatternContractMock = {
@ -104,4 +108,63 @@ describe('createSearchSource', () => {
language: 'lucene',
});
});
it('uses DataViews.get', async () => {
const dataViewMock: DataView = {
toSpec: jest.fn().mockReturnValue(Promise.resolve({})),
getSourceFiltering: jest.fn().mockReturnValue({
excludes: [],
}),
} as unknown as DataView;
const get = jest.fn().mockReturnValue(Promise.resolve(dataViewMock));
const getDataViewLazy = jest.fn();
indexPatternContractMock = {
get,
getDataViewLazy,
} as unknown as jest.Mocked<DataViewsContract>;
createSearchSource = createSearchSourceFactory(indexPatternContractMock, dependencies);
await createSearchSource({
index: '123-456',
highlightAll: true,
query: {
query: '',
language: 'kuery',
},
});
expect(get).toHaveBeenCalledWith('123-456');
expect(getDataViewLazy).not.toHaveBeenCalled();
});
it('uses DataViews.getDataViewLazy when flag is passed', async () => {
const dataViewLazyMock: DataViewLazy = {
toSpec: jest.fn().mockReturnValue(Promise.resolve({})),
getSourceFiltering: jest.fn().mockReturnValue({
excludes: [],
}),
} as unknown as DataViewLazy;
const get = jest.fn();
const getDataViewLazy = jest.fn().mockReturnValue(Promise.resolve(dataViewLazyMock));
indexPatternContractMock = {
get,
getDataViewLazy,
} as unknown as jest.Mocked<DataViewsContract>;
createSearchSource = createSearchSourceFactory(indexPatternContractMock, dependencies);
await createSearchSource(
{
index: '123-456',
highlightAll: true,
query: {
query: '',
language: 'kuery',
},
},
true
);
expect(get).not.toHaveBeenCalled();
expect(getDataViewLazy).toHaveBeenCalledWith('123-456');
});
});

48
src/plugins/data/common/search/search_source/create_search_source.ts
View file

@ -6,7 +6,8 @@
* Side Public License, v 1.
*/
import { DataViewsContract } from '@kbn/data-views-plugin/common';
import { DataViewsContract, DataView, DataViewLazy } from '@kbn/data-views-plugin/common';
import { FieldFormatsStartCommon } from '@kbn/field-formats-plugin/common';
import { migrateLegacyQuery } from './migrate_legacy_query';
import { SearchSource, SearchSourceDependencies } from './search_source';
import { SerializedSearchSourceFields } from '../..';
@ -33,7 +34,11 @@ export const createSearchSource = (
indexPatterns: DataViewsContract,
searchSourceDependencies: SearchSourceDependencies
) => {
const createFields = async (searchSourceFields: SerializedSearchSourceFields = {}) => {
let dataViewLazy: DataViewLazy | undefined;
const createFields = async (
searchSourceFields: SerializedSearchSourceFields = {},
useDataViewLazy = false
) => {
const { index, parent, ...restOfFields } = searchSourceFields;
const fields: SearchSourceFields = {
...restOfFields,
@ -41,10 +46,31 @@ export const createSearchSource = (
// hydrating index pattern
if (searchSourceFields.index) {
if (typeof searchSourceFields.index === 'string') {
fields.index = await indexPatterns.get(searchSourceFields.index);
if (!useDataViewLazy) {
fields.index =
typeof searchSourceFields.index === 'string'
? await indexPatterns.get(searchSourceFields.index)
: await indexPatterns.create(searchSourceFields.index);
} else {
fields.index = await indexPatterns.create(searchSourceFields.index);
dataViewLazy =
typeof searchSourceFields.index === 'string'
? await indexPatterns.getDataViewLazy(searchSourceFields.index)
: await indexPatterns.createDataViewLazy(searchSourceFields.index);
const [spec, shortDotsEnable, metaFields] = await Promise.all([
dataViewLazy.toSpec(),
searchSourceDependencies.dataViews.getShortDotsEnable(),
searchSourceDependencies.dataViews.getMetaFields(),
]);
const dataView = new DataView({
spec,
// field format functionality is not used within search source
fieldFormats: {} as FieldFormatsStartCommon,
shortDotsEnable,
metaFields,
});
fields.index = dataView;
}
}
@ -55,8 +81,11 @@ export const createSearchSource = (
return fields;
};
const createSearchSourceFn = async (searchSourceFields: SerializedSearchSourceFields = {}) => {
const fields = await createFields(searchSourceFields);
const createSearchSourceFn = async (
searchSourceFields: SerializedSearchSourceFields = {},
useDataViewLazy?: boolean
) => {
const fields = await createFields(searchSourceFields, !!useDataViewLazy);
const searchSource = new SearchSource(fields, searchSourceDependencies);
// todo: move to migration script .. create issue
@ -65,6 +94,11 @@ export const createSearchSource = (
if (typeof query !== 'undefined') {
searchSource.setField('query', migrateLegacyQuery(query));
}
// using the dataViewLazy check as a type guard
if (useDataViewLazy && dataViewLazy) {
const dataViewFields = await searchSource.loadDataViewFields(dataViewLazy);
fields.index?.fields.replaceAll(Object.values(dataViewFields).map((fld) => fld.toSpec()));
}
return searchSource;
};

7
src/plugins/data/common/search/search_source/mocks.ts
View file

@ -9,6 +9,7 @@
import { of } from 'rxjs';
import type { MockedKeys } from '@kbn/utility-types-jest';
import { uiSettingsServiceMock } from '@kbn/core/public/mocks';
import { DataViewsContract } from '@kbn/data-views-plugin/common';
import { SearchSource, SearchSourceDependencies } from './search_source';
import { ISearchStartSearchSource, ISearchSource, SearchSourceFields } from './types';
@ -37,10 +38,12 @@ export const searchSourceInstanceMock: MockedKeys<ISearchSource> = {
toExpressionAst: jest.fn(),
getActiveIndexFilter: jest.fn(),
parseActiveIndexPatternFromQueryString: jest.fn(),
loadDataViewFields: jest.fn(),
};
export const searchSourceCommonMock: jest.Mocked<ISearchStartSearchSource> = {
create: jest.fn().mockReturnValue(searchSourceInstanceMock),
createLazy: jest.fn().mockReturnValue(searchSourceInstanceMock),
createEmpty: jest.fn().mockReturnValue(searchSourceInstanceMock),
telemetry: jest.fn(),
getAllMigrations: jest.fn(),
@ -71,4 +74,8 @@ export const createSearchSourceMock = (
),
onResponse: jest.fn().mockImplementation((req, res) => res),
scriptedFieldsEnabled: true,
dataViews: {
getMetaFields: jest.fn(),
getShortDotsEnable: jest.fn(),
} as unknown as DataViewsContract,
});

6
src/plugins/data/common/search/search_source/search_source.test.ts
View file

@ -7,7 +7,7 @@
*/
import Rx, { firstValueFrom, lastValueFrom, of, throwError } from 'rxjs';
import type { DataView } from '@kbn/data-views-plugin/common';
import type { DataView, DataViewsContract } from '@kbn/data-views-plugin/common';
import { buildExpression, ExpressionAstExpression } from '@kbn/expressions-plugin/common';
import type { MockedKeys } from '@kbn/utility-types-jest';
import type { ISearchGeneric } from '@kbn/search-types';
@ -95,6 +95,10 @@ describe('SearchSource', () => {
search: mockSearchMethod,
onResponse: jest.fn().mockImplementation((_, res) => res),
scriptedFieldsEnabled: true,
dataViews: {
getMetaFields: jest.fn(),
getShortDotsEnable: jest.fn(),
} as unknown as jest.Mocked<DataViewsContract>,
};
searchSource = new SearchSource({}, searchSourceDependencies);

48
src/plugins/data/common/search/search_source/search_source.ts
View file

@ -77,13 +77,15 @@ import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import {
buildEsQuery,
Filter,
fromKueryExpression,
isOfQueryType,
isPhraseFilter,
isPhrasesFilter,
getKqlFieldNames,
} from '@kbn/es-query';
import { fieldWildcardFilter } from '@kbn/kibana-utils-plugin/common';
import { getHighlightRequest } from '@kbn/field-formats-plugin/common';
import type { DataView } from '@kbn/data-views-plugin/common';
import { DataView, DataViewLazy, DataViewsContract } from '@kbn/data-views-plugin/common';
import {
ExpressionAstExpression,
buildExpression,
@ -134,6 +136,7 @@ export const searchSourceRequiredUiSettings = [
export interface SearchSourceDependencies extends FetchHandlers {
aggs: AggsStart;
search: ISearchGeneric;
dataViews: DataViewsContract;
scriptedFieldsEnabled: boolean;
}
@ -712,7 +715,7 @@ export class SearchSource {
}
private readonly getFieldName = (fld: SearchFieldValue): string =>
typeof fld === 'string' ? fld : (fld.field as string);
typeof fld === 'string' ? fld : (fld?.field as string);
private getFieldsWithoutSourceFilters(
index: DataView | undefined,
@ -773,6 +776,47 @@ export class SearchSource {
return field;
}
public async loadDataViewFields(dataView: DataViewLazy) {
const request = this.mergeProps(this, { body: {} });
let fields = dataView.timeFieldName ? [dataView.timeFieldName] : [];
const sort = this.getField('sort');
if (sort) {
const sortArr = Array.isArray(sort) ? sort : [sort];
for (const s of sortArr) {
const keys = Object.keys(s);
fields = fields.concat(keys);
}
}
for (const query of request.query) {
if (query.query) {
const nodes = fromKueryExpression(query.query);
const queryFields = getKqlFieldNames(nodes);
fields = fields.concat(queryFields);
}
}
const filters = request.filters;
if (filters) {
const filtersArr = Array.isArray(filters) ? filters : [filters];
for (const f of filtersArr) {
fields = fields.concat(f.meta.key);
}
}
fields = fields.filter((f) => Boolean(f));
if (dataView.getSourceFiltering() && dataView.getSourceFiltering().excludes.length) {
// if source filtering is enabled, we need to fetch all the fields
return (await dataView.getFields({ fieldName: ['*'] })).getFieldMapSorted();
} else if (fields.length) {
return (
await dataView.getFields({
fieldName: fields,
})
).getFieldMapSorted();
}
// no fields needed to be loaded for query
return {};
}
private flatten() {
const { getConfig } = this.dependencies;
const metaFields = getConfig(UI_SETTINGS.META_FIELDS) ?? [];

5
src/plugins/data/common/search/search_source/search_source_service.test.ts
View file

@ -20,6 +20,10 @@ describe('SearchSource service', () => {
search: jest.fn(),
onResponse: jest.fn(),
scriptedFieldsEnabled: true,
dataViews: {
getMetaFields: jest.fn(),
getShortDotsEnable: jest.fn(),
} as unknown as DataViewsContract,
};
});
@ -32,6 +36,7 @@ describe('SearchSource service', () => {
expect(Object.keys(start)).toEqual([
'create',
'createLazy',
'createEmpty',
'extract',
'inject',

4
src/plugins/data/common/search/search_source/search_source_service.ts
View file

@ -61,6 +61,10 @@ export class SearchSourceService {
* creates searchsource based on serialized search source fields
*/
create: createSearchSource(indexPatterns, dependencies),
createLazy: (searchSourceFields: SerializedSearchSourceFields = {}) => {
const fn = createSearchSource(indexPatterns, dependencies);
return fn(searchSourceFields, true);
},
/**
* creates an enpty search source
*/

2
src/plugins/data/common/search/search_source/types.ts
View file

@ -34,6 +34,8 @@ export interface ISearchStartSearchSource
* @param fields
*/
create: (fields?: SerializedSearchSourceFields) => Promise<ISearchSource>;
createLazy: (fields?: SerializedSearchSourceFields) => Promise<ISearchSource>;
/**
* creates empty {@link SearchSource}
*/

1
src/plugins/data/public/search/search_service.ts
View file

@ -264,6 +264,7 @@ export class SearchService implements Plugin<ISearchSetup, ISearchStart> {
aggs,
getConfig: uiSettings.get.bind(uiSettings),
search,
dataViews: indexPatterns,
onResponse: (request, response, options) => {
if (!options.disableWarningToasts) {
const { rawResponse } = response;

1
src/plugins/data/server/search/search_service.ts
View file

@ -316,6 +316,7 @@ export class SearchService implements Plugin<ISearchSetup, ISearchStart> {
getConfig: <T = any>(key: string): T => uiSettingsCache[key],
search: this.asScoped(request).search,
onResponse: (req, res) => res,
dataViews: scopedIndexPatterns,
scriptedFieldsEnabled: true,
};

2
src/plugins/data_views/common/data_views/data_view_lazy.ts
View file

@ -108,7 +108,7 @@ export class DataViewLazy extends AbstractDataView {
return {
getFieldMap: () => fieldMap,
getFieldMapSorted: () => {
getFieldMapSorted: (): Record<string, DataViewField> => {
if (!hasBeenSorted) {
fieldMapSorted = chain(fieldMap).toPairs().sortBy(0).fromPairs().value();
hasBeenSorted = true;

2
src/plugins/saved_search/common/service/get_saved_searches.test.ts
View file

@ -123,6 +123,7 @@ describe('getSavedSearch', () => {
"getSearchRequestBody": [MockFunction],
"getSerializedFields": [MockFunction],
"history": Array [],
"loadDataViewFields": [MockFunction],
"onRequestStart": [MockFunction],
"parseActiveIndexPatternFromQueryString": [MockFunction],
"removeField": [MockFunction],
@ -231,6 +232,7 @@ describe('getSavedSearch', () => {
"getSearchRequestBody": [MockFunction],
"getSerializedFields": [MockFunction],
"history": Array [],
"loadDataViewFields": [MockFunction],
"onRequestStart": [MockFunction],
"parseActiveIndexPatternFromQueryString": [MockFunction],
"removeField": [MockFunction],

4
src/plugins/saved_search/common/service/saved_searches_utils.test.ts
View file

@ -65,6 +65,10 @@ describe('saved_searches_utils', () => {
"aggs": Object {
"createAggConfigs": [MockFunction],
},
"dataViews": Object {
"getMetaFields": [MockFunction],
"getShortDotsEnable": [MockFunction],
},
"getConfig": [MockFunction],
"onResponse": [MockFunction],
"scriptedFieldsEnabled": true,

1
src/plugins/saved_search/public/services/saved_searches/saved_search_attribute_service.test.ts
View file

@ -219,6 +219,7 @@ describe('getSavedSearchAttributeService', () => {
"getSearchRequestBody": [MockFunction],
"getSerializedFields": [MockFunction],
"history": Array [],
"loadDataViewFields": [MockFunction],
"onRequestStart": [MockFunction],
"parseActiveIndexPatternFromQueryString": [MockFunction],
"removeField": [MockFunction],

12
x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.test.ts
View file

@ -84,7 +84,7 @@ describe('fetchSearchSourceQuery', () => {
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { dateStart, dateEnd } = getTimeRange();
const { searchSource, filterToExcludeHitsFromPreviousRun } = updateSearchSource(
const { searchSource, filterToExcludeHitsFromPreviousRun } = await updateSearchSource(
searchSourceInstance,
dataViewMock,
params,
@ -124,7 +124,7 @@ describe('fetchSearchSourceQuery', () => {
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { dateStart, dateEnd } = getTimeRange();
const { searchSource, filterToExcludeHitsFromPreviousRun } = updateSearchSource(
const { searchSource, filterToExcludeHitsFromPreviousRun } = await updateSearchSource(
searchSourceInstance,
dataViewMock,
params,
@ -189,7 +189,7 @@ describe('fetchSearchSourceQuery', () => {
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { dateStart, dateEnd } = getTimeRange();
const { searchSource, filterToExcludeHitsFromPreviousRun } = updateSearchSource(
const { searchSource, filterToExcludeHitsFromPreviousRun } = await updateSearchSource(
searchSourceInstance,
dataViewMock,
params,
@ -229,7 +229,7 @@ describe('fetchSearchSourceQuery', () => {
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { dateStart, dateEnd } = getTimeRange();
const { searchSource, filterToExcludeHitsFromPreviousRun } = updateSearchSource(
const { searchSource, filterToExcludeHitsFromPreviousRun } = await updateSearchSource(
searchSourceInstance,
dataViewMock,
params,
@ -275,7 +275,7 @@ describe('fetchSearchSourceQuery', () => {
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { dateStart, dateEnd } = getTimeRange();
const { searchSource } = updateSearchSource(
const { searchSource } = await updateSearchSource(
searchSourceInstance,
dataViewMock,
params,
@ -346,7 +346,7 @@ describe('fetchSearchSourceQuery', () => {
const searchSourceInstance = createSearchSourceMock({ index: dataViewMock });
const { dateStart, dateEnd } = getTimeRange();
const { filterToExcludeHitsFromPreviousRun } = updateSearchSource(
const { filterToExcludeHitsFromPreviousRun } = await updateSearchSource(
searchSourceInstance,
dataViewMock,
params,

11
x-pack/plugins/stack_alerts/server/rule_types/es_query/lib/fetch_search_source_query.ts
View file

@ -56,11 +56,10 @@ export async function fetchSearchSourceQuery({
const { logger, searchSourceClient } = services;
const isGroupAgg = isGroupAggregation(params.termField);
const isCountAgg = isCountAggregation(params.aggType);
const initialSearchSource = await searchSourceClient.create(params.searchConfiguration);
const initialSearchSource = await searchSourceClient.createLazy(params.searchConfiguration);
const index = initialSearchSource.getField('index') as DataView;
const { searchSource, filterToExcludeHitsFromPreviousRun } = updateSearchSource(
const { searchSource, filterToExcludeHitsFromPreviousRun } = await updateSearchSource(
initialSearchSource,
index,
params,
@ -102,7 +101,7 @@ export async function fetchSearchSourceQuery({
};
}
export function updateSearchSource(
export async function updateSearchSource(
searchSource: ISearchSource,
index: DataView,
params: OnlySearchSourceRuleParams,
@ -110,9 +109,9 @@ export function updateSearchSource(
dateStart: string,
dateEnd: string,
alertLimit?: number
): { searchSource: ISearchSource; filterToExcludeHitsFromPreviousRun: Filter | null } {
): Promise<{ searchSource: ISearchSource; filterToExcludeHitsFromPreviousRun: Filter | null }> {
const isGroupAgg = isGroupAggregation(params.termField);
const timeField = index.getTimeField();
const timeField = await index.getTimeField();
if (!timeField) {
throw new Error(`Data view with ID ${index.id} no longer contains a time field.`);