github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 17:28:26 -04:00
Code Issues Projects Releases Packages Wiki Activity

[8.x] Entity Store Config - Lookback period (#206421) (#208283)

Browse source 
# Backport

This will backport the following commits from `main` to `8.x`:
- [Entity Store Config - Lookback period
(#206421)](https://github.com/elastic/kibana/pull/206421)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Charlotte Alexandra
Wilson","email":"CAWilson94@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-01-24T21:59:44Z","message":"Entity
Store Config - Lookback period (#206421)\n\n## Summary \r\n\r\nThis PR
enables configurability of the lookbackPeriod for Security’s\r\nEntity
Store through the enable API.\r\n\r\n### Testing \r\n\r\n\r\n1. Open the
latest instance of Kibana.\r\n2. Ensure both the Entity Store and Risk
Score are enabled.\r\n3. Navigate to Dev Tools.\r\n4. Use the enable
endpoint with lookbackPeriod, where the lookbackPeriod\r\nmust be a
number followed by one of the supported units (m, s, or h).\r\n\r\n\r\n
```\r\nPOST kbn:api/entity_store/enable\r\n{\r\n \"lookbackPeriod\":
\"72h\" // example value\r\n}\r\n```\r\n5. After enabling, use the
status endpoint to confirm the configuration:\r\n```\r\nGET
kbn:api/entity_store/status\r\n{}\r\n```\r\n\r\n6. Post some documents,
to check the entities available within the lookbackPeriod - confirm you
can only see those within this period, and not outside of this period
e.g. \"5h\" period, post with a timestamp of today (within 5 hours) and
from yesterday and view results. \r\n ```\r\nPOST
lookback-period-test/_doc\r\n{\r\n \"entity_id\": \"LookbackPeriod Test
Today\",\r\n \"entity_name\": \"LookbackPeriod Test Today\",\r\n
\"timestamp\": \"2025-01-17T14:31:00Z\"\r\n}\r\n```\r\n```\r\nPOST
lookback-period-test/_doc\r\n{\r\n \"entity_id\": \"LookbackPeriod Test
Yesterday\",\r\n \"entity_name\": \"LookbackPeriod Test Yesterday\",\r\n
\"timestamp\": \"2025-01-16T14:31:00Z\"\r\n}\r\n```\r\n\r\n7. Can also
go to entity store, check one of the transforms and check\r\nthe JSON
gte ranges from here too, they should show the
new\r\nlookbackPeriod.\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"667040fbecdb084581dec8b6cfe99bd992cd48db","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["v9.0.0","Team:
SecuritySolution","release_note:feature","Team:Entity
Analytics","backport:version","v8.18.0"],"title":"Entity Store Config -
Lookback
period","number":206421,"url":"https://github.com/elastic/kibana/pull/206421","mergeCommit":{"message":"Entity
Store Config - Lookback period (#206421)\n\n## Summary \r\n\r\nThis PR
enables configurability of the lookbackPeriod for Security’s\r\nEntity
Store through the enable API.\r\n\r\n### Testing \r\n\r\n\r\n1. Open the
latest instance of Kibana.\r\n2. Ensure both the Entity Store and Risk
Score are enabled.\r\n3. Navigate to Dev Tools.\r\n4. Use the enable
endpoint with lookbackPeriod, where the lookbackPeriod\r\nmust be a
number followed by one of the supported units (m, s, or h).\r\n\r\n\r\n
```\r\nPOST kbn:api/entity_store/enable\r\n{\r\n \"lookbackPeriod\":
\"72h\" // example value\r\n}\r\n```\r\n5. After enabling, use the
status endpoint to confirm the configuration:\r\n```\r\nGET
kbn:api/entity_store/status\r\n{}\r\n```\r\n\r\n6. Post some documents,
to check the entities available within the lookbackPeriod - confirm you
can only see those within this period, and not outside of this period
e.g. \"5h\" period, post with a timestamp of today (within 5 hours) and
from yesterday and view results. \r\n ```\r\nPOST
lookback-period-test/_doc\r\n{\r\n \"entity_id\": \"LookbackPeriod Test
Today\",\r\n \"entity_name\": \"LookbackPeriod Test Today\",\r\n
\"timestamp\": \"2025-01-17T14:31:00Z\"\r\n}\r\n```\r\n```\r\nPOST
lookback-period-test/_doc\r\n{\r\n \"entity_id\": \"LookbackPeriod Test
Yesterday\",\r\n \"entity_name\": \"LookbackPeriod Test Yesterday\",\r\n
\"timestamp\": \"2025-01-16T14:31:00Z\"\r\n}\r\n```\r\n\r\n7. Can also
go to entity store, check one of the transforms and check\r\nthe JSON
gte ranges from here too, they should show the
new\r\nlookbackPeriod.\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"667040fbecdb084581dec8b6cfe99bd992cd48db"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/206421","number":206421,"mergeCommit":{"message":"Entity
Store Config - Lookback period (#206421)\n\n## Summary \r\n\r\nThis PR
enables configurability of the lookbackPeriod for Security’s\r\nEntity
Store through the enable API.\r\n\r\n### Testing \r\n\r\n\r\n1. Open the
latest instance of Kibana.\r\n2. Ensure both the Entity Store and Risk
Score are enabled.\r\n3. Navigate to Dev Tools.\r\n4. Use the enable
endpoint with lookbackPeriod, where the lookbackPeriod\r\nmust be a
number followed by one of the supported units (m, s, or h).\r\n\r\n\r\n
```\r\nPOST kbn:api/entity_store/enable\r\n{\r\n \"lookbackPeriod\":
\"72h\" // example value\r\n}\r\n```\r\n5. After enabling, use the
status endpoint to confirm the configuration:\r\n```\r\nGET
kbn:api/entity_store/status\r\n{}\r\n```\r\n\r\n6. Post some documents,
to check the entities available within the lookbackPeriod - confirm you
can only see those within this period, and not outside of this period
e.g. \"5h\" period, post with a timestamp of today (within 5 hours) and
from yesterday and view results. \r\n ```\r\nPOST
lookback-period-test/_doc\r\n{\r\n \"entity_id\": \"LookbackPeriod Test
Today\",\r\n \"entity_name\": \"LookbackPeriod Test Today\",\r\n
\"timestamp\": \"2025-01-17T14:31:00Z\"\r\n}\r\n```\r\n```\r\nPOST
lookback-period-test/_doc\r\n{\r\n \"entity_id\": \"LookbackPeriod Test
Yesterday\",\r\n \"entity_name\": \"LookbackPeriod Test Yesterday\",\r\n
\"timestamp\": \"2025-01-16T14:31:00Z\"\r\n}\r\n```\r\n\r\n7. Can also
go to entity store, check one of the transforms and check\r\nthe JSON
gte ranges from here too, they should show the
new\r\nlookbackPeriod.\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"667040fbecdb084581dec8b6cfe99bd992cd48db"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: Charlotte Alexandra Wilson <CAWilson94@users.noreply.github.com>
Co-authored-by: CAWilson94 <charlotte.wilson@elastic.co>
This commit is contained in:
Kibana Machine 2025-01-30 00:47:01 +11:00 • committed by GitHub
parent 28ca7d52d3
commit 2f369dae8c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 128 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

9
oas_docs/output/kibana.serverless.yaml
View file

@ -7464,6 +7464,11 @@ paths:
type: string
indexPattern:
$ref: '#/components/schemas/Security_Entity_Analytics_API_IndexPattern'
lookbackPeriod:
default: 24h
description: The lookback period for the entity store
pattern: '[smdh]$'
type: string
description: Schema for the entity store initialization
required: true
responses:
@ -47566,6 +47571,10 @@ components:
type: string
indexPattern:
$ref: '#/components/schemas/Security_Entity_Analytics_API_IndexPattern'
lookbackPeriod:
default: 24h
pattern: '[smdh]$'
type: string
status:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EngineStatus'
type:

9
oas_docs/output/kibana.yaml
View file

@ -13047,6 +13047,11 @@ paths:
type: string
indexPattern:
$ref: '#/components/schemas/Security_Entity_Analytics_API_IndexPattern'
lookbackPeriod:
default: 24h
description: The lookback period for the entity store
pattern: '[smdh]$'
type: string
description: Schema for the entity store initialization
required: true
responses:
@ -35960,6 +35965,10 @@ components:
type: string
indexPattern:
$ref: '#/components/schemas/Security_Entity_Analytics_API_IndexPattern'
lookbackPeriod:
default: 24h
pattern: '[smdh]$'
type: string
status:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EngineStatus'
type:

4
x-pack/platform/packages/shared/kbn-entities-schema/src/schema/common.ts
View file

@ -44,7 +44,9 @@ export const docCountMetricSchema = z.object({
filter: filterSchema,
});
export const durationSchema = z.string().regex(/^\d+[m|d|s|h]$/);
export const durationSchema = z.string().regex(/^\d+[m|s|d|h]$/, {
message: 'Must be a number followed by one of the units: m, s, d or h',
});
export const durationSchemaWithMinimum = (minimumMinutes: number) =>
durationSchema.refine(

5
x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/entity_store/common.gen.ts
View file

@ -36,6 +36,11 @@ export const EngineDescriptor = z.object({
status: EngineStatus,
filter: z.string().optional(),
fieldHistoryLength: z.number().int(),
lookbackPeriod: z
.string()
.regex(/[smdh]$/)
.optional()
.default('24h'),
error: z.object({}).optional(),
});

4
x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/entity_store/common.schema.yaml
View file

@ -32,6 +32,10 @@ components:
type: string
fieldHistoryLength:
type: integer
lookbackPeriod:
type: string
default: 24h
pattern: '[smdh]$'
error:
type: object

8
x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/entity_store/enable.gen.ts
View file

@ -24,6 +24,14 @@ export const InitEntityStoreRequestBody = z.object({
* The number of historical values to keep for each field.
*/
fieldHistoryLength: z.number().int().optional().default(10),
/**
* The lookback period for the entity store
*/
lookbackPeriod: z
.string()
.regex(/[smdh]$/)
.optional()
.default('24h'),
indexPattern: IndexPattern.optional(),
filter: z.string().optional(),
entityTypes: z.array(EntityType).optional(),

5
x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/entity_store/enable.schema.yaml
View file

@ -23,6 +23,11 @@ paths:
type: integer
description: The number of historical values to keep for each field.
default: 10
lookbackPeriod:
type: string
description: The lookback period for the entity store
default: 24h
pattern: '[smdh]$'
indexPattern:
$ref: './common.schema.yaml#/components/schemas/IndexPattern'
filter:

9
x-pack/solutions/security/plugins/security_solution/docs/openapi/ess/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml
View file

@ -321,6 +321,11 @@ paths:
type: string
indexPattern:
$ref: '#/components/schemas/IndexPattern'
lookbackPeriod:
default: 24h
description: The lookback period for the entity store
pattern: '[smdh]$'
type: string
description: Schema for the entity store initialization
required: true
responses:
@ -937,6 +942,10 @@ components:
type: string
indexPattern:
$ref: '#/components/schemas/IndexPattern'
lookbackPeriod:
default: 24h
pattern: '[smdh]$'
type: string
status:
$ref: '#/components/schemas/EngineStatus'
type:

9
x-pack/solutions/security/plugins/security_solution/docs/openapi/serverless/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml
View file

@ -321,6 +321,11 @@ paths:
type: string
indexPattern:
$ref: '#/components/schemas/IndexPattern'
lookbackPeriod:
default: 24h
description: The lookback period for the entity store
pattern: '[smdh]$'
type: string
description: Schema for the entity store initialization
required: true
responses:
@ -937,6 +942,10 @@ components:
type: string
indexPattern:
$ref: '#/components/schemas/IndexPattern'
lookbackPeriod:
default: 24h
pattern: '[smdh]$'
type: string
status:
$ref: '#/components/schemas/EngineStatus'
type:

1
x-pack/solutions/security/plugins/security_solution/public/entity_analytics/components/entity_store/components/engines_status/components/engine_status_header_action.test.tsx
View file

@ -33,6 +33,7 @@ const defaultEngineResponse: GetEntityStoreStatusResponse['engines'][0] = {
status: 'started',
fieldHistoryLength: 0,
components: [defaultComponent],
lookbackPeriod: '24h',
};
describe('EngineStatusHeaderAction', () => {

64
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.ts
View file

@ -127,6 +127,18 @@ interface SearchEntitiesParams {
sortOrder: SortOrder;
}
export const DEFAULT_INIT_ENTITY_STORE: InitEntityStoreRequestBody = {
indexPattern: '',
lookbackPeriod: '24h',
filter: '',
fieldHistoryLength: 10,
enrichPolicyExecutionInterval: DEFAULT_INTERVAL,
};
const DEFAULT_ENTITY_ENGINE: InitEntityEngineRequestBody & { lookbackPeriod?: string } = {
...DEFAULT_INIT_ENTITY_STORE,
};
export class EntityStoreDataClient {
private engineClient: EngineDescriptorClient;
private assetCriticalityMigrationClient: AssetCriticalityMigrationClient;
@ -201,19 +213,25 @@ export class EntityStoreDataClient {
}
public async enable(
{
indexPattern = '',
filter = '',
fieldHistoryLength = 10,
entityTypes,
enrichPolicyExecutionInterval,
}: InitEntityStoreRequestBody,
requestBodyOverrides: Partial<InitEntityStoreRequestBody> = {},
{ pipelineDebugMode = false }: { pipelineDebugMode?: boolean } = {}
): Promise<InitEntityStoreResponse> {
if (!this.options.taskManager) {
throw new Error('Task Manager is not available');
}
const {
indexPattern,
lookbackPeriod,
filter,
fieldHistoryLength,
entityTypes,
enrichPolicyExecutionInterval,
} = {
...DEFAULT_INIT_ENTITY_STORE,
...requestBodyOverrides,
};
// Immediately defer the initialization to the next tick. This way we don't block on the init preflight checks
const run = <T>(fn: () => Promise<T>) =>
new Promise<T>((resolve) => setTimeout(() => fn().then(resolve), 0));
@ -230,7 +248,13 @@ export class EntityStoreDataClient {
run(() =>
this.init(
entity,
{ indexPattern, filter, fieldHistoryLength, enrichPolicyExecutionInterval },
{
indexPattern,
lookbackPeriod,
filter,
fieldHistoryLength,
enrichPolicyExecutionInterval,
},
{ pipelineDebugMode }
)
)
@ -290,14 +314,21 @@ export class EntityStoreDataClient {
public async init(
entityType: EntityType,
{
indexPattern = '',
filter = '',
fieldHistoryLength = 10,
enrichPolicyExecutionInterval = DEFAULT_INTERVAL,
}: InitEntityEngineRequestBody,
InitEntityEngineRequestBodyOverrides: Partial<typeof DEFAULT_ENTITY_ENGINE> = {},
{ pipelineDebugMode = false }: { pipelineDebugMode?: boolean } = {}
): Promise<InitEntityEngineResponse> {
const mergedRequest = {
...DEFAULT_ENTITY_ENGINE,
...InitEntityEngineRequestBodyOverrides,
} as Required<typeof DEFAULT_ENTITY_ENGINE>;
const {
indexPattern,
filter,
fieldHistoryLength,
lookbackPeriod,
enrichPolicyExecutionInterval,
} = mergedRequest;
const { experimentalFeatures } = this.options;
if (entityType === EntityType.universal && !experimentalFeatures.assetInventoryStoreEnabled) {
@ -344,6 +375,7 @@ export class EntityStoreDataClient {
const descriptor = await this.engineClient.init(entityType, {
filter,
fieldHistoryLength,
lookbackPeriod,
indexPattern,
});
this.log('debug', entityType, `Initialized engine saved object`);
@ -351,6 +383,7 @@ export class EntityStoreDataClient {
this.asyncSetup(
entityType,
fieldHistoryLength,
lookbackPeriod,
enrichPolicyExecutionInterval,
this.options.taskManager,
indexPattern,
@ -367,6 +400,7 @@ export class EntityStoreDataClient {
private async asyncSetup(
entityType: EntityType,
fieldHistoryLength: number,
lookbackPeriod: string,
enrichPolicyExecutionInterval: string,
taskManager: TaskManagerStartContract,
indexPattern: string,
@ -382,7 +416,7 @@ export class EntityStoreDataClient {
const description = createEngineDescription({
entityType,
namespace,
requestParams: { indexPattern, fieldHistoryLength },
requestParams: { indexPattern, fieldHistoryLength, lookbackPeriod },
defaultIndexPatterns,
config,
});

6
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/installation/engine_description.ts
View file

@ -42,6 +42,7 @@ interface EngineDescriptionParams {
requestParams?: {
indexPattern?: string;
fieldHistoryLength?: number;
lookbackPeriod?: string;
};
defaultIndexPatterns: string[];
}
@ -59,7 +60,10 @@ export const createEngineDescription = (options: EngineDescriptionParams) => {
const settings: EntityEngineInstallationDescriptor['settings'] = {
syncDelay: `${config.syncDelay.asSeconds()}s`,
frequency: `${config.frequency.asSeconds()}s`,
lookbackPeriod: description.settings?.lookbackPeriod || DEFAULT_LOOKBACK_PERIOD,
lookbackPeriod:
requestParams?.lookbackPeriod ||
description.settings?.lookbackPeriod ||
DEFAULT_LOOKBACK_PERIOD,
timestampField: description.settings?.timestampField || DEFAULT_TIMESTAMP_FIELD,
};

5
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/saved_object/engine_descriptor.ts
View file

@ -37,7 +37,8 @@ export class EngineDescriptorClient {
filter,
fieldHistoryLength,
indexPattern,
}: { filter: string; fieldHistoryLength: number; indexPattern: string }
lookbackPeriod,
}: { filter: string; fieldHistoryLength: number; indexPattern: string; lookbackPeriod: string }
) {
const engineDescriptor = await this.find(entityType);
@ -54,6 +55,7 @@ export class EngineDescriptorClient {
filter,
fieldHistoryLength,
indexPattern,
lookbackPeriod,
};
await this.deps.soClient.update<EngineDescriptor>(
entityEngineDescriptorTypeName,
@ -73,6 +75,7 @@ export class EngineDescriptorClient {
indexPattern,
filter,
fieldHistoryLength,
lookbackPeriod,
},
{ id: this.getSavedObjectId(entityType) }
);

4
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/entity_store.ts
View file

@ -91,6 +91,7 @@ export default ({ getService }: FtrProviderContext) => {
indexPattern: '',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
});
});
@ -107,6 +108,7 @@ export default ({ getService }: FtrProviderContext) => {
indexPattern: '',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
});
});
});
@ -125,6 +127,7 @@ export default ({ getService }: FtrProviderContext) => {
indexPattern: '',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
},
{
status: 'started',
@ -132,6 +135,7 @@ export default ({ getService }: FtrProviderContext) => {
indexPattern: '',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
},
]);
});

4
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/entity_store_nondefault_spaces.ts
View file

@ -77,6 +77,7 @@ export default ({ getService }: FtrProviderContextWithSpaces) => {
type: 'host',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
indexPattern: '',
});
});
@ -96,6 +97,7 @@ export default ({ getService }: FtrProviderContextWithSpaces) => {
type: 'user',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
indexPattern: '',
});
});
@ -114,6 +116,7 @@ export default ({ getService }: FtrProviderContextWithSpaces) => {
type: 'host',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
indexPattern: '',
},
{
@ -121,6 +124,7 @@ export default ({ getService }: FtrProviderContextWithSpaces) => {
type: 'user',
filter: '',
fieldHistoryLength: 10,
lookbackPeriod: '24h',
indexPattern: '',
},
]);