github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 01:38:56 -04:00
Code Issues Projects Releases Packages Wiki Activity

[8.x] [SecuritySolutions] Create Entity Store 'entities/list' API (#192806) (#193562)

Browse source 
# Backport

This will backport the following commits from `main` to `8.x`:
- [[SecuritySolutions] Create Entity Store 'entities/list' API
(#192806)](https://github.com/elastic/kibana/pull/192806)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pablo
Machado","email":"pablo.nevesmachado@elastic.co"},"sourceCommit":{"committedDate":"2024-09-19T12:54:53Z","message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["v9.0.0","release_note:feature","backport:prev-minor","Theme:
entity_analytics","Feature:Entity Analytics","Team:Entity
Analytics","v8.16.0"],"number":192806,"url":"https://github.com/elastic/kibana/pull/192806","mergeCommit":{"message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192806","number":192806,"mergeCommit":{"message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Pablo Machado 2024-09-23 12:07:53 +02:00 • committed by GitHub
parent caa3c71956
commit 8a6da9bd78
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
31 changed files with 2354 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.buildkite/ftr_security_serverless_configs.yml
View file

@ -78,6 +78,7 @@ disabled:
- x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/exception_lists_items/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/lists_items/trial_license_complete_tier/configs/serverless.config.ts
- x-pack/test/security_solution_api_integration/test_suites/explore/hosts/trial_license_complete_tier/configs/serverless.config.ts

1
.buildkite/ftr_security_stateful_configs.yml
View file

@ -62,6 +62,7 @@ enabled:
- x-pack/test/security_solution_api_integration/test_suites/detections_response/user_roles/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/exception_lists_items/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/lists_items/trial_license_complete_tier/configs/ess.config.ts
- x-pack/test/security_solution_api_integration/test_suites/explore/hosts/trial_license_complete_tier/configs/ess.config.ts

245
oas_docs/output/kibana.serverless.staging.yaml
View file

@ -8321,6 +8321,85 @@ paths:
summary: Stop the Entity Store engine
tags:
- Security Solution Entity Analytics API
/api/entity_store/entities/list:
get:
description: 'List entities records, paging, sorting and filtering as needed.'
operationId: ListEntities
parameters:
- in: query
name: sort_field
required: false
schema:
type: string
- in: query
name: sort_order
required: false
schema:
enum:
- asc
- desc
type: string
- in: query
name: page
required: false
schema:
minimum: 1
type: integer
- in: query
name: per_page
required: false
schema:
maximum: 10000
minimum: 1
type: integer
- description: An ES query to filter by.
in: query
name: filterQuery
required: false
schema:
type: string
- in: query
name: entities_types
required: true
schema:
items:
$ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_EntityType
type: array
responses:
'200':
content:
application/json; Elastic-Api-Version=2023-10-31:
schema:
type: object
properties:
inspect:
$ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_InspectQuery
page:
minimum: 1
type: integer
per_page:
maximum: 1000
minimum: 1
type: integer
records:
items:
$ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_Entity
type: array
total:
minimum: 0
type: integer
required:
- records
- page
- per_page
- total
description: Entities returned successfully
summary: List Entity Store Entities
tags:
- Security Solution Entity Analytics API
/api/exception_lists:
delete:
operationId: DeleteExceptionList
@ -30804,11 +30883,92 @@ components:
- started
- stopped
type: string
Security_Solution_Entity_Analytics_API_Entity:
oneOf:
- $ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_UserEntity
- $ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_HostEntity
Security_Solution_Entity_Analytics_API_EntityType:
enum:
- user
- host
type: string
Security_Solution_Entity_Analytics_API_HostEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
host:
type: object
properties:
architecture:
items:
type: string
type: array
domain:
items:
type: string
type: array
hostname:
items:
type: string
type: array
id:
items:
type: string
type: array
ip:
items:
type: string
type: array
mac:
items:
type: string
type: array
name:
type: string
type:
items:
type: string
type: array
required:
- name
Security_Solution_Entity_Analytics_API_IdField:
enum:
- host.name
@ -30816,6 +30976,20 @@ components:
type: string
Security_Solution_Entity_Analytics_API_IndexPattern:
type: string
Security_Solution_Entity_Analytics_API_InspectQuery:
type: object
properties:
dsl:
items:
type: string
type: array
response:
items:
type: string
type: array
required:
- dsl
- response
Security_Solution_Entity_Analytics_API_RiskEngineScheduleNowErrorResponse:
type: object
properties:
@ -30843,6 +31017,77 @@ components:
required:
- status_code
- message
Security_Solution_Entity_Analytics_API_UserEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
user:
type: object
properties:
domain:
items:
type: string
type: array
email:
items:
type: string
type: array
full_name:
items:
type: string
type: array
hash:
items:
type: string
type: array
id:
items:
type: string
type: array
name:
type: string
roles:
items:
type: string
type: array
required:
- name
Security_Solution_Exceptions_API_CreateExceptionListItemComment:
type: object
properties:

245
oas_docs/output/kibana.staging.yaml
View file

@ -11768,6 +11768,85 @@ paths:
summary: Stop the Entity Store engine
tags:
- Security Solution Entity Analytics API
/api/entity_store/entities/list:
get:
description: 'List entities records, paging, sorting and filtering as needed.'
operationId: ListEntities
parameters:
- in: query
name: sort_field
required: false
schema:
type: string
- in: query
name: sort_order
required: false
schema:
enum:
- asc
- desc
type: string
- in: query
name: page
required: false
schema:
minimum: 1
type: integer
- in: query
name: per_page
required: false
schema:
maximum: 10000
minimum: 1
type: integer
- description: An ES query to filter by.
in: query
name: filterQuery
required: false
schema:
type: string
- in: query
name: entities_types
required: true
schema:
items:
$ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_EntityType
type: array
responses:
'200':
content:
application/json; Elastic-Api-Version=2023-10-31:
schema:
type: object
properties:
inspect:
$ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_InspectQuery
page:
minimum: 1
type: integer
per_page:
maximum: 1000
minimum: 1
type: integer
records:
items:
$ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_Entity
type: array
total:
minimum: 0
type: integer
required:
- records
- page
- per_page
- total
description: Entities returned successfully
summary: List Entity Store Entities
tags:
- Security Solution Entity Analytics API
/api/exception_lists:
delete:
operationId: DeleteExceptionList
@ -38833,11 +38912,92 @@ components:
- started
- stopped
type: string
Security_Solution_Entity_Analytics_API_Entity:
oneOf:
- $ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_UserEntity
- $ref: >-
#/components/schemas/Security_Solution_Entity_Analytics_API_HostEntity
Security_Solution_Entity_Analytics_API_EntityType:
enum:
- user
- host
type: string
Security_Solution_Entity_Analytics_API_HostEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
host:
type: object
properties:
architecture:
items:
type: string
type: array
domain:
items:
type: string
type: array
hostname:
items:
type: string
type: array
id:
items:
type: string
type: array
ip:
items:
type: string
type: array
mac:
items:
type: string
type: array
name:
type: string
type:
items:
type: string
type: array
required:
- name
Security_Solution_Entity_Analytics_API_IdField:
enum:
- host.name
@ -38845,6 +39005,20 @@ components:
type: string
Security_Solution_Entity_Analytics_API_IndexPattern:
type: string
Security_Solution_Entity_Analytics_API_InspectQuery:
type: object
properties:
dsl:
items:
type: string
type: array
response:
items:
type: string
type: array
required:
- dsl
- response
Security_Solution_Entity_Analytics_API_RiskEngineScheduleNowErrorResponse:
type: object
properties:
@ -38872,6 +39046,77 @@ components:
required:
- status_code
- message
Security_Solution_Entity_Analytics_API_UserEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
user:
type: object
properties:
domain:
items:
type: string
type: array
email:
items:
type: string
type: array
full_name:
items:
type: string
type: array
hash:
items:
type: string
type: array
id:
items:
type: string
type: array
name:
type: string
roles:
items:
type: string
type: array
required:
- name
Security_Solution_Exceptions_API_CreateExceptionListItemComment:
type: object
properties:

6
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/common.gen.ts
View file

@ -36,3 +36,9 @@ export const EngineDescriptor = z.object({
status: EngineStatus.optional(),
filter: z.string().optional(),
});
export type InspectQuery = z.infer<typeof InspectQuery>;
export const InspectQuery = z.object({
response: z.array(z.string()),
dsl: z.array(z.string()),
});

19
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/common.schema.yaml
View file

@ -6,7 +6,6 @@ info:
paths: {}
components:
schemas:
EntityType:
type: string
enum:
@ -31,7 +30,21 @@ components:
- installing
- started
- stopped
IndexPattern:
type: string
InspectQuery:
type: object
properties:
response:
type: array
items:
type: string
dsl:
type: array
items:
type: string
required:
- dsl
- response

77
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/common.gen.ts Normal file
View file

@ -0,0 +1,77 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/*
* NOTICE: Do not edit this file manually.
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
*
* info:
* title: Common Entities Schemas
* version: 1
*/
import { z } from '@kbn/zod';
export type UserEntity = z.infer<typeof UserEntity>;
export const UserEntity = z.object({
user: z
.object({
full_name: z.array(z.string()).optional(),
domain: z.array(z.string()).optional(),
roles: z.array(z.string()).optional(),
name: z.string(),
id: z.array(z.string()).optional(),
email: z.array(z.string()).optional(),
hash: z.array(z.string()).optional(),
})
.optional(),
entity: z
.object({
lastSeenTimestamp: z.string().datetime(),
schemaVersion: z.string(),
definitionVersion: z.string(),
displayName: z.string(),
identityFields: z.array(z.string()),
id: z.string(),
type: z.literal('node'),
firstSeenTimestamp: z.string().datetime(),
definitionId: z.string(),
})
.optional(),
});
export type HostEntity = z.infer<typeof HostEntity>;
export const HostEntity = z.object({
host: z
.object({
hostname: z.array(z.string()).optional(),
domain: z.array(z.string()).optional(),
ip: z.array(z.string()).optional(),
name: z.string(),
id: z.array(z.string()).optional(),
type: z.array(z.string()).optional(),
mac: z.array(z.string()).optional(),
architecture: z.array(z.string()).optional(),
})
.optional(),
entity: z
.object({
lastSeenTimestamp: z.string().datetime(),
schemaVersion: z.string(),
definitionVersion: z.string(),
displayName: z.string(),
identityFields: z.array(z.string()),
id: z.string(),
type: z.literal('node'),
firstSeenTimestamp: z.string().datetime(),
definitionId: z.string(),
})
.optional(),
});
export type Entity = z.infer<typeof Entity>;
export const Entity = z.union([UserEntity, HostEntity]);

159
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/common.schema.yaml Normal file
View file

@ -0,0 +1,159 @@
openapi: 3.0.0
info:
title: Common Entities Schemas
description: Common Entities schemas for the Entity Store
version: '1'
paths: {}
components:
schemas:
UserEntity:
type: object
properties:
user:
type: object
properties:
full_name:
type: array
items:
type: string
domain:
type: array
items:
type: string
roles:
type: array
items:
type: string
name:
type: string
id:
type: array
items:
type: string
email:
type: array
items:
type: string
hash:
type: array
items:
type: string
required:
- name
entity:
type: object
properties:
lastSeenTimestamp:
type: string
format: date-time
schemaVersion:
type: string
definitionVersion:
type: string
displayName:
type: string
identityFields:
type: array
items:
type: string
id:
type: string
type:
type: string
enum:
- node
firstSeenTimestamp:
type: string
format: date-time
definitionId:
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
HostEntity:
type: object
properties:
host:
type: object
properties:
hostname:
type: array
items:
type: string
domain:
type: array
items:
type: string
ip:
type: array
items:
type: string
name:
type: string
id:
type: array
items:
type: string
type:
type: array
items:
type: string
mac:
type: array
items:
type: string
architecture:
type: array
items:
type: string
required:
- name
entity:
type: object
properties:
lastSeenTimestamp:
type: string
format: date-time
schemaVersion:
type: string
definitionVersion:
type: string
displayName:
type: string
identityFields:
type: array
items:
type: string
id:
type: string
type:
type: string
enum:
- node
firstSeenTimestamp:
type: string
format: date-time
definitionId:
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
Entity:
oneOf:
- $ref: '#/components/schemas/UserEntity'
- $ref: '#/components/schemas/HostEntity'

44
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/list_entities.gen.ts Normal file
View file

@ -0,0 +1,44 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/*
* NOTICE: Do not edit this file manually.
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
*
* info:
* title: Entities List Schema
* version: 2023-10-31
*/
import { z } from '@kbn/zod';
import { ArrayFromString } from '@kbn/zod-helpers';
import { EntityType, InspectQuery } from '../common.gen';
import { Entity } from './common.gen';
export type ListEntitiesRequestQuery = z.infer<typeof ListEntitiesRequestQuery>;
export const ListEntitiesRequestQuery = z.object({
sort_field: z.string().optional(),
sort_order: z.enum(['asc', 'desc']).optional(),
page: z.coerce.number().int().min(1).optional(),
per_page: z.coerce.number().int().min(1).max(10000).optional(),
/**
* An ES query to filter by.
*/
filterQuery: z.string().optional(),
entities_types: ArrayFromString(EntityType),
});
export type ListEntitiesRequestQueryInput = z.input<typeof ListEntitiesRequestQuery>;
export type ListEntitiesResponse = z.infer<typeof ListEntitiesResponse>;
export const ListEntitiesResponse = z.object({
records: z.array(Entity),
page: z.number().int().min(1),
per_page: z.number().int().min(1).max(1000),
total: z.number().int().min(0),
inspect: InspectQuery.optional(),
});

82
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/list_entities.schema.yaml Normal file
View file

@ -0,0 +1,82 @@
# ⚠️ Updating this file? Also update the public API docs at https://github.com/elastic/security-docs/tree/main/docs/advanced-entity-analytics/api
openapi: 3.0.0
info:
version: '2023-10-31'
title: Entities List Schema
paths:
/api/entity_store/entities/list:
get:
x-labels: [ess, serverless]
x-codegen-enabled: true
operationId: ListEntities
summary: List Entity Store Entities
description: List entities records, paging, sorting and filtering as needed.
parameters:
- name: sort_field
in: query
required: false
schema:
type: string
- name: sort_order
in: query
required: false
schema:
type: string
enum:
- asc
- desc
- name: page
in: query
required: false
schema:
type: integer
minimum: 1
- name: per_page
in: query
required: false
schema:
type: integer
minimum: 1
maximum: 10000
- name: filterQuery
in: query
required: false
schema:
type: string
description: An ES query to filter by.
- name: entities_types
in: query
required: true
schema:
type: array
items:
$ref: '../common.schema.yaml#/components/schemas/EntityType'
responses:
'200':
description: Entities returned successfully
content:
application/json:
schema:
type: object
properties:
records:
type: array
items:
$ref: './common.schema.yaml#/components/schemas/Entity'
page:
type: integer
minimum: 1
per_page:
type: integer
minimum: 1
maximum: 1000
total:
type: integer
minimum: 0
inspect:
$ref: '../common.schema.yaml#/components/schemas/InspectQuery'
required:
- records
- page
- per_page
- total

24
x-pack/plugins/security_solution/common/api/quickstart_client.gen.ts
View file

@ -270,6 +270,10 @@ import type {
StopEntityStoreRequestParamsInput,
StopEntityStoreResponse,
} from './entity_analytics/entity_store/engine/stop.gen';
import type {
ListEntitiesRequestQueryInput,
ListEntitiesResponse,
} from './entity_analytics/entity_store/entities/list_entities.gen';
import type { DisableRiskEngineResponse } from './entity_analytics/risk_engine/engine_disable_route.gen';
import type { EnableRiskEngineResponse } from './entity_analytics/risk_engine/engine_enable_route.gen';
import type { InitRiskEngineResponse } from './entity_analytics/risk_engine/engine_init_route.gen';
@ -1446,6 +1450,23 @@ finalize it.
})
.catch(catchAxiosErrorFormatAndThrow);
}
/**
* List entities records, paging, sorting and filtering as needed.
*/
async listEntities(props: ListEntitiesProps) {
this.log.info(`${new Date().toISOString()} Calling API ListEntities`);
return this.kbnClient
.request<ListEntitiesResponse>({
path: '/api/entity_store/entities/list',
headers: {
[ELASTIC_HTTP_VERSION_HEADER]: '2023-10-31',
},
method: 'GET',
query: props.query,
})
.catch(catchAxiosErrorFormatAndThrow);
}
async listEntityStoreEngines() {
this.log.info(`${new Date().toISOString()} Calling API ListEntityStoreEngines`);
return this.kbnClient
@ -2068,6 +2089,9 @@ export interface InstallPrepackedTimelinesProps {
export interface InternalUploadAssetCriticalityRecordsProps {
attachment: FormData;
}
export interface ListEntitiesProps {
query: ListEntitiesRequestQueryInput;
}
export interface PatchRuleProps {
body: PatchRuleRequestBodyInput;
}

15
x-pack/plugins/security_solution/common/entity_analytics/entity_store/constants.ts Normal file
View file

@ -0,0 +1,15 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/**
* Entity Store routes
*/
export const ENTITY_STORE_URL = '/api/entity_store' as const;
export const ENTITIES_URL = `${ENTITY_STORE_URL}/entities` as const;
export const LIST_ENTITIES_URL = `${ENTITIES_URL}/list` as const;

240
x-pack/plugins/security_solution/docs/openapi/ess/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml
View file

@ -437,6 +437,82 @@ paths:
summary: Stop the Entity Store engine
tags:
- Security Solution Entity Analytics API
/api/entity_store/entities/list:
get:
description: 'List entities records, paging, sorting and filtering as needed.'
operationId: ListEntities
parameters:
- in: query
name: sort_field
required: false
schema:
type: string
- in: query
name: sort_order
required: false
schema:
enum:
- asc
- desc
type: string
- in: query
name: page
required: false
schema:
minimum: 1
type: integer
- in: query
name: per_page
required: false
schema:
maximum: 10000
minimum: 1
type: integer
- description: An ES query to filter by.
in: query
name: filterQuery
required: false
schema:
type: string
- in: query
name: entities_types
required: true
schema:
items:
$ref: '#/components/schemas/EntityType'
type: array
responses:
'200':
content:
application/json:
schema:
type: object
properties:
inspect:
$ref: '#/components/schemas/InspectQuery'
page:
minimum: 1
type: integer
per_page:
maximum: 1000
minimum: 1
type: integer
records:
items:
$ref: '#/components/schemas/Entity'
type: array
total:
minimum: 0
type: integer
required:
- records
- page
- per_page
- total
description: Entities returned successfully
summary: List Entity Store Entities
tags:
- Security Solution Entity Analytics API
/api/risk_score/engine/schedule_now:
post:
operationId: ScheduleRiskEngineNow
@ -549,11 +625,90 @@ components:
- started
- stopped
type: string
Entity:
oneOf:
- $ref: '#/components/schemas/UserEntity'
- $ref: '#/components/schemas/HostEntity'
EntityType:
enum:
- user
- host
type: string
HostEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
host:
type: object
properties:
architecture:
items:
type: string
type: array
domain:
items:
type: string
type: array
hostname:
items:
type: string
type: array
id:
items:
type: string
type: array
ip:
items:
type: string
type: array
mac:
items:
type: string
type: array
name:
type: string
type:
items:
type: string
type: array
required:
- name
IdField:
enum:
- host.name
@ -561,6 +716,20 @@ components:
type: string
IndexPattern:
type: string
InspectQuery:
type: object
properties:
dsl:
items:
type: string
type: array
response:
items:
type: string
type: array
required:
- dsl
- response
RiskEngineScheduleNowErrorResponse:
type: object
properties:
@ -588,6 +757,77 @@ components:
required:
- status_code
- message
UserEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
user:
type: object
properties:
domain:
items:
type: string
type: array
email:
items:
type: string
type: array
full_name:
items:
type: string
type: array
hash:
items:
type: string
type: array
id:
items:
type: string
type: array
name:
type: string
roles:
items:
type: string
type: array
required:
- name
securitySchemes:
BasicAuth:
scheme: basic

240
x-pack/plugins/security_solution/docs/openapi/serverless/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml
View file

@ -437,6 +437,82 @@ paths:
summary: Stop the Entity Store engine
tags:
- Security Solution Entity Analytics API
/api/entity_store/entities/list:
get:
description: 'List entities records, paging, sorting and filtering as needed.'
operationId: ListEntities
parameters:
- in: query
name: sort_field
required: false
schema:
type: string
- in: query
name: sort_order
required: false
schema:
enum:
- asc
- desc
type: string
- in: query
name: page
required: false
schema:
minimum: 1
type: integer
- in: query
name: per_page
required: false
schema:
maximum: 10000
minimum: 1
type: integer
- description: An ES query to filter by.
in: query
name: filterQuery
required: false
schema:
type: string
- in: query
name: entities_types
required: true
schema:
items:
$ref: '#/components/schemas/EntityType'
type: array
responses:
'200':
content:
application/json:
schema:
type: object
properties:
inspect:
$ref: '#/components/schemas/InspectQuery'
page:
minimum: 1
type: integer
per_page:
maximum: 1000
minimum: 1
type: integer
records:
items:
$ref: '#/components/schemas/Entity'
type: array
total:
minimum: 0
type: integer
required:
- records
- page
- per_page
- total
description: Entities returned successfully
summary: List Entity Store Entities
tags:
- Security Solution Entity Analytics API
/api/risk_score/engine/schedule_now:
post:
operationId: ScheduleRiskEngineNow
@ -549,11 +625,90 @@ components:
- started
- stopped
type: string
Entity:
oneOf:
- $ref: '#/components/schemas/UserEntity'
- $ref: '#/components/schemas/HostEntity'
EntityType:
enum:
- user
- host
type: string
HostEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
host:
type: object
properties:
architecture:
items:
type: string
type: array
domain:
items:
type: string
type: array
hostname:
items:
type: string
type: array
id:
items:
type: string
type: array
ip:
items:
type: string
type: array
mac:
items:
type: string
type: array
name:
type: string
type:
items:
type: string
type: array
required:
- name
IdField:
enum:
- host.name
@ -561,6 +716,20 @@ components:
type: string
IndexPattern:
type: string
InspectQuery:
type: object
properties:
dsl:
items:
type: string
type: array
response:
items:
type: string
type: array
required:
- dsl
- response
RiskEngineScheduleNowErrorResponse:
type: object
properties:
@ -588,6 +757,77 @@ components:
required:
- status_code
- message
UserEntity:
type: object
properties:
entity:
type: object
properties:
definitionId:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
user:
type: object
properties:
domain:
items:
type: string
type: array
email:
items:
type: string
type: array
full_name:
items:
type: string
type: array
hash:
items:
type: string
type: array
id:
items:
type: string
type: array
name:
type: string
roles:
items:
type: string
type: array
required:
- name
securitySchemes:
BasicAuth:
scheme: basic

30
x-pack/plugins/security_solution/public/entity_analytics/api/api.ts
View file

@ -6,6 +6,7 @@
*/
import { useMemo } from 'react';
import { LIST_ENTITIES_URL } from '../../../common/entity_analytics/entity_store/constants';
import type { RiskEngineScheduleNowResponse } from '../../../common/api/entity_analytics/risk_engine/engine_schedule_now_route.gen';
import type { DisableRiskEngineResponse } from '../../../common/api/entity_analytics/risk_engine/engine_disable_route.gen';
import type { UploadAssetCriticalityRecordsResponse } from '../../../common/api/entity_analytics/asset_criticality/upload_asset_criticality_csv.gen';
@ -44,6 +45,8 @@ import {
import type { SnakeToCamelCase } from '../common/utils';
import { useKibana } from '../../common/lib/kibana/kibana_react';
import type { ReadRiskEngineSettingsResponse } from '../../../common/api/entity_analytics/risk_engine';
import type { ListEntitiesResponse } from '../../../common/api/entity_analytics/entity_store/entities/list_entities.gen';
import { type ListEntitiesRequestQuery } from '../../../common/api/entity_analytics/entity_store/entities/list_entities.gen';
export interface DeleteAssetCriticalityResponse {
deleted: true;
@ -69,6 +72,30 @@ export const useEntityAnalyticsRoutes = () => {
signal,
});
/**
* Fetches entities from the Entity Store
*/
const fetchEntitiesList = ({
signal,
params,
}: {
signal?: AbortSignal;
params: FetchEntitiesListParams;
}) =>
http.fetch<ListEntitiesResponse>(LIST_ENTITIES_URL, {
version: API_VERSIONS.public.v1,
method: 'GET',
query: {
entities_types: params.entitiesTypes,
sort_field: params.sortField,
sort_order: params.sortOrder,
page: params.page,
per_page: params.perPage,
filterQuery: params.filterQuery,
},
signal,
});
/**
* Fetches risks engine status
*/
@ -256,8 +283,11 @@ export const useEntityAnalyticsRoutes = () => {
getRiskScoreIndexStatus,
fetchRiskEngineSettings,
calculateEntityRiskScore,
fetchEntitiesList,
};
}, [http]);
};
export type AssetCriticality = SnakeToCamelCase<AssetCriticalityRecord>;
export type FetchEntitiesListParams = SnakeToCamelCase<ListEntitiesRequestQuery>;

34
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/__snapshots__/entity_store_data_client.test.ts.snap generated Normal file
View file

@ -0,0 +1,34 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP
exports[`EntityStoreDataClient search entities returns inspect query params 1`] = `
Object {
"dsl": Array [
"{
\\"index\\": [
\\".entities.v1.latest.ea_host_entity_store\\"
],
\\"body\\": {
\\"bool\\": {
\\"filter\\": []
}
}
}",
],
"response": Array [
"{
\\"took\\": 0,
\\"timed_out\\": false,
\\"_shards\\": {
\\"total\\": 0,
\\"successful\\": 0,
\\"skipped\\": 0,
\\"failed\\": 0
},
\\"hits\\": {
\\"total\\": 0,
\\"hits\\": []
}
}",
],
}
`;

2
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/constants.ts
View file

@ -19,3 +19,5 @@ export const ENGINE_STATUS: Record<Uppercase<EngineStatus>, EngineStatus> = {
STARTED: 'started',
STOPPED: 'stopped',
};
export const MAX_SEARCH_RESPONSE_SIZE = 10_000;

92
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/definition.ts
View file

@ -7,50 +7,52 @@
import { entityDefinitionSchema, type EntityDefinition } from '@kbn/entities-schema';
import { ENTITY_STORE_DEFAULT_SOURCE_INDICES } from './constants';
import { getEntityDefinitionId } from './utils/utils';
export const HOST_ENTITY_DEFINITION: EntityDefinition = entityDefinitionSchema.parse({
id: 'ea_host_entity_store',
name: 'EA Host Store',
type: 'host',
indexPatterns: ENTITY_STORE_DEFAULT_SOURCE_INDICES,
identityFields: ['host.name'],
displayNameTemplate: '{{host.name}}',
metadata: [
'host.domain',
'host.hostname',
'host.id',
'host.ip',
'host.mac',
'host.name',
'host.type',
'host.architecture',
],
history: {
timestampField: '@timestamp',
interval: '1m',
},
version: '1.0.0',
});
export const buildHostEntityDefinition = (): EntityDefinition =>
entityDefinitionSchema.parse({
id: getEntityDefinitionId('host'),
name: 'EA Host Store',
type: 'host',
indexPatterns: ENTITY_STORE_DEFAULT_SOURCE_INDICES,
identityFields: ['host.name'],
displayNameTemplate: '{{host.name}}',
metadata: [
'host.domain',
'host.hostname',
'host.id',
'host.ip',
'host.mac',
'host.name',
'host.type',
'host.architecture',
],
history: {
timestampField: '@timestamp',
interval: '1m',
},
version: '1.0.0',
});
export const USER_ENTITY_DEFINITION: EntityDefinition = entityDefinitionSchema.parse({
id: 'ea_user_entity_store',
name: 'EA User Store',
type: 'user',
indexPatterns: ENTITY_STORE_DEFAULT_SOURCE_INDICES,
identityFields: ['user.name'],
displayNameTemplate: '{{user.name}}',
metadata: [
'user.domain',
'user.email',
'user.full_name',
'user.hash',
'user.id',
'user.name',
'user.roles',
],
history: {
timestampField: '@timestamp',
interval: '1m',
},
version: '1.0.0',
});
export const buildUserEntityDefinition = (): EntityDefinition =>
entityDefinitionSchema.parse({
id: getEntityDefinitionId('user'),
name: 'EA User Store',
indexPatterns: ENTITY_STORE_DEFAULT_SOURCE_INDICES,
identityFields: ['user.name'],
displayNameTemplate: '{{user.name}}',
metadata: [
'user.domain',
'user.email',
'user.full_name',
'user.hash',
'user.id',
'user.name',
'user.roles',
],
history: {
timestampField: '@timestamp',
interval: '1m',
},
version: '1.0.0',
});

1
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.mock.ts
View file

@ -15,6 +15,7 @@ const createEntityStoreDataClientMock = () =>
get: jest.fn(),
list: jest.fn(),
delete: jest.fn(),
searchEntities: jest.fn(),
} as unknown as jest.Mocked<EntityStoreDataClient>);
export const entityStoreDataClientMock = { create: createEntityStoreDataClientMock };

138
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.test.ts Normal file
View file

@ -0,0 +1,138 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import {
loggingSystemMock,
elasticsearchServiceMock,
savedObjectsClientMock,
} from '@kbn/core/server/mocks';
import { EntityStoreDataClient } from './entity_store_data_client';
import { EntityClient } from '@kbn/entityManager-plugin/server/lib/entity_client';
import type { SortOrder } from '@elastic/elasticsearch/lib/api/types';
import type { EntityType } from '../../../../common/api/entity_analytics/entity_store/common.gen';
describe('EntityStoreDataClient', () => {
const logger = loggingSystemMock.createLogger();
const mockSavedObjectClient = savedObjectsClientMock.create();
const esClientMock = elasticsearchServiceMock.createScopedClusterClient().asInternalUser;
const loggerMock = loggingSystemMock.createLogger();
const dataClient = new EntityStoreDataClient({
esClient: esClientMock,
logger: loggerMock,
namespace: 'default',
soClient: mockSavedObjectClient,
entityClient: new EntityClient({
esClient: esClientMock,
soClient: mockSavedObjectClient,
logger,
}),
});
const defaultSearchParams = {
entityTypes: ['host'] as EntityType[],
page: 1,
perPage: 10,
sortField: 'hostName',
sortOrder: 'asc' as SortOrder,
};
describe('search entities', () => {
beforeEach(() => {
jest.resetAllMocks();
esClientMock.search.mockResolvedValue({
took: 0,
timed_out: false,
_shards: {
total: 0,
successful: 0,
skipped: 0,
failed: 0,
},
hits: {
total: 0,
hits: [],
},
});
});
it('searches in the entities store indices', async () => {
await dataClient.searchEntities({
...defaultSearchParams,
entityTypes: ['host', 'user'],
});
expect(esClientMock.search).toHaveBeenCalledWith(
expect.objectContaining({
index: [
'.entities.v1.latest.ea_host_entity_store',
'.entities.v1.latest.ea_user_entity_store',
],
})
);
});
it('should filter by filterQuery param', async () => {
await dataClient.searchEntities({
...defaultSearchParams,
filterQuery: '{"match_all":{}}',
});
expect(esClientMock.search).toHaveBeenCalledWith(
expect.objectContaining({ query: { bool: { filter: [{ match_all: {} }] } } })
);
});
it('should paginate', async () => {
await dataClient.searchEntities({
...defaultSearchParams,
page: 3,
perPage: 7,
});
expect(esClientMock.search).toHaveBeenCalledWith(
expect.objectContaining({ from: 14, size: 7 })
);
});
it('should sort', async () => {
await dataClient.searchEntities({
...defaultSearchParams,
sortField: '@timestamp',
sortOrder: 'asc',
});
expect(esClientMock.search).toHaveBeenCalledWith(
expect.objectContaining({ sort: [{ '@timestamp': 'asc' }] })
);
});
it('caps the size to the maximum query size', async () => {
await dataClient.searchEntities({
...defaultSearchParams,
perPage: 999_999,
});
const maxSize = 10_000;
expect(esClientMock.search).toHaveBeenCalledWith(expect.objectContaining({ size: maxSize }));
});
it('ignores an index_not_found_exception if the entity index does not exist', async () => {
await dataClient.searchEntities(defaultSearchParams);
expect(esClientMock.search).toHaveBeenCalledWith(
expect.objectContaining({ ignore_unavailable: true })
);
});
it('returns inspect query params', async () => {
const response = await dataClient.searchEntities(defaultSearchParams);
expect(response.inspect).toMatchSnapshot();
});
});
});

57
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.ts
View file

@ -8,6 +8,9 @@
import type { Logger, ElasticsearchClient, SavedObjectsClientContract } from '@kbn/core/server';
import type { EntityClient } from '@kbn/entityManager-plugin/server/lib/entity_client';
import type { SortOrder } from '@elastic/elasticsearch/lib/api/types';
import type { Entity } from '../../../../common/api/entity_analytics/entity_store/entities/common.gen';
import { createQueryFilterClauses } from '../../../utils/build_query';
import type {
InitEntityStoreRequestBody,
InitEntityStoreResponse,
@ -15,11 +18,12 @@ import type {
import type {
EngineDescriptor,
EntityType,
InspectQuery,
} from '../../../../common/api/entity_analytics/entity_store/common.gen';
import { entityEngineDescriptorTypeName } from './saved_object';
import { EngineDescriptorClient } from './saved_object/engine_descriptor';
import { getEntityDefinition } from './utils/utils';
import { ENGINE_STATUS } from './constants';
import { getEntitiesIndexName, getEntityDefinition } from './utils/utils';
import { ENGINE_STATUS, MAX_SEARCH_RESPONSE_SIZE } from './constants';
interface EntityStoreClientOpts {
logger: Logger;
@ -29,6 +33,15 @@ interface EntityStoreClientOpts {
soClient: SavedObjectsClientContract;
}
interface SearchEntitiesParams {
entityTypes: EntityType[];
filterQuery?: string;
page: number;
perPage: number;
sortField: string;
sortOrder: SortOrder;
}
export class EntityStoreDataClient {
private engineClient: EngineDescriptorClient;
constructor(private readonly options: EntityStoreClientOpts) {
@ -117,4 +130,44 @@ export class EntityStoreDataClient {
return { deleted: true };
}
public async searchEntities(params: SearchEntitiesParams): Promise<{
records: Entity[];
total: number;
inspect: InspectQuery;
}> {
const { page, perPage, sortField, sortOrder, filterQuery, entityTypes } = params;
const index = entityTypes.map(getEntitiesIndexName);
const from = (page - 1) * perPage;
const sort = sortField ? [{ [sortField]: sortOrder }] : undefined;
const filter = [...createQueryFilterClauses(filterQuery)];
const query = {
bool: {
filter,
},
};
const response = await this.options.esClient.search<Entity>({
index,
query,
size: Math.min(perPage, MAX_SEARCH_RESPONSE_SIZE),
from,
sort,
ignore_unavailable: true,
});
const { hits } = response;
const total = typeof hits.total === 'number' ? hits.total : hits.total?.value ?? 0;
const records = hits.hits.map((hit) => hit._source as Entity);
const inspect: InspectQuery = {
dsl: [JSON.stringify({ index, body: query }, null, 2)],
response: [JSON.stringify(response, null, 2)],
};
return { records, total, inspect };
}
}

92
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/routes/entities/list.ts Normal file
View file

@ -0,0 +1,92 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/*
* NOTICE: Do not edit this file manually.
* This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator.
*
* info:
* title: List Entity Store engines
* version: 1
*/
import type { IKibanaResponse, Logger } from '@kbn/core/server';
import { buildSiemResponse } from '@kbn/lists-plugin/server/routes/utils';
import { transformError } from '@kbn/securitysolution-es-utils';
import { buildRouteValidationWithZod } from '@kbn/zod-helpers';
import { LIST_ENTITIES_URL } from '../../../../../../common/entity_analytics/entity_store/constants';
import type { ListEntitiesResponse } from '../../../../../../common/api/entity_analytics/entity_store/entities/list_entities.gen';
import { ListEntitiesRequestQuery } from '../../../../../../common/api/entity_analytics/entity_store/entities/list_entities.gen';
import { APP_ID } from '../../../../../../common';
import { API_VERSIONS } from '../../../../../../common/entity_analytics/constants';
import type { EntityAnalyticsRoutesDeps } from '../../../types';
export const listEntitiesRoute = (router: EntityAnalyticsRoutesDeps['router'], logger: Logger) => {
router.versioned
.get({
access: 'public',
path: LIST_ENTITIES_URL,
options: {
tags: ['access:securitySolution', `access:${APP_ID}-entity-analytics`],
},
})
.addVersion(
{
version: API_VERSIONS.public.v1,
validate: {
request: {
query: buildRouteValidationWithZod(ListEntitiesRequestQuery),
},
},
},
async (context, request, response): Promise<IKibanaResponse<ListEntitiesResponse>> => {
const siemResponse = buildSiemResponse(response);
try {
const {
page = 1,
per_page: perPage = 10,
sort_field: sortField = 'entity.lastSeenTimestamp',
sort_order: sortOrder = 'desc',
entities_types: entityTypes,
filterQuery,
} = request.query;
const securitySolution = await context.securitySolution;
const entityStoreClient = securitySolution.getEntityStoreDataClient();
const { records, total, inspect } = await entityStoreClient.searchEntities({
entityTypes,
filterQuery,
page,
perPage,
sortField,
sortOrder,
});
return response.ok({
body: {
records,
total,
page,
per_page: perPage,
inspect,
},
});
} catch (e) {
logger.error(e);
const error = transformError(e);
return siemResponse.error({
statusCode: error.statusCode,
body: error.message,
});
}
}
);
};

2
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/routes/register_entity_store_routes.ts
View file

@ -7,6 +7,7 @@
import type { EntityAnalyticsRoutesDeps } from '../../types';
import { deleteEntityEngineRoute } from './delete';
import { listEntitiesRoute } from './entities/list';
import { getEntityEngineRoute } from './get';
import { initEntityEngineRoute } from './init';
import { listEntityEnginesRoute } from './list';
@ -20,4 +21,5 @@ export const registerEntityStoreRoutes = ({ router, logger }: EntityAnalyticsRou
deleteEntityEngineRoute(router, logger);
getEntityEngineRoute(router, logger);
listEntityEnginesRoute(router, logger);
listEntitiesRoute(router, logger);
};

21
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/utils/utils.ts
View file

@ -6,16 +6,22 @@
*/
import type { SavedObjectsFindResponse } from '@kbn/core-saved-objects-api-server';
import {
ENTITY_LATEST,
ENTITY_SCHEMA_VERSION_V1,
entitiesIndexPattern,
} from '@kbn/entities-schema';
import type {
EngineDescriptor,
EntityType,
} from '../../../../../common/api/entity_analytics/entity_store/common.gen';
import { HOST_ENTITY_DEFINITION, USER_ENTITY_DEFINITION } from '../definition';
import { buildHostEntityDefinition, buildUserEntityDefinition } from '../definition';
import { entityEngineDescriptorTypeName } from '../saved_object';
export const getEntityDefinition = (entityType: EntityType) => {
if (entityType === 'host') return HOST_ENTITY_DEFINITION;
if (entityType === 'user') return USER_ENTITY_DEFINITION;
if (entityType === 'host') return buildHostEntityDefinition();
if (entityType === 'user') return buildUserEntityDefinition();
throw new Error(`Unsupported entity type: ${entityType}`);
};
@ -31,3 +37,12 @@ export const ensureEngineExists =
export const getByEntityTypeQuery = (entityType: EntityType) => {
return `${entityEngineDescriptorTypeName}.attributes.type: ${entityType}`;
};
export const getEntitiesIndexName = (entityType: EntityType) =>
entitiesIndexPattern({
schemaVersion: ENTITY_SCHEMA_VERSION_V1,
dataset: ENTITY_LATEST,
definitionId: getEntityDefinitionId(entityType),
});
export const getEntityDefinitionId = (entityType: EntityType) => `ea_${entityType}_entity_store`;

15
x-pack/test/api_integration/services/security_solution_api.gen.ts
View file

@ -102,6 +102,7 @@ import {
InitEntityStoreRequestBodyInput,
} from '@kbn/security-solution-plugin/common/api/entity_analytics/entity_store/engine/init.gen';
import { InstallPrepackedTimelinesRequestBodyInput } from '@kbn/security-solution-plugin/common/api/timeline/install_prepackaged_timelines/install_prepackaged_timelines_route.gen';
import { ListEntitiesRequestQueryInput } from '@kbn/security-solution-plugin/common/api/entity_analytics/entity_store/entities/list_entities.gen';
import { PatchRuleRequestBodyInput } from '@kbn/security-solution-plugin/common/api/detection_engine/rule_management/crud/patch_rule/patch_rule_route.gen';
import { PatchTimelineRequestBodyInput } from '@kbn/security-solution-plugin/common/api/timeline/patch_timelines/patch_timeline_route.gen';
import {
@ -844,6 +845,17 @@ finalize it.
.set(ELASTIC_HTTP_VERSION_HEADER, '1')
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana');
},
/**
* List entities records, paging, sorting and filtering as needed.
*/
listEntities(props: ListEntitiesProps) {
return supertest
.get('/api/entity_store/entities/list')
.set('kbn-xsrf', 'true')
.set(ELASTIC_HTTP_VERSION_HEADER, '2023-10-31')
.set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
.query(props.query);
},
listEntityStoreEngines() {
return supertest
.get('/api/entity_store/engines')
@ -1313,6 +1325,9 @@ export interface InitEntityStoreProps {
export interface InstallPrepackedTimelinesProps {
body: InstallPrepackedTimelinesRequestBodyInput;
}
export interface ListEntitiesProps {
query: ListEntitiesRequestQueryInput;
}
export interface PatchRuleProps {
body: PatchRuleRequestBodyInput;
}

85
x-pack/test/functional/es_archives/security_solution/entity_store/data.json Normal file
View file

@ -0,0 +1,85 @@
{
"type": "doc",
"value": {
"id": "a4cf452c1e0375c3d4412cb550ad1783358468a3b3b777da4829d72c7d6fb74f",
"index": ".entities.v1.latest.ea_user_entity_store",
"source": {
"event": {
"ingested": "2024-09-11T11:26:49.706875Z"
},
"user": {
"full_name": [],
"domain": [],
"roles": [],
"name": "hinamatsumoto",
"id": [],
"email": [],
"hash": []
},
"entity": {
"lastSeenTimestamp": "2024-09-11T11:24:15.588Z",
"schemaVersion": "v1",
"definitionVersion": "1.0.0",
"displayName": "hinamatsumoto",
"identityFields": [
"user.name"
],
"id": "LBQAgKHGmpup0Kg9nlKmeQ==",
"type": "node",
"firstSeenTimestamp": "2024-09-11T10:46:00.000Z",
"definitionId": "ea_user_entity_store"
}
}
}
}
{
"type": "doc",
"value": {
"id": "a2cf452c1e0375c3d4412cb550bd1783358468a3b3b777da4829d72c7d6fb71f",
"index": ".entities.v1.latest.ea_host_entity_store",
"source": {
"event": {
"ingested": "2024-09-11T11:26:49.641707Z"
},
"host": {
"hostname": [
"ali-ubuntu-server"
],
"domain": [],
"ip": [
"1050::6:600:300c:326c",
"192.168.1.10",
"1050::5:700:400d:427c",
"10.142.2.222"
],
"name": "ali-ubuntu-server",
"id": [
"new_host_id",
"b123c1d92f3821b748a7218b4e78125f"
],
"type": [],
"mac": [
"42-2b-ff-8e-ac-2f",
"51-3c-ff-9e-ac-2g"
],
"architecture": [
"x86_64"
]
},
"entity": {
"lastSeenTimestamp": "2024-09-11T11:24:15.591Z",
"schemaVersion": "v1",
"definitionVersion": "1.0.0",
"displayName": "ali-ubuntu-server",
"identityFields": [
"host.name"
],
"id": "ZXKm6GEcUJY6NHkMgPPmGQ==",
"type": "node",
"firstSeenTimestamp": "2024-09-11T10:46:00.000Z",
"definitionId": "ea_host_entity_store"
}
}
}
}

303
x-pack/test/functional/es_archives/security_solution/entity_store/mappings.json Normal file
View file

@ -0,0 +1,303 @@
{
"type": "index",
"value": {
"index": ".entities.v1.latest.ea_host_entity_store",
"mappings": {
"date_detection": false,
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"text": {
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
}
}
},
{
"entity_metrics": {
"path_match": "entity.metrics.*",
"match_mapping_type": [
"long",
"double"
],
"mapping": {
"type": "{dynamic_type}"
}
}
}
],
"properties": {
"entity": {
"properties": {
"definitionId": {
"type": "keyword",
"ignore_above": 1024
},
"definitionVersion": {
"type": "keyword",
"ignore_above": 1024
},
"displayName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"firstSeenTimestamp": {
"type": "date"
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"identityFields": {
"type": "keyword"
},
"lastSeenTimestamp": {
"type": "date"
},
"schemaVersion": {
"type": "keyword",
"ignore_above": 1024
},
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"event": {
"properties": {
"ingested": {
"type": "date"
}
}
},
"host": {
"properties": {
"architecture": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"hostname": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"id": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"ip": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"mac": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"name": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"labels": {
"type": "object"
},
"tags": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"settings": {
"index": {
"auto_expand_replicas": "0-1",
"number_of_replicas": "0",
"number_of_shards": "1"
}
}
}
}
{
"type": "index",
"value": {
"index": ".entities.v1.latest.ea_user_entity_store",
"mappings": {
"date_detection": false,
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"text": {
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
}
}
},
{
"entity_metrics": {
"path_match": "entity.metrics.*",
"match_mapping_type": [
"long",
"double"
],
"mapping": {
"type": "{dynamic_type}"
}
}
}
],
"properties": {
"entity": {
"properties": {
"definitionId": {
"type": "keyword",
"ignore_above": 1024
},
"definitionVersion": {
"type": "keyword",
"ignore_above": 1024
},
"displayName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"firstSeenTimestamp": {
"type": "date"
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"identityFields": {
"type": "keyword"
},
"lastSeenTimestamp": {
"type": "date"
},
"schemaVersion": {
"type": "keyword",
"ignore_above": 1024
},
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"event": {
"properties": {
"ingested": {
"type": "date"
}
}
},
"labels": {
"type": "object"
},
"tags": {
"type": "keyword",
"ignore_above": 1024
},
"user": {
"properties": {
"domain": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"email": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"id": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"name": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
},
"settings": {
"index": {
"auto_expand_replicas": "0-1",
"number_of_replicas": "0",
"number_of_shards": "1"
}
}
}
}

28
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/ess.config.ts Normal file
View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FtrConfigProviderContext } from '@kbn/test';
export default async function ({ readConfigFile }: FtrConfigProviderContext) {
const functionalConfig = await readConfigFile(
require.resolve('../../../../../config/ess/config.base.trial')
);
return {
...functionalConfig.getAll(),
kbnTestServer: {
...functionalConfig.get('kbnTestServer'),
serverArgs: [
...functionalConfig.get('kbnTestServer.serverArgs'),
`--xpack.securitySolution.enableExperimental=${JSON.stringify(['entityStoreEnabled'])}`,
],
},
testFiles: [require.resolve('..')],
junit: {
reportName: 'Entity Analytics - Entity Store Integration Tests - ESS Env - Trial License',
},
};
}

24
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/serverless.config.ts Normal file
View file

@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { createTestConfig } from '../../../../../config/serverless/config.base';
export default createTestConfig({
kbnTestServerArgs: [
`--xpack.securitySolution.enableExperimental=${JSON.stringify(['entityStoreEnabled'])}`,
`--xpack.securitySolutionServerless.productTypes=${JSON.stringify([
{ product_line: 'security', product_tier: 'complete' },
{ product_line: 'endpoint', product_tier: 'complete' },
{ product_line: 'cloud', product_tier: 'complete' },
])}`,
],
testFiles: [require.resolve('..')],
junit: {
reportName:
'Entity Analytics - Entity Store Integration Tests - Serverless Env - Complete Tier',
},
});

71
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/entities_list.ts Normal file
View file

@ -0,0 +1,71 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import expect from 'expect';
import { FtrProviderContext } from '../../../../ftr_provider_context';
export default ({ getService }: FtrProviderContext) => {
const securitySolutionApi = getService('securitySolutionApi');
describe('@ess @serverless @skipInServerlessMKI Entity store - Entities list API', () => {
describe('when the entity store is disable', () => {
it("should return response with success status when the index doesn't exist", async () => {
const { body } = await securitySolutionApi.listEntities({
query: { entities_types: ['host'] },
});
expect(body).toEqual(
expect.objectContaining({
total: 0,
records: [],
})
);
});
});
describe('when the entity store is enable', () => {
const esArchiver = getService('esArchiver');
before(async () => {
await esArchiver.load('x-pack/test/functional/es_archives/security_solution/entity_store');
});
after(async () => {
await esArchiver.unload(
'x-pack/test/functional/es_archives/security_solution/entity_store'
);
});
it('should return hosts in the entity store index', async () => {
const { body } = await securitySolutionApi.listEntities({
query: { entities_types: ['host'] },
});
expect(body.total).toEqual(1);
expect(body.records.length).toEqual(1);
});
it('should return users in the entity store index', async () => {
const { body } = await securitySolutionApi.listEntities({
query: { entities_types: ['user'] },
});
expect(body.total).toEqual(1);
expect(body.records.length).toEqual(1);
});
it('should return all entities in the entity store index', async () => {
const { body } = await securitySolutionApi.listEntities({
query: { entities_types: ['user', 'host'] },
});
expect(body.total).toEqual(2);
expect(body.records.length).toEqual(2);
});
});
});
};

14
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/index.ts Normal file
View file

@ -0,0 +1,14 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FtrProviderContext } from '../../../../ftr_provider_context';
export default function ({ loadTestFile }: FtrProviderContext) {
describe('Entity Analytics - Entity Store', function () {
loadTestFile(require.resolve('./entities_list'));
});
}