github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 01:38:56 -04:00
Code Issues Projects Releases Packages Wiki Activity

[8.x] [Entity Analytics] Add Field Retention Enrich Policy and Ingest Pipeline to Entity Engine (#193848) (#195929)

Browse source 
# Backport

This will backport the following commits from `main` to `8.x`:
- [[Entity Analytics] Add Field Retention Enrich Policy and Ingest
Pipeline to Entity Engine
(#193848)](https://github.com/elastic/kibana/pull/193848)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Mark
Hopkin","email":"mark.hopkin@elastic.co"},"sourceCommit":{"committedDate":"2024-10-11T14:04:49Z","message":"[Entity
Analytics] Add Field Retention Enrich Policy and Ingest Pipeline to
Entity Engine (#193848)\n\n## Summary\r\n\r\nAdd the \"Ouroboros\" part
of the entity engine:\r\n\r\n- an enrich policy is created for each
engine\r\n- the enrich policy is executed every 30s by a kibana task,
this will be\r\n1h once we move to a 24h lookback\r\n- create an ingest
pipeline for the latest which performs the specified\r\nfield retention
operations (for more detail see below)\r\n\r\n<img width=\"2112\"
alt=\"Screenshot 2024-10-02 at 13 42
11\"\r\nsrc=\"https://github.com/user-attachments/assets/f727607f-2e0a-4056-a51e-393fb2a97a95\">\r\n\r\n<details>\r\n<summary>
Expand for example host entity </summary>\r\n```\r\n{\r\n
\"@timestamp\": \"2024-10-01T12:10:46.000Z\",\r\n \"host\": {\r\n
\"name\": \"host9\",\r\n \"hostname\": [\r\n \"host9\"\r\n ],\r\n
\"domain\": [\r\n \"test.com\"\r\n ],\r\n \"ip\": [\r\n \"1.1.1.1\",\r\n
\"1.1.1.2\",\r\n \"1.1.1.3\"\r\n ],\r\n \"risk\": {\r\n
\"calculated_score\": \"70.0\",\r\n \"calculated_score_norm\":
\"27.00200653076172\",\r\n \"calculated_level\": \"Low\"\r\n },\r\n
\"id\": [\r\n \"1234567890abcdef\"\r\n ],\r\n \"type\": [\r\n
\"server\"\r\n ],\r\n \"mac\": [\r\n \"AA:AA:AA:AA:AA:AB\",\r\n
\"aa:aa:aa:aa:aa:aa\",\r\n \"AA:AA:AA:AA:AA:AC\"\r\n ],\r\n
\"architecture\": [\r\n \"x86_64\"\r\n ]\r\n },\r\n \"asset\": {\r\n
\"criticality\": \"low_impact\"\r\n },\r\n \"entity\": {\r\n \"name\":
\"host9\",\r\n \"id\": \"kP/jiFHWSwWlO7W0+fGWrg==\",\r\n \"source\":
[\r\n \"risk-score.risk-score-latest-default\",\r\n
\".asset-criticality.asset-criticality-default\",\r\n
\".ds-logs-testlogs1-default-2024.10.01-000001\",\r\n
\".ds-logs-testlogs2-default-2024.10.01-000001\",\r\n
\".ds-logs-testlogs3-default-2024.10.01-000001\"\r\n ],\r\n \"type\":
\"host\"\r\n }\r\n}\r\n```\r\n</details>\r\n\r\n### Field retention
operators\r\n\r\nFirst some terminology:\r\n\r\n- **latest value** - the
value produced by the transform which\r\nrepresents the latest vioew of
a given field in the transform lookback\r\nperiod\r\n- **enrich value**
- the value added to the document by the enrich\r\npolicy, this
represents the last value of a field outiside of the\r\ntransform
lookback window\r\n\r\nWe hope that this will one day be merged into the
entity manager\r\nframework so I've tried to abstract this as much as
possible. A field\r\nretention operator specifies how we should choose a
value for a field\r\nwhen looking at the latest value and the enrich
value.\r\n\r\n### Collect values\r\nCollect unique values in an array,
first taking from the latest values\r\nand then filling with enrich
values up to maxLength.\r\n\r\n```\r\n{\r\n operation:
'collect_values',\r\n field: 'host.ip',\r\n maxLength:
10\r\n}\r\n```\r\n\r\n### Prefer newest value\r\nChoose the latest value
if present, otherwise choose the enrich value.\r\n\r\n```\r\n{\r\n
operation: 'prefer_newest_value',\r\n field:
'asset.criticality'\r\n}\r\n```\r\n\r\n### Prefer oldest value\r\nChoose
the enrich value if it is present, otherwise choose
latest.\r\n```\r\n{\r\n operation: 'prefer_oldest_value',\r\n field:
'first_seen_timestamp'\r\n}\r\n```\r\n\r\n## Test instructions\r\n\r\nWe
currently require extra permissions for the kibana system user
for\r\nthis to work, so we must\r\n\r\n### 1. Get Elasticsearch running
from source\r\nThis prototype requires a custom branch of elasticsearch
in order to\r\ngive the kibana system user more privileges.\r\n\r\n####
Step 1 - Clone the prototype branch\r\nThe elasticsearch branch is
at\r\nhttps://github.com/elastic/elasticsearch/tree/entity-store-permissions.\r\n\r\nOr
you can use [github command line](https://cli.github.com/)
to\r\ncheckout my draft PR:\r\n```\r\ngh pr checkout
113942\r\n```\r\n#### Step 2 - Install Java\r\nInstall
[homebrew](https://brew.sh/) if you do not have it.\r\n\r\n```\r\nbrew
install openjdk@21\r\nsudo ln -sfn
/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk
/Library/Java/JavaVirtualMachines/openjdk-21.jdk\r\n```\r\n\r\n#### Step
3 - Run elasticsearch\r\nThis makes sure your data stays between runs of
elasticsearch, and that\r\nyou have platinum license
features\r\n\r\n```\r\n./gradlew run --data-dir /tmp/elasticsearch-repo
--preserve-data -Drun.license_type=trial\r\n```\r\n\r\n### 2. Get Kibana
Running\r\n\r\n#### Step 1 - Connect kibana to elasticsearch\r\n\r\nSet
this in your kibana config:\r\n\r\n```\r\nelasticsearch.username:
elastic-admin\r\nelasticsearch.password: elastic-password\r\n```\r\nNow
start kibana and you should have connected to the elasticsearch
you\r\nmade.\r\n\r\n### 3. Initialise entity engine and send
data!\r\n\r\n- Initialise the host or user engine (or
both)\r\n\r\n```\r\ncurl -H 'Content-Type: application/json' \\\r\n -X
POST \\ \r\n -H 'kbn-xsrf: true' \\\r\n -H 'elastic-api-version:
2023-10-31' \\\r\n -d '{}' \\\r\n
http:///elastic:changeme@localhost:5601/api/entity_store/engines/host/init
\r\n```\r\n\r\n- use your favourite data generation tool to create data,
maybe\r\nhttps://github.com/elastic/security-documents-generator\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"51312159b0436841e0364d7aac0056757962907c","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:
SecuritySolution","backport:prev-minor","Team:Entity
Analytics"],"title":"[Entity Analytics] Add Field Retention Enrich
Policy and Ingest Pipeline to Entity
Engine","number":193848,"url":"https://github.com/elastic/kibana/pull/193848","mergeCommit":{"message":"[Entity
Analytics] Add Field Retention Enrich Policy and Ingest Pipeline to
Entity Engine (#193848)\n\n## Summary\r\n\r\nAdd the \"Ouroboros\" part
of the entity engine:\r\n\r\n- an enrich policy is created for each
engine\r\n- the enrich policy is executed every 30s by a kibana task,
this will be\r\n1h once we move to a 24h lookback\r\n- create an ingest
pipeline for the latest which performs the specified\r\nfield retention
operations (for more detail see below)\r\n\r\n<img width=\"2112\"
alt=\"Screenshot 2024-10-02 at 13 42
11\"\r\nsrc=\"https://github.com/user-attachments/assets/f727607f-2e0a-4056-a51e-393fb2a97a95\">\r\n\r\n<details>\r\n<summary>
Expand for example host entity </summary>\r\n```\r\n{\r\n
\"@timestamp\": \"2024-10-01T12:10:46.000Z\",\r\n \"host\": {\r\n
\"name\": \"host9\",\r\n \"hostname\": [\r\n \"host9\"\r\n ],\r\n
\"domain\": [\r\n \"test.com\"\r\n ],\r\n \"ip\": [\r\n \"1.1.1.1\",\r\n
\"1.1.1.2\",\r\n \"1.1.1.3\"\r\n ],\r\n \"risk\": {\r\n
\"calculated_score\": \"70.0\",\r\n \"calculated_score_norm\":
\"27.00200653076172\",\r\n \"calculated_level\": \"Low\"\r\n },\r\n
\"id\": [\r\n \"1234567890abcdef\"\r\n ],\r\n \"type\": [\r\n
\"server\"\r\n ],\r\n \"mac\": [\r\n \"AA:AA:AA:AA:AA:AB\",\r\n
\"aa:aa:aa:aa:aa:aa\",\r\n \"AA:AA:AA:AA:AA:AC\"\r\n ],\r\n
\"architecture\": [\r\n \"x86_64\"\r\n ]\r\n },\r\n \"asset\": {\r\n
\"criticality\": \"low_impact\"\r\n },\r\n \"entity\": {\r\n \"name\":
\"host9\",\r\n \"id\": \"kP/jiFHWSwWlO7W0+fGWrg==\",\r\n \"source\":
[\r\n \"risk-score.risk-score-latest-default\",\r\n
\".asset-criticality.asset-criticality-default\",\r\n
\".ds-logs-testlogs1-default-2024.10.01-000001\",\r\n
\".ds-logs-testlogs2-default-2024.10.01-000001\",\r\n
\".ds-logs-testlogs3-default-2024.10.01-000001\"\r\n ],\r\n \"type\":
\"host\"\r\n }\r\n}\r\n```\r\n</details>\r\n\r\n### Field retention
operators\r\n\r\nFirst some terminology:\r\n\r\n- **latest value** - the
value produced by the transform which\r\nrepresents the latest vioew of
a given field in the transform lookback\r\nperiod\r\n- **enrich value**
- the value added to the document by the enrich\r\npolicy, this
represents the last value of a field outiside of the\r\ntransform
lookback window\r\n\r\nWe hope that this will one day be merged into the
entity manager\r\nframework so I've tried to abstract this as much as
possible. A field\r\nretention operator specifies how we should choose a
value for a field\r\nwhen looking at the latest value and the enrich
value.\r\n\r\n### Collect values\r\nCollect unique values in an array,
first taking from the latest values\r\nand then filling with enrich
values up to maxLength.\r\n\r\n```\r\n{\r\n operation:
'collect_values',\r\n field: 'host.ip',\r\n maxLength:
10\r\n}\r\n```\r\n\r\n### Prefer newest value\r\nChoose the latest value
if present, otherwise choose the enrich value.\r\n\r\n```\r\n{\r\n
operation: 'prefer_newest_value',\r\n field:
'asset.criticality'\r\n}\r\n```\r\n\r\n### Prefer oldest value\r\nChoose
the enrich value if it is present, otherwise choose
latest.\r\n```\r\n{\r\n operation: 'prefer_oldest_value',\r\n field:
'first_seen_timestamp'\r\n}\r\n```\r\n\r\n## Test instructions\r\n\r\nWe
currently require extra permissions for the kibana system user
for\r\nthis to work, so we must\r\n\r\n### 1. Get Elasticsearch running
from source\r\nThis prototype requires a custom branch of elasticsearch
in order to\r\ngive the kibana system user more privileges.\r\n\r\n####
Step 1 - Clone the prototype branch\r\nThe elasticsearch branch is
at\r\nhttps://github.com/elastic/elasticsearch/tree/entity-store-permissions.\r\n\r\nOr
you can use [github command line](https://cli.github.com/)
to\r\ncheckout my draft PR:\r\n```\r\ngh pr checkout
113942\r\n```\r\n#### Step 2 - Install Java\r\nInstall
[homebrew](https://brew.sh/) if you do not have it.\r\n\r\n```\r\nbrew
install openjdk@21\r\nsudo ln -sfn
/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk
/Library/Java/JavaVirtualMachines/openjdk-21.jdk\r\n```\r\n\r\n#### Step
3 - Run elasticsearch\r\nThis makes sure your data stays between runs of
elasticsearch, and that\r\nyou have platinum license
features\r\n\r\n```\r\n./gradlew run --data-dir /tmp/elasticsearch-repo
--preserve-data -Drun.license_type=trial\r\n```\r\n\r\n### 2. Get Kibana
Running\r\n\r\n#### Step 1 - Connect kibana to elasticsearch\r\n\r\nSet
this in your kibana config:\r\n\r\n```\r\nelasticsearch.username:
elastic-admin\r\nelasticsearch.password: elastic-password\r\n```\r\nNow
start kibana and you should have connected to the elasticsearch
you\r\nmade.\r\n\r\n### 3. Initialise entity engine and send
data!\r\n\r\n- Initialise the host or user engine (or
both)\r\n\r\n```\r\ncurl -H 'Content-Type: application/json' \\\r\n -X
POST \\ \r\n -H 'kbn-xsrf: true' \\\r\n -H 'elastic-api-version:
2023-10-31' \\\r\n -d '{}' \\\r\n
http:///elastic:changeme@localhost:5601/api/entity_store/engines/host/init
\r\n```\r\n\r\n- use your favourite data generation tool to create data,
maybe\r\nhttps://github.com/elastic/security-documents-generator\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"51312159b0436841e0364d7aac0056757962907c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193848","number":193848,"mergeCommit":{"message":"[Entity
Analytics] Add Field Retention Enrich Policy and Ingest Pipeline to
Entity Engine (#193848)\n\n## Summary\r\n\r\nAdd the \"Ouroboros\" part
of the entity engine:\r\n\r\n- an enrich policy is created for each
engine\r\n- the enrich policy is executed every 30s by a kibana task,
this will be\r\n1h once we move to a 24h lookback\r\n- create an ingest
pipeline for the latest which performs the specified\r\nfield retention
operations (for more detail see below)\r\n\r\n<img width=\"2112\"
alt=\"Screenshot 2024-10-02 at 13 42
11\"\r\nsrc=\"https://github.com/user-attachments/assets/f727607f-2e0a-4056-a51e-393fb2a97a95\">\r\n\r\n<details>\r\n<summary>
Expand for example host entity </summary>\r\n```\r\n{\r\n
\"@timestamp\": \"2024-10-01T12:10:46.000Z\",\r\n \"host\": {\r\n
\"name\": \"host9\",\r\n \"hostname\": [\r\n \"host9\"\r\n ],\r\n
\"domain\": [\r\n \"test.com\"\r\n ],\r\n \"ip\": [\r\n \"1.1.1.1\",\r\n
\"1.1.1.2\",\r\n \"1.1.1.3\"\r\n ],\r\n \"risk\": {\r\n
\"calculated_score\": \"70.0\",\r\n \"calculated_score_norm\":
\"27.00200653076172\",\r\n \"calculated_level\": \"Low\"\r\n },\r\n
\"id\": [\r\n \"1234567890abcdef\"\r\n ],\r\n \"type\": [\r\n
\"server\"\r\n ],\r\n \"mac\": [\r\n \"AA:AA:AA:AA:AA:AB\",\r\n
\"aa:aa:aa:aa:aa:aa\",\r\n \"AA:AA:AA:AA:AA:AC\"\r\n ],\r\n
\"architecture\": [\r\n \"x86_64\"\r\n ]\r\n },\r\n \"asset\": {\r\n
\"criticality\": \"low_impact\"\r\n },\r\n \"entity\": {\r\n \"name\":
\"host9\",\r\n \"id\": \"kP/jiFHWSwWlO7W0+fGWrg==\",\r\n \"source\":
[\r\n \"risk-score.risk-score-latest-default\",\r\n
\".asset-criticality.asset-criticality-default\",\r\n
\".ds-logs-testlogs1-default-2024.10.01-000001\",\r\n
\".ds-logs-testlogs2-default-2024.10.01-000001\",\r\n
\".ds-logs-testlogs3-default-2024.10.01-000001\"\r\n ],\r\n \"type\":
\"host\"\r\n }\r\n}\r\n```\r\n</details>\r\n\r\n### Field retention
operators\r\n\r\nFirst some terminology:\r\n\r\n- **latest value** - the
value produced by the transform which\r\nrepresents the latest vioew of
a given field in the transform lookback\r\nperiod\r\n- **enrich value**
- the value added to the document by the enrich\r\npolicy, this
represents the last value of a field outiside of the\r\ntransform
lookback window\r\n\r\nWe hope that this will one day be merged into the
entity manager\r\nframework so I've tried to abstract this as much as
possible. A field\r\nretention operator specifies how we should choose a
value for a field\r\nwhen looking at the latest value and the enrich
value.\r\n\r\n### Collect values\r\nCollect unique values in an array,
first taking from the latest values\r\nand then filling with enrich
values up to maxLength.\r\n\r\n```\r\n{\r\n operation:
'collect_values',\r\n field: 'host.ip',\r\n maxLength:
10\r\n}\r\n```\r\n\r\n### Prefer newest value\r\nChoose the latest value
if present, otherwise choose the enrich value.\r\n\r\n```\r\n{\r\n
operation: 'prefer_newest_value',\r\n field:
'asset.criticality'\r\n}\r\n```\r\n\r\n### Prefer oldest value\r\nChoose
the enrich value if it is present, otherwise choose
latest.\r\n```\r\n{\r\n operation: 'prefer_oldest_value',\r\n field:
'first_seen_timestamp'\r\n}\r\n```\r\n\r\n## Test instructions\r\n\r\nWe
currently require extra permissions for the kibana system user
for\r\nthis to work, so we must\r\n\r\n### 1. Get Elasticsearch running
from source\r\nThis prototype requires a custom branch of elasticsearch
in order to\r\ngive the kibana system user more privileges.\r\n\r\n####
Step 1 - Clone the prototype branch\r\nThe elasticsearch branch is
at\r\nhttps://github.com/elastic/elasticsearch/tree/entity-store-permissions.\r\n\r\nOr
you can use [github command line](https://cli.github.com/)
to\r\ncheckout my draft PR:\r\n```\r\ngh pr checkout
113942\r\n```\r\n#### Step 2 - Install Java\r\nInstall
[homebrew](https://brew.sh/) if you do not have it.\r\n\r\n```\r\nbrew
install openjdk@21\r\nsudo ln -sfn
/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk
/Library/Java/JavaVirtualMachines/openjdk-21.jdk\r\n```\r\n\r\n#### Step
3 - Run elasticsearch\r\nThis makes sure your data stays between runs of
elasticsearch, and that\r\nyou have platinum license
features\r\n\r\n```\r\n./gradlew run --data-dir /tmp/elasticsearch-repo
--preserve-data -Drun.license_type=trial\r\n```\r\n\r\n### 2. Get Kibana
Running\r\n\r\n#### Step 1 - Connect kibana to elasticsearch\r\n\r\nSet
this in your kibana config:\r\n\r\n```\r\nelasticsearch.username:
elastic-admin\r\nelasticsearch.password: elastic-password\r\n```\r\nNow
start kibana and you should have connected to the elasticsearch
you\r\nmade.\r\n\r\n### 3. Initialise entity engine and send
data!\r\n\r\n- Initialise the host or user engine (or
both)\r\n\r\n```\r\ncurl -H 'Content-Type: application/json' \\\r\n -X
POST \\ \r\n -H 'kbn-xsrf: true' \\\r\n -H 'elastic-api-version:
2023-10-31' \\\r\n -d '{}' \\\r\n
http:///elastic:changeme@localhost:5601/api/entity_store/engines/host/init
\r\n```\r\n\r\n- use your favourite data generation tool to create data,
maybe\r\nhttps://github.com/elastic/security-documents-generator\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>\r\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"51312159b0436841e0364d7aac0056757962907c"}}]}]
BACKPORT-->

Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
This commit is contained in:
Kibana Machine 2024-10-12 02:50:56 +11:00 • committed by GitHub
parent 00988326b5
commit 5229bcacc8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
82 changed files with 3257 additions and 691 deletions
Download patch file Download diff file Expand all files Collapse all files

94
oas_docs/output/kibana.serverless.staging.yaml
View file

@ -8288,6 +8288,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
default: 10
description: The number of historical values to keep for each field.
type: integer
filter:
type: string
indexPattern:
@ -29947,6 +29951,8 @@ components:
Security_Entity_Analytics_API_EngineDescriptor:
type: object
properties:
fieldHistoryLength:
type: integer
filter:
type: string
indexPattern:
@ -29955,6 +29961,11 @@ components:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EngineStatus'
type:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EntityType'
required:
- type
- indexPattern
- status
- fieldHistoryLength
Security_Entity_Analytics_API_EngineStatus:
enum:
- installing
@ -30063,6 +30074,9 @@ components:
Security_Entity_Analytics_API_HostEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -30074,42 +30088,15 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
host:
type: object
properties:
@ -30148,6 +30135,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- host
- entity
Security_Entity_Analytics_API_IdField:
enum:
- host.name
@ -30237,6 +30228,9 @@ components:
Security_Entity_Analytics_API_UserEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -30248,42 +30242,14 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
user:
type: object
@ -30319,6 +30285,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- user
- entity
Security_Exceptions_API_CreateExceptionListItemComment:
type: object
properties:

94
oas_docs/output/kibana.serverless.yaml
View file

@ -8288,6 +8288,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
default: 10
description: The number of historical values to keep for each field.
type: integer
filter:
type: string
indexPattern:
@ -29947,6 +29951,8 @@ components:
Security_Entity_Analytics_API_EngineDescriptor:
type: object
properties:
fieldHistoryLength:
type: integer
filter:
type: string
indexPattern:
@ -29955,6 +29961,11 @@ components:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EngineStatus'
type:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EntityType'
required:
- type
- indexPattern
- status
- fieldHistoryLength
Security_Entity_Analytics_API_EngineStatus:
enum:
- installing
@ -30063,6 +30074,9 @@ components:
Security_Entity_Analytics_API_HostEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -30074,42 +30088,15 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
host:
type: object
properties:
@ -30148,6 +30135,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- host
- entity
Security_Entity_Analytics_API_IdField:
enum:
- host.name
@ -30237,6 +30228,9 @@ components:
Security_Entity_Analytics_API_UserEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -30248,42 +30242,14 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
user:
type: object
@ -30319,6 +30285,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- user
- entity
Security_Exceptions_API_CreateExceptionListItemComment:
type: object
properties:

94
oas_docs/output/kibana.staging.yaml
View file

@ -11716,6 +11716,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
default: 10
description: The number of historical values to keep for each field.
type: integer
filter:
type: string
indexPattern:
@ -37959,6 +37963,8 @@ components:
Security_Entity_Analytics_API_EngineDescriptor:
type: object
properties:
fieldHistoryLength:
type: integer
filter:
type: string
indexPattern:
@ -37967,6 +37973,11 @@ components:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EngineStatus'
type:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EntityType'
required:
- type
- indexPattern
- status
- fieldHistoryLength
Security_Entity_Analytics_API_EngineStatus:
enum:
- installing
@ -38075,6 +38086,9 @@ components:
Security_Entity_Analytics_API_HostEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -38086,42 +38100,15 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
host:
type: object
properties:
@ -38160,6 +38147,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- host
- entity
Security_Entity_Analytics_API_IdField:
enum:
- host.name
@ -38249,6 +38240,9 @@ components:
Security_Entity_Analytics_API_UserEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -38260,42 +38254,14 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
user:
type: object
@ -38331,6 +38297,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- user
- entity
Security_Exceptions_API_CreateExceptionListItemComment:
type: object
properties:

94
oas_docs/output/kibana.yaml
View file

@ -11716,6 +11716,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
default: 10
description: The number of historical values to keep for each field.
type: integer
filter:
type: string
indexPattern:
@ -37959,6 +37963,8 @@ components:
Security_Entity_Analytics_API_EngineDescriptor:
type: object
properties:
fieldHistoryLength:
type: integer
filter:
type: string
indexPattern:
@ -37967,6 +37973,11 @@ components:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EngineStatus'
type:
$ref: '#/components/schemas/Security_Entity_Analytics_API_EntityType'
required:
- type
- indexPattern
- status
- fieldHistoryLength
Security_Entity_Analytics_API_EngineStatus:
enum:
- installing
@ -38075,6 +38086,9 @@ components:
Security_Entity_Analytics_API_HostEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -38086,42 +38100,15 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
host:
type: object
properties:
@ -38160,6 +38147,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- host
- entity
Security_Entity_Analytics_API_IdField:
enum:
- host.name
@ -38249,6 +38240,9 @@ components:
Security_Entity_Analytics_API_UserEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -38260,42 +38254,14 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
user:
type: object
@ -38331,6 +38297,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- user
- entity
Security_Exceptions_API_CreateExceptionListItemComment:
type: object
properties:

1
packages/kbn-check-mappings-update-cli/current_fields.json
View file

@ -313,6 +313,7 @@
"apiKey"
],
"entity-engine-status": [
"fieldHistoryLength",
"filter",
"indexPattern",
"status",

4
packages/kbn-check-mappings-update-cli/current_mappings.json
View file

@ -1060,6 +1060,10 @@
"entity-engine-status": {
"dynamic": false,
"properties": {
"fieldHistoryLength": {
"index": false,
"type": "integer"
},
"filter": {
"type": "keyword"
},

2
src/core/server/integration_tests/ci_checks/saved_objects/check_registered_types.test.ts
View file

@ -93,7 +93,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
"enterprise_search_telemetry": "9ac912e1417fc8681e0cd383775382117c9e3d3d",
"entity-definition": "e3811fd5fbb878d170067c0d6897a2e63010af36",
"entity-discovery-api-key": "c267a65c69171d1804362155c1378365f5acef88",
"entity-engine-status": "0738aa1a06d3361911740f8f166071ea43a00927",
"entity-engine-status": "8cb7dcb13f5e2ea8f2e08dd4af72c110e2051120",
"epm-packages": "8042d4a1522f6c4e6f5486e791b3ffe3a22f88fd",
"epm-packages-assets": "7a3e58efd9a14191d0d1a00b8aaed30a145fd0b1",
"event-annotation-group": "715ba867d8c68f3c9438052210ea1c30a9362582",

7
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/common.gen.ts
View file

@ -31,10 +31,11 @@ export const EngineStatusEnum = EngineStatus.enum;
export type EngineDescriptor = z.infer<typeof EngineDescriptor>;
export const EngineDescriptor = z.object({
type: EntityType.optional(),
indexPattern: IndexPattern.optional(),
status: EngineStatus.optional(),
type: EntityType,
indexPattern: IndexPattern,
status: EngineStatus,
filter: z.string().optional(),
fieldHistoryLength: z.number().int(),
});
export type InspectQuery = z.infer<typeof InspectQuery>;

7
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/common.schema.yaml
View file

@ -14,6 +14,11 @@ components:
EngineDescriptor:
type: object
required:
- type
- indexPattern
- status
- fieldHistoryLength
properties:
type:
$ref: '#/components/schemas/EntityType'
@ -23,6 +28,8 @@ components:
$ref: '#/components/schemas/EngineStatus'
filter:
type: string
fieldHistoryLength:
type: integer
EngineStatus:
type: string

14
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/engine/index.ts Normal file
View file

@ -0,0 +1,14 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './delete.gen';
export * from './get.gen';
export * from './init.gen';
export * from './list.gen';
export * from './start.gen';
export * from './stats.gen';
export * from './stop.gen';

4
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/engine/init.gen.ts
View file

@ -29,6 +29,10 @@ export type InitEntityEngineRequestParamsInput = z.input<typeof InitEntityEngine
export type InitEntityEngineRequestBody = z.infer<typeof InitEntityEngineRequestBody>;
export const InitEntityEngineRequestBody = z.object({
/**
* The number of historical values to keep for each field.
*/
fieldHistoryLength: z.number().int().optional().default(10),
indexPattern: IndexPattern.optional(),
filter: z.string().optional(),
});

4
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/engine/init.schema.yaml
View file

@ -25,6 +25,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
type: integer
description: The number of historical values to keep for each field.
default: 10
indexPattern:
$ref: '../common.schema.yaml#/components/schemas/IndexPattern'
filter:

84
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/common.gen.ts
View file

@ -21,32 +21,21 @@ import { AssetCriticalityLevel } from '../../asset_criticality/common.gen';
export type UserEntity = z.infer<typeof UserEntity>;
export const UserEntity = z.object({
user: z
.object({
full_name: z.array(z.string()).optional(),
domain: z.array(z.string()).optional(),
roles: z.array(z.string()).optional(),
name: z.string(),
id: z.array(z.string()).optional(),
email: z.array(z.string()).optional(),
hash: z.array(z.string()).optional(),
risk: EntityRiskScoreRecord.optional(),
})
.optional(),
entity: z
.object({
lastSeenTimestamp: z.string().datetime(),
schemaVersion: z.string(),
definitionVersion: z.string(),
displayName: z.string(),
identityFields: z.array(z.string()),
id: z.string(),
type: z.literal('node'),
firstSeenTimestamp: z.string().datetime(),
definitionId: z.string(),
source: z.string(),
})
.optional(),
'@timestamp': z.string().datetime(),
entity: z.object({
name: z.string(),
source: z.array(z.string()),
}),
user: z.object({
full_name: z.array(z.string()).optional(),
domain: z.array(z.string()).optional(),
roles: z.array(z.string()).optional(),
name: z.string(),
id: z.array(z.string()).optional(),
email: z.array(z.string()).optional(),
hash: z.array(z.string()).optional(),
risk: EntityRiskScoreRecord.optional(),
}),
asset: z
.object({
criticality: AssetCriticalityLevel,
@ -56,33 +45,22 @@ export const UserEntity = z.object({
export type HostEntity = z.infer<typeof HostEntity>;
export const HostEntity = z.object({
host: z
.object({
hostname: z.array(z.string()).optional(),
domain: z.array(z.string()).optional(),
ip: z.array(z.string()).optional(),
name: z.string(),
id: z.array(z.string()).optional(),
type: z.array(z.string()).optional(),
mac: z.array(z.string()).optional(),
architecture: z.array(z.string()).optional(),
risk: EntityRiskScoreRecord.optional(),
})
.optional(),
entity: z
.object({
lastSeenTimestamp: z.string().datetime(),
schemaVersion: z.string(),
definitionVersion: z.string(),
displayName: z.string(),
identityFields: z.array(z.string()),
id: z.string(),
type: z.literal('node'),
firstSeenTimestamp: z.string().datetime(),
definitionId: z.string(),
source: z.string().optional(),
})
.optional(),
'@timestamp': z.string().datetime(),
entity: z.object({
name: z.string(),
source: z.array(z.string()),
}),
host: z.object({
hostname: z.array(z.string()).optional(),
domain: z.array(z.string()).optional(),
ip: z.array(z.string()).optional(),
name: z.string(),
id: z.array(z.string()).optional(),
type: z.array(z.string()).optional(),
mac: z.array(z.string()).optional(),
architecture: z.array(z.string()).optional(),
risk: EntityRiskScoreRecord.optional(),
}),
asset: z
.object({
criticality: AssetCriticalityLevel,

118
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/common.schema.yaml
View file

@ -8,7 +8,26 @@ components:
schemas:
UserEntity:
type: object
required:
- "@timestamp"
- user
- entity
properties:
"@timestamp":
type: string
format: date-time
entity:
type: object
required:
- name
- source
properties:
name:
type: string
source:
type: array
items:
type: string
user:
type: object
properties:
@ -42,46 +61,6 @@ components:
$ref: '../../common/common.schema.yaml#/components/schemas/EntityRiskScoreRecord'
required:
- name
entity:
type: object
properties:
lastSeenTimestamp:
type: string
format: date-time
schemaVersion:
type: string
definitionVersion:
type: string
displayName:
type: string
identityFields:
type: array
items:
type: string
id:
type: string
type:
type: string
enum:
- node
firstSeenTimestamp:
type: string
format: date-time
definitionId:
type: string
source:
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- source
asset:
type: object
properties:
@ -91,7 +70,26 @@ components:
- criticality
HostEntity:
type: object
required:
- "@timestamp"
- host
- entity
properties:
"@timestamp":
type: string
format: date-time
entity:
type: object
required:
- name
- source
properties:
name:
type: string
source:
type: array
items:
type: string
host:
type: object
properties:
@ -129,46 +127,6 @@ components:
$ref: '../../common/common.schema.yaml#/components/schemas/EntityRiskScoreRecord'
required:
- name
entity:
type: object
properties:
lastSeenTimestamp:
type: string
format: date-time
schemaVersion:
type: string
definitionVersion:
type: string
displayName:
type: string
identityFields:
type: array
items:
type: string
id:
type: string
type:
type: string
enum:
- node
firstSeenTimestamp:
type: string
format: date-time
definitionId:
type: string
source:
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
asset:
type: object
properties:

9
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/entities/index.ts Normal file
View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './common.gen';
export * from './list_entities.gen';

10
x-pack/plugins/security_solution/common/api/entity_analytics/entity_store/index.ts Normal file
View file

@ -0,0 +1,10 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './common.gen';
export * from './engine';
export * from './entities';

1
x-pack/plugins/security_solution/common/api/entity_analytics/index.ts
View file

@ -8,4 +8,5 @@
export * from './asset_criticality';
export * from './risk_engine';
export * from './risk_score';
export * from './entity_store';
export { EntityAnalyticsPrivileges } from './common';

94
x-pack/plugins/security_solution/docs/openapi/ess/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml
View file

@ -340,6 +340,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
default: 10
description: The number of historical values to keep for each field.
type: integer
filter:
type: string
indexPattern:
@ -704,6 +708,8 @@ components:
EngineDescriptor:
type: object
properties:
fieldHistoryLength:
type: integer
filter:
type: string
indexPattern:
@ -712,6 +718,11 @@ components:
$ref: '#/components/schemas/EngineStatus'
type:
$ref: '#/components/schemas/EntityType'
required:
- type
- indexPattern
- status
- fieldHistoryLength
EngineStatus:
enum:
- installing
@ -819,6 +830,9 @@ components:
HostEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -829,42 +843,15 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
host:
type: object
properties:
@ -902,6 +889,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- host
- entity
IdField:
enum:
- host.name
@ -991,6 +982,9 @@ components:
UserEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -1001,42 +995,14 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
user:
type: object
@ -1071,6 +1037,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- user
- entity
securitySchemes:
BasicAuth:
scheme: basic

94
x-pack/plugins/security_solution/docs/openapi/serverless/security_solution_entity_analytics_api_2023_10_31.bundled.schema.yaml
View file

@ -340,6 +340,10 @@ paths:
schema:
type: object
properties:
fieldHistoryLength:
default: 10
description: The number of historical values to keep for each field.
type: integer
filter:
type: string
indexPattern:
@ -704,6 +708,8 @@ components:
EngineDescriptor:
type: object
properties:
fieldHistoryLength:
type: integer
filter:
type: string
indexPattern:
@ -712,6 +718,11 @@ components:
$ref: '#/components/schemas/EngineStatus'
type:
$ref: '#/components/schemas/EntityType'
required:
- type
- indexPattern
- status
- fieldHistoryLength
EngineStatus:
enum:
- installing
@ -819,6 +830,9 @@ components:
HostEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -829,42 +843,15 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
host:
type: object
properties:
@ -902,6 +889,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- host
- entity
IdField:
enum:
- host.name
@ -991,6 +982,9 @@ components:
UserEntity:
type: object
properties:
'@timestamp':
format: date-time
type: string
asset:
type: object
properties:
@ -1001,42 +995,14 @@ components:
entity:
type: object
properties:
definitionId:
name:
type: string
definitionVersion:
type: string
displayName:
type: string
firstSeenTimestamp:
format: date-time
type: string
id:
type: string
identityFields:
source:
items:
type: string
type: array
lastSeenTimestamp:
format: date-time
type: string
schemaVersion:
type: string
source:
type: string
type:
enum:
- node
type: string
required:
- lastSeenTimestamp
- schemaVersion
- definitionVersion
- displayName
- identityFields
- id
- type
- firstSeenTimestamp
- definitionId
- name
- source
user:
type: object
@ -1071,6 +1037,10 @@ components:
type: array
required:
- name
required:
- '@timestamp'
- user
- entity
securitySchemes:
BasicAuth:
scheme: basic

13
x-pack/plugins/security_solution/public/entity_analytics/components/entity_store/entities_list.test.tsx
View file

@ -29,18 +29,11 @@ const responseData: ListEntitiesResponse = {
total: 1,
records: [
{
'@timestamp': '2021-08-02T14:00:00.000Z',
user: { name: entityName },
entity: {
type: 'node',
id: 'entity-id',
lastSeenTimestamp: '2023-01-01T00:00:00Z',
schemaVersion: '1.0',
definitionVersion: '1.0',
displayName: entityName,
identityFields: ['field1', 'field2'],
firstSeenTimestamp: '2023-01-01T00:00:00Z',
definitionId: 'definition-id',
source: 'source',
name: entityName,
source: ['source'],
},
},
],

10
x-pack/plugins/security_solution/public/entity_analytics/components/entity_store/helpers.test.ts
View file

@ -14,9 +14,14 @@ import type {
describe('isUserEntity', () => {
it('should return true if the record is a UserEntity', () => {
const userEntity: UserEntity = {
'@timestamp': '2021-08-02T14:00:00.000Z',
user: {
name: 'test_user',
},
entity: {
name: 'test_user',
source: ['logs-test'],
},
};
expect(isUserEntity(userEntity)).toBe(true);
@ -24,9 +29,14 @@ describe('isUserEntity', () => {
it('should return false if the record is not a UserEntity', () => {
const nonUserEntity: Entity = {
'@timestamp': '2021-08-02T14:00:00.000Z',
host: {
name: 'test_host',
},
entity: {
name: 'test_host',
source: ['logs-test'],
},
};
expect(isUserEntity(nonUserEntity)).toBe(false);

7
x-pack/plugins/security_solution/public/entity_analytics/components/entity_store/hooks/use_entities_list_columns.tsx
View file

@ -45,8 +45,7 @@ export const useEntitiesListColumns = (): EntitiesListColumns => {
),
render: (record: Entity) => {
const field = record.entity?.identityFields[0];
const value = record.entity?.displayName;
const value = record.entity.name;
const onClick = () => {
const id = isUserEntity(record) ? UserPanelKey : HostPanelKey;
const params = {
@ -58,7 +57,7 @@ export const useEntitiesListColumns = (): EntitiesListColumns => {
openRightPanel({ id, params });
};
if (!field || !value) {
if (!value) {
return null;
}
@ -91,7 +90,7 @@ export const useEntitiesListColumns = (): EntitiesListColumns => {
return (
<span>
{isUserEntity(record) ? <EuiIcon type="user" /> : <EuiIcon type="storage" />}
<span css={{ paddingLeft: euiTheme.size.s }}>{record.entity?.displayName}</span>
<span css={{ paddingLeft: euiTheme.size.s }}>{record.entity.name}</span>
</span>
);
},

5
x-pack/plugins/security_solution/server/config.mock.ts
View file

@ -44,6 +44,11 @@ export const createMockConfig = (): ConfigType => {
maxBulkRequestBodySizeBytes: 10_485_760,
},
},
entityStore: {
developer: {
pipelineDebugMode: false,
},
},
},
};
};

5
x-pack/plugins/security_solution/server/config.ts
View file

@ -175,6 +175,11 @@ export const configSchema = schema.object({
maxBulkRequestBodySizeBytes: schema.number({ defaultValue: 100_000 }), // 100KB
}),
}),
entityStore: schema.object({
developer: schema.object({
pipelineDebugMode: schema.boolean({ defaultValue: false }),
}),
}),
}),
});

2
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/__snapshots__/entity_store_data_client.test.ts.snap generated
View file

@ -5,7 +5,7 @@ Object {
"dsl": Array [
"{
\\"index\\": [
\\".entities.v1.latest.ea_default_host_entity_store\\"
\\".entities.v1.latest.security_host_default\\"
]
}",
],

4
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/constants.ts
View file

@ -14,6 +14,10 @@ import { DEFAULT_INDEX_PATTERN } from '../../../../common/constants';
*/
export const ENTITY_STORE_DEFAULT_SOURCE_INDICES = DEFAULT_INDEX_PATTERN;
export const DEFAULT_LOOKBACK_PERIOD = '24h';
export const DEFAULT_INTERVAL = '15s';
export const ENGINE_STATUS: Record<Uppercase<EngineStatus>, EngineStatus> = {
INSTALLING: 'installing',
STARTED: 'started',

51
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/definition.ts
View file

@ -1,51 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { entityDefinitionSchema, type EntityDefinition } from '@kbn/entities-schema';
import { ENTITY_STORE_DEFAULT_SOURCE_INDICES } from './constants';
import { buildEntityDefinitionId } from './utils/utils';
export const buildHostEntityDefinition = (space: string): EntityDefinition =>
entityDefinitionSchema.parse({
id: buildEntityDefinitionId('host', space),
name: 'EA Host Store',
type: 'host',
indexPatterns: ENTITY_STORE_DEFAULT_SOURCE_INDICES,
identityFields: ['host.name'],
displayNameTemplate: '{{host.name}}',
metadata: [
'host.domain',
'host.hostname',
'host.id',
'host.ip',
'host.mac',
'host.name',
'host.type',
'host.architecture',
],
latest: {
timestampField: '@timestamp',
},
version: '1.0.0',
managed: true,
});
export const buildUserEntityDefinition = (space: string): EntityDefinition =>
entityDefinitionSchema.parse({
id: buildEntityDefinitionId('user', space),
name: 'EA User Store',
type: 'user',
indexPatterns: ENTITY_STORE_DEFAULT_SOURCE_INDICES,
identityFields: ['user.name'],
displayNameTemplate: '{{user.name}}',
metadata: ['user.email', 'user.full_name', 'user.hash', 'user.id', 'user.name', 'user.roles'],
latest: {
timestampField: '@timestamp',
},
version: '1.0.0',
managed: true,
});

43
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/component_template.ts Normal file
View file

@ -0,0 +1,43 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { ElasticsearchClient } from '@kbn/core/server';
import type { UnitedEntityDefinition } from '../united_entity_definitions';
const getComponentTemplateName = (definitionId: string) => `${definitionId}-latest@platform`;
interface Options {
unitedDefinition: UnitedEntityDefinition;
esClient: ElasticsearchClient;
}
export const createEntityIndexComponentTemplate = ({ unitedDefinition, esClient }: Options) => {
const { entityManagerDefinition, indexMappings } = unitedDefinition;
const name = getComponentTemplateName(entityManagerDefinition.id);
return esClient.cluster.putComponentTemplate({
name,
body: {
template: {
settings: {
hidden: true,
},
mappings: indexMappings,
},
},
});
};
export const deleteEntityIndexComponentTemplate = ({ unitedDefinition, esClient }: Options) => {
const { entityManagerDefinition } = unitedDefinition;
const name = getComponentTemplateName(entityManagerDefinition.id);
return esClient.cluster.deleteComponentTemplate(
{ name },
{
ignore: [404],
}
);
};

79
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/enrich_policy.ts Normal file
View file

@ -0,0 +1,79 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { ElasticsearchClient, Logger } from '@kbn/core/server';
import type { EnrichPutPolicyRequest } from '@elastic/elasticsearch/lib/api/types';
import { getEntitiesIndexName } from '../utils';
import type { UnitedEntityDefinition } from '../united_entity_definitions';
export const getFieldRetentionEnrichPolicyName = ({
namespace,
entityType,
version,
}: Pick<UnitedEntityDefinition, 'namespace' | 'entityType' | 'version'>): string => {
return `entity_store_field_retention_${entityType}_${namespace}_v${version}`;
};
const getFieldRetentionEnrichPolicy = (
unitedDefinition: UnitedEntityDefinition
): EnrichPutPolicyRequest => {
const { namespace, entityType, fieldRetentionDefinition } = unitedDefinition;
return {
name: getFieldRetentionEnrichPolicyName(unitedDefinition),
match: {
indices: getEntitiesIndexName(entityType, namespace),
match_field: fieldRetentionDefinition.matchField,
enrich_fields: fieldRetentionDefinition.fields.map(({ field }) => field),
},
};
};
export const createFieldRetentionEnrichPolicy = async ({
esClient,
unitedDefinition,
}: {
esClient: ElasticsearchClient;
unitedDefinition: UnitedEntityDefinition;
}) => {
const policy = getFieldRetentionEnrichPolicy(unitedDefinition);
return esClient.enrich.putPolicy(policy);
};
export const executeFieldRetentionEnrichPolicy = async ({
esClient,
unitedDefinition,
logger,
}: {
unitedDefinition: UnitedEntityDefinition;
esClient: ElasticsearchClient;
logger: Logger;
}): Promise<{ executed: boolean }> => {
const name = getFieldRetentionEnrichPolicyName(unitedDefinition);
try {
await esClient.enrich.executePolicy({ name });
return { executed: true };
} catch (e) {
if (e.statusCode === 404) {
return { executed: false };
}
logger.error(
`Error executing field retention enrich policy for ${unitedDefinition.entityType}: ${e.message}`
);
throw e;
}
};
export const deleteFieldRetentionEnrichPolicy = async ({
unitedDefinition,
esClient,
}: {
esClient: ElasticsearchClient;
unitedDefinition: UnitedEntityDefinition;
}) => {
const name = getFieldRetentionEnrichPolicyName(unitedDefinition);
return esClient.enrich.deletePolicy({ name }, { ignore: [404] });
};

42
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/entity_index.ts Normal file
View file

@ -0,0 +1,42 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { ElasticsearchClient, Logger } from '@kbn/core/server';
import type { EntityType } from '../../../../../common/api/entity_analytics';
import { getEntitiesIndexName } from '../utils';
interface Options {
entityType: EntityType;
esClient: ElasticsearchClient;
namespace: string;
logger: Logger;
}
export const createEntityIndex = async ({ entityType, esClient, namespace, logger }: Options) => {
try {
await esClient.indices.create({
index: getEntitiesIndexName(entityType, namespace),
body: {},
});
} catch (e) {
if (e.meta.body.error.type === 'resource_already_exists_exception') {
logger.debug(`Index for ${entityType} already exists, skipping creation.`);
} else {
throw e;
}
}
};
export const deleteEntityIndex = ({ entityType, esClient, namespace }: Options) =>
esClient.indices.delete(
{
index: getEntitiesIndexName(entityType, namespace),
},
{
ignore: [404],
}
);

11
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/index.ts Normal file
View file

@ -0,0 +1,11 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './entity_index';
export * from './ingest_pipeline';
export * from './component_template';
export * from './enrich_policy';

163
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_pipeline.ts Normal file
View file

@ -0,0 +1,163 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { ElasticsearchClient, Logger } from '@kbn/core/server';
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
import type { EntityDefinition } from '@kbn/entities-schema';
import { type FieldRetentionDefinition } from '../field_retention_definition';
import {
debugDeepCopyContextStep,
getDotExpanderSteps,
getRemoveEmptyFieldSteps,
removeEntityDefinitionFieldsStep,
retentionDefinitionToIngestProcessorSteps,
} from './ingest_processor_steps';
import { getIdentityFieldForEntityType } from '../utils';
import { getFieldRetentionEnrichPolicyName } from './enrich_policy';
import type { UnitedEntityDefinition } from '../united_entity_definitions';
const getPlatformPipelineId = (definition: EntityDefinition) => {
return `${definition.id}-latest@platform`;
};
// the field that the enrich processor writes to
export const ENRICH_FIELD = 'historical';
/**
* Builds the ingest pipeline for the field retention policy.
* Broadly the pipeline enriches the entity with the field retention enrich policy,
* then applies the field retention policy to the entity fields, and finally removes
* the enrich field and any empty fields.
*
* While developing, be sure to set debugMode to true this will keep the enrich field
* and the context field in the document to help with debugging.
*/
const buildIngestPipeline = ({
version,
fieldRetentionDefinition,
allEntityFields,
debugMode,
namespace,
}: {
fieldRetentionDefinition: FieldRetentionDefinition;
allEntityFields: string[];
debugMode?: boolean;
namespace: string;
version: string;
}): IngestProcessorContainer[] => {
const { entityType, matchField } = fieldRetentionDefinition;
const enrichPolicyName = getFieldRetentionEnrichPolicyName({
namespace,
entityType,
version,
});
return [
...(debugMode ? [debugDeepCopyContextStep()] : []),
{
enrich: {
policy_name: enrichPolicyName,
field: matchField,
target_field: ENRICH_FIELD,
},
},
{
set: {
field: '@timestamp',
value: '{{entity.lastSeenTimestamp}}',
},
},
{
set: {
field: 'entity.name',
value: `{{${getIdentityFieldForEntityType(entityType)}}}`,
},
},
...getDotExpanderSteps(allEntityFields),
...retentionDefinitionToIngestProcessorSteps(fieldRetentionDefinition, {
enrichField: ENRICH_FIELD,
}),
...getRemoveEmptyFieldSteps([...allEntityFields, 'asset', `${entityType}.risk`]),
removeEntityDefinitionFieldsStep(),
...(!debugMode
? [
{
remove: {
ignore_failure: true,
field: ENRICH_FIELD,
},
},
]
: []),
];
};
// developing the pipeline is a bit tricky, so we have a debug mode
// set xpack.securitySolution.entityAnalytics.entityStore.developer.pipelineDebugMode
// to true to keep the enrich field and the context field in the document to help with debugging.
export const createPlatformPipeline = async ({
unitedDefinition,
logger,
esClient,
debugMode,
}: {
unitedDefinition: UnitedEntityDefinition;
logger: Logger;
esClient: ElasticsearchClient;
debugMode?: boolean;
}) => {
const { fieldRetentionDefinition, entityManagerDefinition } = unitedDefinition;
const allEntityFields: string[] = (entityManagerDefinition?.metadata || []).map((m) => {
if (typeof m === 'string') {
return m;
}
return m.destination;
});
const pipeline = {
id: getPlatformPipelineId(entityManagerDefinition),
body: {
_meta: {
managed_by: 'entity_store',
managed: true,
},
description: `Ingest pipeline for entity defiinition ${entityManagerDefinition.id}`,
processors: buildIngestPipeline({
namespace: unitedDefinition.namespace,
version: unitedDefinition.version,
fieldRetentionDefinition,
allEntityFields,
debugMode,
}),
},
};
logger.debug(`Attempting to create pipeline: ${JSON.stringify(pipeline)}`);
await esClient.ingest.putPipeline(pipeline);
};
export const deletePlatformPipeline = ({
unitedDefinition,
logger,
esClient,
}: {
unitedDefinition: UnitedEntityDefinition;
logger: Logger;
esClient: ElasticsearchClient;
}) => {
const pipelineId = getPlatformPipelineId(unitedDefinition.entityManagerDefinition);
logger.debug(`Attempting to delete pipeline: ${pipelineId}`);
return esClient.ingest.deletePipeline(
{
id: pipelineId,
},
{
ignore: [404],
}
);
};

46
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_processor_steps/debug_deep_copy_context_step.ts Normal file
View file

@ -0,0 +1,46 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
/**
* Deeply copies ctx object to the debug_ctx field for debugging purposes
* Deep copy is necessary because the context is a mutable object and painless copies by ref
*/
export const debugDeepCopyContextStep = (): IngestProcessorContainer => ({
script: {
lang: 'painless',
source: `
Map deepCopy(Map original) {
Map copy = new HashMap();
for (entry in original.entrySet()) {
if (entry.getValue() instanceof Map) {
// Recursively deep copy nested maps
copy.put(entry.getKey(), deepCopy((Map)entry.getValue()));
} else if (entry.getValue() instanceof List) {
// Deep copy lists
List newList = new ArrayList();
for (item in (List)entry.getValue()) {
if (item instanceof Map) {
newList.add(deepCopy((Map)item));
} else {
newList.add(item);
}
}
copy.put(entry.getKey(), newList);
} else {
// Copy by value for other types
copy.put(entry.getKey(), entry.getValue());
}
}
return copy;
}
ctx.debug_ctx = deepCopy(ctx);
`,
},
});

22
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_processor_steps/get_dot_expander_steps.ts Normal file
View file

@ -0,0 +1,22 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
/**
* Returns the ingest processor steps for the dot expander processor for the given fields.
* We need to expand the dot notation fields to be able to use them in the field retention processors.
* Painless treats { "a.b" : "c" } and { "a" : { "b" : "c" } } as different fields.
*/
export const getDotExpanderSteps = (fields: string[]): IngestProcessorContainer[] =>
fields.map((field) => {
return {
dot_expander: {
field,
},
};
});

23
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_processor_steps/get_remove_empty_field_steps.ts Normal file
View file

@ -0,0 +1,23 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
import { isFieldMissingOrEmpty } from '../../painless';
/**
* This function creates an ingest processor step that removes a field if it is missing or empty.
*/
export const getRemoveEmptyFieldSteps = (fields: string[]): IngestProcessorContainer[] =>
fields.map((field) => {
return {
remove: {
if: isFieldMissingOrEmpty(`ctx.${field}`),
field,
ignore_missing: true,
},
};
});

12
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_processor_steps/index.ts Normal file
View file

@ -0,0 +1,12 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { debugDeepCopyContextStep } from './debug_deep_copy_context_step';
export { getDotExpanderSteps } from './get_dot_expander_steps';
export { getRemoveEmptyFieldSteps } from './get_remove_empty_field_steps';
export { removeEntityDefinitionFieldsStep } from './remove_entity_definition_fields_step';
export { retentionDefinitionToIngestProcessorSteps } from './retention_definition_to_ingest_processor_steps';

27
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_processor_steps/remove_entity_definition_fields_step.ts Normal file
View file

@ -0,0 +1,27 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
/**
* This function creates an ingest processor step that removes entity definition fields which
* are not ECS.
*/
export const removeEntityDefinitionFieldsStep = (): IngestProcessorContainer => ({
remove: {
ignore_failure: true,
field: [
'entity.lastSeenTimestamp',
'entity.schemaVersion',
'entity.definitionVersion',
'entity.identityFields',
'entity.definitionId',
'entity.displayName',
'entity.firstSeenTimestamp',
],
},
});

26
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/elasticsearch_assets/ingest_processor_steps/retention_definition_to_ingest_processor_steps.ts Normal file
View file

@ -0,0 +1,26 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
import type {
FieldRetentionDefinition,
FieldRetentionOperatorBuilderOptions,
} from '../../field_retention_definition';
import { fieldOperatorToIngestProcessor } from '../../field_retention_definition';
/**
* Converts a field retention definition to the ingest processor steps
* required to apply the field retention policy.
*/
export const retentionDefinitionToIngestProcessorSteps = (
fieldRetentionDefinition: FieldRetentionDefinition,
options: FieldRetentionOperatorBuilderOptions
): IngestProcessorContainer[] => {
return fieldRetentionDefinition.fields.map((field) =>
fieldOperatorToIngestProcessor(field, options)
);
};

18
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.test.ts
View file

@ -11,13 +11,10 @@ import {
savedObjectsClientMock,
} from '@kbn/core/server/mocks';
import { EntityStoreDataClient } from './entity_store_data_client';
import { EntityClient } from '@kbn/entityManager-plugin/server/lib/entity_client';
import type { SortOrder } from '@elastic/elasticsearch/lib/api/types';
import type { EntityType } from '../../../../common/api/entity_analytics/entity_store/common.gen';
import { AssetCriticalityEcsMigrationClient } from '../asset_criticality/asset_criticality_migration_client';
describe('EntityStoreDataClient', () => {
const logger = loggingSystemMock.createLogger();
const mockSavedObjectClient = savedObjectsClientMock.create();
const esClientMock = elasticsearchServiceMock.createScopedClusterClient().asInternalUser;
const loggerMock = loggingSystemMock.createLogger();
@ -26,16 +23,7 @@ describe('EntityStoreDataClient', () => {
logger: loggerMock,
namespace: 'default',
soClient: mockSavedObjectClient,
entityClient: new EntityClient({
esClient: esClientMock,
soClient: mockSavedObjectClient,
logger,
}),
assetCriticalityMigrationClient: new AssetCriticalityEcsMigrationClient({
esClient: esClientMock,
logger: loggerMock,
auditLogger: undefined,
}),
kibanaVersion: '9.0.0',
});
const defaultSearchParams = {
@ -74,8 +62,8 @@ describe('EntityStoreDataClient', () => {
expect(esClientMock.search).toHaveBeenCalledWith(
expect.objectContaining({
index: [
'.entities.v1.latest.ea_default_host_entity_store',
'.entities.v1.latest.ea_default_user_entity_store',
'.entities.v1.latest.security_host_default',
'.entities.v1.latest.security_user_default',
],
})
);

269
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_store_data_client.ts
View file

@ -5,33 +5,54 @@
* 2.0.
*/
import type { Logger, ElasticsearchClient, SavedObjectsClientContract } from '@kbn/core/server';
import type { EntityClient } from '@kbn/entityManager-plugin/server/lib/entity_client';
import type {
Logger,
ElasticsearchClient,
SavedObjectsClientContract,
AuditLogger,
} from '@kbn/core/server';
import { EntityClient } from '@kbn/entityManager-plugin/server/lib/entity_client';
import type { SortOrder } from '@elastic/elasticsearch/lib/api/types';
import type { TaskManagerStartContract } from '@kbn/task-manager-plugin/server';
import type { Entity } from '../../../../common/api/entity_analytics/entity_store/entities/common.gen';
import type {
InitEntityEngineRequestBody,
InitEntityEngineResponse,
} from '../../../../common/api/entity_analytics/entity_store/engine/init.gen';
import type {
EntityType,
InspectQuery,
} from '../../../../common/api/entity_analytics/entity_store/common.gen';
import { EngineDescriptorClient } from './saved_object/engine_descriptor';
import { getEntitiesIndexName, getEntityDefinition } from './utils/utils';
import { buildEntityDefinitionId, getEntitiesIndexName } from './utils';
import { ENGINE_STATUS, MAX_SEARCH_RESPONSE_SIZE } from './constants';
import type { AssetCriticalityEcsMigrationClient } from '../asset_criticality/asset_criticality_migration_client';
import { AssetCriticalityEcsMigrationClient } from '../asset_criticality/asset_criticality_migration_client';
import { getUnitedEntityDefinition } from './united_entity_definitions';
import {
startEntityStoreFieldRetentionEnrichTask,
removeEntityStoreFieldRetentionEnrichTask,
} from './task';
import {
createEntityIndex,
deleteEntityIndex,
createPlatformPipeline,
deletePlatformPipeline,
createEntityIndexComponentTemplate,
deleteEntityIndexComponentTemplate,
createFieldRetentionEnrichPolicy,
executeFieldRetentionEnrichPolicy,
deleteFieldRetentionEnrichPolicy,
} from './elasticsearch_assets';
import { RiskScoreDataClient } from '../risk_score/risk_score_data_client';
interface EntityStoreClientOpts {
logger: Logger;
esClient: ElasticsearchClient;
entityClient: EntityClient;
assetCriticalityMigrationClient: AssetCriticalityEcsMigrationClient;
namespace: string;
soClient: SavedObjectsClientContract;
taskManager?: TaskManagerStartContract;
auditLogger?: AuditLogger;
kibanaVersion: string;
}
interface SearchEntitiesParams {
@ -45,54 +66,163 @@ interface SearchEntitiesParams {
export class EntityStoreDataClient {
private engineClient: EngineDescriptorClient;
private assetCriticalityMigrationClient: AssetCriticalityEcsMigrationClient;
private entityClient: EntityClient;
private riskScoreDataClient: RiskScoreDataClient;
constructor(private readonly options: EntityStoreClientOpts) {
const { esClient, logger, soClient, auditLogger, kibanaVersion, namespace } = options;
this.entityClient = new EntityClient({
esClient,
soClient,
logger,
});
this.engineClient = new EngineDescriptorClient({
soClient: options.soClient,
namespace: options.namespace,
soClient,
namespace,
});
this.assetCriticalityMigrationClient = new AssetCriticalityEcsMigrationClient({
esClient,
logger,
auditLogger,
});
this.riskScoreDataClient = new RiskScoreDataClient({
soClient,
esClient,
logger,
namespace,
kibanaVersion,
});
}
public async init(
entityType: EntityType,
{ indexPattern = '', filter = '' }: InitEntityEngineRequestBody
{ indexPattern = '', filter = '', fieldHistoryLength = 10 }: InitEntityEngineRequestBody,
{ pipelineDebugMode = false }: { pipelineDebugMode?: boolean } = {}
): Promise<InitEntityEngineResponse> {
const { entityClient, assetCriticalityMigrationClient, logger } = this.options;
const requiresMigration = await assetCriticalityMigrationClient.isEcsDataMigrationRequired();
if (!this.options.taskManager) {
throw new Error('Task Manager is not available');
}
const { logger, esClient, namespace, taskManager } = this.options;
await this.riskScoreDataClient.createRiskScoreLatestIndex();
const requiresMigration =
await this.assetCriticalityMigrationClient.isEcsDataMigrationRequired();
if (requiresMigration) {
throw new Error(
'Asset criticality data migration is required before initializing entity store. If this error persists, please restart Kibana.'
);
}
const definition = getEntityDefinition(entityType, this.options.namespace);
logger.info(
`In namespace ${this.options.namespace}: Initializing entity store for ${entityType}`
);
const debugLog = (message: string) =>
logger.debug(`[Entity Engine] [${entityType}] ${message}`);
const descriptor = await this.engineClient.init(entityType, definition, filter);
await entityClient.createEntityDefinition({
const descriptor = await this.engineClient.init(entityType, {
filter,
fieldHistoryLength,
indexPattern,
});
logger.debug(`Initialized engine for ${entityType}`);
// first create the entity definition without starting it
// so that the index template is created which we can add a component template to
const unitedDefinition = getUnitedEntityDefinition({
entityType,
namespace,
fieldHistoryLength,
});
const { entityManagerDefinition } = unitedDefinition;
await this.entityClient.createEntityDefinition({
definition: {
...definition,
...entityManagerDefinition,
filter,
indexPatterns: indexPattern
? [...definition.indexPatterns, ...indexPattern.split(',')]
: definition.indexPatterns,
? [...entityManagerDefinition.indexPatterns, ...indexPattern.split(',')]
: entityManagerDefinition.indexPatterns,
},
installOnly: true,
});
const updated = await this.engineClient.update(definition.id, ENGINE_STATUS.STARTED);
debugLog(`Created entity definition`);
// the index must be in place with the correct mapping before the enrich policy is created
// this is because the enrich policy will fail if the index does not exist with the correct fields
await createEntityIndexComponentTemplate({
unitedDefinition,
esClient,
});
debugLog(`Created entity index component template`);
await createEntityIndex({
entityType,
esClient,
namespace,
logger,
});
debugLog(`Created entity index`);
// we must create and execute the enrich policy before the pipeline is created
// this is because the pipeline will fail if the enrich index does not exist
await createFieldRetentionEnrichPolicy({
unitedDefinition,
esClient,
});
debugLog(`Created field retention enrich policy`);
await executeFieldRetentionEnrichPolicy({
unitedDefinition,
esClient,
logger,
});
debugLog(`Executed field retention enrich policy`);
await createPlatformPipeline({
debugMode: pipelineDebugMode,
unitedDefinition,
logger,
esClient,
});
debugLog(`Created @platform pipeline`);
// finally start the entity definition now that everything is in place
const updated = await this.start(entityType, { force: true });
debugLog(`Started entity definition`);
// the task will execute the enrich policy on a schedule
await startEntityStoreFieldRetentionEnrichTask({
namespace,
logger,
taskManager,
});
logger.info(`Entity store initialized`);
return { ...descriptor, ...updated };
}
public async start(entityType: EntityType) {
const definition = getEntityDefinition(entityType, this.options.namespace);
public async getExistingEntityDefinition(entityType: EntityType) {
const entityDefinitionId = buildEntityDefinitionId(entityType, this.options.namespace);
const {
definitions: [definition],
} = await this.entityClient.getEntityDefinitions({
id: entityDefinitionId,
});
if (!definition) {
throw new Error(`Unable to find entity definition for ${entityType}`);
}
return definition;
}
public async start(entityType: EntityType, options?: { force: boolean }) {
const descriptor = await this.engineClient.get(entityType);
if (descriptor.status !== ENGINE_STATUS.STOPPED) {
if (!options?.force && descriptor.status !== ENGINE_STATUS.STOPPED) {
throw new Error(
`In namespace ${this.options.namespace}: Cannot start Entity engine for ${entityType} when current status is: ${descriptor.status}`
);
@ -101,14 +231,16 @@ export class EntityStoreDataClient {
this.options.logger.info(
`In namespace ${this.options.namespace}: Starting entity store for ${entityType}`
);
await this.options.entityClient.startEntityDefinition(definition);
return this.engineClient.update(definition.id, ENGINE_STATUS.STARTED);
// startEntityDefinition requires more fields than the engine descriptor
// provides so we need to fetch the full entity definition
const fullEntityDefinition = await this.getExistingEntityDefinition(entityType);
await this.entityClient.startEntityDefinition(fullEntityDefinition);
return this.engineClient.update(entityType, ENGINE_STATUS.STARTED);
}
public async stop(entityType: EntityType) {
const definition = getEntityDefinition(entityType, this.options.namespace);
const descriptor = await this.engineClient.get(entityType);
if (descriptor.status !== ENGINE_STATUS.STARTED) {
@ -120,9 +252,12 @@ export class EntityStoreDataClient {
this.options.logger.info(
`In namespace ${this.options.namespace}: Stopping entity store for ${entityType}`
);
await this.options.entityClient.stopEntityDefinition(definition);
// stopEntityDefinition requires more fields than the engine descriptor
// provides so we need to fetch the full entity definition
const fullEntityDefinition = await this.getExistingEntityDefinition(entityType);
await this.entityClient.stopEntityDefinition(fullEntityDefinition);
return this.engineClient.update(definition.id, ENGINE_STATUS.STOPPED);
return this.engineClient.update(entityType, ENGINE_STATUS.STOPPED);
}
public async get(entityType: EntityType) {
@ -133,17 +268,71 @@ export class EntityStoreDataClient {
return this.engineClient.list();
}
public async delete(entityType: EntityType, deleteData: boolean) {
const { id } = getEntityDefinition(entityType, this.options.namespace);
public async delete(
entityType: EntityType,
taskManager: TaskManagerStartContract,
deleteData: boolean
) {
const { namespace, logger, esClient } = this.options;
const descriptor = await this.engineClient.maybeGet(entityType);
const unitedDefinition = getUnitedEntityDefinition({
entityType,
namespace: this.options.namespace,
fieldHistoryLength: descriptor?.fieldHistoryLength ?? 10,
});
const { entityManagerDefinition } = unitedDefinition;
logger.info(`In namespace ${namespace}: Deleting entity store for ${entityType}`);
try {
try {
await this.entityClient.deleteEntityDefinition({
id: entityManagerDefinition.id,
deleteData,
});
} catch (e) {
logger.error(`Error deleting entity definition for ${entityType}: ${e.message}`);
}
await deleteEntityIndexComponentTemplate({
unitedDefinition,
esClient,
});
await deletePlatformPipeline({
unitedDefinition,
logger,
esClient,
});
await deleteFieldRetentionEnrichPolicy({
unitedDefinition,
esClient,
});
this.options.logger.info(
`In namespace ${this.options.namespace}: Deleting entity store for ${entityType}`
);
if (deleteData) {
await deleteEntityIndex({
entityType,
esClient,
namespace,
logger,
});
}
// if the last engine then stop the task
const { engines } = await this.engineClient.list();
if (engines.length === 0) {
await removeEntityStoreFieldRetentionEnrichTask({
namespace,
logger,
taskManager,
});
}
await this.options.entityClient.deleteEntityDefinition({ id, deleteData });
await this.engineClient.delete(id);
if (descriptor) {
await this.engineClient.delete(entityType);
}
return { deleted: true };
return { deleted: true };
} catch (e) {
logger.error(`Error deleting entity store for ${entityType}: ${e.message}`);
// TODO: should we set the engine status to error here?
throw e;
}
}
public async searchEntities(params: SearchEntitiesParams): Promise<{

53
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/field_retention_definition/collect_values.ts Normal file
View file

@ -0,0 +1,53 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { isFieldMissingOrEmpty } from '../painless';
import type { BaseFieldRetentionOperator, FieldRetentionOperatorBuilder } from './types';
export interface CollectValues extends BaseFieldRetentionOperator {
operation: 'collect_values';
maxLength: number;
}
/**
* A field retention operator that collects up to `maxLength` values of the specified field.
* Values are first collected from the field, then from the enrich field if the field is not present or empty.
*/
export const collectValuesProcessor: FieldRetentionOperatorBuilder<CollectValues> = (
{ field, maxLength },
{ enrichField }
) => {
const ctxField = `ctx.${field}`;
const ctxEnrichField = `ctx.${enrichField}.${field}`;
return {
script: {
lang: 'painless',
source: `
Set uniqueVals = new HashSet();
if (!(${isFieldMissingOrEmpty(ctxField)})) {
if(${ctxField} instanceof List) {
uniqueVals.addAll(${ctxField});
} else {
uniqueVals.add(${ctxField});
}
}
if (uniqueVals.size() < params.max_length && !(${isFieldMissingOrEmpty(ctxEnrichField)})) {
int remaining = params.max_length - uniqueVals.size();
List historicalVals = ${ctxEnrichField}.subList(0, (int) Math.min(remaining, ${ctxEnrichField}.size()));
uniqueVals.addAll(historicalVals);
}
${ctxField} = new ArrayList(uniqueVals).subList(0, (int) Math.min(params.max_length, uniqueVals.size()));
`,
params: {
max_length: maxLength,
},
},
};
};

9
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/field_retention_definition/index.ts Normal file
View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export type * from './types';
export { fieldOperatorToIngestProcessor } from './operator_to_ingest_processor';

28
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/field_retention_definition/operator_to_ingest_processor.ts Normal file
View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { FieldRetentionOperator, FieldRetentionOperatorBuilder } from './types';
import { preferNewestValueProcessor } from './prefer_newest_value';
import { preferOldestValueProcessor } from './prefer_oldest_value';
import { collectValuesProcessor } from './collect_values';
/**
* Converts a field retention operator to an ingest processor.
* An ingest processor is a step that can be added to an ingest pipeline.
*/
export const fieldOperatorToIngestProcessor: FieldRetentionOperatorBuilder<
FieldRetentionOperator
> = (fieldOperator, options) => {
switch (fieldOperator.operation) {
case 'prefer_newest_value':
return preferNewestValueProcessor(fieldOperator, options);
case 'prefer_oldest_value':
return preferOldestValueProcessor(fieldOperator, options);
case 'collect_values':
return collectValuesProcessor(fieldOperator, options);
}
};

34
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/field_retention_definition/prefer_newest_value.ts Normal file
View file

@ -0,0 +1,34 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { BaseFieldRetentionOperator, FieldRetentionOperatorBuilder } from './types';
import { isFieldMissingOrEmpty } from '../painless';
export interface PreferNewestValue extends BaseFieldRetentionOperator {
operation: 'prefer_newest_value';
}
/**
* A field retention operator that prefers the newest value of the specified field.
* If the field is missing or empty, the value from the enrich field is used.
*/
export const preferNewestValueProcessor: FieldRetentionOperatorBuilder<PreferNewestValue> = (
{ field },
{ enrichField }
) => {
const latestField = `ctx.${field}`;
const historicalField = `${enrichField}.${field}`;
return {
set: {
if: `(${isFieldMissingOrEmpty(latestField)}) && !(${isFieldMissingOrEmpty(
`ctx.${historicalField}`
)})`,
field,
value: `{{${historicalField}}}`,
},
};
};

31
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/field_retention_definition/prefer_oldest_value.ts Normal file
View file

@ -0,0 +1,31 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { BaseFieldRetentionOperator, FieldRetentionOperatorBuilder } from './types';
import { isFieldMissingOrEmpty } from '../painless';
export interface PreferOldestValue extends BaseFieldRetentionOperator {
operation: 'prefer_oldest_value';
}
/**
* A field retention operator that prefers the oldest value of the specified field.
* If the historical field is missing or empty, the latest value is used.
*/
export const preferOldestValueProcessor: FieldRetentionOperatorBuilder<PreferOldestValue> = (
{ field },
{ enrichField }
) => {
const historicalField = `${enrichField}.${field}`;
return {
set: {
if: `!(${isFieldMissingOrEmpty(`ctx.${historicalField}`)})`,
field,
value: `{{${historicalField}}}`,
},
};
};

34
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/field_retention_definition/types.ts Normal file
View file

@ -0,0 +1,34 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IngestProcessorContainer } from '@elastic/elasticsearch/lib/api/types';
import type { EntityType } from '../../../../../common/api/entity_analytics/entity_store';
import type { CollectValues } from './collect_values';
import type { PreferNewestValue } from './prefer_newest_value';
import type { PreferOldestValue } from './prefer_oldest_value';
export interface FieldRetentionDefinition {
entityType: EntityType;
matchField: string;
fields: FieldRetentionOperator[];
}
export interface BaseFieldRetentionOperator {
field: string;
operation: string;
}
export interface FieldRetentionOperatorBuilderOptions {
enrichField: string;
}
export type FieldRetentionOperator = PreferNewestValue | PreferOldestValue | CollectValues;
export type FieldRetentionOperatorBuilder<O extends BaseFieldRetentionOperator> = (
operator: O,
options: FieldRetentionOperatorBuilderOptions
) => IngestProcessorContainer;

9
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/painless/index.ts Normal file
View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './is_field_missing_or_empty';
export * from './path_utils';

25
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/painless/is_field_missing_or_empty.ts Normal file
View file

@ -0,0 +1,25 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { getConditionalPath } from './path_utils';
/**
* Given a field, returns a painless script that checks if the field is missing or empty.
*
* @param {string} field The field to check with dot notation e.g ctx.a.b.c
* @return {*} {string} The painless script that checks if the field is missing or empty
*/
export const isFieldMissingOrEmpty = (field: string): string => {
const conditionalPath = getConditionalPath(field);
const classesWithEmptyCheck = ['Collection', 'String', 'Map'];
const emptyCheck = `((${classesWithEmptyCheck
.map((c) => `${field} instanceof ${c}`)
.join(' || ')}) && ${field}.isEmpty())`;
return `${conditionalPath} == null || ${emptyCheck}`;
};

28
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/painless/path_utils.test.ts Normal file
View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { getConditionalPath } from './path_utils';
describe('painless path utils', () => {
describe('getConditionalPath', () => {
it('should do nothing with single value', () => {
const path = 'a';
const result = getConditionalPath(path);
expect(result).toEqual('a');
});
it('should conditionalise path length 2', () => {
const path = 'a.b';
const result = getConditionalPath(path);
expect(result).toEqual('a?.b');
});
it('should conditionalise longer path', () => {
const path = 'a.b.c.d.e.f.g';
const result = getConditionalPath(path);
expect(result).toEqual('a?.b?.c?.d?.e?.f?.g');
});
});
});

16
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/painless/path_utils.ts Normal file
View file

@ -0,0 +1,16 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/**
* Converts a dot notation path to a bracket notation path
* e.g "a.b.c" => "a['b']['c']"
* @param {string} path
* @return {*} {string}
*/
// convert a path like a.b.c to a?.b?.c
export const getConditionalPath = (path: string): string => path.split('.').join('?.');

14
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/routes/delete.ts
View file

@ -17,10 +17,12 @@ import {
} from '../../../../../common/api/entity_analytics/entity_store/engine/delete.gen';
import { API_VERSIONS, APP_ID } from '../../../../../common/constants';
import type { EntityAnalyticsRoutesDeps } from '../../types';
import { TASK_MANAGER_UNAVAILABLE_ERROR } from '../../risk_engine/routes/translations';
export const deleteEntityEngineRoute = (
router: EntityAnalyticsRoutesDeps['router'],
logger: Logger
logger: Logger,
getStartServices: EntityAnalyticsRoutesDeps['getStartServices']
) => {
router.versioned
.delete({
@ -43,12 +45,18 @@ export const deleteEntityEngineRoute = (
async (context, request, response): Promise<IKibanaResponse<DeleteEntityEngineResponse>> => {
const siemResponse = buildSiemResponse(response);
const [_, { taskManager }] = await getStartServices();
if (!taskManager) {
return siemResponse.error({
statusCode: 400,
body: TASK_MANAGER_UNAVAILABLE_ERROR,
});
}
try {
const secSol = await context.securitySolution;
const body = await secSol
.getEntityStoreDataClient()
.delete(request.params.entityType, !!request.query.data);
.delete(request.params.entityType, taskManager, !!request.query.data);
return response.ok({ body });
} catch (e) {

16
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/routes/init.ts
View file

@ -17,10 +17,12 @@ import {
} from '../../../../../common/api/entity_analytics/entity_store/engine/init.gen';
import { API_VERSIONS, APP_ID } from '../../../../../common/constants';
import type { EntityAnalyticsRoutesDeps } from '../../types';
import { checkAndInitAssetCriticalityResources } from '../../asset_criticality/check_and_init_asset_criticality_resources';
export const initEntityEngineRoute = (
router: EntityAnalyticsRoutesDeps['router'],
logger: Logger
logger: Logger,
config: EntityAnalyticsRoutesDeps['config']
) => {
router.versioned
.post({
@ -43,18 +45,22 @@ export const initEntityEngineRoute = (
async (context, request, response): Promise<IKibanaResponse<InitEntityEngineResponse>> => {
const siemResponse = buildSiemResponse(response);
const secSol = await context.securitySolution;
const { pipelineDebugMode } = config.entityAnalytics.entityStore.developer;
await checkAndInitAssetCriticalityResources(context, logger);
try {
const secSol = await context.securitySolution;
const body: InitEntityEngineResponse = await secSol
.getEntityStoreDataClient()
.init(request.params.entityType, request.body);
.init(request.params.entityType, request.body, {
pipelineDebugMode,
});
return response.ok({ body });
} catch (e) {
logger.error('Error in InitEntityEngine:', e);
const error = transformError(e);
logger.error(`Error initialising entity engine: ${error.message}`);
return siemResponse.error({
statusCode: error.statusCode,
body: error.message,

11
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/routes/register_entity_store_routes.ts
View file

@ -14,11 +14,16 @@ import { listEntityEnginesRoute } from './list';
import { startEntityEngineRoute } from './start';
import { stopEntityEngineRoute } from './stop';
export const registerEntityStoreRoutes = ({ router, logger }: EntityAnalyticsRoutesDeps) => {
initEntityEngineRoute(router, logger);
export const registerEntityStoreRoutes = ({
router,
logger,
getStartServices,
config,
}: EntityAnalyticsRoutesDeps) => {
initEntityEngineRoute(router, logger, config);
startEntityEngineRoute(router, logger);
stopEntityEngineRoute(router, logger);
deleteEntityEngineRoute(router, logger);
deleteEntityEngineRoute(router, logger, getStartServices);
getEntityEngineRoute(router, logger);
listEntityEnginesRoute(router, logger);
listEntitiesRoute(router, logger);

41
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/saved_object/engine_descriptor.ts
View file

@ -9,7 +9,6 @@ import type {
SavedObjectsClientContract,
SavedObjectsFindResponse,
} from '@kbn/core-saved-objects-api-server';
import type { EntityDefinition } from '@kbn/entities-schema';
import type {
EngineDescriptor,
EngineStatus,
@ -17,7 +16,7 @@ import type {
} from '../../../../../common/api/entity_analytics/entity_store/common.gen';
import { entityEngineDescriptorTypeName } from './engine_descriptor_type';
import { getByEntityTypeQuery, getEntityDefinition } from '../utils/utils';
import { getByEntityTypeQuery } from '../utils';
import { ENGINE_STATUS } from '../constants';
interface EngineDescriptorDependencies {
@ -28,7 +27,18 @@ interface EngineDescriptorDependencies {
export class EngineDescriptorClient {
constructor(private readonly deps: EngineDescriptorDependencies) {}
async init(entityType: EntityType, definition: EntityDefinition, filter: string) {
getSavedObjectId(entityType: EntityType) {
return `entity-engine-descriptor-${entityType}-${this.deps.namespace}`;
}
async init(
entityType: EntityType,
{
filter,
fieldHistoryLength,
indexPattern,
}: { filter: string; fieldHistoryLength: number; indexPattern: string }
) {
const engineDescriptor = await this.find(entityType);
if (engineDescriptor.total > 0)
@ -39,15 +49,17 @@ export class EngineDescriptorClient {
{
status: ENGINE_STATUS.INSTALLING,
type: entityType,
indexPattern: definition.indexPatterns.join(','),
indexPattern,
filter,
fieldHistoryLength,
},
{ id: definition.id }
{ id: this.getSavedObjectId(entityType) }
);
return attributes;
}
async update(id: string, status: EngineStatus) {
async update(entityType: EntityType, status: EngineStatus) {
const id = this.getSavedObjectId(entityType);
const { attributes } = await this.deps.soClient.update<EngineDescriptor>(
entityEngineDescriptorTypeName,
id,
@ -66,7 +78,7 @@ export class EngineDescriptorClient {
}
async get(entityType: EntityType): Promise<EngineDescriptor> {
const { id } = getEntityDefinition(entityType, this.deps.namespace);
const id = this.getSavedObjectId(entityType);
const { attributes } = await this.deps.soClient.get<EngineDescriptor>(
entityEngineDescriptorTypeName,
@ -76,6 +88,18 @@ export class EngineDescriptorClient {
return attributes;
}
async maybeGet(entityType: EntityType): Promise<EngineDescriptor | undefined> {
try {
const descriptor = await this.get(entityType);
return descriptor;
} catch (e) {
if (e.isBoom && e.output.statusCode === 404) {
return undefined;
}
throw e;
}
}
async list() {
return this.deps.soClient
.find<EngineDescriptor>({
@ -88,7 +112,8 @@ export class EngineDescriptorClient {
}));
}
async delete(id: string) {
async delete(entityType: EntityType) {
const id = this.getSavedObjectId(entityType);
return this.deps.soClient.delete(entityEngineDescriptorTypeName, id);
}
}

29
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/saved_object/engine_descriptor_type.ts
View file

@ -5,6 +5,7 @@
* 2.0.
*/
import type { SavedObjectsModelVersion } from '@kbn/core-saved-objects-server';
import { SECURITY_SOLUTION_SAVED_OBJECT_INDEX } from '@kbn/core-saved-objects-server';
import type { SavedObjectsType } from '@kbn/core/server';
@ -25,12 +26,40 @@ export const entityEngineDescriptorTypeMappings: SavedObjectsType['mappings'] =
status: {
type: 'keyword', // EngineStatus: installing | started | stopped
},
fieldHistoryLength: {
type: 'integer',
index: false,
},
},
};
const version1: SavedObjectsModelVersion = {
changes: [
{
type: 'mappings_addition',
addedMappings: {
fieldHistoryLength: { type: 'integer', index: false },
},
},
{
type: 'data_backfill',
backfillFn: (document) => {
return {
attributes: {
...document.attributes,
fieldHistoryLength: 10,
},
};
},
},
],
};
export const entityEngineDescriptorType: SavedObjectsType = {
name: entityEngineDescriptorTypeName,
indexPattern: SECURITY_SOLUTION_SAVED_OBJECT_INDEX,
hidden: false,
namespaceType: 'multiple-isolated',
mappings: entityEngineDescriptorTypeMappings,
modelVersions: { 1: version1 },
};

12
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/task/constants.ts Normal file
View file

@ -0,0 +1,12 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export const SCOPE = ['securitySolution'];
export const TYPE = 'entity_store:field_retention:enrichment';
export const VERSION = '1.0.0';
export const INTERVAL = '1h';
export const TIMEOUT = '10m';

221
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/task/field_retention_enrichment_task.ts Normal file
View file

@ -0,0 +1,221 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import moment from 'moment';
import { type Logger, SavedObjectsErrorHelpers } from '@kbn/core/server';
import type {
ConcreteTaskInstance,
TaskManagerSetupContract,
TaskManagerStartContract,
} from '@kbn/task-manager-plugin/server';
import type { EntityType } from '../../../../../common/api/entity_analytics/entity_store';
import {
defaultState,
stateSchemaByVersion,
type LatestTaskStateSchema as EntityStoreFieldRetentionTaskState,
} from './state';
import { INTERVAL, SCOPE, TIMEOUT, TYPE, VERSION } from './constants';
import type { EntityAnalyticsRoutesDeps } from '../../types';
import { getAvailableEntityTypes, getUnitedEntityDefinition } from '../united_entity_definitions';
import { executeFieldRetentionEnrichPolicy } from '../elasticsearch_assets';
const logFactory =
(logger: Logger, taskId: string) =>
(message: string): void =>
logger.info(`[task ${taskId}]: ${message}`);
const debugLogFactory =
(logger: Logger, taskId: string) =>
(message: string): void =>
logger.debug(`[task ${taskId}]: ${message}`);
const getTaskName = (): string => TYPE;
const getTaskId = (namespace: string): string => `${TYPE}:${namespace}:${VERSION}`;
type ExecuteEnrichPolicy = (
namespace: string,
entityType: EntityType
) => ReturnType<typeof executeFieldRetentionEnrichPolicy>;
export const registerEntityStoreFieldRetentionEnrichTask = ({
getStartServices,
logger,
taskManager,
}: {
getStartServices: EntityAnalyticsRoutesDeps['getStartServices'];
logger: Logger;
taskManager: TaskManagerSetupContract | undefined;
}): void => {
if (!taskManager) {
logger.info('Task Manager is unavailable; skipping entity store enrich policy registration.');
return;
}
const executeEnrichPolicy: ExecuteEnrichPolicy = async (
namespace: string,
entityType: EntityType
): ReturnType<typeof executeFieldRetentionEnrichPolicy> => {
const [coreStart, _] = await getStartServices();
const esClient = coreStart.elasticsearch.client.asInternalUser;
const unitedDefinition = getUnitedEntityDefinition({
namespace,
entityType,
fieldHistoryLength: 10, // we are not using this value so it can be anything
});
return executeFieldRetentionEnrichPolicy({
unitedDefinition,
esClient,
logger,
});
};
taskManager.registerTaskDefinitions({
[getTaskName()]: {
title: 'Entity Analytics Entity Store - Execute Enrich Policy Task',
timeout: TIMEOUT,
stateSchemaByVersion,
createTaskRunner: createTaskRunnerFactory({
logger,
executeEnrichPolicy,
}),
},
});
};
export const startEntityStoreFieldRetentionEnrichTask = async ({
logger,
namespace,
taskManager,
}: {
logger: Logger;
namespace: string;
taskManager: TaskManagerStartContract;
}) => {
const taskId = getTaskId(namespace);
const log = logFactory(logger, taskId);
log('starting task');
log('attempting to schedule');
try {
await taskManager.ensureScheduled({
id: taskId,
taskType: getTaskName(),
scope: SCOPE,
schedule: {
interval: INTERVAL,
},
state: { ...defaultState, namespace },
params: { version: VERSION },
});
} catch (e) {
logger.warn(`[task ${taskId}]: error scheduling task, received ${e.message}`);
throw e;
}
};
export const removeEntityStoreFieldRetentionEnrichTask = async ({
logger,
namespace,
taskManager,
}: {
logger: Logger;
namespace: string;
taskManager: TaskManagerStartContract;
}) => {
try {
await taskManager.remove(getTaskId(namespace));
} catch (err) {
if (!SavedObjectsErrorHelpers.isNotFoundError(err)) {
logger.error(`Failed to remove entity store enrich policy task: ${err.message}`);
throw err;
}
}
};
export const runTask = async ({
executeEnrichPolicy,
isCancelled,
logger,
taskInstance,
}: {
logger: Logger;
isCancelled: () => boolean;
executeEnrichPolicy: ExecuteEnrichPolicy;
taskInstance: ConcreteTaskInstance;
}): Promise<{
state: EntityStoreFieldRetentionTaskState;
}> => {
const state = taskInstance.state as EntityStoreFieldRetentionTaskState;
const taskId = taskInstance.id;
const log = logFactory(logger, taskId);
const debugLog = debugLogFactory(logger, taskId);
try {
const taskStartTime = moment().utc().toISOString();
log('running task');
const updatedState = {
lastExecutionTimestamp: taskStartTime,
namespace: state.namespace,
runs: state.runs + 1,
};
if (taskId !== getTaskId(state.namespace)) {
log('outdated task; exiting');
return { state: updatedState };
}
const entityTypes = getAvailableEntityTypes();
for (const entityType of entityTypes) {
const start = Date.now();
debugLog(`executing field retention enrich policy for ${entityType}`);
try {
const { executed } = await executeEnrichPolicy(state.namespace, entityType);
if (!executed) {
debugLog(`Field retention encrich policy for ${entityType} does not exist`);
} else {
log(
`Executed field retention enrich policy for ${entityType} in ${Date.now() - start}ms`
);
}
} catch (e) {
log(`error executing field retention enrich policy for ${entityType}: ${e.message}`);
}
}
const taskCompletionTime = moment().utc().toISOString();
const taskDurationInSeconds = moment(taskCompletionTime).diff(moment(taskStartTime), 'seconds');
log(`task run completed in ${taskDurationInSeconds} seconds`);
return {
state: updatedState,
};
} catch (e) {
logger.error(`[task ${taskId}]: error running task, received ${e.message}`);
throw e;
}
};
const createTaskRunnerFactory =
({ logger, executeEnrichPolicy }: { logger: Logger; executeEnrichPolicy: ExecuteEnrichPolicy }) =>
({ taskInstance }: { taskInstance: ConcreteTaskInstance }) => {
let cancelled = false;
const isCancelled = () => cancelled;
return {
run: async () =>
runTask({
executeEnrichPolicy,
isCancelled,
logger,
taskInstance,
}),
cancel: async () => {
cancelled = true;
},
};
};

8
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/task/index.ts Normal file
View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './field_retention_enrichment_task';

44
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/task/state.ts Normal file
View file

@ -0,0 +1,44 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { schema, type TypeOf } from '@kbn/config-schema';
/**
* WARNING: Do not modify the existing versioned schema(s) below, instead define a new version (ex: 2, 3, 4).
* This is required to support zero-downtime upgrades and rollbacks. See https://github.com/elastic/kibana/issues/155764.
*
* As you add a new schema version, don't forget to change latestTaskStateSchema variable to reference the latest schema.
* For example, changing stateSchemaByVersion[1].schema to stateSchemaByVersion[2].schema.
*/
export const stateSchemaByVersion = {
1: {
// A task that was created < 8.10 will go through this "up" migration
// to ensure it matches the v1 schema.
up: (state: Record<string, unknown>) => ({
lastExecutionTimestamp: state.lastExecutionTimestamp || undefined,
runs: state.runs || 0,
namespace: typeof state.namespace === 'string' ? state.namespace : 'default',
scoresWritten: typeof state.scoresWritten === 'number' ? state.scoresWritten : undefined,
}),
schema: schema.object({
lastExecutionTimestamp: schema.maybe(schema.string()),
namespace: schema.string(),
runs: schema.number(),
scoresWritten: schema.maybe(schema.number()),
}),
},
};
const latestTaskStateSchema = stateSchemaByVersion[1].schema;
export type LatestTaskStateSchema = TypeOf<typeof latestTaskStateSchema>;
export const defaultState: LatestTaskStateSchema = {
lastExecutionTimestamp: undefined,
namespace: 'default',
runs: 0,
scoresWritten: undefined,
};

28
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/constants.ts Normal file
View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { MappingProperties } from './types';
export const BASE_ENTITY_INDEX_MAPPING: MappingProperties = {
'@timestamp': {
type: 'date',
},
'asset.criticality': {
type: 'keyword',
},
'entity.name': {
type: 'text',
fields: {
text: {
type: 'keyword',
},
},
},
'entity.source': {
type: 'keyword',
},
};

86
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/definition_utils.ts Normal file
View file

@ -0,0 +1,86 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { MappingProperty } from '@elastic/elasticsearch/lib/api/types';
import type { UnitedDefinitionField } from './types';
export const collectValues = ({
field,
mapping = { type: 'keyword' },
sourceField,
fieldHistoryLength,
}: {
field: string;
mapping?: MappingProperty;
sourceField?: string;
fieldHistoryLength: number;
}): UnitedDefinitionField => ({
field,
definition: {
source: sourceField ?? field,
destination: field,
aggregation: {
type: 'terms',
limit: fieldHistoryLength,
},
},
retention_operator: { operation: 'collect_values', field, maxLength: fieldHistoryLength },
mapping,
});
export const collectValuesWithLength =
(fieldHistoryLength: number) =>
(opts: { field: string; mapping?: MappingProperty; sourceField?: string }) =>
collectValues({ ...opts, fieldHistoryLength });
export const newestValue = ({
field,
mapping = { type: 'keyword' },
sourceField,
}: {
field: string;
mapping?: MappingProperty;
sourceField?: string;
}): UnitedDefinitionField => ({
field,
definition: {
source: sourceField ?? field,
destination: field,
aggregation: {
type: 'top_value',
sort: {
'@timestamp': 'desc',
},
},
},
retention_operator: { operation: 'prefer_newest_value', field },
mapping,
});
export const oldestValue = ({
field,
mapping = { type: 'keyword' },
sourceField,
}: {
field: string;
mapping?: MappingProperty;
sourceField?: string;
}): UnitedDefinitionField => ({
field,
definition: {
source: sourceField ?? field,
destination: field,
aggregation: {
type: 'top_value',
sort: {
'@timestamp': 'asc',
},
},
},
retention_operator: { operation: 'prefer_oldest_value', field },
mapping,
});

47
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/entity_types/common.ts Normal file
View file

@ -0,0 +1,47 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { EntityType } from '../../../../../../common/api/entity_analytics/entity_store';
import { getIdentityFieldForEntityType } from '../../utils';
import { collectValues, newestValue } from '../definition_utils';
import type { UnitedDefinitionField } from '../types';
export const getCommonUnitedFieldDefinitions = ({
entityType,
fieldHistoryLength,
}: {
entityType: EntityType;
fieldHistoryLength: number;
}): UnitedDefinitionField[] => {
const identityField = getIdentityFieldForEntityType(entityType);
return [
collectValues({
sourceField: '_index',
field: 'entity.source',
fieldHistoryLength,
}),
newestValue({ field: 'asset.criticality' }),
newestValue({
field: `${entityType}.risk.calculated_level`,
}),
newestValue({
field: `${entityType}.risk.calculated_score`,
mapping: {
type: 'float',
},
}),
newestValue({
field: `${entityType}.risk.calculated_score_norm`,
mapping: {
type: 'float',
},
}),
{
field: identityField,
},
];
};

31
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/entity_types/host.ts Normal file
View file

@ -0,0 +1,31 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { collectValuesWithLength } from '../definition_utils';
import type { UnitedDefinitionBuilder } from '../types';
export const getHostUnitedDefinition: UnitedDefinitionBuilder = (fieldHistoryLength: number) => {
const collect = collectValuesWithLength(fieldHistoryLength);
return {
entityType: 'host',
version: '1.0.0',
fields: [
collect({ field: 'host.domain' }),
collect({ field: 'host.hostname' }),
collect({ field: 'host.id' }),
collect({
field: 'host.ip',
mapping: {
type: 'ip',
},
}),
collect({ field: 'host.mac' }),
collect({ field: 'host.type' }),
collect({ field: 'host.architecture' }),
],
};
};

10
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/entity_types/index.ts Normal file
View file

@ -0,0 +1,10 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { getHostUnitedDefinition } from './host';
export { getUserUnitedDefinition } from './user';
export { getCommonUnitedFieldDefinitions } from './common';

24
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/entity_types/user.ts Normal file
View file

@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { collectValuesWithLength } from '../definition_utils';
import type { UnitedDefinitionBuilder } from '../types';
export const getUserUnitedDefinition: UnitedDefinitionBuilder = (fieldHistoryLength: number) => {
const collect = collectValuesWithLength(fieldHistoryLength);
return {
entityType: 'user',
version: '1.0.0',
fields: [
collect({ field: 'user.domain' }),
collect({ field: 'user.email' }),
collect({ field: 'user.full_name' }),
collect({ field: 'user.hash' }),
collect({ field: 'user.id' }),
collect({ field: 'user.roles' }),
],
};
};

541
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/get_united_definition.test.ts Normal file
View file

@ -0,0 +1,541 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { getUnitedEntityDefinition } from './get_united_definition';
describe('getUnitedEntityDefinition', () => {
describe('host', () => {
const unitedDefinition = getUnitedEntityDefinition({
entityType: 'host',
namespace: 'test',
fieldHistoryLength: 10,
});
it('mapping', () => {
expect(unitedDefinition.indexMappings).toMatchInlineSnapshot(`
Object {
"properties": Object {
"@timestamp": Object {
"type": "date",
},
"asset.criticality": Object {
"type": "keyword",
},
"entity.name": Object {
"fields": Object {
"text": Object {
"type": "keyword",
},
},
"type": "text",
},
"entity.source": Object {
"type": "keyword",
},
"host.architecture": Object {
"type": "keyword",
},
"host.domain": Object {
"type": "keyword",
},
"host.hostname": Object {
"type": "keyword",
},
"host.id": Object {
"type": "keyword",
},
"host.ip": Object {
"type": "ip",
},
"host.mac": Object {
"type": "keyword",
},
"host.name": Object {
"type": "keyword",
},
"host.risk.calculated_level": Object {
"type": "keyword",
},
"host.risk.calculated_score": Object {
"type": "float",
},
"host.risk.calculated_score_norm": Object {
"type": "float",
},
"host.type": Object {
"type": "keyword",
},
},
}
`);
});
it('fieldRetentionDefinition', () => {
expect(unitedDefinition.fieldRetentionDefinition).toMatchInlineSnapshot(`
Object {
"entityType": "host",
"fields": Array [
Object {
"field": "host.domain",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "host.hostname",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "host.id",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "host.ip",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "host.mac",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "host.type",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "host.architecture",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "entity.source",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "asset.criticality",
"operation": "prefer_newest_value",
},
Object {
"field": "host.risk.calculated_level",
"operation": "prefer_newest_value",
},
Object {
"field": "host.risk.calculated_score",
"operation": "prefer_newest_value",
},
Object {
"field": "host.risk.calculated_score_norm",
"operation": "prefer_newest_value",
},
],
"matchField": "host.name",
}
`);
});
it('entityManagerDefinition', () => {
expect(unitedDefinition.entityManagerDefinition).toMatchInlineSnapshot(`
Object {
"displayNameTemplate": "{{host.name}}",
"id": "security_host_test",
"identityFields": Array [
Object {
"field": "host.name",
"optional": false,
},
],
"indexPatterns": Array [
"apm-*-transaction*",
"auditbeat-*",
"endgame-*",
"filebeat-*",
"logs-*",
"packetbeat-*",
"traces-apm*",
"winlogbeat-*",
"-*elastic-cloud-logs-*",
".asset-criticality.asset-criticality-test",
"risk-score.risk-score-latest-test",
],
"latest": Object {
"lookbackPeriod": "24h",
"timestampField": "@timestamp",
},
"managed": true,
"metadata": Array [
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.domain",
"source": "host.domain",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.hostname",
"source": "host.hostname",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.id",
"source": "host.id",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.ip",
"source": "host.ip",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.mac",
"source": "host.mac",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.type",
"source": "host.type",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "host.architecture",
"source": "host.architecture",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "entity.source",
"source": "_index",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "asset.criticality",
"source": "asset.criticality",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "host.risk.calculated_level",
"source": "host.risk.calculated_level",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "host.risk.calculated_score",
"source": "host.risk.calculated_score",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "host.risk.calculated_score_norm",
"source": "host.risk.calculated_score_norm",
},
],
"name": "Security 'host' Entity Store Definition",
"type": "host",
"version": "1.0.0",
}
`);
});
});
describe('user', () => {
const unitedDefinition = getUnitedEntityDefinition({
entityType: 'user',
namespace: 'test',
fieldHistoryLength: 10,
});
it('mapping', () => {
expect(unitedDefinition.indexMappings).toMatchInlineSnapshot(`
Object {
"properties": Object {
"@timestamp": Object {
"type": "date",
},
"asset.criticality": Object {
"type": "keyword",
},
"entity.name": Object {
"fields": Object {
"text": Object {
"type": "keyword",
},
},
"type": "text",
},
"entity.source": Object {
"type": "keyword",
},
"user.domain": Object {
"type": "keyword",
},
"user.email": Object {
"type": "keyword",
},
"user.full_name": Object {
"type": "keyword",
},
"user.hash": Object {
"type": "keyword",
},
"user.id": Object {
"type": "keyword",
},
"user.name": Object {
"type": "keyword",
},
"user.risk.calculated_level": Object {
"type": "keyword",
},
"user.risk.calculated_score": Object {
"type": "float",
},
"user.risk.calculated_score_norm": Object {
"type": "float",
},
"user.roles": Object {
"type": "keyword",
},
},
}
`);
});
it('fieldRetentionDefinition', () => {
expect(unitedDefinition.fieldRetentionDefinition).toMatchInlineSnapshot(`
Object {
"entityType": "user",
"fields": Array [
Object {
"field": "user.domain",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "user.email",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "user.full_name",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "user.hash",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "user.id",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "user.roles",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "entity.source",
"maxLength": 10,
"operation": "collect_values",
},
Object {
"field": "asset.criticality",
"operation": "prefer_newest_value",
},
Object {
"field": "user.risk.calculated_level",
"operation": "prefer_newest_value",
},
Object {
"field": "user.risk.calculated_score",
"operation": "prefer_newest_value",
},
Object {
"field": "user.risk.calculated_score_norm",
"operation": "prefer_newest_value",
},
],
"matchField": "user.name",
}
`);
});
it('entityManagerDefinition', () => {
expect(unitedDefinition.entityManagerDefinition).toMatchInlineSnapshot(`
Object {
"displayNameTemplate": "{{user.name}}",
"id": "security_user_test",
"identityFields": Array [
Object {
"field": "user.name",
"optional": false,
},
],
"indexPatterns": Array [
"apm-*-transaction*",
"auditbeat-*",
"endgame-*",
"filebeat-*",
"logs-*",
"packetbeat-*",
"traces-apm*",
"winlogbeat-*",
"-*elastic-cloud-logs-*",
".asset-criticality.asset-criticality-test",
"risk-score.risk-score-latest-test",
],
"latest": Object {
"lookbackPeriod": "24h",
"timestampField": "@timestamp",
},
"managed": true,
"metadata": Array [
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "user.domain",
"source": "user.domain",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "user.email",
"source": "user.email",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "user.full_name",
"source": "user.full_name",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "user.hash",
"source": "user.hash",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "user.id",
"source": "user.id",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "user.roles",
"source": "user.roles",
},
Object {
"aggregation": Object {
"limit": 10,
"type": "terms",
},
"destination": "entity.source",
"source": "_index",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "asset.criticality",
"source": "asset.criticality",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "user.risk.calculated_level",
"source": "user.risk.calculated_level",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "user.risk.calculated_score",
"source": "user.risk.calculated_score",
},
Object {
"aggregation": Object {
"sort": Object {
"@timestamp": "desc",
},
"type": "top_value",
},
"destination": "user.risk.calculated_score_norm",
"source": "user.risk.calculated_score_norm",
},
],
"name": "Security 'user' Entity Store Definition",
"type": "user",
"version": "1.0.0",
}
`);
});
});
});

48
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/get_united_definition.ts Normal file
View file

@ -0,0 +1,48 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { memoize } from 'lodash';
import type { EntityType } from '../../../../../common/api/entity_analytics';
import {
getHostUnitedDefinition,
getUserUnitedDefinition,
getCommonUnitedFieldDefinitions,
} from './entity_types';
import type { UnitedDefinitionBuilder } from './types';
import { UnitedEntityDefinition } from './united_entity_definition';
const unitedDefinitionBuilders: Record<EntityType, UnitedDefinitionBuilder> = {
host: getHostUnitedDefinition,
user: getUserUnitedDefinition,
};
interface Options {
entityType: EntityType;
namespace: string;
fieldHistoryLength: number;
}
export const getUnitedEntityDefinition = memoize(
({ entityType, namespace, fieldHistoryLength }: Options): UnitedEntityDefinition => {
const unitedDefinition = unitedDefinitionBuilders[entityType](fieldHistoryLength);
unitedDefinition.fields.push(
...getCommonUnitedFieldDefinitions({
entityType,
fieldHistoryLength,
})
);
return new UnitedEntityDefinition({
...unitedDefinition,
namespace,
});
},
({ entityType, namespace, fieldHistoryLength }: Options) =>
`${entityType}-${namespace}-${fieldHistoryLength}`
);
export const getAvailableEntityTypes = (): EntityType[] =>
Object.keys(unitedDefinitionBuilders) as EntityType[];

10
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/index.ts Normal file
View file

@ -0,0 +1,10 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './get_united_definition';
export { UnitedEntityDefinition } from './united_entity_definition';

30
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/types.ts Normal file
View file

@ -0,0 +1,30 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { MappingProperty, MappingTypeMapping } from '@elastic/elasticsearch/lib/api/types';
import type { EntityDefinition } from '@kbn/entities-schema';
import type { EntityType } from '../../../../../common/api/entity_analytics';
import type { FieldRetentionOperator } from '../field_retention_definition';
export type MappingProperties = NonNullable<MappingTypeMapping['properties']>;
type EntityDefinitionMetadataElement = NonNullable<EntityDefinition['metadata']>[number];
export interface UnitedDefinitionField {
field: string;
retention_operator?: FieldRetentionOperator;
mapping?: MappingProperty;
definition?: EntityDefinitionMetadataElement;
}
export interface UnitedEntityDefinitionConfig {
version: string;
entityType: EntityType;
fields: UnitedDefinitionField[];
}
export type UnitedDefinitionBuilder = (fieldHistoryLength: number) => UnitedEntityDefinitionConfig;

114
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/united_entity_definitions/united_entity_definition.ts Normal file
View file

@ -0,0 +1,114 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { entityDefinitionSchema, type EntityDefinition } from '@kbn/entities-schema';
import type { MappingTypeMapping } from '@elastic/elasticsearch/lib/api/types';
import type { EntityType } from '../../../../../common/api/entity_analytics/entity_store/common.gen';
import { getRiskScoreLatestIndex } from '../../../../../common/entity_analytics/risk_engine';
import { getAssetCriticalityIndex } from '../../../../../common/entity_analytics/asset_criticality';
import {
DEFAULT_INTERVAL,
DEFAULT_LOOKBACK_PERIOD,
ENTITY_STORE_DEFAULT_SOURCE_INDICES,
} from '../constants';
import { buildEntityDefinitionId, getIdentityFieldForEntityType } from '../utils';
import type {
FieldRetentionDefinition,
FieldRetentionOperator,
} from '../field_retention_definition';
import type { MappingProperties, UnitedDefinitionField } from './types';
import { BASE_ENTITY_INDEX_MAPPING } from './constants';
export class UnitedEntityDefinition {
version: string;
entityType: EntityType;
fields: UnitedDefinitionField[];
namespace: string;
entityManagerDefinition: EntityDefinition;
fieldRetentionDefinition: FieldRetentionDefinition;
indexMappings: MappingTypeMapping;
constructor(opts: {
version: string;
entityType: EntityType;
fields: UnitedDefinitionField[];
namespace: string;
}) {
this.version = opts.version;
this.entityType = opts.entityType;
this.fields = opts.fields;
this.namespace = opts.namespace;
this.entityManagerDefinition = this.toEntityManagerDefinition();
this.fieldRetentionDefinition = this.toFieldRetentionDefinition();
this.indexMappings = this.toIndexMappings();
}
private toEntityManagerDefinition(): EntityDefinition {
const { entityType, namespace } = this;
const identityField = getIdentityFieldForEntityType(this.entityType);
const metadata = this.fields
.filter((field) => field.definition)
.map((field) => field.definition!); // eslint-disable-line @typescript-eslint/no-non-null-assertion
return entityDefinitionSchema.parse({
id: buildEntityDefinitionId(entityType, namespace),
name: `Security '${entityType}' Entity Store Definition`,
type: entityType,
indexPatterns: [
...ENTITY_STORE_DEFAULT_SOURCE_INDICES,
getAssetCriticalityIndex(namespace),
getRiskScoreLatestIndex(namespace),
],
identityFields: [identityField],
displayNameTemplate: `{{${identityField}}}`,
metadata,
latest: {
timestampField: '@timestamp',
lookbackPeriod: DEFAULT_LOOKBACK_PERIOD,
interval: DEFAULT_INTERVAL,
},
version: this.version,
managed: true,
});
}
private toFieldRetentionDefinition(): FieldRetentionDefinition {
return {
entityType: this.entityType,
matchField: getIdentityFieldForEntityType(this.entityType),
fields: this.fields
.filter((field) => field.retention_operator !== undefined)
.map((field) => field.retention_operator as FieldRetentionOperator),
};
}
private toIndexMappings(): MappingTypeMapping {
const identityField = getIdentityFieldForEntityType(this.entityType);
const initialMappings: MappingProperties = {
...BASE_ENTITY_INDEX_MAPPING,
[identityField]: {
type: 'keyword',
},
};
const properties = this.fields.reduce((acc, { field, mapping }) => {
if (!mapping) {
return acc;
}
acc[field] = mapping;
return acc;
}, initialMappings);
properties[identityField] = {
type: 'keyword',
};
return {
properties,
};
}
}

11
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/utils/utils.ts → x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/utils/entity_utils.ts
View file

@ -11,15 +11,12 @@ import {
entitiesIndexPattern,
} from '@kbn/entities-schema';
import type { EntityType } from '../../../../../common/api/entity_analytics/entity_store/common.gen';
import { buildHostEntityDefinition, buildUserEntityDefinition } from '../definition';
import { entityEngineDescriptorTypeName } from '../saved_object';
export const getEntityDefinition = (entityType: EntityType, space: string) => {
if (entityType === 'host') return buildHostEntityDefinition(space);
if (entityType === 'user') return buildUserEntityDefinition(space);
export const getIdentityFieldForEntityType = (entityType: EntityType) => {
if (entityType === 'host') return 'host.name';
throw new Error(`Unsupported entity type: ${entityType}`);
return 'user.name';
};
export const getByEntityTypeQuery = (entityType: EntityType) => {
@ -34,5 +31,5 @@ export const getEntitiesIndexName = (entityType: EntityType, namespace: string)
});
export const buildEntityDefinitionId = (entityType: EntityType, space: string) => {
return `ea_${space}_${entityType}_entity_store`;
return `security_${entityType}_${space}`;
};

8
x-pack/plugins/security_solution/server/lib/entity_analytics/entity_store/utils/index.ts Normal file
View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './entity_utils';

20
x-pack/plugins/security_solution/server/lib/entity_analytics/risk_score/risk_score_data_client.ts
View file

@ -87,6 +87,17 @@ export class RiskScoreDataClient {
soClient: this.options.soClient,
});
public createRiskScoreLatestIndex = async () => {
await createOrUpdateIndex({
esClient: this.options.esClient,
logger: this.options.logger,
options: {
index: getRiskScoreLatestIndex(this.options.namespace),
mappings: mappingFromFieldMap(riskScoreFieldMap, false),
},
});
};
public async init() {
const namespace = this.options.namespace;
@ -152,14 +163,7 @@ export class RiskScoreDataClient {
indexPatterns,
});
await createOrUpdateIndex({
esClient,
logger: this.options.logger,
options: {
index: getRiskScoreLatestIndex(namespace),
mappings: mappingFromFieldMap(riskScoreFieldMap, false),
},
});
await this.createRiskScoreLatestIndex();
const transformId = getLatestTransformId(namespace);
await createTransform({

9
x-pack/plugins/security_solution/server/plugin.ts
View file

@ -112,6 +112,7 @@ import {
import { ProductFeaturesService } from './lib/product_features_service/product_features_service';
import { registerRiskScoringTask } from './lib/entity_analytics/risk_score/tasks/risk_scoring_task';
import { registerEntityStoreFieldRetentionEnrichTask } from './lib/entity_analytics/entity_store/task';
import { registerProtectionUpdatesNoteRoutes } from './endpoint/routes/protection_updates_note';
import {
latestRiskScoreIndexPattern,
@ -220,6 +221,14 @@ export class Plugin implements ISecuritySolutionPlugin {
logger.error(`Error scheduling entity analytics migration: ${err}`);
});
if (experimentalFeatures.entityStoreEnabled) {
registerEntityStoreFieldRetentionEnrichTask({
getStartServices: core.getStartServices,
logger: this.logger,
taskManager: plugins.taskManager,
});
}
const requestContextFactory = new RequestContextFactory({
config,
logger,

15
x-pack/plugins/security_solution/server/request_context_factory.ts
View file

@ -10,7 +10,6 @@ import { memoize } from 'lodash';
import type { Logger, KibanaRequest, RequestHandlerContext } from '@kbn/core/server';
import type { BuildFlavor } from '@kbn/config';
import { EntityClient } from '@kbn/entityManager-plugin/server/lib/entity_client';
import { DEFAULT_SPACE_ID } from '../common/constants';
import { AppClientFactory } from './client';
import type { ConfigType } from './config';
@ -33,7 +32,6 @@ import { AssetCriticalityDataClient } from './lib/entity_analytics/asset_critica
import { createDetectionRulesClient } from './lib/detection_engine/rule_management/logic/detection_rules_client/detection_rules_client';
import { buildMlAuthz } from './lib/machine_learning/authz';
import { EntityStoreDataClient } from './lib/entity_analytics/entity_store/entity_store_data_client';
import { AssetCriticalityEcsMigrationClient } from './lib/entity_analytics/asset_criticality/asset_criticality_migration_client';
export interface IRequestContextFactory {
create(
@ -202,16 +200,9 @@ export class RequestContextFactory implements IRequestContextFactory {
esClient,
logger,
soClient,
entityClient: new EntityClient({
esClient,
soClient,
logger,
}),
assetCriticalityMigrationClient: new AssetCriticalityEcsMigrationClient({
logger,
auditLogger: getAuditLogger(),
esClient,
}),
taskManager: startPlugins.taskManager,
auditLogger: getAuditLogger(),
kibanaVersion: options.kibanaVersion,
});
}),
};

41
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/engine.ts
View file

@ -6,15 +6,24 @@
*/
import expect from '@kbn/expect';
import { FtrProviderContext } from '../../../../ftr_provider_context';
import { EntityStoreUtils } from '../../utils';
import { EntityStoreUtils, elasticAssetCheckerFactory } from '../../utils';
export default ({ getService }: FtrProviderContext) => {
const api = getService('securitySolutionApi');
const {
expectTransformExists,
expectTransformNotFound,
expectEnrichPolicyExists,
expectEnrichPolicyNotFound,
expectComponentTemplateExists,
expectComponentTemplateNotFound,
expectIngestPipelineExists,
expectIngestPipelineNotFound,
} = elasticAssetCheckerFactory(getService);
const utils = EntityStoreUtils(getService);
describe('@ess @serverless @skipInServerlessMKI Entity Store Engine APIs', () => {
// TODO: unskip once permissions issue is resolved
describe.skip('@ess @serverless @skipInServerlessMKI Entity Store Engine APIs', () => {
before(async () => {
await utils.cleanEngines();
});
@ -27,17 +36,19 @@ export default ({ getService }: FtrProviderContext) => {
it('should have installed the expected user resources', async () => {
await utils.initEntityEngineForEntityType('user');
const expectedTransforms = ['entities-v1-latest-ea_default_user_entity_store'];
await utils.expectTransformsExist(expectedTransforms);
await expectTransformExists('entities-v1-latest-ea_default_user_entity_store');
await expectEnrichPolicyExists('entity_store_field_retention_user_default_v1');
await expectComponentTemplateExists(`ea_default_user_entity_store-latest@platform`);
await expectIngestPipelineExists(`ea_default_user_entity_store-latest@platform`);
});
it('should have installed the expected host resources', async () => {
await utils.initEntityEngineForEntityType('host');
const expectedTransforms = ['entities-v1-latest-ea_default_host_entity_store'];
await utils.expectTransformsExist(expectedTransforms);
await expectTransformExists('entities-v1-latest-ea_default_host_entity_store');
await expectEnrichPolicyExists('entity_store_field_retention_host_default_v1');
await expectComponentTemplateExists(`ea_default_host_entity_store-latest@platform`);
await expectIngestPipelineExists(`ea_default_host_entity_store-latest@platform`);
});
});
@ -167,7 +178,10 @@ export default ({ getService }: FtrProviderContext) => {
})
.expect(200);
await utils.expectTransformNotFound('entities-v1-latest-ea_host_entity_store');
await expectTransformNotFound('entities-v1-latest-ea_default_host_entity_store');
await expectEnrichPolicyNotFound('entity_store_field_retention_host_default_v1');
await expectComponentTemplateNotFound(`ea_default_host_entity_store-latest@platform`);
await expectIngestPipelineNotFound(`ea_default_host_entity_store-latest@platform`);
});
it('should delete the user entity engine', async () => {
@ -180,7 +194,10 @@ export default ({ getService }: FtrProviderContext) => {
})
.expect(200);
await utils.expectTransformNotFound('entities-v1-latest-ea_user_entity_store');
await expectTransformNotFound('entities-v1-latest-ea_default_user_entity_store');
await expectEnrichPolicyNotFound('entity_store_field_retention_user_default_v1');
await expectComponentTemplateNotFound(`ea_default_user_entity_store-latest@platform`);
await expectIngestPipelineNotFound(`ea_default_user_entity_store-latest@platform`);
});
});
});

3
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/engine_nondefault_spaces.ts
View file

@ -16,7 +16,8 @@ export default ({ getService }: FtrProviderContextWithSpaces) => {
const utils = EntityStoreUtils(getService, namespace);
describe('@ess Entity Store Engine APIs in non-default space', () => {
// TODO: unskip once kibana system user has entity index privileges
describe.skip('@ess Entity Store Engine APIs in non-default space', () => {
before(async () => {
await utils.cleanEngines();
await spaces.create({

4
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/entities_list.ts
View file

@ -10,8 +10,8 @@ import { FtrProviderContext } from '../../../../ftr_provider_context';
export default ({ getService }: FtrProviderContext) => {
const securitySolutionApi = getService('securitySolutionApi');
describe('@ess @serverless @skipInServerlessMKI Entity store - Entities list API', () => {
// TODO: unskip once permissions issue is resolved
describe.skip('@ess @serverless @skipInServerlessMKI Entity store - Entities list API', () => {
describe('when the entity store is disable', () => {
it("should return response with success status when the index doesn't exist", async () => {
const { body } = await securitySolutionApi.listEntities({

378
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/field_retention_operators.ts Normal file
View file

@ -0,0 +1,378 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import expect from '@kbn/expect';
import {
FieldRetentionOperator,
fieldOperatorToIngestProcessor,
} from '@kbn/security-solution-plugin/server/lib/entity_analytics/entity_store/field_retention_definition';
import { FtrProviderContext } from '../../../../ftr_provider_context';
export default ({ getService }: FtrProviderContext) => {
const es = getService('es');
const log = getService('log');
const expectArraysMatchAnyOrder = (a: any[], b: any[]) => {
const aSorted = a.sort();
const bSorted = b.sort();
expect(aSorted).to.eql(bSorted);
};
const applyOperatorToDoc = async (
operator: FieldRetentionOperator,
docSource: any
): Promise<any> => {
const step = fieldOperatorToIngestProcessor(operator, { enrichField: 'historical' });
const doc = {
_index: 'index',
_id: 'id',
_source: docSource,
};
const res = await es.ingest.simulate({
pipeline: {
description: 'test',
processors: [step],
},
docs: [doc],
});
const firstDoc = res.docs?.[0];
// @ts-expect-error error is not in the types
const error = firstDoc?.error;
if (error) {
log.error('Full painless error below: ');
log.error(JSON.stringify(error, null, 2));
throw new Error('Painless error running pipelie see logs for full detail : ' + error?.type);
}
return firstDoc?.doc?._source;
};
describe('@ess @serverless @skipInServerlessMKI Entity store - Field Retention Pipeline Steps', () => {
describe('collect_values operator', () => {
it('should return value if no history', async () => {
const op: FieldRetentionOperator = {
operation: 'collect_values',
field: 'test_field',
maxLength: 10,
};
const doc = {
test_field: ['foo'],
};
const resultDoc = await applyOperatorToDoc(op, doc);
expectArraysMatchAnyOrder(resultDoc.test_field, ['foo']);
});
it('should not take from history if latest field has maxLength values', async () => {
const op: FieldRetentionOperator = {
operation: 'collect_values',
field: 'test_field',
maxLength: 1,
};
const doc = {
test_field: ['foo'],
historical: {
test_field: ['bar', 'baz'],
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expectArraysMatchAnyOrder(resultDoc.test_field, ['foo']);
});
it('should take from history if latest field doesnt have maxLength values', async () => {
const op: FieldRetentionOperator = {
operation: 'collect_values',
field: 'test_field',
maxLength: 10,
};
const doc = {
test_field: ['foo'],
historical: {
test_field: ['bar', 'baz'],
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expectArraysMatchAnyOrder(resultDoc.test_field, ['foo', 'bar', 'baz']);
});
it('should only take from history up to maxLength values', async () => {
const op: FieldRetentionOperator = {
operation: 'collect_values',
field: 'test_field',
maxLength: 2,
};
const doc = {
test_field: ['foo'],
historical: {
test_field: ['bar', 'baz'],
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expectArraysMatchAnyOrder(resultDoc.test_field, ['foo', 'bar']);
});
it('should handle value not being an array', async () => {
const op: FieldRetentionOperator = {
operation: 'collect_values',
field: 'test_field',
maxLength: 2,
};
const doc = {
test_field: 'foo',
historical: {
test_field: ['bar'],
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expectArraysMatchAnyOrder(resultDoc.test_field, ['foo', 'bar']);
});
it('should handle missing values', async () => {
const op: FieldRetentionOperator = {
operation: 'collect_values',
field: 'test_field',
maxLength: 2,
};
const doc = {};
const resultDoc = await applyOperatorToDoc(op, doc);
expectArraysMatchAnyOrder(resultDoc.test_field, []);
});
});
describe('prefer_newest_value operator', () => {
it('should return latest value if no history value', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('latest');
});
it('should return history value if no latest value (undefined)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('historical');
});
it('should return history value if no latest value (empty string)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {
test_field: '',
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('historical');
});
it('should return history value if no latest value (empty array)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {
test_field: [],
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('historical');
});
it('should return history value if no latest value (empty object)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {
test_field: {},
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('historical');
});
it('should return latest value if both latest and history values', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('latest');
});
it('should handle missing values', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_newest_value',
field: 'test_field',
};
const doc = {};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql(undefined);
});
});
describe('prefer_oldest_value operator', () => {
it('should return history value if no latest value', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_oldest_value',
field: 'test_field',
};
const doc = {
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('historical');
});
it('should return latest value if no history value (undefined)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_oldest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
historical: {
test_field: undefined,
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('latest');
});
it('should return latest value if no history value (empty string)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_oldest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
historical: {
test_field: '',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('latest');
});
it('should return latest value if no history value (empty array)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_oldest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
historical: {
test_field: [],
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('latest');
});
it('should return latest value if no history value (empty object)', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_oldest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
historical: {
test_field: {},
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('latest');
});
it('should return history value if both latest and history values', async () => {
const op: FieldRetentionOperator = {
operation: 'prefer_oldest_value',
field: 'test_field',
};
const doc = {
test_field: 'latest',
historical: {
test_field: 'historical',
},
};
const resultDoc = await applyOperatorToDoc(op, doc);
expect(resultDoc.test_field).to.eql('historical');
});
});
});
};

1
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/index.ts
View file

@ -11,6 +11,7 @@ export default function ({ loadTestFile }: FtrProviderContext) {
describe('Entity Analytics - Entity Store', function () {
loadTestFile(require.resolve('./entities_list'));
loadTestFile(require.resolve('./engine'));
loadTestFile(require.resolve('./field_retention_operators'));
loadTestFile(require.resolve('./engine_nondefault_spaces'));
});
}

118
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/utils/elastic_asset_checker.ts Normal file
View file

@ -0,0 +1,118 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FtrProviderContext } from '@kbn/ftr-common-functional-services';
export const elasticAssetCheckerFactory = (getService: FtrProviderContext['getService']) => {
const es = getService('es');
const expectTransformExists = async (transformId: string) => {
return expectTransformStatus(transformId, true);
};
const expectTransformNotFound = async (transformId: string, attempts: number = 5) => {
return expectTransformStatus(transformId, false);
};
const expectTransformStatus = async (
transformId: string,
exists: boolean,
attempts: number = 5,
delayMs: number = 2000
) => {
let currentAttempt = 1;
while (currentAttempt <= attempts) {
try {
await es.transform.getTransform({ transform_id: transformId });
if (!exists) {
throw new Error(`Expected transform ${transformId} to not exist, but it does`);
}
return; // Transform exists, exit the loop
} catch (e) {
if (currentAttempt === attempts) {
if (exists) {
throw new Error(`Expected transform ${transformId} to exist, but it does not: ${e}`);
} else {
return; // Transform does not exist, exit the loop
}
}
await new Promise((resolve) => setTimeout(resolve, delayMs));
currentAttempt++;
}
}
};
const expectEnrichPolicyStatus = async (policyId: string, exists: boolean) => {
try {
await es.enrich.getPolicy({ name: policyId });
if (!exists) {
throw new Error(`Expected enrich policy ${policyId} to not exist, but it does`);
}
} catch (e) {
if (exists) {
throw new Error(`Expected enrich policy ${policyId} to exist, but it does not: ${e}`);
}
}
};
const expectEnrichPolicyExists = async (policyId: string) =>
expectEnrichPolicyStatus(policyId, true);
const expectEnrichPolicyNotFound = async (policyId: string, attempts: number = 5) =>
expectEnrichPolicyStatus(policyId, false);
const expectComponentTemplatStatus = async (templateName: string, exists: boolean) => {
try {
await es.cluster.getComponentTemplate({ name: templateName });
if (!exists) {
throw new Error(`Expected component template ${templateName} to not exist, but it does`);
}
} catch (e) {
if (exists) {
throw new Error(
`Expected component template ${templateName} to exist, but it does not: ${e}`
);
}
}
};
const expectComponentTemplateExists = async (templateName: string) =>
expectComponentTemplatStatus(templateName, true);
const expectComponentTemplateNotFound = async (templateName: string) =>
expectComponentTemplatStatus(templateName, false);
const expectIngestPipelineStatus = async (pipelineId: string, exists: boolean) => {
try {
await es.ingest.getPipeline({ id: pipelineId });
if (!exists) {
throw new Error(`Expected ingest pipeline ${pipelineId} to not exist, but it does`);
}
} catch (e) {
if (exists) {
throw new Error(`Expected ingest pipeline ${pipelineId} to exist, but it does not: ${e}`);
}
}
};
const expectIngestPipelineExists = async (pipelineId: string) =>
expectIngestPipelineStatus(pipelineId, true);
const expectIngestPipelineNotFound = async (pipelineId: string) =>
expectIngestPipelineStatus(pipelineId, false);
return {
expectComponentTemplateExists,
expectComponentTemplateNotFound,
expectEnrichPolicyExists,
expectEnrichPolicyNotFound,
expectIngestPipelineExists,
expectIngestPipelineNotFound,
expectTransformExists,
expectTransformNotFound,
};
};

1
x-pack/test/security_solution_api_integration/test_suites/entity_analytics/utils/index.ts
View file

@ -9,3 +9,4 @@ export * from './risk_engine';
export * from './get_risk_engine_stats';
export * from './asset_criticality';
export * from './entity_store';
export * from './elastic_asset_checker';