github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 17:59:23 -04:00
Code Issues Projects Releases Packages Wiki Activity
kibana/oas_docs
History
Jared Burgett 49946993ec
[8.x] [Entity Store] [Asset Inventory] Universal entity definition (#202888) (#205536) 
# Backport

This will backport the following commits from `main` to `8.x`:
- [[Entity Store] [Asset Inventory] Universal entity definition
(#202888)](https://github.com/elastic/kibana/pull/202888)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Tiago Vila
Verde","email":"tiago.vilaverde@elastic.co"},"sourceCommit":{"committedDate":"2025-01-03T16:43:16Z","message":"[Entity
Store] [Asset Inventory] Universal entity definition (#202888)\n\n##
Summary\r\n\r\nThis PR adds a universal entity definition.\r\nA
universal entity uses `related.entity` as an identifier field
and\r\nincludes an extra processor step that parses the
field\r\n`entities.keyword` and extracts all the entities in said field
(whose\r\noriginal data comes from `related.entities`).\r\n\r\nSee
this\r\n[doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7)\r\nfor
more details.\r\n\r\nTo accomplish this, we need to allow describing an
entity along with\r\nextra entity store resources required for that
entity's engine.\r\nThis PR reworks the current entity store by
introducing an `Entity\r\nDescription`, which has all that required
information. From it, we can\r\nbuild an `EntityEngineDescription` which
adds all the needed data that\r\nmust be computed (as opposed to
hardcoded) and is then used to generate\r\nall the resources needed for
that Entity's engine (entity definition,\r\npipeline, enrich policy,
index mappings, etc).\r\n\r\n<img width=\"3776\"
alt=\"EntityDescriptions\"\r\nsrc=\"https://github.com/user-attachments/assets/bdf7915f-1981-47e6-a815-31163f24ad03\">\r\n\r\nThis
required a refactoring of the current `UnitedEntityDefinition`,\r\nwhich
has now been removed in favour of more contextual functions for\r\nall
the different parts.\r\nThe intention is to decouple the Entity
Description schema from the\r\nschemas required for field retention,
entity manager and pipeline. We\r\ncan then freely expand on our Entity
Description as required, and simply\r\nalter the conversion functions
when needed.\r\n\r\n## How to test\r\n\r\n1. On a fresh ES cluster, add
some entity data\r\n* For hosts and user, use the [security
documents\r\ngenerator](https://github.com/elastic/security-documents-generator)\r\n
* For universal, there are a few more steps:\r\n 1. Create the
`entity.keyword` builder pipeline\r\n 2. Add it to a index template\r\n
3. Post some docs to the corresponding index \r\n2. Initialise the
universal entity engine via:
`POST\r\nkbn:/api/entity_store/engines/universal/init {}`\r\n* Note that
using the UI does not work, as we've specifically removed\r\nthe
Universal engine from the normal Entity Store workflow\r\n3. Check the
status of the store is `running` via
`GET\r\nkbn:/api/entity_store/status`\r\n4. Once the transform runs, you
can query `GET entities*/_search` to see\r\nthe created
entities\r\n\r\nNote that universal entities do not show up in the
dashboard Entities\r\nList.\r\n\r\n\r\n### Code to ingest
data\r\n<details>\r\n<summary>Pipeline</summary>\r\n\r\n```js\r\nPUT
_ingest/pipeline/entities-keyword-builder\r\n{\r\n
\"description\":\"Serialize entities.metadata into a keyword
field\",\r\n \"processors\":[\r\n {\r\n \"script\":{\r\n
\"lang\":\"painless\",\r\n \"source\":\"\"\"\r\nString jsonFromMap(Map
map) {\r\n StringBuilder json = new StringBuilder(\"{\");\r\n boolean
first = true;\r\n\r\n for (entry in map.entrySet()) {\r\n if (!first)
{\r\n json.append(\",\");\r\n }\r\n first = false;\r\n\r\n String key =
entry.getKey().replace(\"\\\"\", \"\\\\\\\"\");\r\n Object value =
entry.getValue();\r\n\r\n
json.append(\"\\\"\").append(key).append(\"\\\":\");\r\n\r\n if (value
instanceof String) {\r\n String escapedValue = ((String)
value).replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\", \":\");\r\n
json.append(\"\\\"\").append(escapedValue).append(\"\\\"\");\r\n } else
if (value instanceof Map) {\r\n json.append(jsonFromMap((Map)
value));\r\n } else if (value instanceof List) {\r\n
json.append(jsonFromList((List) value));\r\n } else if (value instanceof
Boolean || value instanceof Number) {\r\n
json.append(value.toString());\r\n } else {\r\n // For other types,
treat as string\r\n String escapedValue =
value.toString().replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\",
\":\");\r\n
json.append(\"\\\"\").append(escapedValue).append(\"\\\"\");\r\n }\r\n
}\r\n\r\n json.append(\"}\");\r\n return
json.toString();\r\n}\r\n\r\nString jsonFromList(List list) {\r\n\r\n
StringBuilder json = new StringBuilder(\"[\");\r\n boolean first =
true;\r\n\r\n for (item in list) {\r\n if (!first) {\r\n
json.append(\",\");\r\n }\r\n first = false;\r\n\r\n if (item instanceof
String) {\r\n String escapedItem = ((String) item).replace(\"\\\"\",
\"\\\\\\\"\").replace(\"=\", \":\");\r\n
json.append(\"\\\"\").append(escapedItem).append(\"\\\"\");\r\n } else
if (item instanceof Map) {\r\n json.append(jsonFromMap((Map) item));\r\n
} else if (item instanceof List) {\r\n json.append(jsonFromList((List)
item));\r\n } else if (item instanceof Boolean || item instanceof
Number) {\r\n json.append(item.toString());\r\n } else {\r\n // For
other types, treat as string\r\n String escapedItem =
item.toString().replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\",
\":\");\r\n
json.append(\"\\\"\").append(escapedItem).append(\"\\\"\");\r\n }\r\n
}\r\n\r\n json.append(\"]\");\r\n return
json.toString();\r\n}\r\n\r\ndef metadata =
jsonFromMap(ctx['entities']['metadata']);\r\nctx['entities']['keyword']
= metadata;\r\n\"\"\"\r\n\r\n }\r\n }\r\n
]\r\n}\r\n```\r\n</details>\r\n\r\n<details>\r\n<summary>Index
template</summary>\r\n\r\n```js\r\nPUT
/_index_template/entity_store_index_template\r\n{\r\n
\"index_patterns\":[\r\n \"logs-store\"\r\n ],\r\n \"template\":{\r\n
\"settings\":{\r\n \"index\":{\r\n
\"default_pipeline\":\"entities-keyword-builder\"\r\n }\r\n },\r\n
\"mappings\":{\r\n \"properties\":{\r\n \"@timestamp\":{\r\n
\"type\":\"date\"\r\n },\r\n \"message\":{\r\n \"type\":\"text\"\r\n
},\r\n \"event\":{\r\n \"properties\":{\r\n \"action\":{\r\n
\"type\":\"keyword\"\r\n },\r\n \"category\":{\r\n
\"type\":\"keyword\"\r\n },\r\n \"type\":{\r\n \"type\":\"keyword\"\r\n
},\r\n \"outcome\":{\r\n \"type\":\"keyword\"\r\n },\r\n
\"provider\":{\r\n \"type\":\"keyword\"\r\n },\r\n \"ingested\":{\r\n
\"type\": \"date\"\r\n }\r\n }\r\n },\r\n \"related\":{\r\n
\"properties\":{\r\n \"entity\":{\r\n \"type\":\"keyword\"\r\n }\r\n
}\r\n },\r\n \"entities\":{\r\n \"properties\":{\r\n \"metadata\":{\r\n
\"type\":\"flattened\"\r\n },\r\n \"keyword\":{\r\n
\"type\":\"keyword\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n
}\r\n}\r\n```\r\n</details>\r\n\r\n<details>\r\n<summary>Example source
doc</summary>\r\n\r\n```js\r\nPOST /logs-store/_doc/\r\n{\r\n
\"@timestamp\":\"2024-11-29T10:01:00Z\",\r\n \"message\":\"Eddie\",\r\n
\"event\": {\r\n \"type\":[\r\n \"creation\"\r\n ],\r\n \"ingested\":
\"2024-12-03T10:01:00Z\"\r\n },\r\n \"related\":{\r\n \"entity\":[\r\n
\"AKIAI44QH8DHBEXAMPLE\"\r\n ]\r\n },\r\n \"entities\":{\r\n
\"metadata\":{\r\n \"AKIAI44QH8DHBEXAMPLE\":{\r\n \"entity\":{\r\n
\"id\":\"AKIAI44QH8DHBEXAMPLE\",\r\n \"category\":\"Access
Management\",\r\n \"type\":\"AWS IAM Access Key\"\r\n },\r\n
\"cloud\":{\r\n \"account\":{\r\n \"id\":\"444455556666\"\r\n }\r\n
}\r\n }\r\n }\r\n }\r\n}\r\n```\r\n</details>\r\n\r\n### To do\r\n\r\n-
[x] Add/Improve [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\n-
[x] Feature flag\r\n\r\n\r\n----\r\n#### Update:\r\n\r\nAdded
`assetInventoryStoreEnabled` Feature Flag. It is disabled by\r\ndefault
and even when enabled, the `/api/entity_store/enable` route does\r\nnot
initialize the Universal Entity
Engine.\r\n`/api/entity_store/engines/universal/init` needs to be
manually called\r\nto initialize
it\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Rômulo Farias <romulodefarias@gmail.com>\r\nCo-authored-by:
jaredburgettelastic <jared.burgett@elastic.co>\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"c6b0a31d8ec6d423a8071f50a22e55acedd0dee0","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Cloud
Security","ci:cloud-deploy","Theme: entity_analytics","Team:Entity
Analytics","backport:version","v8.18.0"],"number":202888,"url":"https://github.com/elastic/kibana/pull/202888","mergeCommit":{"message":"[Entity
Store] [Asset Inventory] Universal entity definition (#202888)\n\n##
Summary\r\n\r\nThis PR adds a universal entity definition.\r\nA
universal entity uses `related.entity` as an identifier field
and\r\nincludes an extra processor step that parses the
field\r\n`entities.keyword` and extracts all the entities in said field
(whose\r\noriginal data comes from `related.entities`).\r\n\r\nSee
this\r\n[doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7)\r\nfor
more details.\r\n\r\nTo accomplish this, we need to allow describing an
entity along with\r\nextra entity store resources required for that
entity's engine.\r\nThis PR reworks the current entity store by
introducing an `Entity\r\nDescription`, which has all that required
information. From it, we can\r\nbuild an `EntityEngineDescription` which
adds all the needed data that\r\nmust be computed (as opposed to
hardcoded) and is then used to generate\r\nall the resources needed for
that Entity's engine (entity definition,\r\npipeline, enrich policy,
index mappings, etc).\r\n\r\n<img width=\"3776\"
alt=\"EntityDescriptions\"\r\nsrc=\"https://github.com/user-attachments/assets/bdf7915f-1981-47e6-a815-31163f24ad03\">\r\n\r\nThis
required a refactoring of the current `UnitedEntityDefinition`,\r\nwhich
has now been removed in favour of more contextual functions for\r\nall
the different parts.\r\nThe intention is to decouple the Entity
Description schema from the\r\nschemas required for field retention,
entity manager and pipeline. We\r\ncan then freely expand on our Entity
Description as required, and simply\r\nalter the conversion functions
when needed.\r\n\r\n## How to test\r\n\r\n1. On a fresh ES cluster, add
some entity data\r\n* For hosts and user, use the [security
documents\r\ngenerator](https://github.com/elastic/security-documents-generator)\r\n
* For universal, there are a few more steps:\r\n 1. Create the
`entity.keyword` builder pipeline\r\n 2. Add it to a index template\r\n
3. Post some docs to the corresponding index \r\n2. Initialise the
universal entity engine via:
`POST\r\nkbn:/api/entity_store/engines/universal/init {}`\r\n* Note that
using the UI does not work, as we've specifically removed\r\nthe
Universal engine from the normal Entity Store workflow\r\n3. Check the
status of the store is `running` via
`GET\r\nkbn:/api/entity_store/status`\r\n4. Once the transform runs, you
can query `GET entities*/_search` to see\r\nthe created
entities\r\n\r\nNote that universal entities do not show up in the
dashboard Entities\r\nList.\r\n\r\n\r\n### Code to ingest
data\r\n<details>\r\n<summary>Pipeline</summary>\r\n\r\n```js\r\nPUT
_ingest/pipeline/entities-keyword-builder\r\n{\r\n
\"description\":\"Serialize entities.metadata into a keyword
field\",\r\n \"processors\":[\r\n {\r\n \"script\":{\r\n
\"lang\":\"painless\",\r\n \"source\":\"\"\"\r\nString jsonFromMap(Map
map) {\r\n StringBuilder json = new StringBuilder(\"{\");\r\n boolean
first = true;\r\n\r\n for (entry in map.entrySet()) {\r\n if (!first)
{\r\n json.append(\",\");\r\n }\r\n first = false;\r\n\r\n String key =
entry.getKey().replace(\"\\\"\", \"\\\\\\\"\");\r\n Object value =
entry.getValue();\r\n\r\n
json.append(\"\\\"\").append(key).append(\"\\\":\");\r\n\r\n if (value
instanceof String) {\r\n String escapedValue = ((String)
value).replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\", \":\");\r\n
json.append(\"\\\"\").append(escapedValue).append(\"\\\"\");\r\n } else
if (value instanceof Map) {\r\n json.append(jsonFromMap((Map)
value));\r\n } else if (value instanceof List) {\r\n
json.append(jsonFromList((List) value));\r\n } else if (value instanceof
Boolean || value instanceof Number) {\r\n
json.append(value.toString());\r\n } else {\r\n // For other types,
treat as string\r\n String escapedValue =
value.toString().replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\",
\":\");\r\n
json.append(\"\\\"\").append(escapedValue).append(\"\\\"\");\r\n }\r\n
}\r\n\r\n json.append(\"}\");\r\n return
json.toString();\r\n}\r\n\r\nString jsonFromList(List list) {\r\n\r\n
StringBuilder json = new StringBuilder(\"[\");\r\n boolean first =
true;\r\n\r\n for (item in list) {\r\n if (!first) {\r\n
json.append(\",\");\r\n }\r\n first = false;\r\n\r\n if (item instanceof
String) {\r\n String escapedItem = ((String) item).replace(\"\\\"\",
\"\\\\\\\"\").replace(\"=\", \":\");\r\n
json.append(\"\\\"\").append(escapedItem).append(\"\\\"\");\r\n } else
if (item instanceof Map) {\r\n json.append(jsonFromMap((Map) item));\r\n
} else if (item instanceof List) {\r\n json.append(jsonFromList((List)
item));\r\n } else if (item instanceof Boolean || item instanceof
Number) {\r\n json.append(item.toString());\r\n } else {\r\n // For
other types, treat as string\r\n String escapedItem =
item.toString().replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\",
\":\");\r\n
json.append(\"\\\"\").append(escapedItem).append(\"\\\"\");\r\n }\r\n
}\r\n\r\n json.append(\"]\");\r\n return
json.toString();\r\n}\r\n\r\ndef metadata =
jsonFromMap(ctx['entities']['metadata']);\r\nctx['entities']['keyword']
= metadata;\r\n\"\"\"\r\n\r\n }\r\n }\r\n
]\r\n}\r\n```\r\n</details>\r\n\r\n<details>\r\n<summary>Index
template</summary>\r\n\r\n```js\r\nPUT
/_index_template/entity_store_index_template\r\n{\r\n
\"index_patterns\":[\r\n \"logs-store\"\r\n ],\r\n \"template\":{\r\n
\"settings\":{\r\n \"index\":{\r\n
\"default_pipeline\":\"entities-keyword-builder\"\r\n }\r\n },\r\n
\"mappings\":{\r\n \"properties\":{\r\n \"@timestamp\":{\r\n
\"type\":\"date\"\r\n },\r\n \"message\":{\r\n \"type\":\"text\"\r\n
},\r\n \"event\":{\r\n \"properties\":{\r\n \"action\":{\r\n
\"type\":\"keyword\"\r\n },\r\n \"category\":{\r\n
\"type\":\"keyword\"\r\n },\r\n \"type\":{\r\n \"type\":\"keyword\"\r\n
},\r\n \"outcome\":{\r\n \"type\":\"keyword\"\r\n },\r\n
\"provider\":{\r\n \"type\":\"keyword\"\r\n },\r\n \"ingested\":{\r\n
\"type\": \"date\"\r\n }\r\n }\r\n },\r\n \"related\":{\r\n
\"properties\":{\r\n \"entity\":{\r\n \"type\":\"keyword\"\r\n }\r\n
}\r\n },\r\n \"entities\":{\r\n \"properties\":{\r\n \"metadata\":{\r\n
\"type\":\"flattened\"\r\n },\r\n \"keyword\":{\r\n
\"type\":\"keyword\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n
}\r\n}\r\n```\r\n</details>\r\n\r\n<details>\r\n<summary>Example source
doc</summary>\r\n\r\n```js\r\nPOST /logs-store/_doc/\r\n{\r\n
\"@timestamp\":\"2024-11-29T10:01:00Z\",\r\n \"message\":\"Eddie\",\r\n
\"event\": {\r\n \"type\":[\r\n \"creation\"\r\n ],\r\n \"ingested\":
\"2024-12-03T10:01:00Z\"\r\n },\r\n \"related\":{\r\n \"entity\":[\r\n
\"AKIAI44QH8DHBEXAMPLE\"\r\n ]\r\n },\r\n \"entities\":{\r\n
\"metadata\":{\r\n \"AKIAI44QH8DHBEXAMPLE\":{\r\n \"entity\":{\r\n
\"id\":\"AKIAI44QH8DHBEXAMPLE\",\r\n \"category\":\"Access
Management\",\r\n \"type\":\"AWS IAM Access Key\"\r\n },\r\n
\"cloud\":{\r\n \"account\":{\r\n \"id\":\"444455556666\"\r\n }\r\n
}\r\n }\r\n }\r\n }\r\n}\r\n```\r\n</details>\r\n\r\n### To do\r\n\r\n-
[x] Add/Improve [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\n-
[x] Feature flag\r\n\r\n\r\n----\r\n#### Update:\r\n\r\nAdded
`assetInventoryStoreEnabled` Feature Flag. It is disabled by\r\ndefault
and even when enabled, the `/api/entity_store/enable` route does\r\nnot
initialize the Universal Entity
Engine.\r\n`/api/entity_store/engines/universal/init` needs to be
manually called\r\nto initialize
it\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Rômulo Farias <romulodefarias@gmail.com>\r\nCo-authored-by:
jaredburgettelastic <jared.burgett@elastic.co>\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"c6b0a31d8ec6d423a8071f50a22e55acedd0dee0"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/202888","number":202888,"mergeCommit":{"message":"[Entity
Store] [Asset Inventory] Universal entity definition (#202888)\n\n##
Summary\r\n\r\nThis PR adds a universal entity definition.\r\nA
universal entity uses `related.entity` as an identifier field
and\r\nincludes an extra processor step that parses the
field\r\n`entities.keyword` and extracts all the entities in said field
(whose\r\noriginal data comes from `related.entities`).\r\n\r\nSee
this\r\n[doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7)\r\nfor
more details.\r\n\r\nTo accomplish this, we need to allow describing an
entity along with\r\nextra entity store resources required for that
entity's engine.\r\nThis PR reworks the current entity store by
introducing an `Entity\r\nDescription`, which has all that required
information. From it, we can\r\nbuild an `EntityEngineDescription` which
adds all the needed data that\r\nmust be computed (as opposed to
hardcoded) and is then used to generate\r\nall the resources needed for
that Entity's engine (entity definition,\r\npipeline, enrich policy,
index mappings, etc).\r\n\r\n<img width=\"3776\"
alt=\"EntityDescriptions\"\r\nsrc=\"https://github.com/user-attachments/assets/bdf7915f-1981-47e6-a815-31163f24ad03\">\r\n\r\nThis
required a refactoring of the current `UnitedEntityDefinition`,\r\nwhich
has now been removed in favour of more contextual functions for\r\nall
the different parts.\r\nThe intention is to decouple the Entity
Description schema from the\r\nschemas required for field retention,
entity manager and pipeline. We\r\ncan then freely expand on our Entity
Description as required, and simply\r\nalter the conversion functions
when needed.\r\n\r\n## How to test\r\n\r\n1. On a fresh ES cluster, add
some entity data\r\n* For hosts and user, use the [security
documents\r\ngenerator](https://github.com/elastic/security-documents-generator)\r\n
* For universal, there are a few more steps:\r\n 1. Create the
`entity.keyword` builder pipeline\r\n 2. Add it to a index template\r\n
3. Post some docs to the corresponding index \r\n2. Initialise the
universal entity engine via:
`POST\r\nkbn:/api/entity_store/engines/universal/init {}`\r\n* Note that
using the UI does not work, as we've specifically removed\r\nthe
Universal engine from the normal Entity Store workflow\r\n3. Check the
status of the store is `running` via
`GET\r\nkbn:/api/entity_store/status`\r\n4. Once the transform runs, you
can query `GET entities*/_search` to see\r\nthe created
entities\r\n\r\nNote that universal entities do not show up in the
dashboard Entities\r\nList.\r\n\r\n\r\n### Code to ingest
data\r\n<details>\r\n<summary>Pipeline</summary>\r\n\r\n```js\r\nPUT
_ingest/pipeline/entities-keyword-builder\r\n{\r\n
\"description\":\"Serialize entities.metadata into a keyword
field\",\r\n \"processors\":[\r\n {\r\n \"script\":{\r\n
\"lang\":\"painless\",\r\n \"source\":\"\"\"\r\nString jsonFromMap(Map
map) {\r\n StringBuilder json = new StringBuilder(\"{\");\r\n boolean
first = true;\r\n\r\n for (entry in map.entrySet()) {\r\n if (!first)
{\r\n json.append(\",\");\r\n }\r\n first = false;\r\n\r\n String key =
entry.getKey().replace(\"\\\"\", \"\\\\\\\"\");\r\n Object value =
entry.getValue();\r\n\r\n
json.append(\"\\\"\").append(key).append(\"\\\":\");\r\n\r\n if (value
instanceof String) {\r\n String escapedValue = ((String)
value).replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\", \":\");\r\n
json.append(\"\\\"\").append(escapedValue).append(\"\\\"\");\r\n } else
if (value instanceof Map) {\r\n json.append(jsonFromMap((Map)
value));\r\n } else if (value instanceof List) {\r\n
json.append(jsonFromList((List) value));\r\n } else if (value instanceof
Boolean || value instanceof Number) {\r\n
json.append(value.toString());\r\n } else {\r\n // For other types,
treat as string\r\n String escapedValue =
value.toString().replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\",
\":\");\r\n
json.append(\"\\\"\").append(escapedValue).append(\"\\\"\");\r\n }\r\n
}\r\n\r\n json.append(\"}\");\r\n return
json.toString();\r\n}\r\n\r\nString jsonFromList(List list) {\r\n\r\n
StringBuilder json = new StringBuilder(\"[\");\r\n boolean first =
true;\r\n\r\n for (item in list) {\r\n if (!first) {\r\n
json.append(\",\");\r\n }\r\n first = false;\r\n\r\n if (item instanceof
String) {\r\n String escapedItem = ((String) item).replace(\"\\\"\",
\"\\\\\\\"\").replace(\"=\", \":\");\r\n
json.append(\"\\\"\").append(escapedItem).append(\"\\\"\");\r\n } else
if (item instanceof Map) {\r\n json.append(jsonFromMap((Map) item));\r\n
} else if (item instanceof List) {\r\n json.append(jsonFromList((List)
item));\r\n } else if (item instanceof Boolean || item instanceof
Number) {\r\n json.append(item.toString());\r\n } else {\r\n // For
other types, treat as string\r\n String escapedItem =
item.toString().replace(\"\\\"\", \"\\\\\\\"\").replace(\"=\",
\":\");\r\n
json.append(\"\\\"\").append(escapedItem).append(\"\\\"\");\r\n }\r\n
}\r\n\r\n json.append(\"]\");\r\n return
json.toString();\r\n}\r\n\r\ndef metadata =
jsonFromMap(ctx['entities']['metadata']);\r\nctx['entities']['keyword']
= metadata;\r\n\"\"\"\r\n\r\n }\r\n }\r\n
]\r\n}\r\n```\r\n</details>\r\n\r\n<details>\r\n<summary>Index
template</summary>\r\n\r\n```js\r\nPUT
/_index_template/entity_store_index_template\r\n{\r\n
\"index_patterns\":[\r\n \"logs-store\"\r\n ],\r\n \"template\":{\r\n
\"settings\":{\r\n \"index\":{\r\n
\"default_pipeline\":\"entities-keyword-builder\"\r\n }\r\n },\r\n
\"mappings\":{\r\n \"properties\":{\r\n \"@timestamp\":{\r\n
\"type\":\"date\"\r\n },\r\n \"message\":{\r\n \"type\":\"text\"\r\n
},\r\n \"event\":{\r\n \"properties\":{\r\n \"action\":{\r\n
\"type\":\"keyword\"\r\n },\r\n \"category\":{\r\n
\"type\":\"keyword\"\r\n },\r\n \"type\":{\r\n \"type\":\"keyword\"\r\n
},\r\n \"outcome\":{\r\n \"type\":\"keyword\"\r\n },\r\n
\"provider\":{\r\n \"type\":\"keyword\"\r\n },\r\n \"ingested\":{\r\n
\"type\": \"date\"\r\n }\r\n }\r\n },\r\n \"related\":{\r\n
\"properties\":{\r\n \"entity\":{\r\n \"type\":\"keyword\"\r\n }\r\n
}\r\n },\r\n \"entities\":{\r\n \"properties\":{\r\n \"metadata\":{\r\n
\"type\":\"flattened\"\r\n },\r\n \"keyword\":{\r\n
\"type\":\"keyword\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n
}\r\n}\r\n```\r\n</details>\r\n\r\n<details>\r\n<summary>Example source
doc</summary>\r\n\r\n```js\r\nPOST /logs-store/_doc/\r\n{\r\n
\"@timestamp\":\"2024-11-29T10:01:00Z\",\r\n \"message\":\"Eddie\",\r\n
\"event\": {\r\n \"type\":[\r\n \"creation\"\r\n ],\r\n \"ingested\":
\"2024-12-03T10:01:00Z\"\r\n },\r\n \"related\":{\r\n \"entity\":[\r\n
\"AKIAI44QH8DHBEXAMPLE\"\r\n ]\r\n },\r\n \"entities\":{\r\n
\"metadata\":{\r\n \"AKIAI44QH8DHBEXAMPLE\":{\r\n \"entity\":{\r\n
\"id\":\"AKIAI44QH8DHBEXAMPLE\",\r\n \"category\":\"Access
Management\",\r\n \"type\":\"AWS IAM Access Key\"\r\n },\r\n
\"cloud\":{\r\n \"account\":{\r\n \"id\":\"444455556666\"\r\n }\r\n
}\r\n }\r\n }\r\n }\r\n}\r\n```\r\n</details>\r\n\r\n### To do\r\n\r\n-
[x] Add/Improve [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\n-
[x] Feature flag\r\n\r\n\r\n----\r\n#### Update:\r\n\r\nAdded
`assetInventoryStoreEnabled` Feature Flag. It is disabled by\r\ndefault
and even when enabled, the `/api/entity_store/enable` route does\r\nnot
initialize the Universal Entity
Engine.\r\n`/api/entity_store/engines/universal/init` needs to be
manually called\r\nto initialize
it\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Rômulo Farias <romulodefarias@gmail.com>\r\nCo-authored-by:
jaredburgettelastic <jared.burgett@elastic.co>\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"c6b0a31d8ec6d423a8071f50a22e55acedd0dee0"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Tiago Vila Verde <tiago.vilaverde@elastic.co>
2025-01-03 15:10:56 -06:00
..
examples [8.x] [DOCS] Remove inference connector docs (#198633) (#199846) 2024-11-12 13:56:59 -06:00
linters [8.x] [OpenAPI] Add redocly lint configuration (#199360) (#199554) 2024-11-08 17:22:00 -05:00
output [8.x] [Entity Store] [Asset Inventory] Universal entity definition (#202888) (#205536) 2025-01-03 15:10:56 -06:00
overlays [8.x] backport "Sustainable Kibana Architecture: Move modules owned by @elastic/obs-ux-logs-team" (#205234) 2024-12-30 13:05:58 +01:00
scripts [8.x] Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-data-discovery (#205267) 2024-12-30 13:03:12 -06:00
bundle.json [8.x] [Discover] Rename Saved Search to Discover Session (#202217) (#204818) 2024-12-19 21:38:57 +11:00
kibana.info.yaml [8.x] [OpenAPI] Fix Serverless API base URL (#202373) (#202583) 2024-12-04 14:15:28 -06:00
makefile [8.x] [OAS] Publish OAS bundles to bump.sh (#197482) (#200135) 2024-11-20 10:16:01 +01:00
package-lock.json [8.x] Update dependency @redocly/cli to ^1.26.0 (main) (#204435) (#204517) 2024-12-17 00:30:44 -06:00
package.json [8.x] Update dependency @redocly/cli to ^1.26.0 (main) (#204435) (#204517) 2024-12-17 00:30:44 -06:00
README.md [8.x] [OAS] Publish OAS bundles to bump.sh (#197482) (#200135) 2024-11-20 10:16:01 +01:00

README.md

Kibana API reference documentation

Documentation about our OpenAPI bundling workflow and configuration. See Kibana's hosted stateful and serverless docs.

Workflow

The final goal of this workflow is to produce an OpenAPI bundle containing all Kibana's public APIs.

Step 0

OAS from Kibana's APIs are continuously extracted and captured in bundle.json and bundle.serverless.json as fully formed OAS documentation. See node scripts/capture_oas_snapshot --help for more info.

These bundles form the basis of our OpenAPI bundles to which we append and layer extra information before publishing.

Step 1

Append pre-existing bundles not extracted from code using kbn-openapi-bundler to produce the final resulting bundles.

To add more files into the final bundle, edit the appropriate oas_docs/scripts/merge*.js files.

Step 2

Apply any final overalys to the document that might include examples or final tweaks (see the "Scripts" section for more details).

Scripts

The oas_docs/scripts folder contains scripts that point to the source domain-specific OpenAPI bundles and specify additional parameters for producing the final output bundle. Currently, there are the following scripts:

  • merge_ess_oas.js script produces production an output bundle for ESS

  • merge_serverless_oas.js script produces production an output bundle for Serverless

Output Kibana OpenAPI bundles

The oas_docs/output folder contains the final resulting Kibana OpenAPI bundles

  • kibana.yaml production ready ESS OpenAPI bundle
  • kibana.serverless.yaml production ready Serverless OpenAPI bundle

Bundling commands

Besides the scripts in the oas_docs/scripts folder, there is an oas_docs/makefile to simplify the workflow. Use make help to see available commands.