github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-24 17:59:23 -04:00
Code Issues Projects Releases Packages Wiki Activity
kibana/oas_docs
History
Tiago Vila Verde c6b0a31d8e
[Entity Store] [Asset Inventory] Universal entity definition (#202888) 
## Summary

This PR adds a universal entity definition.
A universal entity uses `related.entity` as an identifier field and
includes an extra processor step that parses the field
`entities.keyword` and extracts all the entities in said field (whose
original data comes from `related.entities`).

See this
[doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7)
for more details.

To accomplish this, we need to allow describing an entity along with
extra entity store resources required for that entity's engine.
This PR reworks the current entity store by introducing an `Entity
Description`, which has all that required information. From it, we can
build an `EntityEngineDescription` which adds all the needed data that
must be computed (as opposed to hardcoded) and is then used to generate
all the resources needed for that Entity's engine (entity definition,
pipeline, enrich policy, index mappings, etc).

<img width="3776" alt="EntityDescriptions"
src="https://github.com/user-attachments/assets/bdf7915f-1981-47e6-a815-31163f24ad03">

This required a refactoring of the current `UnitedEntityDefinition`,
which has now been removed in favour of more contextual functions for
all the different parts.
The intention is to decouple the Entity Description schema from the
schemas required for field retention, entity manager and pipeline. We
can then freely expand on our Entity Description as required, and simply
alter the conversion functions when needed.

## How to test

1. On a fresh ES cluster, add some entity data
* For hosts and user, use the [security documents
generator](https://github.com/elastic/security-documents-generator)
   * For universal, there are a few more steps:
      1. Create the `entity.keyword` builder pipeline
      2. Add it to a index template
      3. Post some docs to the corresponding index 
2. Initialise the universal entity engine via: `POST
kbn:/api/entity_store/engines/universal/init {}`
* Note that using the UI does not work, as we've specifically removed
the Universal engine from the normal Entity Store workflow
3. Check the status of the store is `running` via `GET
kbn:/api/entity_store/status`
4. Once the transform runs, you can query `GET entities*/_search` to see
the created entities

Note that universal entities do not show up in the dashboard Entities
List.


### Code to ingest data
<details>
<summary>Pipeline</summary>

```js
PUT _ingest/pipeline/entities-keyword-builder
{
   "description":"Serialize entities.metadata into a keyword field",
   "processors":[
      {
         "script":{
            "lang":"painless",
            "source":"""
String jsonFromMap(Map map) {
    StringBuilder json = new StringBuilder("{");
    boolean first = true;

    for (entry in map.entrySet()) {
        if (!first) {
            json.append(",");
        }
        first = false;

        String key = entry.getKey().replace("\"", "\\\"");
        Object value = entry.getValue();

        json.append("\"").append(key).append("\":");

        if (value instanceof String) {
            String escapedValue = ((String) value).replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedValue).append("\"");
        } else if (value instanceof Map) {
            json.append(jsonFromMap((Map) value));
        } else if (value instanceof List) {
            json.append(jsonFromList((List) value));
        } else if (value instanceof Boolean || value instanceof Number) {
            json.append(value.toString());
        } else {
            // For other types, treat as string
            String escapedValue = value.toString().replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedValue).append("\"");
        }
    }

    json.append("}");
    return json.toString();
}

String jsonFromList(List list) {

    StringBuilder json = new StringBuilder("[");
    boolean first = true;

    for (item in list) {
        if (!first) {
            json.append(",");
        }
        first = false;

        if (item instanceof String) {
            String escapedItem = ((String) item).replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedItem).append("\"");
        } else if (item instanceof Map) {
            json.append(jsonFromMap((Map) item));
        } else if (item instanceof List) {
            json.append(jsonFromList((List) item));
        } else if (item instanceof Boolean || item instanceof Number) {
            json.append(item.toString());
        } else {
            // For other types, treat as string
            String escapedItem = item.toString().replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedItem).append("\"");
        }
    }

    json.append("]");
    return json.toString();
}

def metadata = jsonFromMap(ctx['entities']['metadata']);
ctx['entities']['keyword'] = metadata;
"""

            }
        }
    ]
}
```
</details>

<details>
<summary>Index template</summary>

```js
PUT /_index_template/entity_store_index_template
{
   "index_patterns":[
      "logs-store"
   ],
   "template":{
      "settings":{
         "index":{
            "default_pipeline":"entities-keyword-builder"
         }
      },
      "mappings":{
         "properties":{
            "@timestamp":{
               "type":"date"
            },
            "message":{
               "type":"text"
            },
            "event":{
               "properties":{
                  "action":{
                     "type":"keyword"
                  },
                  "category":{
                     "type":"keyword"
                  },
                  "type":{
                     "type":"keyword"
                  },
                  "outcome":{
                     "type":"keyword"
                  },
                  "provider":{
                     "type":"keyword"
                  },
                  "ingested":{
                    "type": "date"
                  }
               }
            },
            "related":{
               "properties":{
                  "entity":{
                     "type":"keyword"
                  }
               }
            },
            "entities":{
               "properties":{
                  "metadata":{
                     "type":"flattened"
                  },
                  "keyword":{
                     "type":"keyword"
                  }
               }
            }
         }
      }
   }
}
```
</details>

<details>
<summary>Example source doc</summary>

```js
POST /logs-store/_doc/
{
   "@timestamp":"2024-11-29T10:01:00Z",
   "message":"Eddie",
   "event": {
      "type":[
         "creation"
      ],
      "ingested": "2024-12-03T10:01:00Z"
   },
   "related":{
      "entity":[
         "AKIAI44QH8DHBEXAMPLE"
      ]
   },
   "entities":{
      "metadata":{
         "AKIAI44QH8DHBEXAMPLE":{
            "entity":{
               "id":"AKIAI44QH8DHBEXAMPLE",
               "category":"Access Management",
               "type":"AWS IAM Access Key"
            },
            "cloud":{
               "account":{
                  "id":"444455556666"
               }
            }
         }
      }
   }
}
```
</details>

### To do

- [x] Add/Improve [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
- [x] Feature flag


----
#### Update:

Added `assetInventoryStoreEnabled` Feature Flag. It is disabled by
default and even when enabled, the `/api/entity_store/enable` route does
not initialize the Universal Entity Engine.
`/api/entity_store/engines/universal/init` needs to be manually called
to initialize it

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Rômulo Farias <romulodefarias@gmail.com>
Co-authored-by: jaredburgettelastic <jared.burgett@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2025-01-03 10:43:16 -06:00
..
examples [DOCS] Remove inference connector docs (#198633) 2024-11-12 11:56:37 -06:00
linters [OpenAPI] Add redocly lint configuration (#199360) 2024-11-08 14:07:55 -06:00
output [Entity Store] [Asset Inventory] Universal entity definition (#202888) 2025-01-03 10:43:16 -06:00
overlays Sustainable Kibana Architecture: Move modules owned by @elastic/obs-ux-infra_services-team (#202830) 2024-12-29 09:58:37 +01:00
scripts Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-data-discovery (#203152) 2024-12-30 13:23:47 +01:00
bundle.json [Fleet] fix response schema for cancel upgrade (#205493) 2025-01-03 10:34:00 -06:00
bundle.serverless.json [Fleet] fix response schema for cancel upgrade (#205493) 2025-01-03 10:34:00 -06:00
kibana.info.serverless.yaml [DOCS] Remove technical preview from serverless APIs (#201054) 2024-11-21 09:45:10 +01:00
kibana.info.yaml [OpenAPI] Fix Serverless API base URL (#202373) 2024-12-02 12:09:03 -08:00
makefile [OAS] Publish OAS bundles to bump.sh (#197482) 2024-11-14 09:15:47 +01:00
package-lock.json Update dependency @redocly/cli to ^1.26.0 (main) (#204435) 2024-12-16 22:41:31 -06:00
package.json Update dependency @redocly/cli to ^1.26.0 (main) (#204435) 2024-12-16 22:41:31 -06:00
README.md [OAS] Publish OAS bundles to bump.sh (#197482) 2024-11-14 09:15:47 +01:00

README.md

Kibana API reference documentation

Documentation about our OpenAPI bundling workflow and configuration. See Kibana's hosted stateful and serverless docs.

Workflow

The final goal of this workflow is to produce an OpenAPI bundle containing all Kibana's public APIs.

Step 0

OAS from Kibana's APIs are continuously extracted and captured in bundle.json and bundle.serverless.json as fully formed OAS documentation. See node scripts/capture_oas_snapshot --help for more info.

These bundles form the basis of our OpenAPI bundles to which we append and layer extra information before publishing.

Step 1

Append pre-existing bundles not extracted from code using kbn-openapi-bundler to produce the final resulting bundles.

To add more files into the final bundle, edit the appropriate oas_docs/scripts/merge*.js files.

Step 2

Apply any final overalys to the document that might include examples or final tweaks (see the "Scripts" section for more details).

Scripts

The oas_docs/scripts folder contains scripts that point to the source domain-specific OpenAPI bundles and specify additional parameters for producing the final output bundle. Currently, there are the following scripts:

  • merge_ess_oas.js script produces production an output bundle for ESS

  • merge_serverless_oas.js script produces production an output bundle for Serverless

Output Kibana OpenAPI bundles

The oas_docs/output folder contains the final resulting Kibana OpenAPI bundles

  • kibana.yaml production ready ESS OpenAPI bundle
  • kibana.serverless.yaml production ready Serverless OpenAPI bundle

Bundling commands

Besides the scripts in the oas_docs/scripts folder, there is an oas_docs/makefile to simplify the workflow. Use make help to see available commands.