github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-06-27 10:40:07 -04:00
Code Issues Projects Releases Packages Wiki Activity
Your window into the Elastic Stack
82832 commits 499 branches 518 tags 21 GiB
Find a file
Tiago Vila Verde b07849cc5a
[Entity Store][Asset Inventory] Dynamic field retention for universal entity (#206419) 
## Summary

This PR improves upon the Universal entity definition and entity store
work being done to support Asset Inventory by introducing a flag
`dynamic` to the definition.

The entity store uses an enrich policy in order to retain observed data
that falls outside of a `lookbackPeriod` used by the transform that runs
the aggregations on the source fields.
Normally, we have to specify a retention strategy per each field defined
in an entity definition. However, for universal entities, (some of) the
fields are dynamically generated based on the JSON extractor pipeline
processor, which means we cannot define which strategy to use in the
definition itself.
To account for this, when `dynamic` is set to `true`, we run an extra
ingest pipeline step to process _any field which does not show up in the
entity definition_ (ie, has been dynamically generated). At the moment,
this pipeline step uses a strategy that always picks the latest value,
although int he future, this might need to be configurable, mimicking
the ability to choose strategies for "static" fields.

See this
[doc](https://docs.google.com/document/d/1D8xDtn3HHP65i1Y3eIButacD6ZizyjZZRJB7mxlXzQY/edit?tab=t.0#heading=h.9fz3qtlfzjg7)
for more details and [this
Figma](https://www.figma.com/board/17dpxrztlM4O120p9qMcNw/Entity-descriptions?node-id=0-1&t=JLcB84l9NxCnudAs-1)
for information regarding Entity Store architecture.



## How to test:

### Setup

1. Ensure the default Security Data View exists by navigating to some
Security solution UI.
2. Set up the `entity.keyword` builder pipeline
* Add it to an index that matches any of the default index patterns in
the security data view (eg: `logs-store`)
* Make sure and ingested doc contains both `event.ingested` and
`@timestamp`.
      * Easiest way is to add `set` processors to the builder pipeline.
3. Because of the async nature of the field retention process, it is
recommended to change some of the default values (explained below)
4. Enable `debugging` by adding
`xpack.securitySolution.entityAnalytics.entityStore.developer.pipelineDebugMode:
true` to your `kibana.dev.yml`
5. Enable the `assetInventoryStoreEnabled` FF: 
```
xpack.securitySolution.enableExperimental:
  - assetInventoryStoreEnabled
```

### Interacting with the store

In Kibana dev tools:

#### Phase 1
1. `POST` some of the example docs to the `logs-store` index
2. Confirm the `entity.keyword` field is being added by the builder
pipeline via `GET logs-store/_search`.
3. Initialise the universal entity engine via: `POST
kbn:/api/entity_store/engines/universal/init {}`
* In order to properly test field retention, it's advisable to reduce
the `lookbackPeriod` setting, which means some of the docs in the index
might fall out of the window if it takes too long to initialize the
engine. Any docs posted when the engine is running should be picked up.
* Note that using the UI does not work, as we've specifically removed
the Universal engine from the normal Entity Store workflow
4. Check the status of the store is `running` via `GET
kbn:/api/entity_store/status`
5. Check that the transform has ran by querying the store index: `GET
.entities.v1.latest.security_universal*/_search`
* There should be one entity per `related.entity` found in the source
index
* The fields in the JSON string in `entities.keyword` should appear as
fields in the target documents
* There should also be a `debug` field and potentially a `historical`
field, if enough time has passed for the enrich policy to run. These are
normally hidden, but show up when in `debug mode`.

#### Phase 2

1. Wait some time (the `INTERVAL` constant) for the enrich policy to
populate the `.enrich` indices with the latest data from the store index
* Ideally, this will mean that any docs in the source index now fall
outside of `lookbackPeriod` of the transform.
* Alternatively, you can manually run the enrich poly via: `PUT
/_enrich/policy/entity_store_field_retention_universal_default_v1.0.0/_execute`.
* It's also possible to update the source docs' timestamps and
`event.ingested` to ensure they're outside the `lookbackPeriod`
3. `POST` a new doc to the source index (eg: `logs-store`)
* The new doc should either have a new, not yet observed property in
`entities.metadata`, or the same fields but with different, new values.
4. Query the store index again. 
* The entity in question should now reflect the new changes _but
preserve the old data too!_
   * Existing fields should have been updated to new values
* New fields should have been `recursively` merged. Ie, nested fields
should not be an issue.
* The `historical` field should show the "previous state" of the entity
doc. This is useful to confirm that a field's value is, in fact, the
"latest" value, whether that comes from a new doc that falls in the
lookback window of the transform, or from this `historical` "cache".
 


### Code

#### Default values:

* in
[`server/lib/entity_analytics/entity_store/entity_definition/universal.ts#L75-L76`](6686d57ce5/x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_definitions/entity_descriptions/universal.ts (L75-L76)):
    * Add the following fields to `settings`: 
    ```ts
      { frequency: '2s', lookbackPeriod: '1m', syncDelay: '2s'}
    ```
* in
[`server/lib/entity_analytics/entity_store/task/constants.ts#L11-L13`](6686d57ce5/x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/task/constants.ts (L11-L13))
   * Change the following defaults: 
   ```ts
     export const INTERVAL = '1m';
     export const TIMEOUT = '30s';
   ```

#### Ingest pipeline

<details>
<summary>Pipeline</summary>

```js
PUT _ingest/pipeline/entities-keyword-builder
{
   "description":"Serialize entities.metadata into a keyword field",
   "processors":[
      {
         "set": {
           "field": "event.ingested",
           "value": "{{_ingest.timestamp}}"
         }
      },
      {
         "set": {
           "field": "@timestamp",
           "value": "{{_ingest.timestamp}}"
         }
      },
      {
         "script":{
            "lang":"painless",
            "source":"""
String jsonFromMap(Map map) {
    StringBuilder json = new StringBuilder("{");
    boolean first = true;

    for (entry in map.entrySet()) {
        if (!first) {
            json.append(",");
        }
        first = false;

        String key = entry.getKey().replace("\"", "\\\"");
        Object value = entry.getValue();

        json.append("\"").append(key).append("\":");

        if (value instanceof String) {
            String escapedValue = ((String) value).replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedValue).append("\"");
        } else if (value instanceof Map) {
            json.append(jsonFromMap((Map) value));
        } else if (value instanceof List) {
            json.append(jsonFromList((List) value));
        } else if (value instanceof Boolean || value instanceof Number) {
            json.append(value.toString());
        } else {
            // For other types, treat as string
            String escapedValue = value.toString().replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedValue).append("\"");
        }
    }

    json.append("}");
    return json.toString();
}

String jsonFromList(List list) {

    StringBuilder json = new StringBuilder("[");
    boolean first = true;

    for (item in list) {
        if (!first) {
            json.append(",");
        }
        first = false;

        if (item instanceof String) {
            String escapedItem = ((String) item).replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedItem).append("\"");
        } else if (item instanceof Map) {
            json.append(jsonFromMap((Map) item));
        } else if (item instanceof List) {
            json.append(jsonFromList((List) item));
        } else if (item instanceof Boolean || item instanceof Number) {
            json.append(item.toString());
        } else {
            // For other types, treat as string
            String escapedItem = item.toString().replace("\"", "\\\"").replace("=", ":");
            json.append("\"").append(escapedItem).append("\"");
        }
    }

    json.append("]");
    return json.toString();
}

def metadata = jsonFromMap(ctx['entities']['metadata']);
ctx['entities']['keyword'] = metadata;
"""

            }
        }
    ]
}
```
</details>


<details>
<summary>Index template</summary>

```js
PUT /_index_template/entity_store_index_template
{
   "index_patterns":[
      "logs-store"
   ],
   "template":{
      "settings":{
         "index":{
            "default_pipeline":"entities-keyword-builder"
         }
      },
      "mappings":{
         "properties":{
            "@timestamp":{
               "type":"date"
            },
            "message":{
               "type":"text"
            },
            "event":{
               "properties":{
                  "action":{
                     "type":"keyword"
                  },
                  "category":{
                     "type":"keyword"
                  },
                  "type":{
                     "type":"keyword"
                  },
                  "outcome":{
                     "type":"keyword"
                  },
                  "provider":{
                     "type":"keyword"
                  },
                  "ingested":{
                    "type": "date"
                  }
               }
            },
            "related":{
               "properties":{
                  "entity":{
                     "type":"keyword"
                  }
               }
            },
            "entities":{
               "properties":{
                  "metadata":{
                     "type":"flattened"
                  },
                  "keyword":{
                     "type":"keyword"
                  }
               }
            }
         }
      }
   }
}
```
</details>

<details>
<summary>Example source docs</summary>

#### Phase 1:

```js
POST /logs-store/_doc/
{
   "related":{
      "entity":[
         "test-id"
      ]
   },
   "entities":{
      "metadata":{
         "test-id":{
            "okta":{
               "foo": {
                  "baz": {
                     "qux": 1
                  }

               }
            },
            "cloud": {
               "super": 123
            }
         }
      }
   }
}
```

```js
POST /logs-store/_doc/
{
   "related":{
      "entity":[
         "test-id"
      ]
   },
   "entities":{
      "metadata":{
         "test-id":{
            "cloud":{
               "host": "me"
            }
         }
      }
   }
}
```

#### Phase 2:

```js
POST /logs-store/_doc/
{
   "related":{
      "entity":[
         "test-id"
      ]
   },
   "entities":{
      "metadata":{
         "test-id":{
            "cloud":{
               "host": "me",
               "super": 1111111,
            },
            "okta":{
               "foo": {
                  "baz": {
                     "qux": 99,
                     "hello": "world"
                  },
                  "hello": "world"
               },
               "hello": "world"
            }
         }
      }
   }
}
```

</details>
2025-01-19 23:37:10 +00:00
.buildkite Add keyword builder pipeline 2025-01-19 14:38:32 +02:00
.devcontainer Removing experimental for the FIPS mode config (#200734) 2024-11-19 15:23:20 -05:00
.github Add keyword builder pipeline 2025-01-19 14:38:32 +02:00
api_docs [api-docs] 2025-01-19 Daily api_docs build (#207150) 2025-01-19 07:04:06 +00:00
config [Onboarding] [Stack] Add Onboarding experience into Stack (#204351) 2025-01-15 17:03:25 -07:00
dev_docs Adding fips docs to nav (#206935) 2025-01-17 01:38:28 +01:00
docs [Reporting Docs for inspecting the query used for CSV export (#207001) 2025-01-17 09:48:15 +01:00
examples [Dashboard][Collapsable Panels] Swap react-grid-layout for kbn-grid-layout (#205341) 2025-01-14 14:51:14 -07:00
kbn_pm Sustainable Kibana Architecture: Move CodeEditor related packages #205587 (#205738) 2025-01-08 15:25:47 +01:00
legacy_rfcs Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-security (#202748) 2025-01-05 12:57:01 +01:00
licenses Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
oas_docs [Fleet] Send Agentless API resources (#206042) 2025-01-19 12:52:10 +02:00
packages [Fleet] Send Agentless API resources (#206042) 2025-01-19 12:52:10 +02:00
plugins
scripts Update styled_components_files.js to include all files that import styled-components (#205011) 2025-01-05 16:54:17 +01:00
src [Fleet] Send Agentless API resources (#206042) 2025-01-19 12:52:10 +02:00
test [Discover] Remove redundant data fetching when hiding/showing the histogram/chart (#206389) 2025-01-17 10:49:37 +00:00
typings make emotion typing global (#200958) 2024-12-05 12:20:43 -06:00
x-pack [Entity Store][Asset Inventory] Dynamic field retention for universal entity (#206419) 2025-01-19 23:37:10 +00:00
.backportrc.json chore(NA): adds 8.17 into backportrc (#201065) 2024-11-21 06:05:29 +00:00
.bazelignore Remove references to deleted .ci folder (#177168) 2024-02-20 19:54:21 +01:00
.bazeliskversion chore(NA): upgrade bazelisk into v1.11.0 (#125070) 2022-02-09 20:43:57 +00:00
.bazelrc chore(NA): use new and more performant BuildBuddy servers (#130350) 2022-04-18 02:01:38 +01:00
.bazelrc.common Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
.bazelversion chore(NA): revert bazel upgrade for v5.2.0 (#135096) 2022-06-24 03:57:21 +01:00
.browserslistrc Add Firefox ESR to browserlistrc (#184462) 2024-05-29 17:53:18 -05:00
.editorconfig
.eslintignore Sustainable Kibana Architecture: Move CodeEditor related packages #205587 (#205738) 2025-01-08 15:25:47 +01:00
.eslintrc.js Reenable ESLint helpers for Observability (#206585) 2025-01-14 17:02:33 +01:00
.gitattributes
.gitignore [Investigate App] add MVP evaluation framework for AI root cause analysis integration (#204634) 2025-01-17 12:16:10 -05:00
.i18nrc.json SKA: Relocate @kbn/grid-layout (#206821) 2025-01-15 17:39:41 +00:00
.node-version Upgrade Node.js to 20.15.1 (#187791) 2024-07-15 12:34:07 -05:00
.npmrc [npmrc] Fix puppeteer_skip_download configuration (#177673) 2024-02-22 18:59:01 -07:00
.nvmrc Upgrade Node.js to 20.15.1 (#187791) 2024-07-15 12:34:07 -05:00
.prettierignore
.prettierrc
.puppeteerrc Add .puppeteerrc (#179847) 2024-04-03 09:14:39 -05:00
.stylelintignore
.stylelintrc Bump stylelint to ^14 (#136693) 2022-07-20 10:11:00 -05:00
.telemetryrc.json Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-core (#201653) 2025-01-04 11:47:24 -07:00
.yarnrc
BUILD.bazel Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
catalog-info.yaml [sonarqube] Disable cron (#190611) 2024-08-15 09:19:09 -05:00
CODE_OF_CONDUCT.md
CONTRIBUTING.md
FAQ.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
fleet_packages.json [main] Sync bundled packages with Package Storage (#192007) 2024-09-03 12:26:57 -05:00
github_checks_reporter.json
kibana.d.ts Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
LICENSE.txt Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
NOTICE.txt [api-docs] 2025-01-01 Daily api_docs build (#205342) 2025-01-01 01:37:13 -06:00
package.json Update dependency oas to ^25.2.1 (main) (#206997) 2025-01-17 23:00:23 +00:00
preinstall_check.js Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
README.md
renovate.json Add check to fail CI if any dependencies are unowned (#206679) 2025-01-16 09:59:04 -05:00
RISK_MATRIX.mdx
run_fleet_setup_parallel.sh Sustainable Kibana Architecture: Move modules owned by @elastic/fleet (#202422) 2024-12-24 15:32:43 +01:00
SECURITY.md
sonar-project.properties [sonarqube] update memory, cpu (#190547) 2024-09-09 16:16:30 -05:00
STYLEGUIDE.mdx [styleguide] update path to scss theme (#140742) 2022-09-15 10:41:14 -04:00
tsconfig.base.json SKA: Relocate @kbn/grid-layout (#206821) 2025-01-15 17:39:41 +00:00
tsconfig.browser.json
tsconfig.browser_bazel.json
tsconfig.json Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
TYPESCRIPT.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
updatecli-compose.yaml deps(updatecli): bump all policies (#195865) 2024-10-15 07:37:12 -05:00
versions.json chore(NA): update versions after v7.17.28 bump (#206648) 2025-01-15 00:28:08 +00:00
WORKSPACE.bazel chore(NA): remove usage of re2 and replace it with a non native module (#188134) 2024-07-15 20:33:28 +01:00
yarn.lock Update dependency oas to ^25.2.1 (main) (#206997) 2025-01-17 23:00:23 +00:00

README.md

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 7.15.1 7.15.1 💚 OK
ES patch number is newer. 7.15.0 7.15.1 ⚠️ Logged warning
ES minor number is newer. 7.14.2 7.15.0 ⚠️ Logged warning
ES major number is newer. 7.15.1 8.0.0 🚫 Fatal error
ES patch number is older. 7.15.1 7.15.0 ⚠️ Logged warning
ES minor number is older. 7.15.1 7.14.2 🚫 Fatal error
ES major number is older. 8.0.0 7.15.1 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.