Your window into the Elastic Stack
Find a file
Marco Antonio Ghiani 0a0853bef9
[Spacetime] Fields metadata services (#183806)
## 📓 Summary

Closes https://github.com/elastic/observability-dev/issues/3331

Given the needs described in the linked issue about having a centralized
and async way to consume field metadata across Kibana, this work focuses
on providing server/client services to consume field metadata on demand
from static ECS definition and integration manifests, with the chance to
extend further the possible resolution sources.


3b2d9027-5c29-4081-ab17-1b43618c62a7

## 💡 Reviewers hints

This PR got quite long as it involves and touches different parts of the
codebase, so I'll break down the interesting parts for an easier review.

More details, code examples and mechanics description can be found in
the README file for the plugin.

### `@kbn/fields-metadata-plugin`

To avoid bundling and consuming the whole ECS static definition
client-side, a new plugin `@kbn/fields-metadata-plugin` is created to
expose the server/client services which enable retrieving only the
fields needed on a use-case basis.

### FieldsMetadataService server side

A `FieldsMetadataService` is instantiated on the plugin setup/start
server lifecycle, exposing a client to consume the fields and setup
tools for registering external dependencies.

The start contract exposes a `FieldsMetadataClient` instance. With this,
any application in Kibana can query for some fields using the available
methods, currently:
- `FieldsMetadataClient.prototype.getByName()`: retrieves a single
`FieldMetadata` instance.
- `FieldsMetadataClient.prototype.find()`: retrieves a record of
matching `FieldMetadata` instances.

`FieldsMetadataClient` is instantiated with the source repositories
dependencies. They act as encapsulated sources which are responsible for
fetching fields from their related source. Currently, there are 2 field
repository sources used in the resolution step, but we can use this
concept to extend the resolution step in future with more sources (LLM,
OTel, ...).
The currently used sources are:
- `EcsFieldsRepository`: allows fetching static ECS field metadata.
- `IntegrationFieldsRepository`: allows fetching fields from an
integration package from EPR, where the fields metadata are stored. To
correctly consume these fields, the `fleet` plugin must be enabled,
otherwise, the service won't be able to access the registered fields
extractor implemented with the fleet services.
As this service performs a more expensive retrieval process than the
`EcsFieldsRepository` constant complexity access, a caching layer is
applied to the retrieved results from the external source to minimize
latency.

### Fields metadata API

To expose this service to the client, a first API endpoint is created to
find field metadata and filter the results to minimize the served
payload.
- `GET /internal/fields_metadata/find` supports some initial query
parameters to narrow the fields' search.

### FieldsMetadataService client side

As we have a server-side `FieldsMetadataService`, we need a client
counterpart to consume the exposed API safely and go through the
validation steps.

The client `FieldsMetadataService` works similarly to the server-side
one, exposing a client which is returned by the public start contract of
the plugin, allowing any other to directly use fields metadata
client-side.

This client would work well with existing state management solutions, as
it's not decoupled from any library.

### useFieldsMetadata

For simpler use cases where we need a quick and easy way to consume
fields metadata client-side, the plugin start contract also exposes a
`useFieldsMetadata` react custom hook, which is pre-created accessing
the FieldsMetadataService client described above. It is important to
retrieve the hook from the start contract of this plugin, as it already
gets all the required dependencies injected minimizing the effort on the
consumer side.

The `UnifiedDocViewer` plugin changes exemplify how we can use this hook
to access and filter fields' metadata quickly.

### `registerIntegrationFieldsExtractor` (@elastic/fleet)

Getting fields from an integration dataset is more complex than
accessing a static dictionary of ECS fields, and to achieve that we need
access to the PackageService implemented by the fleet team.

To get access to the package, maintain a proper separation of concerns
and avoid a direct dependency on the fleet plugin, some actions were
taken:
- the `PackageService.prototype.getPackageFieldsMetadata()` method is
implemented to keep the knowledge about retrieving package details on
this service instead of mixing it on parallel services.
- a fleet `registerIntegrationFieldsExtractor` service is created and
used during the fleet plugin setup to register a callback that accesses
the service as an internal user and retrieves the fields by the given
parameters.
- the fields metadata plugin returns a
`registerIntegrationFieldsExtractor` function from its server setup so
that we can use it to register the above-mentioned callback that
retrieves fields from an integration.

This inverts the dependency between `fields_metadata` and `fleet`
plugins so that the `fields_metadata` plugin keeps zero dependencies on
external apps.

## Adoption

We currently have places where the `@elastic/ecs` package is directly
accessed and where we might be able to refactor the codebase to consume
this service.

**[EcsFlat usages in
Kibana](https://github.com/search?q=repo%3Aelastic%2Fkibana%20EcsFlat&type=code)**

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-06-05 09:51:50 +02:00
.buildkite [SecuritySolution] Relocate endpoint tests (#183611) 2024-06-04 13:04:01 +01:00
.github [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
api_docs [api-docs] 2024-06-05 Daily api_docs build (#184793) 2024-06-05 05:08:49 +00:00
config Disable bfetch in serverless (#183096) 2024-05-30 16:19:12 -07:00
dev_docs Add documentation for kibana:plugin_render_time (#184206) 2024-05-29 10:09:52 +03:00
docs [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
examples [data.search] Improve SearchSource types (part 1) (#184531) 2024-06-01 08:23:55 -07:00
kbn_pm
legacy_rfcs
licenses build: remove requirement to clone open-source repo (#180715) 2024-04-15 15:10:46 -05:00
oas_docs [OAS] Support tags (#184320) 2024-06-04 17:45:12 +02:00
packages [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
plugins
scripts [HTTP/OAS] Commit OAS snapshot (#183338) 2024-05-30 06:02:19 -07:00
src [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
test skip flaky suite (#181884) 2024-06-03 21:29:53 +01:00
typings Remove legacy kibana react code editor (#171047) 2024-01-05 14:35:09 +01:00
x-pack [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
.backportrc.json chore(NA): adds 8.15 into backportrc (#181082) 2024-04-17 21:28:21 +01:00
.bazelignore Remove references to deleted .ci folder (#177168) 2024-02-20 19:54:21 +01:00
.bazeliskversion
.bazelrc
.bazelrc.common
.bazelversion
.browserslistrc Add Firefox ESR to browserlistrc (#184462) 2024-05-29 17:53:18 -05:00
.editorconfig
.eslintignore [ES|QL] New @kbn/esql-services package (#179029) 2024-03-27 14:39:48 +01:00
.eslintrc.js [i18n][system upgrade] Upgrade Intl Packages from v2 to v6 (#179506) 2024-06-02 16:50:33 +03:00
.gitattributes
.gitignore [Moving] Move APM and APM_Data_Access folders into /x-pack/observability_solution/ (#177433) 2024-02-23 09:56:21 -07:00
.i18nrc.json [ResponseOps][Security Solution] Transfer ownership of the alerts grouping package to ResponseOps (#184131) 2024-05-28 11:15:04 +02:00
.node-version Upgrade Node.js to v20.13.1 (#183345) 2024-05-14 12:11:56 -07:00
.npmrc [npmrc] Fix puppeteer_skip_download configuration (#177673) 2024-02-22 18:59:01 -07:00
.nvmrc Upgrade Node.js to v20.13.1 (#183345) 2024-05-14 12:11:56 -07:00
.prettierignore
.prettierrc
.puppeteerrc Add .puppeteerrc (#179847) 2024-04-03 09:14:39 -05:00
.stylelintignore
.stylelintrc
.telemetryrc.json
.yarnrc
BUILD.bazel
catalog-info.yaml [BK] Add template for pipeline defs (#180189) 2024-04-08 11:21:28 +02:00
CODE_OF_CONDUCT.md
CONTRIBUTING.md
FAQ.md
fleet_packages.json [main] Sync bundled packages with Package Storage (#184748) 2024-06-04 09:17:30 -07:00
github_checks_reporter.json
kibana.d.ts
LICENSE.txt
nav-kibana-dev.docnav.json [dev docs] How to set up cross cluster search locally (#182025) 2024-04-30 12:46:21 -05:00
NOTICE.txt Copy assets from appropriate directory for kbn-monaco (#178669) 2024-03-21 16:29:20 +01:00
package.json [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
preinstall_check.js
README.md
renovate.json [Search] Renaming the search frontend group (#184565) 2024-06-03 13:14:49 -07:00
RISK_MATRIX.mdx
run_fleet_setup_parallel.sh [Fleet] Prevent concurrent runs of Fleet setup (#183636) 2024-05-31 16:38:51 +02:00
SECURITY.md
sonar-project.properties [ci] Run sonarqube daily (#173961) 2024-01-03 15:43:29 -06:00
STYLEGUIDE.mdx
tsconfig.base.json [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00
tsconfig.browser.json
tsconfig.browser_bazel.json
tsconfig.json
TYPESCRIPT.md
versions.json chore(NA): add back 8.13 branch into versions.json (#183631) 2024-05-16 13:20:54 +01:00
WORKSPACE.bazel Upgrade Node.js to v20.13.1 (#183345) 2024-05-14 12:11:56 -07:00
yarn.lock [Spacetime] Fields metadata services (#183806) 2024-06-05 09:51:50 +02:00

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 7.15.1 7.15.1 💚 OK
ES patch number is newer. 7.15.0 7.15.1 ⚠️ Logged warning
ES minor number is newer. 7.14.2 7.15.0 ⚠️ Logged warning
ES major number is newer. 7.15.1 8.0.0 🚫 Fatal error
ES patch number is older. 7.15.1 7.15.0 ⚠️ Logged warning
ES minor number is older. 7.15.1 7.14.2 🚫 Fatal error
ES major number is older. 8.0.0 7.15.1 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.