github-mirrors/kibana
1
0
Fork 0
mirror of https://github.com/elastic/kibana.git synced 2025-04-23 17:28:26 -04:00
Code Issues Projects Releases Packages Wiki Activity

[Spacetime] Fields metadata services (#183806)

Browse source 
## 📓 Summary

Closes https://github.com/elastic/observability-dev/issues/3331

Given the needs described in the linked issue about having a centralized
and async way to consume field metadata across Kibana, this work focuses
on providing server/client services to consume field metadata on demand
from static ECS definition and integration manifests, with the chance to
extend further the possible resolution sources.


3b2d9027-5c29-4081-ab17-1b43618c62a7

## 💡 Reviewers hints

This PR got quite long as it involves and touches different parts of the
codebase, so I'll break down the interesting parts for an easier review.

More details, code examples and mechanics description can be found in
the README file for the plugin.

### `@kbn/fields-metadata-plugin`

To avoid bundling and consuming the whole ECS static definition
client-side, a new plugin `@kbn/fields-metadata-plugin` is created to
expose the server/client services which enable retrieving only the
fields needed on a use-case basis.

### FieldsMetadataService server side

A `FieldsMetadataService` is instantiated on the plugin setup/start
server lifecycle, exposing a client to consume the fields and setup
tools for registering external dependencies.

The start contract exposes a `FieldsMetadataClient` instance. With this,
any application in Kibana can query for some fields using the available
methods, currently:
- `FieldsMetadataClient.prototype.getByName()`: retrieves a single
`FieldMetadata` instance.
- `FieldsMetadataClient.prototype.find()`: retrieves a record of
matching `FieldMetadata` instances.

`FieldsMetadataClient` is instantiated with the source repositories
dependencies. They act as encapsulated sources which are responsible for
fetching fields from their related source. Currently, there are 2 field
repository sources used in the resolution step, but we can use this
concept to extend the resolution step in future with more sources (LLM,
OTel, ...).
The currently used sources are:
- `EcsFieldsRepository`: allows fetching static ECS field metadata.
- `IntegrationFieldsRepository`: allows fetching fields from an
integration package from EPR, where the fields metadata are stored. To
correctly consume these fields, the `fleet` plugin must be enabled,
otherwise, the service won't be able to access the registered fields
extractor implemented with the fleet services.
As this service performs a more expensive retrieval process than the
`EcsFieldsRepository` constant complexity access, a caching layer is
applied to the retrieved results from the external source to minimize
latency.

### Fields metadata API

To expose this service to the client, a first API endpoint is created to
find field metadata and filter the results to minimize the served
payload.
- `GET /internal/fields_metadata/find` supports some initial query
parameters to narrow the fields' search.

### FieldsMetadataService client side

As we have a server-side `FieldsMetadataService`, we need a client
counterpart to consume the exposed API safely and go through the
validation steps.

The client `FieldsMetadataService` works similarly to the server-side
one, exposing a client which is returned by the public start contract of
the plugin, allowing any other to directly use fields metadata
client-side.

This client would work well with existing state management solutions, as
it's not decoupled from any library.

### useFieldsMetadata

For simpler use cases where we need a quick and easy way to consume
fields metadata client-side, the plugin start contract also exposes a
`useFieldsMetadata` react custom hook, which is pre-created accessing
the FieldsMetadataService client described above. It is important to
retrieve the hook from the start contract of this plugin, as it already
gets all the required dependencies injected minimizing the effort on the
consumer side.

The `UnifiedDocViewer` plugin changes exemplify how we can use this hook
to access and filter fields' metadata quickly.

### `registerIntegrationFieldsExtractor` (@elastic/fleet)

Getting fields from an integration dataset is more complex than
accessing a static dictionary of ECS fields, and to achieve that we need
access to the PackageService implemented by the fleet team.

To get access to the package, maintain a proper separation of concerns
and avoid a direct dependency on the fleet plugin, some actions were
taken:
- the `PackageService.prototype.getPackageFieldsMetadata()` method is
implemented to keep the knowledge about retrieving package details on
this service instead of mixing it on parallel services.
- a fleet `registerIntegrationFieldsExtractor` service is created and
used during the fleet plugin setup to register a callback that accesses
the service as an internal user and retrieves the fields by the given
parameters.
- the fields metadata plugin returns a
`registerIntegrationFieldsExtractor` function from its server setup so
that we can use it to register the above-mentioned callback that
retrieves fields from an integration.

This inverts the dependency between `fields_metadata` and `fleet`
plugins so that the `fields_metadata` plugin keeps zero dependencies on
external apps.

## Adoption

We currently have places where the `@elastic/ecs` package is directly
accessed and where we might be able to refactor the codebase to consume
this service.

**[EcsFlat usages in
Kibana](https://github.com/search?q=repo%3Aelastic%2Fkibana%20EcsFlat&type=code)**

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Marco Antonio Ghiani 2024-06-05 09:51:50 +02:00 • committed by GitHub
parent 7bb934b0bf
commit 0a0853bef9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
76 changed files with 2398 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.github/CODEOWNERS vendored
View file

@ -448,6 +448,7 @@ examples/field_formats_example @elastic/kibana-data-discovery
src/plugins/field_formats @elastic/kibana-data-discovery
packages/kbn-field-types @elastic/kibana-data-discovery
packages/kbn-field-utils @elastic/kibana-data-discovery
x-pack/plugins/fields_metadata @elastic/obs-ux-logs-team
x-pack/plugins/file_upload @elastic/kibana-gis
examples/files_example @elastic/appex-sharedux
src/plugins/files_management @elastic/appex-sharedux

4
docs/developer/plugin-list.asciidoc
View file

@ -582,6 +582,10 @@ activities.
|The features plugin enhance Kibana with a per-feature privilege system.
|{kib-repo}blob/{branch}/x-pack/plugins/fields_metadata/README.md[fieldsMetadata]
|The @kbn/fields-metadata-plugin is designed to provide a centralized and asynchronous way to consume field metadata across Kibana. This plugin addresses the need for on-demand retrieval of field metadata from static ECS definitions and integration manifests, with the flexibility to extend to additional resolution sources in the future.
|{kib-repo}blob/{branch}/x-pack/plugins/file_upload[fileUpload]
|WARNING: Missing README.

1
package.json
View file

@ -492,6 +492,7 @@
"@kbn/field-formats-plugin": "link:src/plugins/field_formats",
"@kbn/field-types": "link:packages/kbn-field-types",
"@kbn/field-utils": "link:packages/kbn-field-utils",
"@kbn/fields-metadata-plugin": "link:x-pack/plugins/fields_metadata",
"@kbn/file-upload-plugin": "link:x-pack/plugins/file_upload",
"@kbn/files-example-plugin": "link:examples/files_example",
"@kbn/files-management-plugin": "link:src/plugins/files_management",

1
packages/kbn-io-ts-utils/index.ts
View file

@ -9,6 +9,7 @@
export type { IndexPattern } from './src/index_pattern_rt';
export type { NonEmptyString, NonEmptyStringBrand } from './src/non_empty_string_rt';
export { arrayToStringRt } from './src/array_to_string_rt';
export { deepExactRt } from './src/deep_exact_rt';
export { indexPatternRt } from './src/index_pattern_rt';
export { jsonRt } from './src/json_rt';

52
packages/kbn-io-ts-utils/src/array_to_string_rt/index.test.ts Normal file
View file

@ -0,0 +1,52 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import * as rt from 'io-ts';
import { arrayToStringRt } from '.';
import { isRight, Either, fold } from 'fp-ts/lib/Either';
import { identity } from 'fp-ts/lib/function';
function getValueOrThrow(either: Either<unknown, any>) {
return fold(() => {
throw new Error('Cannot get right value of left');
}, identity)(either);
}
describe('arrayToStringRt', () => {
it('should validate strings', () => {
expect(isRight(arrayToStringRt.decode(''))).toBe(true);
expect(isRight(arrayToStringRt.decode('message'))).toBe(true);
expect(isRight(arrayToStringRt.decode('message,event.original'))).toBe(true);
expect(isRight(arrayToStringRt.decode({}))).toBe(false);
expect(isRight(arrayToStringRt.decode(true))).toBe(false);
});
it('should return array of strings when decoding', () => {
expect(getValueOrThrow(arrayToStringRt.decode(''))).toEqual(['']);
expect(getValueOrThrow(arrayToStringRt.decode('message'))).toEqual(['message']);
expect(getValueOrThrow(arrayToStringRt.decode('message,event.original'))).toEqual([
'message',
'event.original',
]);
});
it('should be pipable', () => {
const piped = arrayToStringRt.pipe(rt.array(rt.string));
const validInput = ['message', 'event.original'];
const invalidInput = {};
const valid = piped.decode(validInput.join(','));
const invalid = piped.decode(invalidInput);
expect(isRight(valid)).toBe(true);
expect(getValueOrThrow(valid)).toEqual(validInput);
expect(isRight(invalid)).toBe(false);
});
});

24
packages/kbn-io-ts-utils/src/array_to_string_rt/index.ts Normal file
View file

@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import { either } from 'fp-ts/lib/Either';
import * as rt from 'io-ts';
export const arrayToStringRt = new rt.Type<unknown[], string, unknown>(
'arrayToString',
rt.array(rt.unknown).is,
(input, context) =>
either.chain(rt.string.validate(input, context), (str) => {
try {
return rt.success(str.split(','));
} catch (e) {
return rt.failure(input, context);
}
}),
(arr) => arr.join(',')
);

1
packages/kbn-optimizer/limits.yml
View file

@ -62,6 +62,7 @@ pageLoadAssetSize:
expressionXY: 45000
features: 21723
fieldFormats: 65209
fieldsMetadata: 21885
files: 22673
filesManagement: 18683
fileUpload: 25664

1
src/plugins/unified_doc_viewer/kibana.jsonc
View file

@ -9,5 +9,6 @@
"browser": true,
"requiredBundles": ["kibanaUtils"],
"requiredPlugins": ["data", "discoverShared", "fieldFormats"],
"optionalPlugins": ["fieldsMetadata"]
}
}

2
src/plugins/unified_doc_viewer/public/__mocks__/services.ts
View file

@ -10,6 +10,7 @@ import { analyticsServiceMock } from '@kbn/core-analytics-browser-mocks';
import { dataPluginMock } from '@kbn/data-plugin/public/mocks';
import { discoverSharedPluginMock } from '@kbn/discover-shared-plugin/public/mocks';
import { fieldFormatsMock } from '@kbn/field-formats-plugin/common/mocks';
import { fieldsMetadataPluginPublicMock } from '@kbn/fields-metadata-plugin/public/mocks';
import { uiSettingsServiceMock } from '@kbn/core-ui-settings-browser-mocks';
import type { UnifiedDocViewerServices, UnifiedDocViewerStart } from '../types';
import { Storage } from '@kbn/kibana-utils-plugin/public';
@ -24,6 +25,7 @@ export const mockUnifiedDocViewerServices: jest.Mocked<UnifiedDocViewerServices>
data: dataPluginMock.createStartContract(),
discoverShared: discoverSharedPluginMock.createStartContract(),
fieldFormats: fieldFormatsMock,
fieldsMetadata: fieldsMetadataPluginPublicMock.createStartContract(),
storage: new Storage(localStorage),
uiSettings: uiSettingsServiceMock.createStartContract(),
unifiedDocViewer: mockUnifiedDocViewer,

39
src/plugins/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview_highlights.tsx
View file

@ -13,6 +13,7 @@ import { i18n } from '@kbn/i18n';
import { DataTableRecord, LogDocumentOverview, fieldConstants } from '@kbn/discover-utils';
import { HighlightField } from './sub_components/highlight_field';
import { HighlightSection } from './sub_components/highlight_section';
import { getUnifiedDocViewerServices } from '../../plugin';
export function LogsOverviewHighlights({
formattedDoc,
@ -21,6 +22,30 @@ export function LogsOverviewHighlights({
formattedDoc: LogDocumentOverview;
flattenedDoc: DataTableRecord['flattened'];
}) {
const {
fieldsMetadata: { useFieldsMetadata },
} = getUnifiedDocViewerServices();
const { fieldsMetadata = {} } = useFieldsMetadata({
attributes: ['flat_name', 'short', 'type'],
fieldNames: [
fieldConstants.SERVICE_NAME_FIELD,
fieldConstants.HOST_NAME_FIELD,
fieldConstants.TRACE_ID_FIELD,
fieldConstants.ORCHESTRATOR_CLUSTER_NAME_FIELD,
fieldConstants.ORCHESTRATOR_RESOURCE_ID_FIELD,
fieldConstants.CLOUD_PROVIDER_FIELD,
fieldConstants.CLOUD_REGION_FIELD,
fieldConstants.CLOUD_AVAILABILITY_ZONE_FIELD,
fieldConstants.CLOUD_PROJECT_ID_FIELD,
fieldConstants.CLOUD_INSTANCE_ID_FIELD,
fieldConstants.LOG_FILE_PATH_FIELD,
fieldConstants.DATASTREAM_DATASET_FIELD,
fieldConstants.DATASTREAM_NAMESPACE_FIELD,
fieldConstants.AGENT_NAME_FIELD,
],
});
const getHighlightProps = (field: keyof LogDocumentOverview) => ({
field,
formattedValue: formattedDoc[field],
@ -38,6 +63,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewService"
label={serviceLabel}
fieldMetadata={fieldsMetadata[fieldConstants.SERVICE_NAME_FIELD]}
{...getHighlightProps(fieldConstants.SERVICE_NAME_FIELD)}
/>
)}
@ -45,6 +71,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewHostName"
label={hostNameLabel}
fieldMetadata={fieldsMetadata[fieldConstants.HOST_NAME_FIELD]}
{...getHighlightProps(fieldConstants.HOST_NAME_FIELD)}
/>
)}
@ -52,6 +79,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewTrace"
label={traceLabel}
fieldMetadata={fieldsMetadata[fieldConstants.TRACE_ID_FIELD]}
{...getHighlightProps(fieldConstants.TRACE_ID_FIELD)}
/>
)}
@ -59,6 +87,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewClusterName"
label={orchestratorClusterNameLabel}
fieldMetadata={fieldsMetadata[fieldConstants.ORCHESTRATOR_CLUSTER_NAME_FIELD]}
{...getHighlightProps(fieldConstants.ORCHESTRATOR_CLUSTER_NAME_FIELD)}
/>
)}
@ -66,6 +95,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewResourceId"
label={orchestratorResourceIdLabel}
fieldMetadata={fieldsMetadata[fieldConstants.ORCHESTRATOR_RESOURCE_ID_FIELD]}
{...getHighlightProps(fieldConstants.ORCHESTRATOR_RESOURCE_ID_FIELD)}
/>
)}
@ -79,6 +109,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewCloudProvider"
label={cloudProviderLabel}
fieldMetadata={fieldsMetadata[fieldConstants.CLOUD_PROVIDER_FIELD]}
icon={
<CloudProviderIcon
cloudProvider={first(
@ -93,6 +124,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewCloudRegion"
label={cloudRegionLabel}
fieldMetadata={fieldsMetadata[fieldConstants.CLOUD_REGION_FIELD]}
{...getHighlightProps(fieldConstants.CLOUD_REGION_FIELD)}
/>
)}
@ -100,6 +132,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewCloudAz"
label={cloudAvailabilityZoneLabel}
fieldMetadata={fieldsMetadata[fieldConstants.CLOUD_AVAILABILITY_ZONE_FIELD]}
{...getHighlightProps(fieldConstants.CLOUD_AVAILABILITY_ZONE_FIELD)}
/>
)}
@ -107,6 +140,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewCloudProjectId"
label={cloudProjectIdLabel}
fieldMetadata={fieldsMetadata[fieldConstants.CLOUD_PROJECT_ID_FIELD]}
{...getHighlightProps(fieldConstants.CLOUD_PROJECT_ID_FIELD)}
/>
)}
@ -114,6 +148,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewCloudInstanceId"
label={cloudInstanceIdLabel}
fieldMetadata={fieldsMetadata[fieldConstants.CLOUD_INSTANCE_ID_FIELD]}
{...getHighlightProps(fieldConstants.CLOUD_INSTANCE_ID_FIELD)}
/>
)}
@ -127,6 +162,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewLogPathFile"
label={logPathFileLabel}
fieldMetadata={fieldsMetadata[fieldConstants.LOG_FILE_PATH_FIELD]}
{...getHighlightProps(fieldConstants.LOG_FILE_PATH_FIELD)}
/>
)}
@ -134,6 +170,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewDataset"
label={datasetLabel}
fieldMetadata={fieldsMetadata[fieldConstants.DATASTREAM_DATASET_FIELD]}
{...getHighlightProps(fieldConstants.DATASTREAM_DATASET_FIELD)}
/>
)}
@ -141,6 +178,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewNamespace"
label={namespaceLabel}
fieldMetadata={fieldsMetadata[fieldConstants.DATASTREAM_NAMESPACE_FIELD]}
useBadge
{...getHighlightProps(fieldConstants.DATASTREAM_NAMESPACE_FIELD)}
/>
@ -149,6 +187,7 @@ export function LogsOverviewHighlights({
<HighlightField
data-test-subj="unifiedDocViewLogsOverviewLogShipper"
label={shipperLabel}
fieldMetadata={fieldsMetadata[fieldConstants.AGENT_NAME_FIELD]}
{...getHighlightProps(fieldConstants.AGENT_NAME_FIELD)}
/>
)}

7
src/plugins/unified_doc_viewer/public/components/doc_viewer_logs_overview/sub_components/highlight_field.tsx
View file

@ -11,12 +11,14 @@ import { css } from '@emotion/react';
import React, { ReactNode } from 'react';
import { dynamic } from '@kbn/shared-ux-utility';
import { euiThemeVars } from '@kbn/ui-theme';
import { PartialFieldMetadataPlain } from '@kbn/fields-metadata-plugin/common';
import { HoverActionPopover } from './hover_popover_action';
const HighlightFieldDescription = dynamic(() => import('./highlight_field_description'));
interface HighlightFieldProps {
field: string;
fieldMetadata?: PartialFieldMetadataPlain;
formattedValue?: string;
icon?: ReactNode;
label: string;
@ -26,6 +28,7 @@ interface HighlightFieldProps {
export function HighlightField({
field,
fieldMetadata,
formattedValue,
icon,
label,
@ -33,13 +36,15 @@ export function HighlightField({
value,
...props
}: HighlightFieldProps) {
const hasFieldDescription = !!fieldMetadata?.short;
return formattedValue && value ? (
<div {...props}>
<EuiFlexGroup responsive={false} alignItems="center" gutterSize="xs">
<EuiTitle css={fieldNameStyle} size="xxxs">
<span>{label}</span>
</EuiTitle>
<HighlightFieldDescription fieldName={field} />
{hasFieldDescription ? <HighlightFieldDescription fieldMetadata={fieldMetadata} /> : null}
</EuiFlexGroup>
<HoverActionPopover title={value} value={value} field={field}>
<EuiFlexGroup

12
src/plugins/unified_doc_viewer/public/components/doc_viewer_logs_overview/sub_components/highlight_field_description.tsx
View file

@ -7,14 +7,16 @@
*/
import { EuiFlexGroup, EuiIconTip } from '@elastic/eui';
import { EcsFlat } from '@elastic/ecs';
import { PartialFieldMetadataPlain } from '@kbn/fields-metadata-plugin/common';
import { FieldIcon } from '@kbn/react-field';
import React from 'react';
export function HighlightFieldDescription({ fieldName }: { fieldName: string }) {
const { short, type } = EcsFlat[fieldName as keyof typeof EcsFlat] ?? {};
if (!short) return null;
export function HighlightFieldDescription({
fieldMetadata,
}: {
fieldMetadata: PartialFieldMetadataPlain;
}) {
const { flat_name: fieldName, short, type } = fieldMetadata;
const title = (
<EuiFlexGroup alignItems="center" gutterSize="s">

5
src/plugins/unified_doc_viewer/public/plugin.tsx
View file

@ -18,6 +18,7 @@ import { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
import { CoreStart } from '@kbn/core/public';
import { dynamic } from '@kbn/shared-ux-utility';
import { DiscoverSharedPublicStart } from '@kbn/discover-shared-plugin/public';
import { FieldsMetadataPublicStart } from '@kbn/fields-metadata-plugin/public';
import type { UnifiedDocViewerServices } from './types';
export const [getUnifiedDocViewerServices, setUnifiedDocViewerServices] =
@ -50,6 +51,7 @@ export interface UnifiedDocViewerStartDeps {
data: DataPublicPluginStart;
discoverShared: DiscoverSharedPublicStart;
fieldFormats: FieldFormatsStart;
fieldsMetadata: FieldsMetadataPublicStart;
}
export class UnifiedDocViewerPublicPlugin
@ -118,7 +120,7 @@ export class UnifiedDocViewerPublicPlugin
public start(core: CoreStart, deps: UnifiedDocViewerStartDeps) {
const { analytics, uiSettings } = core;
const { data, discoverShared, fieldFormats } = deps;
const { data, discoverShared, fieldFormats, fieldsMetadata } = deps;
const storage = new Storage(localStorage);
const unifiedDocViewer = {
registry: this.docViewsRegistry,
@ -128,6 +130,7 @@ export class UnifiedDocViewerPublicPlugin
data,
discoverShared,
fieldFormats,
fieldsMetadata,
storage,
uiSettings,
unifiedDocViewer,

2
src/plugins/unified_doc_viewer/public/types.ts
View file

@ -14,6 +14,7 @@ import type { AnalyticsServiceStart } from '@kbn/core-analytics-browser';
import type { DataPublicPluginStart } from '@kbn/data-plugin/public';
import type { DiscoverSharedPublicStart } from '@kbn/discover-shared-plugin/public';
import type { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
import type { FieldsMetadataPublicStart } from '@kbn/fields-metadata-plugin/public';
import type { Storage } from '@kbn/kibana-utils-plugin/public';
import type { IUiSettingsClient } from '@kbn/core-ui-settings-browser';
import type { UnifiedDocViewerStart } from './plugin';
@ -23,6 +24,7 @@ export interface UnifiedDocViewerServices {
data: DataPublicPluginStart;
discoverShared: DiscoverSharedPublicStart;
fieldFormats: FieldFormatsStart;
fieldsMetadata: FieldsMetadataPublicStart;
storage: Storage;
uiSettings: IUiSettingsClient;
unifiedDocViewer: UnifiedDocViewerStart;

3
src/plugins/unified_doc_viewer/tsconfig.json
View file

@ -29,7 +29,8 @@
"@kbn/custom-icons",
"@kbn/react-field",
"@kbn/ui-theme",
"@kbn/discover-shared-plugin"
"@kbn/discover-shared-plugin",
"@kbn/fields-metadata-plugin"
],
"exclude": [
"target/**/*",

2
tsconfig.base.json
View file

@ -890,6 +890,8 @@
"@kbn/field-types/*": ["packages/kbn-field-types/*"],
"@kbn/field-utils": ["packages/kbn-field-utils"],
"@kbn/field-utils/*": ["packages/kbn-field-utils/*"],
"@kbn/fields-metadata-plugin": ["x-pack/plugins/fields_metadata"],
"@kbn/fields-metadata-plugin/*": ["x-pack/plugins/fields_metadata/*"],
"@kbn/file-upload-plugin": ["x-pack/plugins/file_upload"],
"@kbn/file-upload-plugin/*": ["x-pack/plugins/file_upload/*"],
"@kbn/files-example-plugin": ["examples/files_example"],

171
x-pack/plugins/fields_metadata/README.md Executable file
View file

@ -0,0 +1,171 @@
# Fields Metadata Plugin
The `@kbn/fields-metadata-plugin` is designed to provide a centralized and asynchronous way to consume field metadata across Kibana. This plugin addresses the need for on-demand retrieval of field metadata from static ECS definitions and integration manifests, with the flexibility to extend to additional resolution sources in the future.
## Components and Mechanisms
### FieldsMetadataService (Server-side)
The `FieldsMetadataService` is instantiated during the plugin setup/start lifecycle on the server side. It exposes a client that can be used to consume field metadata and provides tools for registering external dependencies.
#### Start Contract
The start contract exposes a `FieldsMetadataClient` instance, which offers the following methods:
- `getByName(name: string, params? {integration: string, dataset?: string})`: Retrieves a single `FieldMetadata` instance by name.
```ts
const timestampField = await client.getByName('@timestamp')
/*
{
dashed_name: 'timestamp',
type: 'date',
...
}
*/
```
- `find(params?: {fieldNames?: string[], integration?: string, dataset?: string})`: Retrieves a record of matching `FieldMetadata` instances based on the query parameters.
**Parameters**
| Name | Type | Example | Optional |
|---|---|---|---|
| fieldNames | <EcsFieldName \| string>[] | ['@timestamp', 'onepassword.client.platform_version'] | ✅ |
| integration | string | 1password | ✅ |
| dataset | string | 1password.item_usages | ✅ |
```ts
const fields = await client.find({
fieldNames: ['@timestamp', 'onepassword.client.platform_version'],
integration: '1password',
dataset: '*'
})
/*
{
'@timestamp': {
dashed_name: 'timestamp',
type: 'date',
...
},
'onepassword.client.platform_version': {
name: 'platform_version',
type: 'keyword',
...
},
}
*/
```
> N.B. Passing the `dataset` name parameter to `.find` helps narrowing the scope of the integration assets that need to be fetched, increasing the performance of the request.
In case the exact dataset for a field is unknown, is it still possible to pass a `*` value as `dataset` parameter to access all the integration datasets' fields.
Still, is recommended always passing the `dataset` as well if known or unless the required fields come from different datasets of the same integration.
> N.B. In case the `fieldNames` parameter is not passed to `.find`, the result will give the whole list of ECS fields by default. This should be avoided as much as possible, although it helps covering cases where we might need the whole ECS fields list.
#### Source Repositories
The `FieldsMetadataClient` relies on source repositories to fetch field metadata. Currently, there are two repository sources:
- `EcsFieldsRepository`: Fetches static ECS field metadata.
- `IntegrationFieldsRepository`: Fetches fields from an integration package from the Elastic Package Registry (EPR). This requires the `fleet` plugin to be enabled to access the registered fields extractor.
To improve performance, a caching layer is applied to the results retrieved from external sources, minimizing latency and enhancing efficiency.
### Fields Metadata API
A REST API endpoint is exposed to facilitate the retrieval of field metadata:
- `GET /internal/fields_metadata/find`: Supports query parameters to filter and find field metadata, optimizing the payload served to the client.
**Parameters**
| Name | Type | Example | Optional |
|---|---|---|---|
| fieldNames | <EcsFieldName \| string>[] | ['@timestamp', 'onepassword.client.platform_version'] | ✅ |
| attributes | FieldAttribute[] | ['type', 'description', 'name'] | ✅ |
| integration | string | 1password | ✅ |
| dataset | string | 1password.item_usages | ✅ |
### FieldsMetadataService (Client-side)
The client-side counterpart of the `FieldsMetadataService` ensures safe consumption of the exposed API and performs necessary validation steps. The client is returned by the public start contract of the plugin, allowing other parts of Kibana to use fields metadata directly.
With this client request/response validation, error handling and client-side caching are all handled out of the box.
Typical use cases for this client are integrating fields metadata on existing state management solutions or early metadata retrieval on initialization.
```ts
export class FieldsMetadataPlugin implements Plugin {
...
public start(core: CoreStart, plugins) {
const myFieldsMetadata = plugins.fieldsMetadata.client.find(/* ... */);
...
}
}
```
### useFieldsMetadata (React Hook)
For simpler use cases, the `useFieldsMetadata` React custom hook is provided. This hook is pre-configured with the required dependencies and allows quick access to field metadata client-side. It is essential to retrieve this hook from the start contract of the plugin to ensure proper dependency injection.
**Parameters**
| Name | Type | Example | Optional |
|---|---|---|---|
| fieldNames | <EcsFieldName \| string>[] | ['@timestamp', 'onepassword.client.platform_version'] | ✅ |
| attributes | FieldAttribute[] | ['type', 'description', 'name'] | ✅ |
| integration | string | 1password | ✅ |
| dataset | string | 1password.item_usages | ✅ |
It also accepts a second argument, an array of dependencies to determine when the hook should update the retrieved data.
```ts
const FieldsComponent = () => {
const {
fieldsMetadata: { useFieldsMetadata },
} = useServices(); // Or useKibana and any other utility to get the plugin deps
const { fieldsMetadata, error, loading } = useFieldsMetadata({
fieldsName: ['@timestamp', 'agent.name'],
attributes: ['name', 'type']
}, []);
if (loading) {
return <div>Loading...</div>;
}
if (error) {
return <div>Error: {error.message}</div>;
}
return (
<div>
{fieldsMetadata.map(field => (
<div key={field.name}>{field.name}: {field.type}</div>
))}
</div>
);
};
```
### registerIntegrationFieldsExtractor
To handle the complexity of fetching fields from an integration dataset, the `PackageService.prototype.getPackageFieldsMetadata()` method is implemented. This method maintains the separation of concerns and avoids direct dependency on the fleet plugin. During the fleet plugin setup, a `registerIntegrationFieldsExtractor` service is created to register a callback that retrieves fields by given parameters.
```ts
import { registerIntegrationFieldsExtractor } from '@kbn/fields-metadata-plugin/server';
registerIntegrationFieldsExtractor((params) => {
// Custom logic to retrieve fields from an integration
const fields = getFieldsFromIntegration(params);
return fields;
});
```
```ts
export class FleetPluginServer implements Plugin {
public setup(core: CoreStart, plugins) {
plugins.fieldsMetadata.registerIntegrationFieldsExtractor((params) => {
// Custom logic to retrieve fields from an integration
const fields = getFieldsFromIntegration(params);
return fields;
});
}
}
```

9
x-pack/plugins/fields_metadata/common/fields_metadata/common.ts Normal file
View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export const FIND_FIELDS_METADATA_URL = '/internal/fields_metadata';
export const ANY_DATASET = '*';

24
x-pack/plugins/fields_metadata/common/fields_metadata/errors.ts Normal file
View file

@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/* eslint-disable max-classes-per-file */
export class FetchFieldsMetadataError extends Error {
constructor(message: string, public cause?: Error) {
super(message);
Object.setPrototypeOf(this, new.target.prototype);
this.name = 'FetchFieldsMetadataError';
}
}
export class DecodeFieldsMetadataError extends Error {
constructor(message: string, public cause?: Error) {
super(message);
Object.setPrototypeOf(this, new.target.prototype);
this.name = 'DecodeFieldsMetadataError';
}
}

10
x-pack/plugins/fields_metadata/common/fields_metadata/index.ts Normal file
View file

@ -0,0 +1,10 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './common';
export * from './errors';
export * from './types';

53
x-pack/plugins/fields_metadata/common/fields_metadata/models/field_metadata.ts Normal file
View file

@ -0,0 +1,53 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
/* eslint-disable @typescript-eslint/naming-convention */
import pick from 'lodash/pick';
import { FieldAttribute, FieldMetadataPlain, PartialFieldMetadataPlain } from '../types';
// Use class/interface merging to define instance properties from FieldMetadataPlain.
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldMetadata extends FieldMetadataPlain {}
export class FieldMetadata {
private constructor(fieldMetadata: FieldMetadataPlain) {
Object.assign(this, fieldMetadata);
}
public pick(props: FieldAttribute[]): PartialFieldMetadataPlain {
return pick(this, props);
}
public toPlain(): FieldMetadataPlain {
return Object.assign({}, this);
}
public static create(fieldMetadata: FieldMetadataPlain) {
const flat_name = fieldMetadata.flat_name ?? fieldMetadata.name;
const dashed_name = fieldMetadata.dashed_name ?? FieldMetadata.toDashedName(flat_name);
const normalize = fieldMetadata.normalize ?? [];
const short = fieldMetadata.short ?? fieldMetadata.description;
const source = fieldMetadata.source ?? 'unknown';
const type = fieldMetadata.type ?? 'unknown';
const fieldMetadataProps = {
...fieldMetadata,
dashed_name,
flat_name,
normalize,
short,
source,
type,
};
return new FieldMetadata(fieldMetadataProps);
}
private static toDashedName(flatName: string) {
return flatName.split('.').join('-');
}
}

28
x-pack/plugins/fields_metadata/common/fields_metadata/models/fields_metadata_dictionary.ts Normal file
View file

@ -0,0 +1,28 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import mapValues from 'lodash/mapValues';
import { FieldAttribute, FieldMetadataPlain, PartialFieldMetadataPlain } from '../types';
import { FieldMetadata } from './field_metadata';
export type FieldsMetadataMap = Record<string, FieldMetadata>;
export class FieldsMetadataDictionary {
private constructor(private readonly fields: FieldsMetadataMap) {}
pick(attributes: FieldAttribute[]): Record<string, PartialFieldMetadataPlain> {
return mapValues(this.fields, (field) => field.pick(attributes));
}
toPlain(): Record<string, FieldMetadataPlain> {
return mapValues(this.fields, (field) => field.toPlain());
}
public static create(fields: FieldsMetadataMap) {
return new FieldsMetadataDictionary(fields);
}
}

91
x-pack/plugins/fields_metadata/common/fields_metadata/types.ts Normal file
View file

@ -0,0 +1,91 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { EcsFlat } from '@elastic/ecs';
import * as rt from 'io-ts';
export const fieldSourceRT = rt.keyof({
ecs: null,
integration: null,
unknown: null,
});
export const allowedValueRT = rt.intersection([
rt.type({
description: rt.string,
name: rt.string,
}),
rt.partial({
expected_event_types: rt.array(rt.string),
beta: rt.string,
}),
]);
export const multiFieldRT = rt.type({
flat_name: rt.string,
name: rt.string,
type: rt.string,
});
const requiredBaseMetadataPlainRT = rt.type({
name: rt.string,
});
const optionalBaseMetadataPlainRT = rt.partial(requiredBaseMetadataPlainRT.props);
const optionalMetadataPlainRT = rt.partial({
allowed_values: rt.array(allowedValueRT),
beta: rt.string,
dashed_name: rt.string,
description: rt.string,
doc_values: rt.boolean,
example: rt.unknown,
expected_values: rt.array(rt.string),
flat_name: rt.string,
format: rt.string,
ignore_above: rt.number,
index: rt.boolean,
input_format: rt.string,
level: rt.string,
multi_fields: rt.array(multiFieldRT),
normalize: rt.array(rt.string),
object_type: rt.string,
original_fieldset: rt.string,
output_format: rt.string,
output_precision: rt.number,
pattern: rt.string,
required: rt.boolean,
scaling_factor: rt.number,
short: rt.string,
source: fieldSourceRT,
type: rt.string,
});
export const partialFieldMetadataPlainRT = rt.intersection([
optionalBaseMetadataPlainRT,
optionalMetadataPlainRT,
]);
export const fieldMetadataPlainRT = rt.intersection([
requiredBaseMetadataPlainRT,
optionalMetadataPlainRT,
]);
export const fieldAttributeRT = rt.union([
rt.keyof(requiredBaseMetadataPlainRT.props),
rt.keyof(optionalMetadataPlainRT.props),
]);
export type TEcsFields = typeof EcsFlat;
export type EcsFieldName = keyof TEcsFields;
export type IntegrationFieldName = string;
export type FieldName = EcsFieldName | (IntegrationFieldName & {});
export type FieldMetadataPlain = rt.TypeOf<typeof fieldMetadataPlainRT>;
export type PartialFieldMetadataPlain = rt.TypeOf<typeof partialFieldMetadataPlainRT>;
export type FieldAttribute = rt.TypeOf<typeof fieldAttributeRT>;

65
x-pack/plugins/fields_metadata/common/fields_metadata/v1/find_fields_metadata.ts Normal file
View file

@ -0,0 +1,65 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { arrayToStringRt } from '@kbn/io-ts-utils';
import { either } from 'fp-ts/lib/Either';
import * as rt from 'io-ts';
import { ANY_DATASET } from '../common';
import { FetchFieldsMetadataError } from '../errors';
import { FieldAttribute, fieldAttributeRT, FieldName, partialFieldMetadataPlainRT } from '../types';
const baseFindFieldsMetadataRequestQueryRT = rt.exact(
rt.partial({
attributes: arrayToStringRt.pipe(rt.array(fieldAttributeRT)),
fieldNames: arrayToStringRt.pipe(rt.array(rt.string)),
integration: rt.string,
dataset: rt.string,
})
);
// Define a refinement that enforces the constraint
export const findFieldsMetadataRequestQueryRT = new rt.Type(
'FindFieldsMetadataRequestQuery',
(query): query is rt.TypeOf<typeof baseFindFieldsMetadataRequestQueryRT> =>
baseFindFieldsMetadataRequestQueryRT.is(query) &&
(query.integration ? query.dataset !== undefined : true),
(input, context) =>
either.chain(baseFindFieldsMetadataRequestQueryRT.validate(input, context), (query) => {
try {
if (query.integration && !query.dataset) {
throw new FetchFieldsMetadataError('dataset is required if integration is provided');
}
return rt.success(query);
} catch (error) {
return rt.failure(query, context, error.message);
}
}),
baseFindFieldsMetadataRequestQueryRT.encode
);
export const findFieldsMetadataResponsePayloadRT = rt.type({
fields: rt.record(rt.string, partialFieldMetadataPlainRT),
});
export type FindFieldsMetadataRequestQuery =
| {
attributes?: FieldAttribute[];
fieldNames?: FieldName[];
integration?: undefined;
dataset?: undefined;
}
| {
attributes?: FieldAttribute[];
fieldNames?: FieldName[];
integration: string;
dataset: typeof ANY_DATASET | (string & {});
};
export type FindFieldsMetadataResponsePayload = rt.TypeOf<
typeof findFieldsMetadataResponsePayloadRT
>;

8
x-pack/plugins/fields_metadata/common/fields_metadata/v1/index.ts Normal file
View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './find_fields_metadata';

46
x-pack/plugins/fields_metadata/common/hashed_cache.ts Normal file
View file

@ -0,0 +1,46 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import LRUCache from 'lru-cache';
import hash from 'object-hash';
export interface IHashedCache<KeyType, ValueType> {
get(key: KeyType): ValueType | undefined;
set(key: KeyType, value: ValueType): boolean;
has(key: KeyType): boolean;
reset(): void;
}
export class HashedCache<KeyType, ValueType> {
private cache: LRUCache<string, ValueType>;
constructor(options: LRUCache.Options<string, ValueType> = { max: 500 }) {
this.cache = new LRUCache<string, ValueType>(options);
}
public get(key: KeyType): ValueType | undefined {
const serializedKey = this.getHashedKey(key);
return this.cache.get(serializedKey);
}
public set(key: KeyType, value: ValueType) {
const serializedKey = this.getHashedKey(key);
return this.cache.set(serializedKey, value);
}
public has(key: KeyType): boolean {
const serializedKey = this.getHashedKey(key);
return this.cache.has(serializedKey);
}
public reset() {
return this.cache.reset();
}
private getHashedKey(key: KeyType) {
return hash(key, { unorderedArrays: true });
}
}

20
x-pack/plugins/fields_metadata/common/index.ts Normal file
View file

@ -0,0 +1,20 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { fieldMetadataPlainRT } from './fields_metadata/types';
export type {
EcsFieldName,
FieldAttribute,
FieldMetadataPlain,
FieldName,
IntegrationFieldName,
PartialFieldMetadataPlain,
TEcsFields,
} from './fields_metadata/types';
export { FieldMetadata } from './fields_metadata/models/field_metadata';
export { FieldsMetadataDictionary } from './fields_metadata/models/fields_metadata_dictionary';

8
x-pack/plugins/fields_metadata/common/latest.ts Normal file
View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './fields_metadata/v1';

29
x-pack/plugins/fields_metadata/common/runtime_types.ts Normal file
View file

@ -0,0 +1,29 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { RouteValidationFunction } from '@kbn/core/server';
import { createPlainError, decodeOrThrow, formatErrors, throwErrors } from '@kbn/io-ts-utils';
import { fold } from 'fp-ts/lib/Either';
import { pipe } from 'fp-ts/lib/pipeable';
import { Errors, Type } from 'io-ts';
export { createPlainError, decodeOrThrow, formatErrors, throwErrors };
type ValdidationResult<Value> = ReturnType<RouteValidationFunction<Value>>;
export const createValidationFunction =
<DecodedValue, EncodedValue, InputValue>(
runtimeType: Type<DecodedValue, EncodedValue, InputValue>
): RouteValidationFunction<DecodedValue> =>
(inputValue, { badRequest, ok }) =>
pipe(
runtimeType.decode(inputValue),
fold<Errors, DecodedValue, ValdidationResult<DecodedValue>>(
(errors: Errors) => badRequest(formatErrors(errors)),
(result: DecodedValue) => ok(result)
)
);

17
x-pack/plugins/fields_metadata/jest.config.js Normal file
View file

@ -0,0 +1,17 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
module.exports = {
preset: '@kbn/test',
rootDir: '../../..',
roots: ['<rootDir>/x-pack/plugins/fields_metadata'],
coverageDirectory: '<rootDir>/target/kibana-coverage/jest/x-pack/plugins/fields_metadata',
coverageReporters: ['text', 'html'],
collectCoverageFrom: [
'<rootDir>/x-pack/plugins/fields_metadata/{common,public,server}/**/*.{ts,tsx}',
],
};

14
x-pack/plugins/fields_metadata/kibana.jsonc Normal file
View file

@ -0,0 +1,14 @@
{
"type": "plugin",
"id": "@kbn/fields-metadata-plugin",
"owner": "@elastic/obs-ux-logs-team",
"description": "Exposes services for async usage and search of fields metadata.",
"plugin": {
"id": "fieldsMetadata",
"server": true,
"browser": true,
"configPath": ["xpack", "fields_metadata"],
"requiredPlugins": [],
"requiredBundles": [],
}
}

8
x-pack/plugins/fields_metadata/public/hooks/use_fields_metadata/index.ts Normal file
View file

@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './use_fields_metadata';

16
x-pack/plugins/fields_metadata/public/hooks/use_fields_metadata/use_fields_metadata.mock.ts Normal file
View file

@ -0,0 +1,16 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { UseFieldsMetadataHook } from './use_fields_metadata';
export const createUseFieldsMetadataHookMock = (): jest.Mocked<UseFieldsMetadataHook> =>
jest.fn(() => ({
fieldsMetadata: undefined,
loading: false,
error: undefined,
reload: jest.fn(),
}));

88
x-pack/plugins/fields_metadata/public/hooks/use_fields_metadata/use_fields_metadata.test.ts Normal file
View file

@ -0,0 +1,88 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { renderHook } from '@testing-library/react-hooks';
import { createUseFieldsMetadataHook, UseFieldsMetadataParams } from './use_fields_metadata';
import { FindFieldsMetadataResponsePayload } from '../../../common/latest';
import { createFieldsMetadataServiceStartMock } from '../../services/fields_metadata/fields_metadata_service.mock';
import { IFieldsMetadataClient } from '../../services/fields_metadata';
const fields: FindFieldsMetadataResponsePayload['fields'] = {
'@timestamp': {
dashed_name: 'timestamp',
description:
'Date/time when the event originated.\nThis is the date/time extracted from the event, typically representing when the event was generated by the source.\nIf the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.\nRequired field for all events.',
example: '2016-05-23T08:05:34.853Z',
flat_name: '@timestamp',
level: 'core',
name: '@timestamp',
normalize: [],
short: 'Date/time when the event originated.',
type: 'date',
source: 'ecs',
},
};
const mockedFieldsMetadataResponse = { fields };
const fieldsMetadataService = createFieldsMetadataServiceStartMock();
const useFieldsMetadata = createUseFieldsMetadataHook({ fieldsMetadataService });
describe('useFieldsMetadata', () => {
let fieldsMetadataClient: jest.Mocked<IFieldsMetadataClient>;
beforeEach(async () => {
fieldsMetadataClient = await fieldsMetadataService.getClient();
});
afterEach(() => {
jest.clearAllMocks();
});
it('should return the fieldsMetadata value from the API', async () => {
fieldsMetadataClient.find.mockResolvedValue(mockedFieldsMetadataResponse);
const { result, waitForNextUpdate } = renderHook(() => useFieldsMetadata());
expect(result.current.loading).toBe(true);
expect(result.current.fieldsMetadata).toEqual(undefined);
await waitForNextUpdate();
const { fieldsMetadata, loading, error } = result.current;
expect(fieldsMetadata).toEqual(fields);
expect(loading).toBeFalsy();
expect(error).toBeFalsy();
});
it('should call the fieldsMetadata service with the passed parameters', async () => {
fieldsMetadataClient.find.mockResolvedValue(mockedFieldsMetadataResponse);
const params: UseFieldsMetadataParams = {
attributes: ['description', 'short'],
fieldNames: ['@timestamp', 'agent.name'],
integration: 'integration_name',
dataset: 'dataset_name',
};
const { waitForNextUpdate } = renderHook(() => useFieldsMetadata(params));
await waitForNextUpdate();
expect(fieldsMetadataClient.find).toHaveBeenCalledWith(params);
});
it('should return an error if the API call fails', async () => {
const error = new Error('Fetch fields metadata Failed');
fieldsMetadataClient.find.mockRejectedValueOnce(error);
const { result, waitForNextUpdate } = renderHook(() => useFieldsMetadata());
await waitForNextUpdate();
expect(result.current.error?.message).toMatch(error.message);
});
});

49
x-pack/plugins/fields_metadata/public/hooks/use_fields_metadata/use_fields_metadata.ts Normal file
View file

@ -0,0 +1,49 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { useEffect } from 'react';
import useAsyncFn from 'react-use/lib/useAsyncFn';
import {
FindFieldsMetadataRequestQuery,
FindFieldsMetadataResponsePayload,
} from '../../../common/latest';
import { FieldsMetadataServiceStart } from '../../services/fields_metadata';
interface UseFieldsMetadataFactoryDeps {
fieldsMetadataService: FieldsMetadataServiceStart;
}
export type UseFieldsMetadataParams = FindFieldsMetadataRequestQuery;
export interface UseFieldsMetadataReturnType {
fieldsMetadata: FindFieldsMetadataResponsePayload['fields'] | undefined;
loading: boolean;
error: Error | undefined;
reload: ReturnType<typeof useAsyncFn>[1];
}
export type UseFieldsMetadataHook = (
params?: UseFieldsMetadataParams,
deps?: Parameters<typeof useAsyncFn>[1]
) => UseFieldsMetadataReturnType;
export const createUseFieldsMetadataHook = ({
fieldsMetadataService,
}: UseFieldsMetadataFactoryDeps): UseFieldsMetadataHook => {
return (params = {}, deps) => {
const [{ error, loading, value }, load] = useAsyncFn(async () => {
const client = await fieldsMetadataService.getClient();
return client.find(params);
}, deps);
useEffect(() => {
load();
}, [load]);
return { fieldsMetadata: value?.fields, loading, error, reload: load };
};
};

19
x-pack/plugins/fields_metadata/public/index.ts Normal file
View file

@ -0,0 +1,19 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FieldsMetadataPlugin } from './plugin';
export function plugin() {
return new FieldsMetadataPlugin();
}
export type {
FieldsMetadataPublicSetup,
FieldsMetadataPublicStart,
FieldsMetadataPublicSetupDeps,
FieldsMetadataPublicStartDeps,
} from './types';

18
x-pack/plugins/fields_metadata/public/mocks.tsx Normal file
View file

@ -0,0 +1,18 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { createUseFieldsMetadataHookMock } from './hooks/use_fields_metadata/use_fields_metadata.mock';
import { createFieldsMetadataServiceStartMock } from './services/fields_metadata/fields_metadata_service.mock';
const createFieldsMetadataPublicStartMock = () => ({
getClient: createFieldsMetadataServiceStartMock().getClient,
useFieldsMetadata: createUseFieldsMetadataHookMock(),
});
export const fieldsMetadataPluginPublicMock = {
createStartContract: createFieldsMetadataPublicStartMock,
};

38
x-pack/plugins/fields_metadata/public/plugin.ts Normal file
View file

@ -0,0 +1,38 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { CoreStart } from '@kbn/core/public';
import { createUseFieldsMetadataHook } from './hooks/use_fields_metadata';
import { FieldsMetadataService } from './services/fields_metadata';
import { FieldsMetadataClientPluginClass } from './types';
export class FieldsMetadataPlugin implements FieldsMetadataClientPluginClass {
private fieldsMetadata: FieldsMetadataService;
constructor() {
this.fieldsMetadata = new FieldsMetadataService();
}
public setup() {
this.fieldsMetadata.setup();
return {};
}
public start(core: CoreStart) {
const { http } = core;
const fieldsMetadataService = this.fieldsMetadata.start({ http });
const useFieldsMetadata = createUseFieldsMetadataHook({ fieldsMetadataService });
return {
getClient: fieldsMetadataService.getClient,
useFieldsMetadata,
};
}
}

12
x-pack/plugins/fields_metadata/public/services/fields_metadata/fields_metadata_client.mock.ts Normal file
View file

@ -0,0 +1,12 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { IFieldsMetadataClient } from './types';
export const createFieldsMetadataClientMock = (): jest.Mocked<IFieldsMetadataClient> => ({
find: jest.fn(),
});

74
x-pack/plugins/fields_metadata/public/services/fields_metadata/fields_metadata_client.ts Normal file
View file

@ -0,0 +1,74 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { HttpStart } from '@kbn/core/public';
import { HashedCache } from '../../../common/hashed_cache';
import {
FindFieldsMetadataRequestQuery,
findFieldsMetadataRequestQueryRT,
FindFieldsMetadataResponsePayload,
findFieldsMetadataResponsePayloadRT,
} from '../../../common/latest';
import {
DecodeFieldsMetadataError,
FetchFieldsMetadataError,
FieldName,
FIND_FIELDS_METADATA_URL,
} from '../../../common/fields_metadata';
import { decodeOrThrow } from '../../../common/runtime_types';
import { IFieldsMetadataClient } from './types';
export class FieldsMetadataClient implements IFieldsMetadataClient {
private cache: HashedCache<FindFieldsMetadataRequestQuery, FindFieldsMetadataResponsePayload>;
constructor(private readonly http: HttpStart) {
this.cache = new HashedCache();
}
public async find(
params: FindFieldsMetadataRequestQuery
): Promise<FindFieldsMetadataResponsePayload> {
// Initially lookup for existing results given request parameters
if (this.cache.has(params)) {
return this.cache.get(params) as FindFieldsMetadataResponsePayload;
}
const query = findFieldsMetadataRequestQueryRT.encode(params);
const response = await this.http
.get(FIND_FIELDS_METADATA_URL, { query, version: '1' })
.catch((error) => {
throw new FetchFieldsMetadataError(
`Failed to fetch fields ${truncateFieldNamesList(params.fieldNames)}: ${error.message}`
);
});
const data = decodeOrThrow(
findFieldsMetadataResponsePayloadRT,
(message: string) =>
new DecodeFieldsMetadataError(
`Failed decoding fields ${truncateFieldNamesList(params.fieldNames)}: ${message}`
)
)(response);
// Store cached results for given request parameters
this.cache.set(params, data);
return data;
}
}
const truncateFieldNamesList = (fieldNames?: FieldName[]) => {
if (!fieldNames || fieldNames.length === 0) return '';
const visibleFields = fieldNames.slice(0, 3);
const additionalFieldsCount = fieldNames.length - visibleFields.length;
return visibleFields
.join()
.concat(additionalFieldsCount > 0 ? `+${additionalFieldsCount} fields` : '');
};

18
x-pack/plugins/fields_metadata/public/services/fields_metadata/fields_metadata_service.mock.ts Normal file
View file

@ -0,0 +1,18 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { createFieldsMetadataClientMock } from './fields_metadata_client.mock';
import { IFieldsMetadataClient } from './types';
interface FieldsMetadataServiceStartMock {
getClient: () => Promise<jest.Mocked<IFieldsMetadataClient>>;
}
export const createFieldsMetadataServiceStartMock =
(): jest.Mocked<FieldsMetadataServiceStartMock> => ({
getClient: jest.fn().mockResolvedValue(createFieldsMetadataClientMock()),
});

37
x-pack/plugins/fields_metadata/public/services/fields_metadata/fields_metadata_service.ts Normal file
View file

@ -0,0 +1,37 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type {
FieldsMetadataServiceStartDeps,
FieldsMetadataServiceSetup,
FieldsMetadataServiceStart,
IFieldsMetadataClient,
} from './types';
export class FieldsMetadataService {
private client?: IFieldsMetadataClient;
public setup(): FieldsMetadataServiceSetup {
return {};
}
public start({ http }: FieldsMetadataServiceStartDeps): FieldsMetadataServiceStart {
return {
getClient: () => this.getClient({ http }),
};
}
private async getClient({ http }: FieldsMetadataServiceStartDeps) {
if (!this.client) {
const { FieldsMetadataClient } = await import('./fields_metadata_client');
const client = new FieldsMetadataClient(http);
this.client = client;
}
return this.client;
}
}

9
x-pack/plugins/fields_metadata/public/services/fields_metadata/index.ts Normal file
View file

@ -0,0 +1,9 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export * from './fields_metadata_service';
export * from './types';

27
x-pack/plugins/fields_metadata/public/services/fields_metadata/types.ts Normal file
View file

@ -0,0 +1,27 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { HttpStart } from '@kbn/core/public';
import {
FindFieldsMetadataRequestQuery,
FindFieldsMetadataResponsePayload,
} from '../../../common/latest';
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataServiceSetup {}
export interface FieldsMetadataServiceStart {
getClient: () => Promise<IFieldsMetadataClient>;
}
export interface FieldsMetadataServiceStartDeps {
http: HttpStart;
}
export interface IFieldsMetadataClient {
find(params: FindFieldsMetadataRequestQuery): Promise<FindFieldsMetadataResponsePayload>;
}

41
x-pack/plugins/fields_metadata/public/types.ts Normal file
View file

@ -0,0 +1,41 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { CoreSetup, CoreStart, Plugin as PluginClass } from '@kbn/core/public';
import type { UseFieldsMetadataHook } from './hooks/use_fields_metadata/use_fields_metadata';
import type { FieldsMetadataServiceStart } from './services/fields_metadata';
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataPublicSetup {}
export interface FieldsMetadataPublicStart {
getClient: FieldsMetadataServiceStart['getClient'];
useFieldsMetadata: UseFieldsMetadataHook;
}
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataPublicSetupDeps {}
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataPublicStartDeps {}
export type FieldsMetadataClientCoreSetup = CoreSetup<
FieldsMetadataPublicStartDeps,
FieldsMetadataPublicStart
>;
export type FieldsMetadataClientCoreStart = CoreStart;
export type FieldsMetadataClientPluginClass = PluginClass<
FieldsMetadataPublicSetup,
FieldsMetadataPublicStart,
FieldsMetadataPublicSetupDeps,
FieldsMetadataPublicStartDeps
>;
export type FieldsMetadataPublicStartServicesAccessor =
FieldsMetadataClientCoreSetup['getStartServices'];
export type FieldsMetadataPublicStartServices =
ReturnType<FieldsMetadataPublicStartServicesAccessor>;

13
x-pack/plugins/fields_metadata/server/fields_metadata_server.ts Normal file
View file

@ -0,0 +1,13 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FieldsMetadataBackendLibs } from './lib/shared_types';
import { initFieldsMetadataRoutes } from './routes/fields_metadata';
export const initFieldsMetadataServer = (libs: FieldsMetadataBackendLibs) => {
initFieldsMetadataRoutes(libs);
};

21
x-pack/plugins/fields_metadata/server/index.ts Normal file
View file

@ -0,0 +1,21 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { PluginInitializerContext } from '@kbn/core/server';
export type { FieldsMetadataServerSetup, FieldsMetadataServerStart } from './types';
export type {
IntegrationName,
DatasetName,
ExtractedIntegrationFields,
ExtractedDatasetFields,
} from './services/fields_metadata/types';
export async function plugin(context: PluginInitializerContext) {
const { FieldsMetadataPlugin } = await import('./plugin');
return new FieldsMetadataPlugin(context);
}

21
x-pack/plugins/fields_metadata/server/lib/shared_types.ts Normal file
View file

@ -0,0 +1,21 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { Logger } from '@kbn/logging';
import { RequestHandlerContext } from '@kbn/core-http-request-handler-context-server';
import { IRouter } from '@kbn/core-http-server';
import {
FieldsMetadataPluginStartServicesAccessor,
FieldsMetadataServerPluginSetupDeps,
} from '../types';
export interface FieldsMetadataBackendLibs {
getStartServices: FieldsMetadataPluginStartServicesAccessor;
logger: Logger;
plugins: FieldsMetadataServerPluginSetupDeps;
router: IRouter<RequestHandlerContext>;
}

26
x-pack/plugins/fields_metadata/server/mocks.ts Normal file
View file

@ -0,0 +1,26 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import {
createFieldsMetadataServiceSetupMock,
createFieldsMetadataServiceStartMock,
} from './services/fields_metadata/fields_metadata_service.mock';
import { FieldsMetadataServerSetup, FieldsMetadataServerStart } from './types';
const createFieldsMetadataServerSetupMock = (): jest.Mocked<FieldsMetadataServerSetup> => ({
registerIntegrationFieldsExtractor:
createFieldsMetadataServiceSetupMock().registerIntegrationFieldsExtractor,
});
const createFieldsMetadataServerStartMock = (): jest.Mocked<FieldsMetadataServerStart> => ({
getClient: createFieldsMetadataServiceStartMock().getClient,
});
export const fieldsMetadataPluginServerMock = {
createSetupContract: createFieldsMetadataServerSetupMock,
createStartContract: createFieldsMetadataServerStartMock,
};

63
x-pack/plugins/fields_metadata/server/plugin.ts Normal file
View file

@ -0,0 +1,63 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { PluginInitializerContext, CoreStart, Plugin, Logger } from '@kbn/core/server';
import {
FieldsMetadataPluginCoreSetup,
FieldsMetadataServerSetup,
FieldsMetadataServerStart,
FieldsMetadataServerPluginSetupDeps,
FieldsMetadataServerPluginStartDeps,
} from './types';
import { initFieldsMetadataServer } from './fields_metadata_server';
import { FieldsMetadataService } from './services/fields_metadata';
import { FieldsMetadataBackendLibs } from './lib/shared_types';
export class FieldsMetadataPlugin
implements
Plugin<
FieldsMetadataServerSetup,
FieldsMetadataServerStart,
FieldsMetadataServerPluginSetupDeps,
FieldsMetadataServerPluginStartDeps
>
{
private readonly logger: Logger;
private libs!: FieldsMetadataBackendLibs;
private fieldsMetadataService: FieldsMetadataService;
constructor(context: PluginInitializerContext) {
this.logger = context.logger.get();
this.fieldsMetadataService = new FieldsMetadataService(this.logger);
}
public setup(core: FieldsMetadataPluginCoreSetup, plugins: FieldsMetadataServerPluginSetupDeps) {
const fieldsMetadata = this.fieldsMetadataService.setup();
this.libs = {
getStartServices: () => core.getStartServices(),
logger: this.logger,
plugins,
router: core.http.createRouter(),
};
// Register server side APIs
initFieldsMetadataServer(this.libs);
return {
registerIntegrationFieldsExtractor: fieldsMetadata.registerIntegrationFieldsExtractor,
};
}
public start(_core: CoreStart, _plugins: FieldsMetadataServerPluginStartDeps) {
const fieldsMetadata = this.fieldsMetadataService.start();
return { getClient: fieldsMetadata.getClient };
}
}

75
x-pack/plugins/fields_metadata/server/routes/fields_metadata/find_fields_metadata.ts Normal file
View file

@ -0,0 +1,75 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { createValidationFunction } from '../../../common/runtime_types';
import { FIND_FIELDS_METADATA_URL } from '../../../common/fields_metadata';
import * as fieldsMetadataV1 from '../../../common/fields_metadata/v1';
import { FieldsMetadataBackendLibs } from '../../lib/shared_types';
import { FindFieldsMetadataResponsePayload } from '../../../common/fields_metadata/v1';
import { PackageNotFoundError } from '../../services/fields_metadata/errors';
export const initFindFieldsMetadataRoute = ({
router,
getStartServices,
}: FieldsMetadataBackendLibs) => {
router.versioned
.get({
access: 'internal',
path: FIND_FIELDS_METADATA_URL,
})
.addVersion(
{
version: '1',
validate: {
request: {
query: createValidationFunction(fieldsMetadataV1.findFieldsMetadataRequestQueryRT),
},
},
},
async (_requestContext, request, response) => {
const { attributes, fieldNames, integration, dataset } = request.query;
const [_core, _startDeps, startContract] = await getStartServices();
const fieldsMetadataClient = startContract.getClient();
try {
const fieldsDictionary = await fieldsMetadataClient.find({
fieldNames,
integration,
dataset,
});
const responsePayload: FindFieldsMetadataResponsePayload = { fields: {} };
if (attributes) {
responsePayload.fields = fieldsDictionary.pick(attributes);
} else {
responsePayload.fields = fieldsDictionary.toPlain();
}
return response.ok({
body: fieldsMetadataV1.findFieldsMetadataResponsePayloadRT.encode(responsePayload),
});
} catch (error) {
if (error instanceof PackageNotFoundError) {
return response.badRequest({
body: {
message: error.message,
},
});
}
return response.customError({
statusCode: error.statusCode ?? 500,
body: {
message: error.message ?? 'An unexpected error occurred',
},
});
}
}
);
};

12
x-pack/plugins/fields_metadata/server/routes/fields_metadata/index.ts Normal file
View file

@ -0,0 +1,12 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FieldsMetadataBackendLibs } from '../../lib/shared_types';
import { initFindFieldsMetadataRoute } from './find_fields_metadata';
export const initFieldsMetadataRoutes = (libs: FieldsMetadataBackendLibs) => {
initFindFieldsMetadataRoute(libs);
};

14
x-pack/plugins/fields_metadata/server/services/fields_metadata/errors.ts Normal file
View file

@ -0,0 +1,14 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export class PackageNotFoundError extends Error {
constructor(message: string, public cause?: Error) {
super(message);
Object.setPrototypeOf(this, new.target.prototype);
this.name = 'PackageNotFoundError';
}
}

13
x-pack/plugins/fields_metadata/server/services/fields_metadata/fields_metadata_client.mock.ts Normal file
View file

@ -0,0 +1,13 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { IFieldsMetadataClient } from './types';
export const createFieldsMetadataClientMock = (): jest.Mocked<IFieldsMetadataClient> => ({
getByName: jest.fn(),
find: jest.fn(),
});

171
x-pack/plugins/fields_metadata/server/services/fields_metadata/fields_metadata_client.test.ts Normal file
View file

@ -0,0 +1,171 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FieldMetadata, TEcsFields } from '../../../common';
import { loggerMock } from '@kbn/logging-mocks';
import { FieldsMetadataClient } from './fields_metadata_client';
import { EcsFieldsRepository } from './repositories/ecs_fields_repository';
import { IntegrationFieldsRepository } from './repositories/integration_fields_repository';
const ecsFields = {
'@timestamp': {
dashed_name: 'timestamp',
description:
'Date/time when the event originated.\nThis is the date/time extracted from the event, typically representing when the event was generated by the source.\nIf the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline.\nRequired field for all events.',
example: '2016-05-23T08:05:34.853Z',
flat_name: '@timestamp',
level: 'core',
name: '@timestamp',
normalize: [],
required: !0,
short: 'Date/time when the event originated.',
type: 'date',
},
} as TEcsFields;
const integrationFields = {
'1password.item_usages': {
'onepassword.client.platform_version': {
name: 'platform_version',
type: 'keyword',
description:
'The version of the browser or computer where the 1Password app is installed, or the CPU of the machine where the 1Password command-line tool is installed',
flat_name: 'onepassword.client.platform_version',
source: 'integration',
dashed_name: 'onepassword-client-platform_version',
normalize: [],
short:
'The version of the browser or computer where the 1Password app is installed, or the CPU of the machine where the 1Password command-line tool is installed',
},
},
};
describe('FieldsMetadataClient class', () => {
const logger = loggerMock.create();
const ecsFieldsRepository = EcsFieldsRepository.create({ ecsFields });
const integrationFieldsExtractor = jest.fn();
integrationFieldsExtractor.mockImplementation(() => Promise.resolve(integrationFields));
let integrationFieldsRepository: IntegrationFieldsRepository;
let fieldsMetadataClient: FieldsMetadataClient;
beforeEach(() => {
integrationFieldsExtractor.mockClear();
integrationFieldsRepository = IntegrationFieldsRepository.create({
integrationFieldsExtractor,
});
fieldsMetadataClient = FieldsMetadataClient.create({
ecsFieldsRepository,
integrationFieldsRepository,
logger,
});
});
describe('#getByName', () => {
it('should resolve a single ECS FieldMetadata instance by default', async () => {
const timestampFieldInstance = await fieldsMetadataClient.getByName('@timestamp');
expect(integrationFieldsExtractor).not.toHaveBeenCalled();
expectToBeDefined(timestampFieldInstance);
expect(timestampFieldInstance).toBeInstanceOf(FieldMetadata);
const timestampField = timestampFieldInstance.toPlain();
expect(timestampField.hasOwnProperty('dashed_name')).toBeTruthy();
expect(timestampField.hasOwnProperty('description')).toBeTruthy();
expect(timestampField.hasOwnProperty('example')).toBeTruthy();
expect(timestampField.hasOwnProperty('flat_name')).toBeTruthy();
expect(timestampField.hasOwnProperty('level')).toBeTruthy();
expect(timestampField.hasOwnProperty('name')).toBeTruthy();
expect(timestampField.hasOwnProperty('normalize')).toBeTruthy();
expect(timestampField.hasOwnProperty('required')).toBeTruthy();
expect(timestampField.hasOwnProperty('short')).toBeTruthy();
expect(timestampField.hasOwnProperty('type')).toBeTruthy();
});
it('should attempt resolving the field from an integration if it does not exist in ECS and the integration and dataset params are provided', async () => {
const onePasswordFieldInstance = await fieldsMetadataClient.getByName(
'onepassword.client.platform_version',
{ integration: '1password', dataset: '1password.item_usages' }
);
expect(integrationFieldsExtractor).toHaveBeenCalled();
expectToBeDefined(onePasswordFieldInstance);
expect(onePasswordFieldInstance).toBeInstanceOf(FieldMetadata);
const onePasswordField = onePasswordFieldInstance.toPlain();
expect(onePasswordField.hasOwnProperty('name')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('type')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('description')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('flat_name')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('source')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('dashed_name')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('normalize')).toBeTruthy();
expect(onePasswordField.hasOwnProperty('short')).toBeTruthy();
});
it('should not resolve the field from an integration if the integration and dataset params are not provided', async () => {
const onePasswordFieldInstance = await fieldsMetadataClient.getByName(
'onepassword.client.platform_version'
);
expect(integrationFieldsExtractor).not.toHaveBeenCalled();
expect(onePasswordFieldInstance).toBeUndefined();
});
});
describe('#find', () => {
it('should resolve a FieldsMetadataDictionary of matching fields', async () => {
const fieldsDictionaryInstance = await fieldsMetadataClient.find({
fieldNames: ['@timestamp'],
});
expect(integrationFieldsExtractor).not.toHaveBeenCalled();
const fields = fieldsDictionaryInstance.toPlain();
expect(fields.hasOwnProperty('@timestamp')).toBeTruthy();
});
it('should resolve a FieldsMetadataDictionary of matching fields, including integration fields when integration and dataset params are provided', async () => {
const fieldsDictionaryInstance = await fieldsMetadataClient.find({
fieldNames: ['@timestamp', 'onepassword.client.platform_version'],
integration: '1password',
dataset: '1password.item_usages',
});
expect(integrationFieldsExtractor).toHaveBeenCalled();
const fields = fieldsDictionaryInstance.toPlain();
expect(fields.hasOwnProperty('@timestamp')).toBeTruthy();
expect(fields.hasOwnProperty('onepassword.client.platform_version')).toBeTruthy();
});
it('should resolve a FieldsMetadataDictionary of matching fields, skipping unmatched fields', async () => {
const fieldsDictionaryInstance = await fieldsMetadataClient.find({
fieldNames: ['@timestamp', 'onepassword.client.platform_version', 'not-existing-field'],
integration: '1password',
dataset: '1password.item_usages',
});
expect(integrationFieldsExtractor).toHaveBeenCalled();
const fields = fieldsDictionaryInstance.toPlain();
expect(fields.hasOwnProperty('@timestamp')).toBeTruthy();
expect(fields.hasOwnProperty('onepassword.client.platform_version')).toBeTruthy();
expect(fields.hasOwnProperty('not-existing-field')).toBeFalsy();
});
});
});
function expectToBeDefined<T>(value: T | undefined): asserts value is T {
expect(value).toBeDefined();
}

73
x-pack/plugins/fields_metadata/server/services/fields_metadata/fields_metadata_client.ts Normal file
View file

@ -0,0 +1,73 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { Logger } from '@kbn/core/server';
import { FieldName, FieldMetadata, FieldsMetadataDictionary } from '../../../common';
import { EcsFieldsRepository } from './repositories/ecs_fields_repository';
import { IntegrationFieldsRepository } from './repositories/integration_fields_repository';
import { IntegrationFieldsSearchParams } from './repositories/types';
import { FindFieldsMetadataOptions, IFieldsMetadataClient } from './types';
interface FieldsMetadataClientDeps {
logger: Logger;
ecsFieldsRepository: EcsFieldsRepository;
integrationFieldsRepository: IntegrationFieldsRepository;
}
export class FieldsMetadataClient implements IFieldsMetadataClient {
private constructor(
private readonly logger: Logger,
private readonly ecsFieldsRepository: EcsFieldsRepository,
private readonly integrationFieldsRepository: IntegrationFieldsRepository
) {}
async getByName<TFieldName extends FieldName>(
fieldName: TFieldName,
{ integration, dataset }: Partial<IntegrationFieldsSearchParams> = {}
): Promise<FieldMetadata | undefined> {
this.logger.debug(`Retrieving field metadata for: ${fieldName}`);
// 1. Try resolving from ecs static metadata
let field = this.ecsFieldsRepository.getByName(fieldName);
// 2. Try searching for the fiels in the Elastic Package Registry
if (!field && integration) {
field = await this.integrationFieldsRepository.getByName(fieldName, { integration, dataset });
}
return field;
}
async find({
fieldNames,
integration,
dataset,
}: FindFieldsMetadataOptions = {}): Promise<FieldsMetadataDictionary> {
if (!fieldNames) {
return this.ecsFieldsRepository.find();
}
const fields: Record<string, FieldMetadata> = {};
for (const fieldName of fieldNames) {
const field = await this.getByName(fieldName, { integration, dataset });
if (field) {
fields[fieldName] = field;
}
}
return FieldsMetadataDictionary.create(fields);
}
public static create({
logger,
ecsFieldsRepository,
integrationFieldsRepository,
}: FieldsMetadataClientDeps) {
return new FieldsMetadataClient(logger, ecsFieldsRepository, integrationFieldsRepository);
}
}

19
x-pack/plugins/fields_metadata/server/services/fields_metadata/fields_metadata_service.mock.ts Normal file
View file

@ -0,0 +1,19 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { createFieldsMetadataClientMock } from './fields_metadata_client.mock';
import { FieldsMetadataServiceSetup, FieldsMetadataServiceStart } from './types';
export const createFieldsMetadataServiceSetupMock =
(): jest.Mocked<FieldsMetadataServiceSetup> => ({
registerIntegrationFieldsExtractor: jest.fn(),
});
export const createFieldsMetadataServiceStartMock =
(): jest.Mocked<FieldsMetadataServiceStart> => ({
getClient: jest.fn(() => createFieldsMetadataClientMock()),
});

47
x-pack/plugins/fields_metadata/server/services/fields_metadata/fields_metadata_service.ts Normal file
View file

@ -0,0 +1,47 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { EcsFlat as ecsFields } from '@elastic/ecs';
import { Logger } from '@kbn/core/server';
import { FieldsMetadataClient } from './fields_metadata_client';
import { EcsFieldsRepository } from './repositories/ecs_fields_repository';
import { IntegrationFieldsRepository } from './repositories/integration_fields_repository';
import { IntegrationFieldsExtractor } from './repositories/types';
import { FieldsMetadataServiceSetup, FieldsMetadataServiceStart } from './types';
export class FieldsMetadataService {
private integrationFieldsExtractor: IntegrationFieldsExtractor = () => Promise.resolve({});
constructor(private readonly logger: Logger) {}
public setup(): FieldsMetadataServiceSetup {
return {
registerIntegrationFieldsExtractor: (extractor: IntegrationFieldsExtractor) => {
this.integrationFieldsExtractor = extractor;
},
};
}
public start(): FieldsMetadataServiceStart {
const { logger, integrationFieldsExtractor } = this;
const ecsFieldsRepository = EcsFieldsRepository.create({ ecsFields });
const integrationFieldsRepository = IntegrationFieldsRepository.create({
integrationFieldsExtractor,
});
return {
getClient() {
return FieldsMetadataClient.create({
logger,
ecsFieldsRepository,
integrationFieldsRepository,
});
},
};
}
}

14
x-pack/plugins/fields_metadata/server/services/fields_metadata/index.ts Normal file
View file

@ -0,0 +1,14 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export { FieldsMetadataService } from './fields_metadata_service';
export { FieldsMetadataClient } from './fields_metadata_client';
export type {
FieldsMetadataServiceSetup,
FieldsMetadataServiceStart,
FieldsMetadataServiceStartDeps,
} from './types';

54
x-pack/plugins/fields_metadata/server/services/fields_metadata/repositories/ecs_fields_repository.ts Normal file
View file

@ -0,0 +1,54 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import mapValues from 'lodash/mapValues';
import { FieldsMetadataDictionary } from '../../../../common/fields_metadata/models/fields_metadata_dictionary';
import { FieldMetadata, FieldName, TEcsFields } from '../../../../common';
interface EcsFieldsRepositoryDeps {
ecsFields: TEcsFields;
}
interface FindOptions {
fieldNames?: FieldName[];
}
export class EcsFieldsRepository {
private readonly ecsFields: Record<FieldName, FieldMetadata>;
private constructor(ecsFields: TEcsFields) {
this.ecsFields = mapValues(ecsFields, (field) =>
FieldMetadata.create({ ...field, source: 'ecs' })
);
}
getByName(fieldName: FieldName): FieldMetadata | undefined {
return this.ecsFields[fieldName];
}
find({ fieldNames }: FindOptions = {}): FieldsMetadataDictionary {
if (!fieldNames) {
return FieldsMetadataDictionary.create(this.ecsFields);
}
const fields = fieldNames.reduce((fieldsMetadata, fieldName) => {
const field = this.getByName(fieldName);
if (field) {
fieldsMetadata[fieldName] = field;
}
return fieldsMetadata;
}, {} as Record<FieldName, FieldMetadata>);
return FieldsMetadataDictionary.create(fields);
}
public static create({ ecsFields }: EcsFieldsRepositoryDeps) {
return new EcsFieldsRepository(ecsFields);
}
}

135
x-pack/plugins/fields_metadata/server/services/fields_metadata/repositories/integration_fields_repository.ts Normal file
View file

@ -0,0 +1,135 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { ANY_DATASET } from '../../../../common/fields_metadata';
import { HashedCache } from '../../../../common/hashed_cache';
import { FieldMetadata, IntegrationFieldName } from '../../../../common';
import {
ExtractedIntegrationFields,
IntegrationFieldsExtractor,
IntegrationFieldsSearchParams,
IntegrationName,
} from './types';
import { PackageNotFoundError } from '../errors';
interface IntegrationFieldsRepositoryDeps {
integrationFieldsExtractor: IntegrationFieldsExtractor;
}
type DatasetFieldsMetadata = Record<string, FieldMetadata>;
type IntegrationFieldsMetadataTree = Record<IntegrationName, DatasetFieldsMetadata>;
export class IntegrationFieldsRepository {
private cache: HashedCache<IntegrationFieldsSearchParams, IntegrationFieldsMetadataTree>;
private constructor(private readonly fieldsExtractor: IntegrationFieldsExtractor) {
this.cache = new HashedCache();
}
async getByName(
fieldName: IntegrationFieldName,
{ integration, dataset }: IntegrationFieldsSearchParams
): Promise<FieldMetadata | undefined> {
let field = this.getCachedField(fieldName, { integration, dataset });
if (!field) {
try {
await this.extractFields({ integration, dataset });
} catch (error) {
throw new PackageNotFoundError(error.message);
}
field = this.getCachedField(fieldName, { integration, dataset });
}
return field;
}
public static create({ integrationFieldsExtractor }: IntegrationFieldsRepositoryDeps) {
return new IntegrationFieldsRepository(integrationFieldsExtractor);
}
private async extractFields({
integration,
dataset,
}: IntegrationFieldsSearchParams): Promise<void> {
const cacheKey = this.getCacheKey({ integration, dataset });
const cachedIntegration = this.cache.get(cacheKey);
if (cachedIntegration) {
return undefined;
}
return this.fieldsExtractor({ integration, dataset })
.then(this.mapExtractedFieldsToFieldMetadataTree)
.then((fieldMetadataTree) => this.storeFieldsInCache(cacheKey, fieldMetadataTree));
}
private getCachedField(
fieldName: IntegrationFieldName,
{ integration, dataset }: IntegrationFieldsSearchParams
): FieldMetadata | undefined {
const cacheKey = this.getCacheKey({ integration, dataset });
const cachedIntegration = this.cache.get(cacheKey);
const datasetName = dataset === ANY_DATASET ? null : dataset;
// 1. Integration fields were never fetched
if (!cachedIntegration) {
return undefined;
}
// 2. Dataset is passed but was never fetched before
if (datasetName && !cachedIntegration.hasOwnProperty(datasetName)) {
return undefined;
}
// 3. Dataset is passed and it was previously fetched, should return the field
if (datasetName && cachedIntegration.hasOwnProperty(datasetName)) {
const targetDataset = cachedIntegration[datasetName];
return targetDataset[fieldName];
}
// 4. Dataset is not passed, we attempt search on all stored datasets
if (!datasetName) {
// Merge all the available datasets into a unique field list. Overriding fields might occur in the process.
const cachedDatasetsFields = Object.assign({}, ...Object.values(cachedIntegration));
return cachedDatasetsFields[fieldName];
}
}
private storeFieldsInCache = (
cacheKey: IntegrationFieldsSearchParams,
extractedFieldsMetadata: IntegrationFieldsMetadataTree
): void => {
const cachedIntegration = this.cache.get(cacheKey);
if (!cachedIntegration) {
this.cache.set(cacheKey, extractedFieldsMetadata);
} else {
this.cache.set(cacheKey, { ...cachedIntegration, ...extractedFieldsMetadata });
}
};
private getCacheKey = (params: IntegrationFieldsSearchParams) => params;
private mapExtractedFieldsToFieldMetadataTree = (extractedFields: ExtractedIntegrationFields) => {
const datasetGroups = Object.entries(extractedFields);
return datasetGroups.reduce((integrationGroup, [datasetName, datasetGroup]) => {
const datasetFieldsEntries = Object.entries(datasetGroup);
integrationGroup[datasetName] = datasetFieldsEntries.reduce(
(datasetFields, [fieldName, field]) => {
datasetFields[fieldName] = FieldMetadata.create({ ...field, source: 'integration' });
return datasetFields;
},
{} as DatasetFieldsMetadata
);
return integrationGroup;
}, {} as IntegrationFieldsMetadataTree);
};
}

22
x-pack/plugins/fields_metadata/server/services/fields_metadata/repositories/types.ts Normal file
View file

@ -0,0 +1,22 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { FieldMetadataPlain } from '../../../../common';
export interface IntegrationFieldsSearchParams {
integration: string;
dataset?: string;
}
export type IntegrationName = string;
export type DatasetName = string;
export type ExtractedIntegrationFields = Record<IntegrationName, ExtractedDatasetFields>;
export type ExtractedDatasetFields = Record<DatasetName, FieldMetadataPlain>;
export type IntegrationFieldsExtractor = (
params: IntegrationFieldsSearchParams
) => Promise<ExtractedIntegrationFields>;

31
x-pack/plugins/fields_metadata/server/services/fields_metadata/types.ts Normal file
View file

@ -0,0 +1,31 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { FieldName, FieldMetadata, FieldsMetadataDictionary } from '../../../common';
import { IntegrationFieldsExtractor, IntegrationFieldsSearchParams } from './repositories/types';
export * from './repositories/types';
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataServiceStartDeps {}
export interface FieldsMetadataServiceSetup {
registerIntegrationFieldsExtractor: (extractor: IntegrationFieldsExtractor) => void;
}
export interface FieldsMetadataServiceStart {
getClient(): IFieldsMetadataClient;
}
export interface FindFieldsMetadataOptions extends Partial<IntegrationFieldsSearchParams> {
fieldNames?: FieldName[];
}
export interface IFieldsMetadataClient {
getByName(fieldName: FieldName): Promise<FieldMetadata | undefined>;
find(params: FindFieldsMetadataOptions): Promise<FieldsMetadataDictionary>;
}

34
x-pack/plugins/fields_metadata/server/types.ts Normal file
View file

@ -0,0 +1,34 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { CoreSetup } from '@kbn/core/server';
import {
FieldsMetadataServiceSetup,
FieldsMetadataServiceStart,
} from './services/fields_metadata/types';
export type FieldsMetadataPluginCoreSetup = CoreSetup<
FieldsMetadataServerPluginStartDeps,
FieldsMetadataServerStart
>;
export type FieldsMetadataPluginStartServicesAccessor =
FieldsMetadataPluginCoreSetup['getStartServices'];
export interface FieldsMetadataServerSetup {
registerIntegrationFieldsExtractor: FieldsMetadataServiceSetup['registerIntegrationFieldsExtractor'];
}
export interface FieldsMetadataServerStart {
getClient: FieldsMetadataServiceStart['getClient'];
}
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataServerPluginSetupDeps {}
// eslint-disable-next-line @typescript-eslint/no-empty-interface
export interface FieldsMetadataServerPluginStartDeps {}

22
x-pack/plugins/fields_metadata/tsconfig.json Normal file
View file

@ -0,0 +1,22 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"outDir": "target/types"
},
"include": [
"../../../typings/**/*",
"common/**/*",
"public/**/*",
"server/**/*",
"types/**/*"
],
"exclude": ["target/**/*"],
"kbn_references": [
"@kbn/core",
"@kbn/io-ts-utils",
"@kbn/logging",
"@kbn/core-http-request-handler-context-server",
"@kbn/core-http-server",
"@kbn/logging-mocks",
]
}

3
x-pack/plugins/fleet/kibana.jsonc
View file

@ -23,7 +23,8 @@
"taskManager",
"files",
"uiActions",
"dashboard"
"dashboard",
"fieldsMetadata"
],
"optionalPlugins": [
"features",

8
x-pack/plugins/fleet/server/plugin.ts
View file

@ -41,6 +41,7 @@ import type {
SecurityPluginStart,
} from '@kbn/security-plugin/server';
import type { PluginSetupContract as FeaturesPluginSetup } from '@kbn/features-plugin/server';
import type { FieldsMetadataServerSetup } from '@kbn/fields-metadata-plugin/server';
import type {
TaskManagerSetupContract,
TaskManagerStartContract,
@ -125,6 +126,7 @@ import { PolicyWatcher } from './services/agent_policy_watch';
import { getPackageSpecTagId } from './services/epm/kibana/assets/tag_assets';
import { FleetMetricsTask } from './services/metrics/fleet_metrics_task';
import { fetchAgentMetrics } from './services/metrics/fetch_agent_metrics';
import { registerIntegrationFieldsExtractor } from './services/register_integration_fields_extractor';
export interface FleetSetupDeps {
security: SecurityPluginSetup;
@ -135,6 +137,7 @@ export interface FleetSetupDeps {
spaces?: SpacesPluginStart;
telemetry?: TelemetryPluginSetup;
taskManager: TaskManagerSetupContract;
fieldsMetadata: FieldsMetadataServerSetup;
}
export interface FleetStartDeps {
@ -279,7 +282,7 @@ export class FleetPlugin
});
}
public setup(core: CoreSetup, deps: FleetSetupDeps) {
public setup(core: CoreSetup<FleetStartDeps, FleetStartContract>, deps: FleetSetupDeps) {
this.httpSetup = core.http;
this.encryptedSavedObjectsSetup = deps.encryptedSavedObjects;
this.cloud = deps.cloud;
@ -574,6 +577,9 @@ export class FleetPlugin
taskManager: deps.taskManager,
logFactory: this.initializerContext.logger,
});
// Register fields metadata extractor
registerIntegrationFieldsExtractor({ core, fieldsMetadata: deps.fieldsMetadata });
}
public start(core: CoreStart, plugins: FleetStartDeps): FleetStartContract {

1
x-pack/plugins/fleet/server/services/epm/package_service.mock.ts
View file

@ -16,6 +16,7 @@ const createClientMock = (): jest.Mocked<PackageClient> => ({
readBundledPackage: jest.fn(),
getAgentPolicyInputs: jest.fn(),
getPackage: jest.fn(),
getPackageFieldsMetadata: jest.fn(),
getPackages: jest.fn(),
reinstallEsAssets: jest.fn(),
});

17
x-pack/plugins/fleet/server/services/epm/package_service.test.ts
View file

@ -44,6 +44,7 @@ const testKeys = [
'ensureInstalledPackage',
'fetchFindLatestPackage',
'getPackage',
'getPackageFieldsMetadata',
'reinstallEsAssets',
'readBundledPackage',
];
@ -127,6 +128,20 @@ function getTest(
};
break;
case testKeys[4]:
test = {
method: mocks.packageClient.getPackageFieldsMetadata.bind(mocks.packageClient),
args: [{ packageName: 'package_name', datasetName: 'dataset_name' }],
spy: jest.spyOn(epmRegistry, 'getPackageFieldsMetadata'),
spyArgs: [{ packageName: 'package_name', datasetName: 'dataset_name' }, undefined],
spyResponse: {
dataset_name: { field_1: { flat_name: 'field_1', type: 'keyword' } },
},
expectedReturnValue: {
dataset_name: { field_1: { flat_name: 'field_1', type: 'keyword' } },
},
};
break;
case testKeys[5]:
const pkg: InstallablePackage = {
format_version: '1.0.0',
name: 'package name',
@ -172,7 +187,7 @@ function getTest(
],
};
break;
case testKeys[5]:
case testKeys[6]:
const bundledPackage = {
name: 'package name',
version: '8.0.0',

19
x-pack/plugins/fleet/server/services/epm/package_service.ts
View file

@ -43,6 +43,7 @@ import { appContextService } from '..';
import type { CustomPackageDatasetConfiguration } from './packages/install';
import type { FetchFindLatestPackageOptions } from './registry';
import { getPackageFieldsMetadata } from './registry';
import * as Registry from './registry';
import { fetchFindLatestPackageOrThrow, getPackage } from './registry';
@ -100,8 +101,14 @@ export interface PackageClient {
getPackage(
packageName: string,
packageVersion: string
): Promise<{ packageInfo: ArchivePackage; paths: string[] }>;
packageVersion: string,
options?: Parameters<typeof getPackage>['2']
): ReturnType<typeof getPackage>;
getPackageFieldsMetadata(
params: Parameters<typeof getPackageFieldsMetadata>['0'],
options?: Parameters<typeof getPackageFieldsMetadata>['1']
): ReturnType<typeof getPackageFieldsMetadata>;
getPackages(params?: {
excludeInstallStatus?: false;
@ -312,6 +319,14 @@ class PackageClientImpl implements PackageClient {
return getPackage(packageName, packageVersion, options);
}
public async getPackageFieldsMetadata(
params: Parameters<typeof getPackageFieldsMetadata>['0'],
options?: Parameters<typeof getPackageFieldsMetadata>['1']
) {
await this.#runPreflight(READ_PACKAGE_INFO_AUTHZ);
return getPackageFieldsMetadata(params, options);
}
public async getPackages(params?: {
excludeInstallStatus?: false;
category?: CategoryId;

88
x-pack/plugins/fleet/server/services/epm/packages/utils.test.ts Normal file
View file

@ -0,0 +1,88 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import { dump } from 'js-yaml';
import type { AssetsMap } from '../../../../common/types';
import type { RegistryDataStream } from '../../../../common';
import { resolveDataStreamFields } from './utils';
describe('resolveDataStreamFields', () => {
const statusAssetYml = dump([
{
name: 'apache.status',
type: 'group',
fields: [
{
name: 'total_accesses',
type: 'long',
description: 'Total number of access requests.\n',
metric_type: 'counter',
},
{
name: 'uptime',
type: 'group',
fields: [
{
name: 'server_uptime',
type: 'long',
description: 'Server uptime in seconds.\n',
metric_type: 'counter',
},
{
name: 'uptime',
type: 'long',
description: 'Server uptime.\n',
metric_type: 'counter',
},
],
},
],
},
]);
const dataStream = {
dataset: 'apache.status',
path: 'status',
} as RegistryDataStream;
const assetsMap = new Map([
['apache-1.18.0/data_stream/status/fields/fields.yml', Buffer.from(statusAssetYml)],
]) as AssetsMap;
const expectedResult = {
'apache.status': {
'apache.status.total_accesses': {
name: 'total_accesses',
type: 'long',
description: 'Total number of access requests.\n',
metric_type: 'counter',
flat_name: 'apache.status.total_accesses',
},
'apache.status.uptime.server_uptime': {
name: 'server_uptime',
type: 'long',
description: 'Server uptime in seconds.\n',
metric_type: 'counter',
flat_name: 'apache.status.uptime.server_uptime',
},
'apache.status.uptime.uptime': {
name: 'uptime',
type: 'long',
description: 'Server uptime.\n',
metric_type: 'counter',
flat_name: 'apache.status.uptime.uptime',
},
},
};
it('should load and resolve fields for the passed data stream', () => {
expect(resolveDataStreamFields({ dataStream, assetsMap })).toEqual(expectedResult);
});
});

113
x-pack/plugins/fleet/server/services/epm/packages/utils.ts
View file

@ -6,6 +6,119 @@
*/
import { withSpan } from '@kbn/apm-utils';
import type { FieldMetadataPlain } from '@kbn/fields-metadata-plugin/common';
import type { ExtractedDatasetFields } from '@kbn/fields-metadata-plugin/server';
import { load } from 'js-yaml';
import type { RegistryDataStream } from '../../../../common';
import type { AssetsMap } from '../../../../common/types';
type InputField =
| FieldMetadataPlain
| {
name: string;
type: 'group';
fields: InputField[];
};
export const withPackageSpan = <T>(stepName: string, func: () => Promise<T>) =>
withSpan({ name: stepName, type: 'package' }, func);
const normalizeFields = (fields: InputField[], prefix = ''): ExtractedDatasetFields => {
return fields.reduce((normalizedFields, field) => {
const flatName = prefix ? `${prefix}.${field.name}` : field.name;
// Recursively resolve field groups
if (isGroupField(field)) {
return Object.assign(normalizedFields, normalizeFields(field.fields || [], flatName));
}
normalizedFields[flatName] = createIntegrationField(field, flatName);
return normalizedFields;
}, {} as ExtractedDatasetFields);
};
const createIntegrationField = (
field: Omit<FieldMetadataPlain, 'flat_name'>,
flatName: string
) => ({
...field,
flat_name: flatName,
});
const isGroupField = (field: InputField): field is Extract<InputField, { type: 'group' }> => {
return field.type === 'group';
};
export const resolveDataStreamsMap = (
dataStreams?: RegistryDataStream[]
): Map<string, RegistryDataStream> => {
if (!dataStreams) return new Map();
return dataStreams.reduce((dataStreamsMap, dataStream) => {
dataStreamsMap.set(dataStream.dataset, dataStream);
return dataStreamsMap;
}, new Map() as Map<string, RegistryDataStream>);
};
export const resolveDataStreamFields = ({
dataStream,
assetsMap,
excludedFieldsAssets,
}: {
dataStream: RegistryDataStream;
assetsMap: AssetsMap;
excludedFieldsAssets?: string[];
}) => {
const { dataset, path } = dataStream;
const dataStreamFieldsAssetPaths = getDataStreamFieldsAssetPaths(
assetsMap,
path,
excludedFieldsAssets
);
/**
* We want to create a single dictionary with fields taken from all the dataset /fields assets.
* This step
* - reads the files buffer
* - normalizes the fields data structure for each file
* - finally merge the fields from each file into a single dictionary
*/
const fields = dataStreamFieldsAssetPaths.reduce((dataStreamFields, fieldsAssetPath) => {
const fieldsAssetBuffer = assetsMap.get(fieldsAssetPath);
if (fieldsAssetBuffer) {
const fieldsAssetJSON = load(fieldsAssetBuffer.toString('utf8'));
const normalizedFields = normalizeFields(fieldsAssetJSON);
Object.assign(dataStreamFields, normalizedFields);
}
return dataStreamFields;
}, {} as ExtractedDatasetFields);
return {
[dataset]: fields,
};
};
const isFieldsAsset = (
assetPath: string,
dataStreamPath: string,
excludedFieldsAssets: string[] = []
) => {
return new RegExp(
`.*\/data_stream\/${dataStreamPath}\/fields\/(?!(${excludedFieldsAssets.join('|')})$).*\.yml`,
'i'
).test(assetPath);
};
const getDataStreamFieldsAssetPaths = (
assetsMap: AssetsMap,
dataStreamPath: string,
excludedFieldsAssets?: string[]
) => {
return [...assetsMap.keys()].filter((path) =>
isFieldsAsset(path, dataStreamPath, excludedFieldsAssets)
);
};

47
x-pack/plugins/fleet/server/services/epm/registry/index.ts
View file

@ -13,6 +13,8 @@ import semverGte from 'semver/functions/gte';
import type { Response } from 'node-fetch';
import type { Logger } from '@kbn/logging';
import type { ExtractedIntegrationFields } from '@kbn/fields-metadata-plugin/server';
import { splitPkgKey as split } from '../../../../common/services';
import { KibanaAssetType } from '../../../types';
@ -48,7 +50,7 @@ import {
import { getBundledPackageByName } from '../packages/bundled_packages';
import { withPackageSpan } from '../packages/utils';
import { resolveDataStreamFields, resolveDataStreamsMap, withPackageSpan } from '../packages/utils';
import { verifyPackageArchiveSignature } from '../packages/package_verification';
@ -352,6 +354,49 @@ export async function getPackage(
return { paths, packageInfo, assetsMap, verificationResult };
}
export async function getPackageFieldsMetadata(
params: { packageName: string; datasetName?: string },
options: { excludedFieldsAssets?: string[] } = {}
): Promise<ExtractedIntegrationFields> {
const { packageName, datasetName } = params;
const { excludedFieldsAssets = ['ecs.yml'] } = options;
// Attempt retrieving latest package name and version
const latestPackage = await fetchFindLatestPackageOrThrow(packageName);
const { name, version } = latestPackage;
// Attempt retrieving latest package
const resolvedPackage = await getPackage(name, version);
// We need to collect all the available data streams for the package.
// In case a dataset is specified from the parameter, it will load the fields only for that specific dataset.
// As a fallback case, we'll try to read the fields for all the data streams in the package.
const dataStreamsMap = resolveDataStreamsMap(resolvedPackage.packageInfo.data_streams);
const { assetsMap } = resolvedPackage;
const dataStream = datasetName ? dataStreamsMap.get(datasetName) : null;
if (dataStream) {
// Resolve a single data stream fields when the `datasetName` parameter is specified
return resolveDataStreamFields({ dataStream, assetsMap, excludedFieldsAssets });
} else {
// Resolve and merge all the integration data streams fields otherwise
return [...dataStreamsMap.values()].reduce(
(packageDataStreamsFields, currentDataStream) =>
Object.assign(
packageDataStreamsFields,
resolveDataStreamFields({
dataStream: currentDataStream,
assetsMap,
excludedFieldsAssets,
})
),
{}
);
}
}
function ensureContentType(archivePath: string) {
const contentType = mime.lookup(archivePath);

27
x-pack/plugins/fleet/server/services/register_integration_fields_extractor.ts Normal file
View file

@ -0,0 +1,27 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import type { CoreSetup } from '@kbn/core/server';
import type { FieldsMetadataServerSetup } from '@kbn/fields-metadata-plugin/server';
import type { FleetStartContract, FleetStartDeps } from '../plugin';
interface RegistrationDeps {
core: CoreSetup<FleetStartDeps, FleetStartContract>;
fieldsMetadata: FieldsMetadataServerSetup;
}
export const registerIntegrationFieldsExtractor = ({ core, fieldsMetadata }: RegistrationDeps) => {
fieldsMetadata.registerIntegrationFieldsExtractor(async ({ integration, dataset }) => {
const [_core, _startDeps, { packageService }] = await core.getStartServices();
return packageService.asInternalUser.getPackageFieldsMetadata({
packageName: integration,
datasetName: dataset,
});
});
};

1
x-pack/plugins/fleet/tsconfig.json
View file

@ -108,6 +108,7 @@
"@kbn/zod-helpers",
"@kbn/react-kibana-mount",
"@kbn/react-kibana-context-render",
"@kbn/fields-metadata-plugin",
"@kbn/test-jest-helpers",
]
}

4
yarn.lock
View file

@ -4948,6 +4948,10 @@
version "0.0.0"
uid ""
"@kbn/fields-metadata-plugin@link:x-pack/plugins/fields_metadata":
version "0.0.0"
uid ""
"@kbn/file-upload-plugin@link:x-pack/plugins/file_upload":
version "0.0.0"
uid ""