kibana/dev_docs/tutorials/saved_objects.mdx
Michael Dokolin a65cd356aa
[Migrations] Add support of deferred migrations (#153117)
* Add deferred migrations parameter.
* Update outdated documents query to take into account deferred migrations.
* Update outdated documents query to take into account the core migration version.
* Update read operations in the saved objects repository to perform deferred migrations.
2023-05-22 11:17:41 +02:00

327 lines
12 KiB
Text
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
id: kibDevTutorialSavedObject
slug: /kibana-dev-docs/tutorials/saved-objects
title: Register a new saved object type
description: Learn how to register a new saved object type.
date: 2021-02-05
tags: ['kibana','onboarding', 'dev', 'architecture', 'tutorials']
---
Saved Object type definitions should be defined in their own `my_plugin/server/saved_objects` directory.
The folder should contain a file per type, named after the snake_case name of the type, and an index.ts file exporting all the types.
**src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts**
```ts
import { SavedObjectsType } from 'src/core/server';
export const dashboardVisualization: SavedObjectsType = {
name: 'dashboard_visualization', [1]
hidden: true,
namespaceType: 'multiple-isolated', [2]
mappings: {
dynamic: false,
properties: {
description: {
type: 'text',
},
hits: {
type: 'integer',
},
},
},
migrations: {
'1.0.0': migratedashboardVisualizationToV1,
'2.0.0': migratedashboardVisualizationToV2,
},
};
```
[1] Since the name of a Saved Object type forms part of the URL path for the public Saved Objects HTTP API,
these should follow our API URL path convention and always be written in snake case.
[2] This field determines "space behavior" -- whether these objects can exist in one space, multiple spaces, or all spaces. This value means
that objects of this type can only exist in a single space. See
<DocLink id="kibDevDocsSavedObjectsIntro" section="sharing-saved-objects" text="Sharing Saved Objects"/> for more information.
**src/plugins/my_plugin/server/saved_objects/index.ts**
```ts
export { dashboardVisualization } from './dashboard_visualization';
export { dashboard } from './dashboard';
```
**src/plugins/my_plugin/server/plugin.ts**
```ts
import { dashboard, dashboardVisualization } from './saved_objects';
export class MyPlugin implements Plugin {
setup({ savedObjects }) {
savedObjects.registerType(dashboard);
savedObjects.registerType(dashboardVisualization);
}
}
```
## Mappings
Each Saved Object type can define its own Elasticsearch field mappings. Because multiple Saved Object
types can share the same index, mappings defined by a type will be nested under a top-level field that matches the type name.
For example, the mappings defined by the dashboard_visualization Saved Object type:
**src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts**
```ts
import { SavedObjectsType } from 'src/core/server';
export const dashboardVisualization: SavedObjectsType = {
name: 'dashboard_visualization',
...
mappings: {
properties: {
dynamic: false,
description: {
type: 'text',
},
hits: {
type: 'integer',
},
},
},
migrations: { ... },
};
```
Will result in the following mappings being applied to the .kibana index:
```ts
{
"mappings": {
"dynamic": "strict",
"properties": {
...
"dashboard_vizualization": {
"dynamic": false,
"properties": {
"description": {
"type": "text",
},
"hits": {
"type": "integer",
},
},
}
}
}
}
```
Do not use field mappings like you would use data types for the columns of a SQL database. Instead, field mappings are analogous to a
SQL index. Only specify field mappings for the fields you wish to search on or query. By specifying `dynamic: false`
in any level of your mappings, Elasticsearch will accept and store any other fields even if they are not specified in your mappings.
Since Elasticsearch has a default limit of 1000 fields per index, plugins should carefully consider the
fields they add to the mappings. Similarly, Saved Object types should never use `dynamic: true` as this can cause an arbitrary
amount of fields to be added to the .kibana index.
## References
Declare <DocLink id="kibDevDocsSavedObjectsIntro" section="References" text="Saved Object references"/> by adding an id, type and name to the
`references` array.
```ts
router.get(
{ path: '/some-path', validate: false },
async (context, req, res) => {
const object = await context.core.savedObjects.client.create(
'dashboard',
{
title: 'my dashboard',
panels: [
{ visualization: 'vis1' }, [1]
],
indexPattern: 'indexPattern1'
},
{ references: [
{ id: '...', type: 'visualization', name: 'vis1' },
{ id: '...', type: 'index_pattern', name: 'indexPattern1' },
]
}
)
...
}
);
```
[1] Note how `dashboard.panels[0].visualization` stores the name property of the reference (not the id directly) to be able to uniquely
identify this reference. This guarantees that the id the reference points to always remains up to date. If a
visualization id was directly stored in `dashboard.panels[0].visualization` there is a risk that this id gets updated without
updating the reference in the references array.
## Migrations
Saved Objects support schema changes between Kibana versions, which we call migrations. Migrations are
applied when a Kibana installation is upgraded from one version to a newer version, when exports are imported via
the Saved Objects Management UI, or when a new object is created via the HTTP API.
### Writing migrations
Each Saved Object type may define migrations for its schema. Migrations are specified by the Kibana version number, receive an input document,
and must return the fully migrated document to be persisted to Elasticsearch.
Lets say we want to define two migrations: - In version 1.1.0, we want to drop the subtitle field and append it to the title - In version
1.4.0, we want to add a new id field to every panel with a newly generated UUID.
First, the current mappings should always reflect the latest or "target" schema. Next, we should define a migration function for each step in the schema evolution:
**src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts**
```ts
import { SavedObjectsType, SavedObjectMigrationFn } from 'src/core/server';
import uuid from 'uuid';
interface DashboardVisualizationPre110 {
title: string;
subtitle: string;
panels: Array<{}>;
}
interface DashboardVisualization110 {
title: string;
panels: Array<{}>;
}
interface DashboardVisualization140 {
title: string;
panels: Array<{ id: string }>;
}
const migrateDashboardVisualization110: SavedObjectMigrationFn<
DashboardVisualizationPre110, [1]
DashboardVisualization110
> = (doc) => {
const { subtitle, ...attributesWithoutSubtitle } = doc.attributes;
return {
...doc, [2]
attributes: {
...attributesWithoutSubtitle,
title: `${doc.attributes.title} - ${doc.attributes.subtitle}`,
},
};
};
const migrateDashboardVisualization140: SavedObjectMigrationFn<
DashboardVisualization110,
DashboardVisualization140
> = (doc) => {
const outPanels = doc.attributes.panels?.map((panel) => {
return { ...panel, id: uuid.v4() };
});
return {
...doc,
attributes: {
...doc.attributes,
panels: outPanels,
},
};
};
export const dashboardVisualization: SavedObjectsType = {
name: 'dashboard_visualization', [1]
/** ... */
migrations: {
// Takes a pre 1.1.0 doc, and converts it to 1.1.0
'1.1.0': migrateDashboardVisualization110,
// Takes a 1.1.0 doc, and converts it to 1.4.0
'1.4.0': migrateDashboardVisualization140, [3]
},
};
```
[1] It is useful to define an interface for each version of the schema. This allows TypeScript to ensure that you are properly handling the input and output
types correctly as the schema evolves.
[2] Returning a shallow copy is necessary to avoid type errors when using different types for the input and output shape.
[3] Migrations do not have to be defined for every version. The version number of a migration must always be the earliest Kibana version
in which this migration was released. So if you are creating a migration which will
be part of the v7.10.0 release, but will also be backported and released as v7.9.3, the migration version should be: 7.9.3.
Migrations should be written defensively, an exception in a migration function will prevent a Kibana upgrade from succeeding and will cause downtime for our users.
Having said that, if a document is encountered that is not in the expected shape, migrations are encouraged to throw an exception to abort the upgrade. In most scenarios, it is better to
fail an upgrade than to silently ignore a corrupt document which can cause unexpected behaviour at some future point in time. When such a scenario is encountered,
the error should be verbose and informative so that the corrupt document can be corrected, if possible.
**WARNING:** Do not attempt to change the `typeMigrationVersion`, `id`, or `type` fields within a migration function, this is not supported.
### Deferred Migrations
Usually, migrations run during the upgrade process, and sometimes that may block it if there is a huge amount of outdated objects.
In this case, it is recommended to mark some of the migrations to defer their execution.
```ts
export const dashboardVisualization: SavedObjectsType = {
name: 'dashboard_visualization', [1]
/** ... */
migrations: {
// Takes a pre 1.1.0 doc, and converts it to 1.1.0
'1.1.0': {
deferred: true,
transform: migrateDashboardVisualization110,
},
},
};
```
By default, all the migrations are not deferred, and in order to make them so, the `deferred` flag should be explicitly set to `true`.
In this case, the documents with only pending deferred migrations will not be migrated during the upgrade process.
But whenever they are accessed via Saved Object API or repository, all the migrations will be applied to them on the fly:
- On read operations, the stored objects remain untouched and only transformed before returning the result.
If there are some failures during the migration, an exception or 500 server error will be thrown,
so that it is guaranteed that all the returned objects will be up to date.
- On write operations, the objects will be migrated to the latest version before writing them.
In other words, this flag postpones the write operation until the objects are explicitly modified.
One important notice: if there is a few pending migrations for a document and not all of them can be deferred,
the document will be migrated during the upgrade process, and all pending migrations will be applied.
### Testing Migrations
Bugs in a migration function cause downtime for our users and therefore have a very high impact. Follow the <DocLink id="kibDevTutorialTestingPlugins" section="saved-objects-migrations" text="Saved Object migrations section in the plugin testing guide"/>.
### How to opt-out of the global savedObjects APIs?
There are 2 options, depending on the amount of flexibility you need:
For complete control over your HTTP APIs and custom handling, declare your type as `hidden`, as shown in the example.
The other option that allows you to build your own HTTP APIs and still use the client as-is is to declare your type as hidden from the global saved objects HTTP APIs as `hiddenFromHttpApis: true`
```ts
import { SavedObjectsType } from 'src/core/server';
export const foo: SavedObjectsType = {
name: 'foo',
hidden: false, [1]
hiddenFromHttpApis: true, [2]
namespaceType: 'multiple-isolated',
mappings: {
dynamic: false,
properties: {
description: {
type: 'text',
},
hits: {
type: 'integer',
},
},
},
migrations: {
'1.0.0': migratedashboardVisualizationToV1,
'2.0.0': migratedashboardVisualizationToV2,
},
};
```
[1] Needs to be `false` to use the `hiddenFromHttpApis` option
[2] Set this to `true` to build your own HTTP API and have complete control over the route handler.