kibana/docs/developer/architecture/core/saved-objects-service.asciidoc

[[saved-objects-service]]
== Saved Objects service

NOTE: The Saved Objects service is available both server and client side.

`Saved Objects service` allows {kib} plugins to use {es} like a primary
database. Think of it as an Object Document Mapper for {es}. Once a
plugin has registered one or more Saved Object types, the Saved Objects client
can be used to query or perform create, read, update and delete operations on
each type.

By using Saved Objects your plugin can take advantage of the following
features:
 
* Migrations can evolve your document's schema by transforming documents and
ensuring that the field mappings on the index are always up to date.
* a <<saved-objects-api,HTTP API>> is automatically exposed for each type (unless
`hidden=true` is specified).
* a Saved Objects client that can be used from both the server and the browser.
* Users can import or export Saved Objects using the Saved Objects management
UI or the Saved Objects import/export API.
* By declaring `references`, an object's entire reference graph will be
exported. This makes it easy for users to export e.g. a `dashboard` object and
have all the `visualization` objects required to display the dashboard
included in the export.
* When the X-Pack security and spaces plugins are enabled these transparently
provide RBAC access control and the ability to organize Saved Objects into
spaces.

This document contains developer guidelines and best-practices for plugins
wanting to use Saved Objects.

=== Server side usage

[[saved-objects-type-registration]]
==== Registering a Saved Object type
Saved object type definitions should be defined in their own `my_plugin/server/saved_objects` directory.

The folder should contain a file per type, named after the snake_case name of the type, and an `index.ts` file exporting all the types.

.src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts
[source,typescript]
----
import { SavedObjectsType } from 'src/core/server';

export const dashboardVisualization: SavedObjectsType = {
  name: 'dashboard_visualization', // <1>
  hidden: true,
  namespaceType: 'multiple-isolated', // <2>
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    1: modelVersion1,
    2: modelVersion2,
  },
  mappings: {
    dynamic: false,
    properties: {
      description: {
        type: 'text',
      },
      hits: {
        type: 'integer',
      },
    },
  },
  // ...other mandatory properties
};
----
<1> Since the name of a Saved Object type may form part of the URL path for the
public Saved Objects HTTP API, these should follow our API URL path convention
and always be written in snake case.
<2> This field determines "space behavior" -- whether these objects can exist in one space, multiple spaces, or all spaces. This value means
that objects of this type can only exist in a single space. See <<sharing-saved-objects,Sharing Saved Objects>> for more information.

.src/plugins/my_plugin/server/saved_objects/index.ts
[source,typescript]
----
export { dashboardVisualization } from './dashboard_visualization';
export { dashboard } from './dashboard'; 
----

.src/plugins/my_plugin/server/plugin.ts
[source,typescript]
----
import { dashboard, dashboardVisualization } from './saved_objects';

export class MyPlugin implements Plugin {
  setup({ savedObjects }) {
    savedObjects.registerType(dashboard);
    savedObjects.registerType(dashboardVisualization);
  }
}
----

==== Mappings
Each Saved Object type can define it's own {es} field mappings.
Because multiple Saved Object types can share the same index, mappings defined
by a type will be nested under a top-level field that matches the type name.

For example, the mappings defined by the `search` Saved
Object type:

https://github.com/elastic/kibana/blob/main/src/platform/plugins/shared/saved_search/server/saved_objects/search.ts#L19-L70[.src/platform/plugins/shared/saved_search/server/saved_objects/search.ts]
[source,typescript]
----
import { SavedObjectsType } from 'src/core/server';
// ... other imports
export function getSavedSearchObjectType: SavedObjectsType = { // <1>
  name: 'search',
  hidden: false,
  namespaceType: 'multiple-isolated',
  mappings: {
    dynamic: false,
    properties: {
      title: { type: 'text' },
      description: { type: 'text' },
    },
  },
  modelVersions: { ... },
  // ...other optional properties
};
----
<1> Simplification

Will result in the following mappings being applied to the `.kibana_analytics` index:
[source,json]
----
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      ...
      "search": {
        "dynamic": false,
        "properties": {
          "title": {
            "type": "text",
          },
          "description": {
            "type": "text",
          },
        },
      }
    }
  }
}
----

Do not use field mappings like you would use data types for the columns of a
SQL database. Instead, field mappings are analogous to a SQL index. Only
specify field mappings for the fields you wish to search on or query. By
specifying `dynamic: false` in any level of your mappings, {es} will
accept and store any other fields even if they are not specified in your mappings.

Since {es} has a default limit of 1000 fields per index, plugins
should carefully consider the fields they add to the mappings. Similarly,
Saved Object types should never use `dynamic: true` as this can cause an
arbitrary amount of fields to be added to the `.kibana` index.

[[saved-objects-service-writing-migrations]]
==== Writing Migrations by defining model versions

Saved Objects support changes using `modelVersions``. The modelVersion API is a new way to define transformations
(_``migrations''_) for your savedObject types, and will replace the
``legacy'' migration API after Kibana version `8.10.0`. The legacy migration API has been deprecated, meaning it is no longer possible to register migrations using the legacy system.

Model versions are decoupled from the stack version and satisfy the requirements for zero downtime and backward-compatibility.

Each Saved Object type may define model versions for its schema and are bound to a given https://github.com/elastic/kibana/blob/9b330e493216e8dde3166451e4714966f63f5ab7/packages/core/saved-objects/core-saved-objects-server/src/saved_objects_type.ts#L22-L27[savedObject type]. Changes to a saved object type are
specified by defining a new model.

=== Defining model versions

As for old migrations, model versions are bound to a given
https://github.com/elastic/kibana/blob/9b330e493216e8dde3166451e4714966f63f5ab7/packages/core/saved-objects/core-saved-objects-server/src/saved_objects_type.ts#L22-L27[savedObject
type]

When registering a SO type, a new
https://github.com/elastic/kibana/blob/9a6a2ccdff619f827b31c40dd9ed30cb27203da7/packages/core/saved-objects/core-saved-objects-server/src/saved_objects_type.ts#L138-L177[modelVersions]
property is available. This attribute is a map of
https://github.com/elastic/kibana/blob/9a6a2ccdff619f827b31c40dd9ed30cb27203da7/packages/core/saved-objects/core-saved-objects-server/src/model_version/model_version.ts#L12-L20[SavedObjectsModelVersion]
which is the top-level type/container to define model versions.

This map follows a similar `{ [version number] => version definition }`
format as the old migration map, however a given SO type’s model version
is now identified by a single integer.

The first version must be numbered as version 1, incrementing by one for
each new version.

That way: - SO type versions are decoupled from stack versioning - SO
type versions are independent between types

_a *valid* version numbering:_

[source,ts]
----
const myType: SavedObjectsType = {
  name: 'test',
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    1: modelVersion1, // valid: start with version 1
    2: modelVersion2, // valid: no gap between versions
  },
  // ...other mandatory properties
};
----

_an *invalid* version numbering:_

[source,ts]
----
const myType: SavedObjectsType = {
  name: 'test',
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    2: modelVersion2, // invalid: first version must be 1
    4: modelVersion3, // invalid: skipped version 3
  },
  // ...other mandatory properties
};
----

=== Structure of a model version

https://github.com/elastic/kibana/blob/9b330e493216e8dde3166451e4714966f63f5ab7/packages/core/saved-objects/core-saved-objects-server/src/model_version/model_version.ts#L12-L20[Model
versions] are not just functions as the previous migrations were, but
structured objects describing how the version behaves and what changed
since the last one.

_A base example of what a model version can look like:_

[source,ts]
----
const myType: SavedObjectsType = {
  name: 'test',
  switchToModelVersionAt: '8.10.0',
  modelVersions: {
    1: {
      changes: [
        {
          type: 'mappings_addition',
          addedMappings: {
            someNewField: { type: 'text' },
          },
        },
        {
          type: 'data_backfill',
          transform: someBackfillFunction,
        },
      ],
      schemas: {
        forwardCompatibility: fcSchema,
        create: createSchema,
      },
    },
  },
  // ...other mandatory properties
};
----

*Note:* Having multiple changes of the same type for a given version is
supported by design to allow merging different sources (to prepare for
an eventual higher-level API)

_This definition would be perfectly valid:_

[source,ts]
----
const version1: SavedObjectsModelVersion = {
  changes: [
    {
      type: 'mappings_addition',
      addedMappings: {
        someNewField: { type: 'text' },
      },
    },
    {
      type: 'mappings_addition',
      addedMappings: {
        anotherNewField: { type: 'text' },
      },
    },
  ],
};
----

It’s currently composed of two main properties:

==== changes

https://github.com/elastic/kibana/blob/9b330e493216e8dde3166451e4714966f63f5ab7/packages/core/saved-objects/core-saved-objects-server/src/model_version/model_version.ts#L21-L51[link
to the TS doc for `changes`]

Describes the list of changes applied during this version.

*Important:* This is the part that replaces the old migration system,
and allows defining when a version adds new mapping, mutates the
documents, or other type-related changes.

The current types of changes are:

===== - mappings_addition

Used to define new mappings introduced in a given version.

_Usage example:_

[source,ts]
----
const change: SavedObjectsModelMappingsAdditionChange = {
  type: 'mappings_addition',
  addedMappings: {
    newField: { type: 'text' },
    existingNestedField: {
      properties: {
        newNestedProp: { type: 'keyword' },
      },
    },
  },
};
----

*note:* _When adding mappings, the root `type.mappings` must also be
updated accordingly (as it was done previously)._

===== - mappings_deprecation

Used to flag mappings as no longer being used and ready to be removed.

_Usage example:_

[source,ts]
----
let change: SavedObjectsModelMappingsDeprecationChange = {
  type: 'mappings_deprecation',
  deprecatedMappings: ['someDeprecatedField', 'someNested.deprecatedField'],
};
----

*note:* _It is currently not possible to remove fields from an existing
index’s mapping (without reindexing into another index), so the mappings
flagged with this change type won’t be deleted for now, but this should
still be used to allow our system to clean the mappings once upstream
(ES) unblock us._

===== - data_backfill

Used to populate fields (indexed or not) added in the same version.

_Usage example:_

[source,ts]
----
let change: SavedObjectsModelDataBackfillChange = {
  type: 'data_backfill',
  transform: (document) => {
    return { attributes: { someAddedField: 'defaultValue' } };
  },
};
----

*note:* _Even if no check is performed to ensure it, this type of model
change should only be used to backfill newly introduced fields._

===== - data_removal

Used to remove data (unset fields) from all documents of the type.

_Usage example:_

[source,ts]
----
let change: SavedObjectsModelDataRemovalChange = {
  type: 'data_removal',
  attributePaths: ['someRootAttributes', 'some.nested.attribute'],
};
----

*note:* _Due to backward compatibility, field utilization must be
stopped in a prior release before actual data removal (in case of
rollback). Please refer to the field removal migration example below in
this document_

===== - unsafe_transform

Used to execute an arbitrary transformation function.

_Usage example:_

[source,ts]
----
let change: SavedObjectsModelUnsafeTransformChange = {
  type: 'unsafe_transform',
  transformFn: (document) => {
    document.attributes.someAddedField = 'defaultValue';
    return { document };
  },
};
----

*note:* _Using such transformations is potentially unsafe, given the
migration system will have no knowledge of which kind of operations will
effectively be executed against the documents. Those should only be used
when there’s no other way to cover one’s migration needs._ *Please reach
out to the development team if you think you need to use this, as you
theoretically shouldn’t.*

==== schemas

https://github.com/elastic/kibana/blob/9b330e493216e8dde3166451e4714966f63f5ab7/packages/core/saved-objects/core-saved-objects-server/src/model_version/schemas.ts#L11-L16[link
to the TS doc for `schemas`]

The schemas associated with this version. Schemas are used to validate
or convert SO documents at various stages of their lifecycle.

The currently available schemas are:

===== forwardCompatibility

This is a new concept introduced by model versions. This schema is used
for inter-version compatibility.

When retrieving a savedObject document from an index, if the version of
the document is higher than the latest version known of the Kibana
instance, the document will go through the `forwardCompatibility` schema
of the associated model version.

*Important:* These conversion mechanism shouldn’t assert the data
itself, and only strip unknown fields to convert the document to the
*shape* of the document at the given version.

Basically, this schema should keep all the known fields of a given
version, and remove all the unknown fields, without throwing.

Forward compatibility schema can be implemented in two different ways.

[arabic]
. Using `config-schema`

_Example of schema for a version having two fields: someField and
anotherField_

[source,ts]
----
const versionSchema = schema.object(
  {
    someField: schema.maybe(schema.string()),
    anotherField: schema.maybe(schema.string()),
  },
  { unknowns: 'ignore' }
);
----

*Important:* Note the `{ unknowns: 'ignore' }` in the schema’s options.
This is required when using `config-schema` based schemas, as this what
will evict the additional fields without throwing an error.

[arabic, start=2]
. Using a plain javascript function

_Example of schema for a version having two fields: someField and
anotherField_

[source,ts]
----
const versionSchema: SavedObjectModelVersionEvictionFn = (attributes) => {
  const knownFields = ['someField', 'anotherField'];
  return pick(attributes, knownFields);
}
----

*note:* _Even if highly recommended, implementing this schema is not
strictly required. Type owners can manage unknown fields and
inter-version compatibility themselves in their service layer instead._

===== create

This is a direct replacement for
https://github.com/elastic/kibana/blob/9b330e493216e8dde3166451e4714966f63f5ab7/packages/core/saved-objects/core-saved-objects-server/src/saved_objects_type.ts#L75-L82[the
old SavedObjectType.schemas] definition, now directly included in the
model version definition.

As a refresher the `create` schema is a `@kbn/config-schema` object-type
schema, and is used to validate the properties of the document during
`create` and `bulkCreate` operations.

*note:* _Implementing this schema is optional, but still recommended, as
otherwise there will be no validating when importing objects_

For implementation examples, refer to <<saved-objects-service-use-case-examples, Use case examples>>.

include::saved-objects-service-use-case-examples.asciidoc[leveloffset=+1]