Developer guide - index patterns key concepts (#96017)

* add index patterns key concepts

dev_docs/key_concepts/data_views.mdx

@@ -0,0 +1,32 @@
+---
+id: kibDataViewsKeyConcepts
+slug: /kibana-dev-docs/data-view-intro
+title: Data Views
+summary: Data views are the central method of defining queryable data sets in Kibana
+date: 2021-08-11
+tags: ['kibana','dev', 'contributor', 'api docs']
+---
+
+*Note: Kibana index patterns are currently being renamed to data views. There will be some naming inconsistencies until the transition is complete.*
+
+Data views (formerly Kibana index patterns or KIPs) are the central method of describing sets of indices for queries. Usage is strongly recommended
+as a number of high level <DocLink id="kibBuildingBlocks" text="building blocks"/> rely on them. Further, they provide a consistent view of data across 
+a variety Kibana apps.
+
+Data views are defined by a wildcard string (an index pattern) which matches indices, data streams, and index aliases, optionally specify a
+timestamp field for time series data, and are stored as a <DocLink id="kibDevDocsSavedObjectsIntro"
+text="saved object"/>. They have a field list which comprises all the fields in matching indices plus fields defined specifically
+on the data view via runtime fields. Schema-on-read functionality is provided by data view defined runtime fields.
+
+![image](../assets/data_view_diagram.png)
+
+
+
+The data view API is made available via the data plugin (`data.indexPatterns`, soon to be renamed) and most commonly used with <DocLink id="kibDevTutorialDataSearchAndSessions" section="high-level-search" text="SearchSource" />
+(`data.search.search.SearchSource`) to perform queries. SearchSource will apply existing filters and queries from the search bar UI.
+
+Users can create data views via [Data view management](https://www.elastic.co/guide/en/kibana/current/index-patterns.html).
+Additionally, they can be created through the data view API.
+
+Data views also allow formatters and custom labels to be defined for fields.
+

dev_docs/tutorials/data_views.mdx

@@ -0,0 +1,86 @@
+---
+id: kibDevTutorialDataViews
+slug: /kibana-dev-docs/tutorials/data-views
+title: Data views API
+summary: Data views API
+date: 2021-08-11
+tags: ['kibana', 'onboarding', 'dev', 'architecture']
+---
+
+*Note: Kibana index patterns are currently being renamed to data views. There will be some naming inconsistencies until the transition is complete.*
+
+### Data views API
+
+- Get list of data views
+- Get default data view and examine fields
+- Get data view by id
+- Find data view by title
+- Create data view
+- Create data view and save it
+- Modify data view and save it
+- Delete data view
+
+#### Get list of data view titles and ids
+
+```
+const idsAndTitles = await data.indexPatterns.getIdsWithTitle();
+idsAndTitles.forEach(({id, title}) => console.log(`Data view id: ${id} title: ${title}`));
+```
+
+#### Get default data view and examine fields
+
+```
+const defaultDataView = await data.indexPatterns.getDefault();
+defaultDataView.fields.forEach(({name}) => { console.log(name); })
+```
+
+#### Get data view by id
+
+```
+const id = 'xxxxxx-xxx-xxxxxx';
+const dataView = await data.indexPatterns.get(id);
+```
+
+#### Find data view by title
+
+```
+const title = 'kibana-*';
+const [dataView] = await data.indexPatterns.find(title);
+```
+
+#### Create data view
+
+```
+const dataView = await data.indexPatterns.create({ title: 'kibana-*' });
+```
+
+#### Create data view and save it immediately
+
+```
+const dataView = await data.indexPatterns.createAndSave({ title: 'kibana-*' });
+```
+
+#### Create data view, modify, and save
+
+```
+const dataView = await data.indexPatterns.create({ title: 'kibana-*' });
+dataView.setFieldCustomLabel('customer_name', 'Customer Name');
+data.indexPatterns.createSavedObject(dataView);
+```
+
+#### Modify data view and save it
+
+```
+dataView.setFieldCustomLabel('customer_name', 'Customer Name');
+await data.indexPatterns.updateSavedObject(dataView);
+```
+
+#### Delete index pattern
+
+```
+await data.indexPatterns.delete(dataViewId);
+```
+
+### Data view HTTP API
+
+Rest-like HTTP CRUD+ API - [docs](https://www.elastic.co/guide/en/kibana/master/index-patterns-api.html)

src/plugins/data/README.mdx

@@ -49,471 +49,14 @@ This is helpful when you want to provide a user with options, for example when c
 
 ```
 
-## Index Patterns
+## Data Views
 
-The Index Patterns API provides a consistent method of structuring and formatting documents
+The data views API provides a consistent method of structuring and formatting documents
 and field lists across the various Kibana apps. Its typically used in conjunction with
-SearchSource for composing queries.
+<DocLink id="kibDevTutorialDataSearchAndSessions" section="high-level-search" text="SearchSource" /> for composing queries.
 
-### Index Patterns API
+*Note: Kibana index patterns are currently being renamed to data views. There will be some naming inconsistencies until the transition is complete.*
 
-- Get list of index patterns
-- Get default index pattern and examine fields
-- Get index pattern by id
-- Find index pattern by title
-- Create index pattern
-- Create index pattern and save it
-- Modify index pattern and save it
-- Delete index pattern
-
-#### Get list of index pattern titles and ids
-
-```
-const idsAndTitles = await data.indexPatterns.getIdsWithTitle();
-idsAndTitles.forEach(({id, title}) => console.log(`Index pattern id: ${id} title: ${title}`));
-```
-
-#### Get default index pattern and examine fields
-
-```
-const defaultIndexPattern = await data.indexPatterns.getDefault();
-defaultIndexPattern.fields.forEach(({name}) => { console.log(name); })
-```
-
-#### Get index pattern by id
-
-```
-const id = 'xxxxxx-xxx-xxxxxx';
-const indexPattern = await data.indexPatterns.get(id);
-```
-
-#### Find index pattern by title
-
-```
-const title = 'kibana-*';
-const [indexPattern] = await data.indexPatterns.find(title);
-```
-
-#### Create index pattern
-
-```
-const indexPattern = await data.indexPatterns.create({ title: 'kibana-*' });
-```
-
-#### Create index pattern and save it immediately
-
-```
-const indexPattern = await data.indexPatterns.createAndSave({ title: 'kibana-*' });
-```
-
-#### Create index pattern, modify, and save
-
-```
-const indexPattern = await data.indexPatterns.create({ title: 'kibana-*' });
-indexPattern.setFieldCustomLabel('customer_name', 'Customer Name');
-data.indexPatterns.createSavedObject(indexPattern);
-```
-
-#### Modify index pattern and save it
-
-```
-indexPattern.setFieldCustomLabel('customer_name', 'Customer Name');
-await data.indexPatterns.updateSavedObject(indexPattern);
-```
-
-#### Delete index pattern
-
-```
-await data.indexPatterns.delete(indexPatternId);
-```
-
-### Index Patterns HTTP API
-
-Index patterns provide Rest-like HTTP CRUD+ API with the following endpoints:
-
-- Index Patterns API
-  - Create an index pattern &mdash; `POST /api/index_patterns/index_pattern`
-  - Fetch an index pattern by `{id}` &mdash; `GET /api/index_patterns/index_pattern/{id}`
-  - Delete an index pattern by `{id}` &mdash; `DELETE /api/index_patterns/index_pattern/{id}`
-  - Partially update an index pattern by `{id}` &mdash; `POST /api/index_patterns/index_pattern/{id}`
-- Fields API
-  - Update field &mdash; `POST /api/index_patterns/index_pattern/{id}/fields`
-- Scripted Fields API
-  - Create a scripted field &mdash; `POST /api/index_patterns/index_pattern/{id}/scripted_field`
-  - Upsert a scripted field &mdash; `PUT /api/index_patterns/index_pattern/{id}/scripted_field`
-  - Fetch a scripted field &mdash; `GET /api/index_patterns/index_pattern/{id}/scripted_field/{name}`
-  - Remove a scripted field &mdash; `DELETE /api/index_patterns/index_pattern/{id}/scripted_field/{name}`
-  - Update a scripted field &mdash; `POST /api/index_patterns/index_pattern/{id}/scripted_field/{name}`
-
-### Index Patterns API
-
-Index Patterns REST API allows you to create, retrieve and delete index patterns. I also
-exposes an update endpoint which allows you to update specific fields without changing
-the rest of the index pattern object.
-
-#### Create an index pattern
-
-Create an index pattern with a custom title.
-
-```
-POST /api/index_patterns/index_pattern
-{
-    "index_pattern": {
-        "title": "hello"
-    }
-}
-```
-
-Customize creation behavior with:
-
-- `override` &mdash; if set to `true`, replaces an existing index pattern if an
-  index pattern with the provided title already exists. Defaults to `false`.
-- `refresh_fields` &mdash; if set to `true` reloads index pattern fields after
-  the index pattern is stored. Defaults to `false`.
-
-```
-POST /api/index_patterns/index_pattern
-{
-    "override": false,
-    "refresh_fields": true,
-    "index_pattern": {
-        "title": "hello"
-    }
-}
-```
-
-At creation all index pattern fields are option and you can provide them.
-
-```
-POST /api/index_patterns/index_pattern
-{
-    "index_pattern": {
-        "id": "...",
-        "version": "...",
-        "title": "...",
-        "type": "...",
-        "intervalName": "...",
-        "timeFieldName": "...",
-        "sourceFilters": [],
-        "fields": {},
-        "typeMeta": {},
-        "fieldFormats": {},
-        "fieldAttrs": {}
-    }
-}
-```
-
-The endpoint returns the created index pattern object.
-
-```json
-{
-  "index_pattern": {}
-}
-```
-
-#### Fetch an index pattern by ID
-
-Retrieve an index pattern by its ID.
-
-```
-GET /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-```
-
-Returns an index pattern object.
-
-```json
-{
-  "index_pattern": {
-    "id": "...",
-    "version": "...",
-    "title": "...",
-    "type": "...",
-    "intervalName": "...",
-    "timeFieldName": "...",
-    "sourceFilters": [],
-    "fields": {},
-    "typeMeta": {},
-    "fieldFormats": {},
-    "fieldAttrs": {}
-  }
-}
-```
-
-#### Delete an index pattern by ID
-
-Delete and index pattern by its ID.
-
-```
-DELETE /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-```
-
-Returns an '200 OK` response with empty body on success.
-
-#### Partially update an index pattern by ID
-
-Update part of an index pattern. Only provided fields will be updated on the
-index pattern, missing fields will stay as they are persisted.
-
-These fields can be update partially:
-
-- `title`
-- `timeFieldName`
-- `intervalName`
-- `fields` (optionally refresh fields)
-- `sourceFilters`
-- `fieldFormatMap`
-- `type`
-- `typeMeta`
-
-Update a title of an index pattern.
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-{
-    "index_pattern": {
-        "title": "new_title"
-    }
-}
-```
-
-All update fields are optional, you can specify the following fields.
-
-```
-POST /api/index_patterns/index_pattern
-{
-    "index_pattern": {
-        "title": "...",
-        "timeFieldName": "...",
-        "intervalName": "...",
-        "sourceFilters": [],
-        "fieldFormats": {},
-        "type": "...",
-        "typeMeta": {},
-        "fields": {}
-    }
-}
-```
-
-- `refresh_fields` &mdash; if set to `true` reloads index pattern fields after
-  the index pattern is stored. Defaults to `false`.
-
-```
-POST /api/index_patterns/index_pattern
-{
-    "refresh_fields": true,
-    "index_pattern": {
-        "fields": {}
-    }
-}
-```
-
-This endpoint returns the updated index pattern object.
-
-```json
-{
-  "index_pattern": {}
-}
-```
-
-### Fields API
-
-Fields API allows to change field metadata, such as `count`, `customLabel`, and `format`.
-
-#### Update fields
-
-Update endpoint allows you to update fields presentation metadata, such as `count`,
-`customLabel`, and `format`. You can update multiple fields in one request. Updates
-are merges with persisted metadata. To remove existing metadata specify `null` as value.
-
-Set popularity `count` for field `foo`:
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
-{
-    "fields": {
-        "foo": {
-            "count": 123
-        }
-    }
-}
-```
-
-Update multiple metadata values and fields in one request:
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
-{
-    "fields": {
-        "foo": {
-            "count": 123,
-            "customLabel": "Foo"
-        },
-        "bar": {
-            "customLabel": "Bar"
-        }
-    }
-}
-```
-
-Use `null` value to delete metadata:
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
-{
-    "fields": {
-        "foo": {
-            "customLabel": null
-        }
-    }
-}
-```
-
-This endpoint returns the updated index pattern object.
-
-```json
-{
-  "index_pattern": {}
-}
-```
-
-### Scripted Fields API
-
-Scripted Fields API provides CRUD API for scripted fields of an index pattern.
-
-#### Create a scripted field
-
-Create a field by simply specifying its name, will default to `string` type. Returns
-an error if a field with the provided name already exists.
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
-{
-    "field": {
-        "name": "my_field"
-    }
-}
-```
-
-Create a field by specifying all field properties.
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
-{
-    "field": {
-        "name": "",
-        "type": "",
-        "searchable": false,
-        "aggregatable": false,
-        "count": 0,
-        "script": "",
-        "scripted": false,
-        "lang": "",
-        "conflictDescriptions": {},
-        "format": {},
-        "esTypes": [],
-        "readFromDocValues": false,
-        "subType": {},
-        "indexed": false,
-        "customLabel": "",
-        "shortDotsEnable": false
-    }
-}
-```
-
-#### Upsert a scripted field
-
-Creates a new field or updates an existing one, if one already exists with the same name.
-
-Create a field by simply specifying its name.
-
-```
-PUT /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
-{
-    "field": {
-        "name": "my_field"
-    }
-}
-```
-
-Create a field by specifying all field properties.
-
-```
-PUT /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
-{
-    "field": {
-        "name": "",
-        "type": "",
-        "searchable": false,
-        "aggregatable": false,
-        "count": 0,
-        "script": "",
-        "scripted": false,
-        "lang": "",
-        "conflictDescriptions": {},
-        "format": {},
-        "esTypes": [],
-        "readFromDocValues": false,
-        "subType": {},
-        "indexed": false,
-        "customLabel": "",
-        "shortDotsEnable": false
-    }
-}
-```
-
-#### Fetch a scripted field
-
-Fetch an existing index pattern field by field name.
-
-```
-GET /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
-```
-
-Returns the field object.
-
-```json
-{
-  "field": {}
-}
-```
-
-#### Delete a scripted field
-
-Delete a field of an index pattern.
-
-```
-DELETE /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
-```
-
-#### Update a an existing scripted field
-
-Updates an exiting field by mergin provided properties with the existing ones. If
-there is no existing field with the specified name, returns a `404 Not Found` error.
-
-You can specify any field properties, except `name` which is specified in the URL path.
-
-```
-POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
-{
-    "field": {
-        "type": "",
-        "searchable": false,
-        "aggregatable": false,
-        "count": 0,
-        "script": "",
-        "scripted": false,
-        "lang": "",
-        "conflictDescriptions": {},
-        "format": {},
-        "esTypes": [],
-        "readFromDocValues": false,
-        "subType": {},
-        "indexed": false,
-        "customLabel": "",
-        "shortDotsEnable": false
-    }
-}
-```
 
 ## Query