diff --git a/docs/reference/data-streams/change-mappings-and-settings.asciidoc b/docs/reference/data-streams/change-mappings-and-settings.asciidoc index 329bc4de49cf..7f596be6edfb 100644 --- a/docs/reference/data-streams/change-mappings-and-settings.asciidoc +++ b/docs/reference/data-streams/change-mappings-and-settings.asciidoc @@ -2,7 +2,7 @@ [[data-streams-change-mappings-and-settings]] == Change mappings and settings for a data stream -Each data stream has a <>. Mappings and index settings from this template are applied to new backing indices created for the stream. This includes the stream's first backing index, which is auto-generated when the stream is created. @@ -417,7 +417,7 @@ mappings and settings you'd like to apply to the new data stream's backing indices. + This index template must meet the -<>. It +<>. It should also contain your previously chosen name or index pattern in the `index_patterns` property. + @@ -471,7 +471,7 @@ PUT /_index_template/new-data-stream-template create the new data stream. The name of the data stream must match the index pattern defined in the new template's `index_patterns` property. + -We do not recommend <>. Later, you will reindex older data from an existing data stream into this new stream. This could result in one or more backing indices that contains a mix of new and old data. diff --git a/docs/reference/data-streams/data-streams.asciidoc b/docs/reference/data-streams/data-streams.asciidoc index 26874290fae5..813319716426 100644 --- a/docs/reference/data-streams/data-streams.asciidoc +++ b/docs/reference/data-streams/data-streams.asciidoc @@ -27,7 +27,7 @@ backing indices. image::images/data-streams/data-streams-diagram.svg[align="center"] -Each data stream requires a matching <>. The +A data stream requires a matching <>. The template contains the mappings and settings used to configure the stream's backing indices. diff --git a/docs/reference/data-streams/set-up-a-data-stream.asciidoc b/docs/reference/data-streams/set-up-a-data-stream.asciidoc index 280bbb06fb9d..6a51013a9779 100644 --- a/docs/reference/data-streams/set-up-a-data-stream.asciidoc +++ b/docs/reference/data-streams/set-up-a-data-stream.asciidoc @@ -4,48 +4,74 @@ To set up a data stream, follow these steps: -. <>. -. <>. -. <>. -. <>. +* <> +* <> +* <> +* <> +* <> -You can also <>. +IMPORTANT: If you use {fleet} or {agent}, skip this tutorial. {fleet} and +{agent} set up data streams for you. See {fleet-guide}/data-streams.html[Data +streams] in the {fleet} Guide. + [discrete] -[[configure-a-data-stream-ilm-policy]] -=== Optional: Configure an {ilm-init} lifecycle policy +[[create-index-lifecycle-policy]] +=== Step 1. Create an index lifecycle policy -While optional, we recommend you configure an <> to automate the management of your data stream's backing -indices. +While optional, we recommend using {ilm-init} to automate the management of your +data stream's backing indices. {ilm-init} requires an index lifecycle policy. -In {kib}, open the menu and go to *Stack Management > Index Lifecycle Policies*. -Click *Create policy*. +To create an index lifecycle policy in {kib}, open the main menu and go to +*Stack Management > Index Lifecycle Policies*. Click *Create policy*. -[role="screenshot"] -image::images/ilm/create-policy.png[Create Policy page] - -[%collapsible] -.API example -==== -Use the <> to configure a policy: +You can also use the <>. [source,console] ---- -PUT /_ilm/policy/my-data-stream-policy +PUT _ilm/policy/my-lifecycle-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { - "max_primary_shard_size": "25GB" + "max_age": "30d", + "max_primary_shard_size": "50gb" + } + } + }, + "warm": { + "min_age": "30d", + "actions": { + "shrink": { + "number_of_shards": 1 + }, + "forcemerge": { + "max_num_segments": 1 + } + } + }, + "cold": { + "min_age": "60d", + "actions": { + "searchable_snapshot": { + "snapshot_repository": "my-snapshot-repo" + } + } + }, + "frozen": { + "min_age": "90d", + "actions": { + "searchable_snapshot": { + "snapshot_repository": "my-snapshot-repo" } } }, "delete": { - "min_age": "30d", + "min_age": "735d", "actions": { "delete": {} } @@ -54,139 +80,158 @@ PUT /_ilm/policy/my-data-stream-policy } } ---- -==== [discrete] -[[create-a-data-stream-template]] -=== Create an index template +[[create-component-templates]] +=== Step 2. Create component templates -. In {kib}, open the menu and go to *Stack Management > Index Management*. -. In the *Index Templates* tab, click *Create template*. -. In the Create template wizard, use the *Data stream* toggle to indicate the -template is used for data streams. -. Use the wizard to finish defining your template. Specify: +A data stream requires a matching index template. In most cases, you compose +this index template using one or more component templates. You typically use +separate component templates for mappings and index settings. This lets you +reuse the component templates in multiple index templates. -* One or more index patterns that match the data stream's name. + -include::{es-repo-dir}/indices/create-data-stream.asciidoc[tag=data-stream-name] +When creating your component templates, include: -* Mappings and settings for the stream's backing indices. +* A <> or <> mapping for the `@timestamp` +field. If you don't specify a mapping, {es} maps `@timestamp` as a `date` field +with default options. -* A priority for the index template -+ -include::{es-repo-dir}/indices/index-templates.asciidoc[tag=built-in-index-templates] +* Your lifecycle policy in the `index.lifecycle.name` index setting. -[[elastic-data-stream-naming-scheme]] -.The Elastic data stream naming scheme -**** -The {agent} uses the Elastic data stream naming scheme to name its data streams. -To help you organize your data consistently and avoid naming collisions, we -recommend you also use the Elastic naming scheme for your other data streams. +To create a component template in {kib}, open the main menu and go to *Stack +Management > Index Management*. In the *Index Templates* view, click *Create a +component template*. -The naming scheme splits data into different data streams based on the following -components. Each component corresponds to a -<> field defined in the -{ecs-ref}[Elastic Common Schema (ECS)]. - -`type`:: -Generic type describing the data, such as `logs`, `metrics`, or `synthetics`. -Corresponds to the `data_stream.type` field. - -`dataset`:: -Describes the ingested data and its structure. Corresponds to the -`data_stream.dataset` field. Defaults to `generic`. - -`namespace`:: -User-configurable arbitrary grouping. Corresponds to the `data_stream.dataset` -field. Defaults to `default`. - -The naming scheme separates these components with a `-` character: - -``` --- -``` - -For example, the {agent} uses the `logs-nginx.access-production` data -stream to store data with a type of `logs`, a dataset of `nginx.access`, and a -namespace of `production`. If you use the {agent} to ingest a log file, it -stores the data in the `logs-generic-default` data stream. - -For more information about the naming scheme and its benefits, see our -https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[An -introduction to the Elastic data stream naming scheme] blog post. -**** - -include::{es-repo-dir}/data-streams/data-streams.asciidoc[tag=timestamp-reqs] - -If using {ilm-init}, specify your lifecycle policy in the `index.lifecycle.name` -setting. - -TIP: Carefully consider your template's mappings and settings. Later changes may -require reindexing. See <>. - -[role="screenshot"] -image::images/data-streams/create-index-template.png[Create template page] - -[%collapsible] -.API example -==== -Use the <> to create -an index template. The template must include a `data_stream` object, indicating -it's used for data streams. +You can also use the <>. [source,console] ---- -PUT /_index_template/my-data-stream-template +# Creates a component template for mappings +PUT _component_template/my-mappings +{ + "template": { + "mappings": { + "properties": { + "@timestamp": { + "type": "date", + "format": "date_optional_time||epoch_millis" + }, + "message": { + "type": "wildcard" + } + } + } + }, + "_meta": { + "description": "Mappings for @timestamp and message fields", + "my-custom-meta-field": "More arbitrary metadata" + } +} + +# Creates a component template for index settings +PUT _component_template/my-settings { - "index_patterns": [ "my-data-stream*" ], - "data_stream": { }, - "priority": 500, "template": { "settings": { - "index.lifecycle.name": "my-data-stream-policy" + "index.lifecycle.name": "my-lifecycle-policy" } + }, + "_meta": { + "description": "Settings for ILM", + "my-custom-meta-field": "More arbitrary metadata" } } ---- // TEST[continued] -==== [discrete] -[[create-a-data-stream]] -=== Create the data stream +[[create-index-template]] +=== Step 3. Create an index template -To automatically create the data stream, submit an -<> to the stream. The stream's -name must match one of your template's index patterns. +Use your component templates to create an index template. Specify: + +* One or more index patterns that match the data stream's name. We recommend +using our {fleet-guide}/data-streams.html#data-streams-naming-scheme[data stream +naming scheme]. + +* That the template is data stream enabled. + +* Any component templates that contain your mappings and index settings. + +* A priority higher than `200` to avoid collisions with built-in templates. +See <>. + +To create an index template in {kib}, open the main menu and go to *Stack +Management > Index Management*. In the *Index Templates* view, click *Create +template*. + +You can also use the <>. +Include the `data_stream` object to enable data streams. [source,console] ---- -POST /my-data-stream/_doc/ +PUT _index_template/my-index-template { - "@timestamp": "2099-03-07T11:04:05.000Z", - "user": { - "id": "vlb44hny" - }, - "message": "Login attempt failed" + "index_patterns": ["my-data-stream*"], + "data_stream": { }, + "composed_of": [ "my-mappings", "my-settings" ], + "priority": 500, + "_meta": { + "description": "Template for my time series data", + "my-custom-meta-field": "More arbitrary metadata" + } } ---- // TEST[continued] -You can also use the <> to -manually create the data stream. The stream's name must match one of your -template's index patterns. +[discrete] +[[create-data-stream]] +=== Step 4. Create the data stream + +To automatically create the data stream, submit an +<> that targets the stream's +name. This name must match one of your index template's index patterns. The +request must use an `op_type` of `create`. Documents must include a `@timestamp` +field. [source,console] ---- -PUT /_data_stream/my-data-stream +PUT my-data-stream/_bulk +{ "create":{ } } +{ "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" } +{ "create":{ } } +{ "@timestamp": "2099-05-06T16:25:42.000Z", "message": "192.0.2.255 - - [06/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } + +POST my-data-stream/_doc +{ + "@timestamp": "2099-05-06T16:21:15.000Z", + "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" +} +---- +// TEST[continued] + +You can also manually create the stream using the +<>. The stream's name must +still match one of your template's index patterns. + +[source,console] +---- +PUT _data_stream/my-data-stream ---- // TEST[continued] // TEST[s/my-data-stream/my-data-stream-alt/] -When you create a data stream, {es} automatically creates a backing index for -the stream. This index also acts as the stream's first write index. +[discrete] +[[secure-data-stream]] +=== Step 5. Secure the data stream + +include::{xes-repo-dir}/security/authorization/alias-privileges.asciidoc[tag=data-stream-security] + +For an example, see <>. [discrete] -[[convert-an-index-alias-to-a-data-stream]] +[[convert-index-alias-to-data-stream]] === Convert an index alias to a data stream // tag::time-series-alias-tip[] @@ -196,12 +241,11 @@ functionality, require less maintenance, and automatically integrate with <>. // end::time-series-alias-tip[] -To convert an index alias with a write index to a new data stream with the same +To convert an index alias with a write index to a data stream with the same name, use the <>. During conversion, the alias’s indices become hidden backing indices for the -stream. The alias’s write index becomes the stream’s write index. Note the data -stream still requires a matching <>. +stream. The alias’s write index becomes the stream’s write index. The stream +still requires a matching index template with data stream enabled. //// [source,console] @@ -218,7 +262,7 @@ POST idx2/_doc/ "@timestamp" : "2099-01-01" } -POST /_aliases +POST _aliases { "actions": [ { @@ -237,7 +281,7 @@ POST /_aliases ] } -PUT /_index_template/template +PUT _index_template/template { "index_patterns": ["my-time-series-data"], "data_stream": { } @@ -248,79 +292,58 @@ PUT /_index_template/template [source,console] ---- -POST /_data_stream/_migrate/my-time-series-data +POST _data_stream/_migrate/my-time-series-data ---- // TEST[continued] [discrete] -[[secure-a-data-stream]] -=== Secure the data stream - -To control access to the data stream and its -data, use <>. - -[discrete] -[[get-info-about-a-data-stream]] +[[get-info-about-data-stream]] === Get information about a data stream -In {kib}, open the menu and go to *Stack Management > Index Management*. In the -*Data Streams* tab, click the data stream's name. +To get information about a data stream in {kib}, open the main menu and go to +*Stack Management > Index Management*. In the *Data Streams* view, click the +data stream's name. -[role="screenshot"] -image::images/data-streams/data-streams-list.png[Data Streams tab] - -[%collapsible] -.API example -==== -Use the <> to retrieve information -about one or more data streams: +You can also use the <>. //// [source,console] ---- -POST /my-data-stream/_rollover/ +POST my-data-stream/_rollover/ ---- // TEST[continued] //// [source,console] ---- -GET /_data_stream/my-data-stream +GET _data_stream/my-data-stream ---- // TEST[continued] -==== [discrete] -[[delete-a-data-stream]] +[[delete-data-stream]] === Delete a data stream -To delete a data stream and its backing indices, open the {kib} menu and go to -*Stack Management > Index Management*. In the *Data Streams* tab, click the -trash icon. The trash icon only displays if you have the `delete_index` +To delete a data stream and its backing indices in {kib}, open the main menu and +go to *Stack Management > Index Management*. In the *Data Streams* view, click +the trash icon. The icon only displays if you have the `delete_index` <> for the data stream. -[role="screenshot"] -image::images/data-streams/data-streams-no-delete.png[Data Streams tab] - -[%collapsible] -.API example -==== -Use the <> to delete a data -stream and its backing indices: +You can also use the <>. [source,console] ---- -DELETE /_data_stream/my-data-stream +DELETE _data_stream/my-data-stream ---- // TEST[continued] -==== //// [source,console] ---- -DELETE /_data_stream/* -DELETE /_index_template/* -DELETE /_ilm/policy/my-data-stream-policy +DELETE _data_stream/* +DELETE _index_template/* +DELETE _component_template/my-* +DELETE _ilm/policy/my-lifecycle-policy ---- // TEST[continued] //// diff --git a/docs/reference/docs/index_.asciidoc b/docs/reference/docs/index_.asciidoc index 5172af5d9ad8..3a71c4331a5f 100644 --- a/docs/reference/docs/index_.asciidoc +++ b/docs/reference/docs/index_.asciidoc @@ -58,7 +58,7 @@ stream enabled. See <>. (Required, string) Name of the data stream or index to target. + If the target doesn't exist and matches the name or wildcard (`*`) pattern of an -<>, this request creates the data stream. See <>. + @@ -195,7 +195,7 @@ exist. To update an existing document, you must use the `_doc` resource. ===== Automatically create data streams and indices If request's target doesn't exist and matches an -<>, the index operation automatically creates the data stream. See <>. diff --git a/docs/reference/eql/detect-threats-with-eql.asciidoc b/docs/reference/eql/detect-threats-with-eql.asciidoc index 900f0f2377b8..e1bc14e6d835 100644 --- a/docs/reference/eql/detect-threats-with-eql.asciidoc +++ b/docs/reference/eql/detect-threats-with-eql.asciidoc @@ -37,7 +37,7 @@ events imitating a Squiblydoo attack. The data has been mapped to To get started: . Create an <> with -<>: +<>: + //// [source,console] diff --git a/docs/reference/indices/create-data-stream.asciidoc b/docs/reference/indices/create-data-stream.asciidoc index a0b3b338eaa7..e97e9973f106 100644 --- a/docs/reference/indices/create-data-stream.asciidoc +++ b/docs/reference/indices/create-data-stream.asciidoc @@ -53,11 +53,8 @@ See <>. ``:: + -- -(Required, string) Name of the data stream to create. - -// tag::data-stream-name[] -We recommend using the <>. Data stream names must meet the following criteria: +(Required, string) Name of the data stream to create. Data stream names must +meet the following criteria: - Lowercase only - Cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, `,`, `#`, `:`, or a @@ -66,6 +63,5 @@ space character - Cannot be `.` or `..` - Cannot be longer than 255 bytes. Multi-byte characters count towards this limit faster. -// end::data-stream-name[] -- diff --git a/docs/reference/indices/delete-data-stream.asciidoc b/docs/reference/indices/delete-data-stream.asciidoc index 3add84dbf757..eb970f0f29d4 100644 --- a/docs/reference/indices/delete-data-stream.asciidoc +++ b/docs/reference/indices/delete-data-stream.asciidoc @@ -6,7 +6,7 @@ ++++ Deletes one or more <> and their backing -indices. See <>. +indices. See <>. //// [source,console] diff --git a/docs/reference/indices/get-data-stream.asciidoc b/docs/reference/indices/get-data-stream.asciidoc index aacaa0e4ecc5..3038c96ed50e 100644 --- a/docs/reference/indices/get-data-stream.asciidoc +++ b/docs/reference/indices/get-data-stream.asciidoc @@ -6,7 +6,7 @@ ++++ Retrieves information about one or more <>. -See <>. +See <>. //// [source,console] @@ -157,7 +157,7 @@ acts as a cumulative count of the stream's rollovers, starting at `1`. `_meta`:: (object) Custom metadata for the stream, copied from the `_meta` object of the -stream's matching <>. If empty, +stream's matching <>. If empty, the response omits this property. `status`:: @@ -186,7 +186,7 @@ One or more primary shards are unassigned, so some data is unavailable. Name of the index template used to create the data stream's backing indices. + The template's index pattern must match the name of this data stream. See -<>. +<>. `ilm_policy`:: (string) diff --git a/docs/reference/indices/index-templates.asciidoc b/docs/reference/indices/index-templates.asciidoc index cc75c9e76757..8fa96498c17e 100644 --- a/docs/reference/indices/index-templates.asciidoc +++ b/docs/reference/indices/index-templates.asciidoc @@ -31,6 +31,7 @@ templates. * If a new data stream or index matches more than one index template, the index template with the highest priority is used. +[[avoid-index-pattern-collisions]] .Avoid index pattern collisions **** // tag::built-in-index-templates[] diff --git a/docs/reference/indices/put-index-template.asciidoc b/docs/reference/indices/put-index-template.asciidoc index 6d1074214150..36c6d1726eb2 100644 --- a/docs/reference/indices/put-index-template.asciidoc +++ b/docs/reference/indices/put-index-template.asciidoc @@ -98,7 +98,7 @@ If this object is included, the template is used to create data streams and their backing indices. Supports an empty object: `data_stream: { }` + Data streams require a matching index template with a `data_stream` object. -See <>. +See <>. + .Properties of `data_stream` [%collapsible%open] @@ -294,7 +294,7 @@ To check the `_meta`, you can use the <>. +See <>. [source,console] -------------------------------------------------- diff --git a/docs/reference/indices/update-settings.asciidoc b/docs/reference/indices/update-settings.asciidoc index 6ed97736313e..d624607b28cd 100644 --- a/docs/reference/indices/update-settings.asciidoc +++ b/docs/reference/indices/update-settings.asciidoc @@ -156,7 +156,7 @@ and reopen the index. You cannot close the write index of a data stream. To update the analyzer for a data stream's write index and future backing -indices, update the analyzer in the <>. Then <> to apply the new analyzer to the stream’s write index and future backing indices. This affects searches and any new data added to the diff --git a/docs/reference/ingest.asciidoc b/docs/reference/ingest.asciidoc index ba749fd66c05..160fd6c12b81 100644 --- a/docs/reference/ingest.asciidoc +++ b/docs/reference/ingest.asciidoc @@ -317,7 +317,7 @@ PUT _ingest/pipeline/logs-my_app-default . Create an <> that includes your pipeline in the <> or <> index setting. Ensure the -template is <>. The +template is <>. The template's index pattern should match `logs--*`. + -- diff --git a/docs/reference/ingest/common-log-format-example.asciidoc b/docs/reference/ingest/common-log-format-example.asciidoc index 4f9e9be41632..70dbcdf740d2 100644 --- a/docs/reference/ingest/common-log-format-example.asciidoc +++ b/docs/reference/ingest/common-log-format-example.asciidoc @@ -156,7 +156,7 @@ pipeline**. You’re now ready to index the logs data to a <>. . Create an <> with -<>. +<>. + [source,console] ---- diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 4581049862a9..4fb73da2d518 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -38,14 +38,14 @@ by default as well. [WARNING] ==== Each data stream requires a matching -<>. The stream uses this +<>. The stream uses this template to create new backing indices. When restoring a data stream, ensure a matching template exists for the stream. You can do this using one of the following methods: * Check for existing templates that match the stream. If no matching template - exists, <>. + exists, <>. * Restore a global cluster state that includes a matching template for the stream. @@ -158,7 +158,7 @@ indices. The `index_settings` and `ignore_index_settings` parameters affect restored backing indices only. New backing indices created for a stream use the index settings specified in the stream's matching -<>. +<>. If you change index settings during a restore, we recommend you make similar changes in the stream's matching index template. This ensures new backing diff --git a/docs/reference/snapshot-restore/take-snapshot.asciidoc b/docs/reference/snapshot-restore/take-snapshot.asciidoc index d7c49de215f6..22f06455baff 100644 --- a/docs/reference/snapshot-restore/take-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/take-snapshot.asciidoc @@ -128,7 +128,7 @@ By setting `include_global_state` to `false` it's possible to prevent the cluste the snapshot. IMPORTANT: The global cluster state includes the cluster's index -templates, such as those <>. If your snapshot includes data streams, we recommend storing the global state as part of the snapshot. This lets you later restored any templates required for a data stream. diff --git a/x-pack/docs/en/security/authorization/alias-privileges.asciidoc b/x-pack/docs/en/security/authorization/alias-privileges.asciidoc index b05b5a4b370b..d482303e14dc 100644 --- a/x-pack/docs/en/security/authorization/alias-privileges.asciidoc +++ b/x-pack/docs/en/security/authorization/alias-privileges.asciidoc @@ -8,14 +8,12 @@ [[data-stream-privileges]] ==== Data stream privileges -A data stream consists of one or more backing indices, which store the stream's -data. Most requests sent to a data stream are routed to one or more of these -backing indices. +// tag::data-stream-security[] -Similar to an index, you can use <> -to control access to a data stream. Any role or user granted privileges to a -data stream are automatically granted the same privileges to its backing -indices. +Use <> to control access to +a data stream. Any role or user granted privileges to a data +stream are automatically granted the same privileges to its backing indices. +// end::data-stream-security[] For example, `my-data-stream` consists of two backing indices: `.ds-my-data-stream-2099.03.07-000001` and @@ -39,7 +37,7 @@ backing indices, the user can retrieve a document directly from //// [source,console] ---- -PUT /my-index/_doc/2 +PUT my-index/_doc/2 { "my-field": "foo" } @@ -48,7 +46,7 @@ PUT /my-index/_doc/2 [source,console] ---- -GET /.ds-my-data-stream-2099.03.08-000002/_doc/2 +GET .ds-my-data-stream-2099.03.08-000002/_doc/2 ---- // TEST[continued] // TEST[s/.ds-my-data-stream-2099.03.08-000002/my-index/] @@ -60,7 +58,7 @@ documents directly from `.ds-my-data-stream-2099.03.09-000003`: [source,console] ---- -GET /.ds-my-data-stream-2099.03.09-000003/_doc/2 +GET .ds-my-data-stream-2099.03.09-000003/_doc/2 ---- // TEST[continued] // TEST[s/.ds-my-data-stream-2099.03.09-000003/my-index/]