elasticsearch/docs/reference/indices/index-templates.asciidoc
Nicolas Ruflin 9f4d7fafad
Add ignore_missing_component_templates config option (#92436)
This change introduces the configuration option `ignore_missing_component_templates` as discussed in https://github.com/elastic/elasticsearch/issues/92426 The implementation [option 6](https://github.com/elastic/elasticsearch/issues/92426#issuecomment-1372675683) was picked with a slight adjustment meaning no patterns are allowed.

## Implementation

During the creation of an index template, the list of component templates is checked if all component templates exist. This check is extended to skip any component templates which are listed under `ignore_missing_component_templates`. An index template that skips the check for the component template `logs-foo@custom` looks as following:


```
PUT _index_template/logs-foo
{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}
```

The component template `logs-foo@package` has to exist before creation. It can be created with:

```
PUT _component_template/logs-foo@custom
{
  "template": {
    "mappings": {
      "properties": {
        "host.ip": {
          "type": "ip"
        }
      }
    }
  }
}
```

## Testing

For manual testing, different scenarios can be tested. To simplify testing, the commands from `.http` file are added. Before each test run, a clean cluster is expected.

### New behaviour, missing component template

With the new config option, it must be possible to create an index template with a missing component templates without getting an error:

```
### Add logs-foo@package component template

PUT http://localhost:9200/
    _component_template/logs-foo@package
Authorization: Basic elastic password
Content-Type: application/json

{
  "template": {
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword"
        }
      }
    }
  }
}

### Add logs-foo index template

PUT http://localhost:9200/
    _index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json

{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}

### Create data stream

PUT http://localhost:9200/
    _data_stream/logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json

### Check if mappings exist

GET http://localhost:9200/
    logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json
```

It is checked if all templates could be created and data stream mappings are correct.

### Old behaviour, with all component templates

In the following, a component template is made optional but it already exists. It is checked, that it will show up in the mappings:

```
### Add logs-foo@package component template

PUT http://localhost:9200/
    _component_template/logs-foo@package
Authorization: Basic elastic password
Content-Type: application/json

{
  "template": {
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword"
        }
      }
    }
  }
}

### Add logs-foo@custom component template

PUT http://localhost:9200/
    _component_template/logs-foo@custom
Authorization: Basic elastic password
Content-Type: application/json

{
  "template": {
    "mappings": {
      "properties": {
        "host.ip": {
          "type": "ip"
        }
      }
    }
  }
}

### Add logs-foo index template

PUT http://localhost:9200/
    _index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json

{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}

### Create data stream

PUT http://localhost:9200/
    _data_stream/logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json

### Check if mappings exist

GET http://localhost:9200/
    logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json
```

### Check old behaviour

Ensure, that the old behaviour still exists when a component template is used that is not part of `ignore_missing_component_templates`: 

```
### Add logs-foo index template

PUT http://localhost:9200/
    _index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json

{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}
```

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2023-01-31 08:40:29 -07:00

165 lines
5 KiB
Text

[[index-templates]]
= Index templates
NOTE: This topic describes the composable index templates introduced in {es} 7.8.
For information about how index templates worked previously,
see the <<indices-templates-v1,legacy template documentation>>.
[[getting]]
An index template is a way to tell {es} how to configure an index when it is
created. For data streams, the index template configures the stream's backing
indices as they are created. Templates are configured
*prior to index creation*. When an index is created - either manually or
through indexing a document - the template settings are used as a basis for
creating the index.
There are two types of templates: index templates and <<indices-component-template,component templates>>. Component templates are reusable building
blocks that configure mappings, settings, and aliases. While you can use
component templates to construct index templates, they aren't directly applied
to a set of indices. Index templates can contain a collection of component
templates, as well as directly specify settings, mappings, and aliases.
The following conditions apply to index templates:
* Composable templates take precedence over legacy templates. If no composable
template matches a given index, a legacy template may still match and be
applied.
* If an index is created with explicit settings and also matches an index
template, the settings from the <<indices-create-index,create index>> request
take precedence over settings specified in the index template and its component
templates.
* If a new data stream or index matches more than one index template, the index
template with the highest priority is used.
[[avoid-index-pattern-collisions]]
.Avoid index pattern collisions
****
{es} has built-in index templates, each with a priority of `100`, for the
following index patterns:
// tag::built-in-index-template-patterns[]
- `logs-*-*`
- `metrics-*-*`
- `synthetics-*-*`
// end::built-in-index-template-patterns[]
{fleet-guide}/fleet-overview.html[{agent}] uses these templates to create
data streams. Index templates created by {fleet} integrations use similar
overlapping index patterns and have a priority up to `200`.
If you use {fleet} or {agent}, assign your index templates a priority lower than
`100` to avoid overriding these templates. Otherwise, to avoid accidentally
applying the templates, do one or more of the following:
- To disable all built-in index and component templates, set
<<stack-templates-enabled,`stack.templates.enabled`>> to `false` using the
<<cluster-update-settings,cluster update settings API>>.
- Use a non-overlapping index pattern.
- Assign templates with an overlapping pattern a `priority` higher than `200`.
For example, if you don't use {fleet} or {agent} and want to create a template
for the `logs-*` index pattern, assign your template a priority of `500`. This
ensures your template is applied instead of the built-in template for
`logs-*-*`.
****
[discrete]
[[create-index-templates]]
== Create index template
Use the <<indices-put-template,index template>> and <<indices-component-template,put component template>> APIs to create and update index templates.
You can also <<index-mgmt,manage index templates>> from Stack
Management in {kib}.
The following requests create two component templates.
[source,console]
----
PUT _component_template/component_template1
{
"template": {
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
}
}
}
}
}
PUT _component_template/runtime_component_template
{
"template": {
"mappings": {
"runtime": { <1>
"day_of_week": {
"type": "keyword",
"script": {
"source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
}
}
}
}
}
}
----
<1> This component template adds a <<runtime-mapping-fields,runtime field>>
named `day_of_week` to the mappings when a new index matches the template.
The following request creates an index template that is _composed of_ these
component templates.
[source,console]
----
PUT _index_template/template_1
{
"index_patterns": ["te*", "bar*"],
"template": {
"settings": {
"number_of_shards": 1
},
"mappings": {
"_source": {
"enabled": true
},
"properties": {
"host_name": {
"type": "keyword"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z yyyy"
}
}
},
"aliases": {
"mydata": { }
}
},
"priority": 500,
"composed_of": ["component_template1", "runtime_component_template"], <1>
"version": 3,
"_meta": {
"description": "my custom"
}
}
----
// TEST[continued]
////
[source,console]
----
DELETE _index_template/template_1
DELETE _component_template/runtime_component_template
DELETE _component_template/component_template1
----
// TEST[continued]
////
include::simulate-multi-component-templates.asciidoc[]
include::ignore-missing-component-templates.asciidoc[]