mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-30 02:13:33 -04:00
This change introduces the configuration option `ignore_missing_component_templates` as discussed in https://github.com/elastic/elasticsearch/issues/92426 The implementation [option 6](https://github.com/elastic/elasticsearch/issues/92426#issuecomment-1372675683) was picked with a slight adjustment meaning no patterns are allowed. ## Implementation During the creation of an index template, the list of component templates is checked if all component templates exist. This check is extended to skip any component templates which are listed under `ignore_missing_component_templates`. An index template that skips the check for the component template `logs-foo@custom` looks as following: ``` PUT _index_template/logs-foo { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ``` The component template `logs-foo@package` has to exist before creation. It can be created with: ``` PUT _component_template/logs-foo@custom { "template": { "mappings": { "properties": { "host.ip": { "type": "ip" } } } } } ``` ## Testing For manual testing, different scenarios can be tested. To simplify testing, the commands from `.http` file are added. Before each test run, a clean cluster is expected. ### New behaviour, missing component template With the new config option, it must be possible to create an index template with a missing component templates without getting an error: ``` ### Add logs-foo@package component template PUT http://localhost:9200/ _component_template/logs-foo@package Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.name": { "type": "keyword" } } } } } ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ### Create data stream PUT http://localhost:9200/ _data_stream/logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ### Check if mappings exist GET http://localhost:9200/ logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ``` It is checked if all templates could be created and data stream mappings are correct. ### Old behaviour, with all component templates In the following, a component template is made optional but it already exists. It is checked, that it will show up in the mappings: ``` ### Add logs-foo@package component template PUT http://localhost:9200/ _component_template/logs-foo@package Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.name": { "type": "keyword" } } } } } ### Add logs-foo@custom component template PUT http://localhost:9200/ _component_template/logs-foo@custom Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.ip": { "type": "ip" } } } } } ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ### Create data stream PUT http://localhost:9200/ _data_stream/logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ### Check if mappings exist GET http://localhost:9200/ logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ``` ### Check old behaviour Ensure, that the old behaviour still exists when a component template is used that is not part of `ignore_missing_component_templates`: ``` ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ``` Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
165 lines
5 KiB
Text
165 lines
5 KiB
Text
[[index-templates]]
|
|
= Index templates
|
|
|
|
NOTE: This topic describes the composable index templates introduced in {es} 7.8.
|
|
For information about how index templates worked previously,
|
|
see the <<indices-templates-v1,legacy template documentation>>.
|
|
|
|
[[getting]]
|
|
An index template is a way to tell {es} how to configure an index when it is
|
|
created. For data streams, the index template configures the stream's backing
|
|
indices as they are created. Templates are configured
|
|
*prior to index creation*. When an index is created - either manually or
|
|
through indexing a document - the template settings are used as a basis for
|
|
creating the index.
|
|
|
|
There are two types of templates: index templates and <<indices-component-template,component templates>>. Component templates are reusable building
|
|
blocks that configure mappings, settings, and aliases. While you can use
|
|
component templates to construct index templates, they aren't directly applied
|
|
to a set of indices. Index templates can contain a collection of component
|
|
templates, as well as directly specify settings, mappings, and aliases.
|
|
|
|
The following conditions apply to index templates:
|
|
|
|
* Composable templates take precedence over legacy templates. If no composable
|
|
template matches a given index, a legacy template may still match and be
|
|
applied.
|
|
* If an index is created with explicit settings and also matches an index
|
|
template, the settings from the <<indices-create-index,create index>> request
|
|
take precedence over settings specified in the index template and its component
|
|
templates.
|
|
* If a new data stream or index matches more than one index template, the index
|
|
template with the highest priority is used.
|
|
|
|
[[avoid-index-pattern-collisions]]
|
|
.Avoid index pattern collisions
|
|
****
|
|
{es} has built-in index templates, each with a priority of `100`, for the
|
|
following index patterns:
|
|
|
|
// tag::built-in-index-template-patterns[]
|
|
- `logs-*-*`
|
|
- `metrics-*-*`
|
|
- `synthetics-*-*`
|
|
// end::built-in-index-template-patterns[]
|
|
|
|
{fleet-guide}/fleet-overview.html[{agent}] uses these templates to create
|
|
data streams. Index templates created by {fleet} integrations use similar
|
|
overlapping index patterns and have a priority up to `200`.
|
|
|
|
If you use {fleet} or {agent}, assign your index templates a priority lower than
|
|
`100` to avoid overriding these templates. Otherwise, to avoid accidentally
|
|
applying the templates, do one or more of the following:
|
|
|
|
- To disable all built-in index and component templates, set
|
|
<<stack-templates-enabled,`stack.templates.enabled`>> to `false` using the
|
|
<<cluster-update-settings,cluster update settings API>>.
|
|
|
|
- Use a non-overlapping index pattern.
|
|
|
|
- Assign templates with an overlapping pattern a `priority` higher than `200`.
|
|
For example, if you don't use {fleet} or {agent} and want to create a template
|
|
for the `logs-*` index pattern, assign your template a priority of `500`. This
|
|
ensures your template is applied instead of the built-in template for
|
|
`logs-*-*`.
|
|
****
|
|
|
|
[discrete]
|
|
[[create-index-templates]]
|
|
== Create index template
|
|
|
|
Use the <<indices-put-template,index template>> and <<indices-component-template,put component template>> APIs to create and update index templates.
|
|
You can also <<index-mgmt,manage index templates>> from Stack
|
|
Management in {kib}.
|
|
|
|
The following requests create two component templates.
|
|
|
|
[source,console]
|
|
----
|
|
PUT _component_template/component_template1
|
|
{
|
|
"template": {
|
|
"mappings": {
|
|
"properties": {
|
|
"@timestamp": {
|
|
"type": "date"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
PUT _component_template/runtime_component_template
|
|
{
|
|
"template": {
|
|
"mappings": {
|
|
"runtime": { <1>
|
|
"day_of_week": {
|
|
"type": "keyword",
|
|
"script": {
|
|
"source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----
|
|
<1> This component template adds a <<runtime-mapping-fields,runtime field>>
|
|
named `day_of_week` to the mappings when a new index matches the template.
|
|
|
|
The following request creates an index template that is _composed of_ these
|
|
component templates.
|
|
|
|
[source,console]
|
|
----
|
|
PUT _index_template/template_1
|
|
{
|
|
"index_patterns": ["te*", "bar*"],
|
|
"template": {
|
|
"settings": {
|
|
"number_of_shards": 1
|
|
},
|
|
"mappings": {
|
|
"_source": {
|
|
"enabled": true
|
|
},
|
|
"properties": {
|
|
"host_name": {
|
|
"type": "keyword"
|
|
},
|
|
"created_at": {
|
|
"type": "date",
|
|
"format": "EEE MMM dd HH:mm:ss Z yyyy"
|
|
}
|
|
}
|
|
},
|
|
"aliases": {
|
|
"mydata": { }
|
|
}
|
|
},
|
|
"priority": 500,
|
|
"composed_of": ["component_template1", "runtime_component_template"], <1>
|
|
"version": 3,
|
|
"_meta": {
|
|
"description": "my custom"
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
|
|
////
|
|
|
|
[source,console]
|
|
----
|
|
DELETE _index_template/template_1
|
|
DELETE _component_template/runtime_component_template
|
|
DELETE _component_template/component_template1
|
|
----
|
|
// TEST[continued]
|
|
|
|
////
|
|
|
|
include::simulate-multi-component-templates.asciidoc[]
|
|
|
|
include::ignore-missing-component-templates.asciidoc[]
|