elasticsearch/docs/reference
Keith Massey 37125f265d
Adding index_template_substitutions to the simulate ingest API (#114128) (#114374)
This adds support for a new `index_template_substitutions` field to the
body of an ingest simulate API request. These substitutions can be used
to change the pipeline(s) used for ingest, or to change the mappings
used for validation. It is similar to the
`component_template_substitutions` added in #113276. Here is an example
that shows both of those usages working together:

```
## First, add a couple of pipelines that set a field to a boolean:
PUT /_ingest/pipeline/foo-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "foo",
        "value": true
      }
    }
  ]
}

PUT /_ingest/pipeline/bar-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "bar",
        "value": true
      }
    }
  ]
}

## Now, create three component templates. One provides a mapping enforces that the only field is "foo"
## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting
## that makes "foo-pipeline" the default pipeline.
## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates
## together would cause a validation exception. These could be in the same template, but are provided
## separately just so that later we can show how multiple templates can be overridden.
PUT _component_template/mappings_template
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "foo": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT _component_template/mappings_template_with_bar
{
    "template": {
      "mappings": {
        "dynamic": "strict",
        "properties": {
          "foo": {
            "type": "keyword"
          },
          "bar": {
            "type": "boolean"
          }
        }
      }
    }
}

PUT _component_template/settings_template
{
  "template": {
    "settings": {
      "index": {
        "default_pipeline": "foo-pipeline"
      }
    }
  }
}

## Here we create an index template  pulling in both of the component templates above
PUT _index_template/template_1
{
  "index_patterns": ["foo*"],
  "composed_of": ["mappings_template", "settings_template"]
}

## We can index a document here to create the index, or not. Either way the simulate call ought to work the same
POST foo-1/_doc
{
  "foo": "FOO"
}

## This will not blow up with validation exceptions because the substitute "index_template_substitutions"
## uses `mappings_template_with_bar`, which adds the bar field.
## And the bar-pipeline is executed rather than the foo-pipeline because the substitute
## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo"
## does not get set to an invalid type.
POST _ingest/_simulate?pretty&index=foo-1
{
  "docs": [
    {
      "_id": "asdf",
      "_source": {
        "foo": "foo",
        "bar": "bar"
      }
    }
  ],
  "component_template_substitutions": {
    "settings_template": {
      "template": {
        "settings": {
          "index": {
            "default_pipeline": "bar-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "template_1": {
      "index_patterns": ["foo*"],
      "composed_of": ["mappings_template_with_bar", "settings_template"]
    }
  }
}
```
2024-10-09 13:04:23 +11:00
..
aggregations Deprecate legacy params from range query (#113286) (#113610) 2024-09-27 01:10:46 +10:00
analysis Deprecate dutch_kp and lovins stemmer as they are removed in Lucene 10 (#113143) (#113815) 2024-10-02 01:57:10 +10:00
autoscaling (Doc+) Link API doc to parent object - part1 (#111951) 2024-08-20 14:58:18 -06:00
behavioral-analytics/apis (Doc+) Link API doc to parent object - part1 (#111951) 2024-08-20 14:58:18 -06:00
cat Use deprecation logger for CLDR date format specifiers (#112917) 2024-09-19 15:50:37 +01:00
ccr (Doc+) Link API doc to parent object - part1 (#111951) 2024-08-20 14:58:18 -06:00
cluster Add more dense_vector details for cluster stats field stats (#113607) (#113884) 2024-10-03 23:16:22 +10:00
commands fix typos (#113329) (#113400) 2024-09-24 02:05:57 +10:00
connector [Zoom] Update existing scopes with granular scopes (#113994) (#114238) 2024-10-08 01:15:21 +11:00
data-management [DOC] auto migrate only for default template (#82043) 2022-05-10 11:35:19 -04:00
data-streams Expose global retention settings via data stream lifecycle API (#112210) 2024-09-02 18:40:08 +10:00
docs (+Doc) Link setting reference for action.auto_create_index (#110558) 2024-07-11 07:57:18 -06:00
eql (Doc+) Link API doc to parent object - part1 (#111951) 2024-08-20 14:58:18 -06:00
esql Docs: ESQL doesn't preserve nulls in a list (#114335) (#114339) 2024-10-09 03:53:24 +11:00
features/apis Simplify reset-features API (#110866) 2024-07-15 13:23:21 +01:00
fleet [Fleet] Fix Fleet API docs (#97981) 2023-07-27 17:22:44 +02:00
graph Fix a couple of docs typos (#99028) 2023-08-30 09:01:34 -04:00
health [DOCS] Fixes the description of 'affected_resources' in health API documentation (#111833) 2024-08-13 18:51:49 +02:00
high-availability Round up shard allocation / recovery / relocation concepts (#109943) 2024-07-25 14:44:57 -04:00
how-to [DOCS] Fix typo in knn tuning guide (#113880) (#113922) 2024-10-02 18:24:09 +10:00
ilm #101193 Preserve Step Info Across ILM Auto Retries (#113187) (#113786) 2024-09-30 21:49:19 +10:00
images [DOCS] Update Using ESQL in Kibana doc (#108715) 2024-05-17 12:36:04 +02:00
index-modules #101472 Updates default index.translog.flush_threshold_size value (#112052) 2024-09-10 11:08:53 +02:00
indices Adding index_template_substitutions to the simulate ingest API (#114128) (#114374) 2024-10-09 13:04:23 +11:00
inference [DOCS] Documents completion task type for the AlibabaCloud AI Searc inference service. (#113845) (#113851) 2024-10-01 22:08:50 +10:00
ingest Adding index_template_substitutions to the simulate ingest API (#114128) (#114374) 2024-10-09 13:04:23 +11:00
licensing [DOCS] Remove testenv annotations from doc snippet tests (#80023) 2021-11-05 18:38:50 -04:00
mapping [8.16] Change default locale of date mappers to ENGLISH (#112799) (#114210) 2024-10-07 15:51:36 +01:00
migration Add some information on locale database to the ES docs (#113587) 2024-09-30 09:28:13 +01:00
ml [DOCS] Adds DeBERTa v2 tokenization params to infer trained model API docs (#114242) (#114277) 2024-10-08 18:05:23 +11:00
modules Upgrade to Lucene 9.12.0 (#113333) (#113835) 2024-10-01 13:55:02 +01:00
monitoring [Docs] Update Monitoring docs with integration assets info (#112164) 2024-09-04 16:07:45 -04:00
query-dsl Revert semantic query passage ranking documentation (#113982) (#113984) 2024-10-03 07:44:04 +10:00
query-rules/apis [Query rules] Add exclude query rule type (#111420) 2024-08-02 08:03:54 -04:00
quickstart [DOCS][101] Update first quick start with mappings examples (#113558) (#113972) 2024-10-03 04:28:07 +10:00
release-notes Forward port release notes for v8.15.2 (#114067) 2024-10-04 12:00:50 +02:00
repositories-metering-api [DOCS] Refactor book-scoped variables in docs/reference/index.asciidoc (#107413) 2024-04-17 14:37:07 +02:00
reranking [DOCS] Update re-ranking intro to remove confusion about stages (#114302) (#114306) 2024-10-08 23:41:58 +11:00
rest-api Document that ?wait_for_active_shards=0 is permitted (#114091) (#114273) 2024-10-08 17:24:15 +11:00
rollup Disallow new rollup jobs in clusters with no rollup usage. (#108624) 2024-05-21 09:38:10 +02:00
scripting Aggs: Scripted metric allow list docs (#109635) 2024-07-11 14:27:51 +02:00
search [DOCS] Removes link from semantic text tutorial. (#114038) (#114045) 2024-10-04 04:16:27 +10:00
search-application/apis Allow typed_keys for search application Search API (#108007) 2024-04-30 11:31:45 +02:00
searchable-snapshots Updates-warning-about-mounting-snapshots (#112057) 2024-08-22 12:22:32 +02:00
security (Doc+) Update example SAML blog for Okta (#112934) (#113098) 2024-09-18 20:30:59 +10:00
settings fix typos (#113329) (#113400) 2024-09-24 02:05:57 +10:00
setup Clarify support for bundled JDK (#113993) (#114208) 2024-10-07 20:36:01 +11:00
shutdown/apis [DOCS] Align docs to implementation for timeout parameters (#108593) 2024-05-16 13:05:39 +10:00
slm/apis Add interval based SLM scheduling (#110847) 2024-08-22 21:15:29 -05:00
snapshot-restore Add max_multipart_parts setting to S3 repository (#113989) (#114161) 2024-10-04 12:14:03 -07:00
sql Removing server from the version compatibility table (#109168) 2024-05-29 11:57:46 -06:00
synonyms/apis Clarify synonyms docs (#110822) 2024-07-18 10:20:26 +02:00
tab-widgets [DOCS] Create Elasticsearch basics section, refactor quickstarts section (#112436) (#113543) 2024-09-26 01:55:19 +10:00
text-structure/apis [DOCS] Refactor book-scoped variables in docs/reference/index.asciidoc (#107413) 2024-04-17 14:37:07 +02:00
transform [DOCS] Adds an admonition to the transform painless examples. (#113706) (#113762) 2024-09-30 17:57:33 +10:00
troubleshooting (Doc+) Avoid search pile up by setting default timeout (#112846) (#113621) 2024-09-27 01:31:18 +10:00
upgrade (Doc+) Link Gateway Settings to Full Restart (#110902) 2024-07-25 09:10:19 -06:00
vectors Adds new bit element_type for dense_vectors (#110059) 2024-06-27 04:48:41 +10:00
watcher Deprecate legacy params from range query (#113286) (#113610) 2024-09-27 01:10:46 +10:00
aggregations.asciidoc
alias.asciidoc Mention alias filters don't apply for get-by-id in docs (#108433) 2024-05-08 15:28:25 -06:00
analysis.asciidoc Update Lucene analysis base url (#84094) 2022-02-17 12:44:12 +01:00
api-conventions.asciidoc Add known issue docs for #111854 (#111978) 2024-08-20 07:25:55 +01:00
cat.asciidoc Add 'dataset' size to cat indices and cat shards (#98622) 2023-08-22 15:36:32 -06:00
cluster.asciidoc Fix voting_only:true in node spec docs (#99099) 2023-08-31 17:41:50 +01:00
data-management.asciidoc [DOCS] Update tech preview copy (#101606) 2023-10-31 10:31:07 -04:00
data-rollup-transform.asciidoc [DOCS] Deprecate rollups (#101265) 2023-10-25 16:52:25 -04:00
datatiers.asciidoc (Doc+) Flush out Data Tiers (#107981) 2024-07-18 14:35:41 -06:00
dependencies-versions.asciidoc [DOCS] Replace dependencies list with a link. Closes #84863 (#90694) 2022-11-09 14:37:55 -08:00
docs.asciidoc
geospatial-analysis.asciidoc Simple addition of ES|QL to geo overview page (#111158) 2024-07-23 12:00:05 +02:00
gs-index.asciidoc
high-availability.asciidoc [DOCS] Overhaul snapshot and restore docs (#79081) 2021-11-15 12:45:07 -05:00
how-to.asciidoc Add guide for tuning kNN search (#89782) 2022-10-12 14:53:53 -07:00
index-modules.asciidoc [8.x] Remove zstd feature flag for index codec best compression. (#112665) (#112857) 2024-09-14 02:48:37 +10:00
index.asciidoc [DOCS] Port connector docs from Enterprise Search guide (#112953) (#113771) 2024-09-30 19:01:25 +10:00
index.x.asciidoc
indices.asciidoc Resolve Cluster API (#102726) 2024-02-08 10:50:05 -05:00
ingest.asciidoc [DOCS] Port connector docs from Enterprise Search guide (#112953) (#113771) 2024-09-30 19:01:25 +10:00
intro.asciidoc [DOCS] Update URL (#114292) (#114304) 2024-10-08 23:18:42 +11:00
landing-page.asciidoc [DOCS] Create Elasticsearch basics section, refactor quickstarts section (#112436) (#113543) 2024-09-26 01:55:19 +10:00
links.asciidoc
mapping.asciidoc [DOCS][101] Update first quick start with mappings examples (#113558) (#113972) 2024-10-03 04:28:07 +10:00
query-dsl.asciidoc Group vector queries into new section (#110722) 2024-07-11 14:45:35 +02:00
redirects.asciidoc [DOCS] Add ES quickstart (#102226) 2023-11-24 15:27:41 +01:00
release-notes.asciidoc Forward port release notes for v8.15.2 (#114067) 2024-10-04 12:00:50 +02:00
run-elasticsearch-locally.asciidoc [8.x] [DOCS] Update local dev instructions to use start-local (#113848) (#113928) 2024-10-02 16:01:43 +02:00
scripting.asciidoc [DOCS] Add documentation for Painless field API (#83388) 2022-02-03 15:15:38 -05:00
search.asciidoc Add retrievers using the parser-only approach (#105470) 2024-03-12 10:11:55 -07:00
setup.asciidoc [DOCS] Create Elasticsearch basics section, refactor quickstarts section (#112436) (#113543) 2024-09-26 01:55:19 +10:00
troubleshooting.asciidoc (Doc+) Capture Elasticsearch diagnostic (#108259) 2024-05-09 10:27:19 -06:00
upgrade.asciidoc [DOCS] Refactor book-scoped variables in docs/reference/index.asciidoc (#107413) 2024-04-17 14:37:07 +02:00