The documentation specifies the possible values for the status of the response. This endpoint is inconsistent with most others that expose the health status as it returns the values as uppercase strings rather than lowercase.
This PR fixes the cases in the documentation to align with the actual values returned in the response body.
This change introduces the configuration option `ignore_missing_component_templates` as discussed in https://github.com/elastic/elasticsearch/issues/92426 The implementation [option 6](https://github.com/elastic/elasticsearch/issues/92426#issuecomment-1372675683) was picked with a slight adjustment meaning no patterns are allowed.
## Implementation
During the creation of an index template, the list of component templates is checked if all component templates exist. This check is extended to skip any component templates which are listed under `ignore_missing_component_templates`. An index template that skips the check for the component template `logs-foo@custom` looks as following:
```
PUT _index_template/logs-foo
{
"index_patterns": ["logs-foo-*"],
"data_stream": { },
"composed_of": ["logs-foo@package", "logs-foo@custom"],
"ignore_missing_component_templates": ["logs-foo@custom"],
"priority": 500
}
```
The component template `logs-foo@package` has to exist before creation. It can be created with:
```
PUT _component_template/logs-foo@custom
{
"template": {
"mappings": {
"properties": {
"host.ip": {
"type": "ip"
}
}
}
}
}
```
## Testing
For manual testing, different scenarios can be tested. To simplify testing, the commands from `.http` file are added. Before each test run, a clean cluster is expected.
### New behaviour, missing component template
With the new config option, it must be possible to create an index template with a missing component templates without getting an error:
```
### Add logs-foo@package component template
PUT http://localhost:9200/
_component_template/logs-foo@package
Authorization: Basic elastic password
Content-Type: application/json
{
"template": {
"mappings": {
"properties": {
"host.name": {
"type": "keyword"
}
}
}
}
}
### Add logs-foo index template
PUT http://localhost:9200/
_index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json
{
"index_patterns": ["logs-foo-*"],
"data_stream": { },
"composed_of": ["logs-foo@package", "logs-foo@custom"],
"ignore_missing_component_templates": ["logs-foo@custom"],
"priority": 500
}
### Create data stream
PUT http://localhost:9200/
_data_stream/logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json
### Check if mappings exist
GET http://localhost:9200/
logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json
```
It is checked if all templates could be created and data stream mappings are correct.
### Old behaviour, with all component templates
In the following, a component template is made optional but it already exists. It is checked, that it will show up in the mappings:
```
### Add logs-foo@package component template
PUT http://localhost:9200/
_component_template/logs-foo@package
Authorization: Basic elastic password
Content-Type: application/json
{
"template": {
"mappings": {
"properties": {
"host.name": {
"type": "keyword"
}
}
}
}
}
### Add logs-foo@custom component template
PUT http://localhost:9200/
_component_template/logs-foo@custom
Authorization: Basic elastic password
Content-Type: application/json
{
"template": {
"mappings": {
"properties": {
"host.ip": {
"type": "ip"
}
}
}
}
}
### Add logs-foo index template
PUT http://localhost:9200/
_index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json
{
"index_patterns": ["logs-foo-*"],
"data_stream": { },
"composed_of": ["logs-foo@package", "logs-foo@custom"],
"ignore_missing_component_templates": ["logs-foo@custom"],
"priority": 500
}
### Create data stream
PUT http://localhost:9200/
_data_stream/logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json
### Check if mappings exist
GET http://localhost:9200/
logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json
```
### Check old behaviour
Ensure, that the old behaviour still exists when a component template is used that is not part of `ignore_missing_component_templates`:
```
### Add logs-foo index template
PUT http://localhost:9200/
_index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json
{
"index_patterns": ["logs-foo-*"],
"data_stream": { },
"composed_of": ["logs-foo@package", "logs-foo@custom"],
"ignore_missing_component_templates": ["logs-foo@custom"],
"priority": 500
}
```
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Mentions that we only report on recoveries for shard copies that
actually exist in the cluster, so you don't see all historical data if,
e.g., the shard copy relocates elsewhere.
We tell users not to force merge unless their index is read-only. This PR
proposes to soften the warning and make it more precise. This way, more users
can consider force merging for their use case, like those with append-only
indices, or those with a small number of updates that can regularly perform a
force merge.
This change adds support for kNN vector fields to the `_disk_usage` API. The
strategy:
* Iterate the vector values (using the same strategy as for doc values) to
estimate the vector data size
* Run some random vector searches to estimate the vector index size
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Closes#84801
* Revert "Revert "[DOCS] Add TSDS docs (#86905)" (#87702)"
This reverts commit 0c86d7b9b2.
* First fix to tests
* Add data_stream object to index template
* small rewording
* Add enable data stream object in gradle example setup
* Add bullet about data stream must be enabled in template
* [DOCS] Add TSDB docs
* Update docs/build.gradle
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Address Nik's comments, part 1
* Address Nik's comments, part deux
* Reword write index
* Add feature flags
* Wrap one more section in feature flag
* Small fixes
* set index.routing_path to optional
* Update storage reduction value
* Update create index template code example
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
The get settings api has accepts the expand_wildcards option. The docs
state the default value is `all`, but it is actually now `open` (which
does not include hidden or closed indices by default). This commit
changes the docs to match the existing behavior.
"Add" was out of the hyperlink context which I have fixed it.
Earlier line 71 was like : * *Add* <<set-up-lifecycle-policy,*lifecycle policy*>>
After rectifying line 71 is like : * <<set-up-lifecycle-policy,*Add lifecycle policy*>>
(cherry picked from commit 3b8d51c696)
Co-authored-by: Tapomoy Bhowmik <99604828+TapomoyBhowmik@users.noreply.github.com>
This commit adds tracking for desired nodes cluster membership.
When desired nodes are updated they are matched against the current
cluster members. Additionally when a node joins the cluster the
desired nodes cluster membership is updated.
* Resolve indices api: add 'system' attribute
* Update docs/changelog/85042.yaml
* Remove magic strings for attribute values
* Update API docs to provide possible resolved index attributes
Add a new rollover condition with the name `max_primary_shard_docs`.
Triggers rollover when the largest primary shard in the index reaches a certain number of documents.
The simulate template api should include the settings that registered IndexSettingProvider generate.
Currently, these settings are not included in the simulate template api response,
but are only used to create a dummy IndexService instance to validate aliases.
Reverts an anchor change from #46711.
Previous versions of the docs use the `_shrinking_an_index` anchor for this
section. Preserving that anchor will prevent doc build breaks in future releases.
Force merge action is a very costly action. It may take several hours to run for big indices. But current force merge rest api do not support wait_for_completion parameter.
This adds support for the wait_for_completion parameter.
Internally we already kept track of whether a data stream is replicated by CCR.
It is part of the `DataStream` class. This just adds it to the xcontent serialization
of the get data stream api response class.
Relates to elastic/kibana#118899
Removes `testenv` annotations and related code. These annotations originally let you skip x-pack snippet tests in the docs. However, that's no longer possible.
Relates to #79309, #31619
We document that `GET /_index_template/...` accepts a comma-separated
list of template names but in fact today this API accepts only a single
name or pattern. Likewise `GET /_cat/templates/...` (at least it didn't
until #78829 but that's not released yet). This commit fixes the docs to
indicate these APIs accept only a single template name and also adds
some extra validation to reject requests containing a `,` since such a
request cannot match any actual templates.
It also adjusts `GET /_cat/templates` to use the filtering built into
`TransportGetComposableIndexTemplateAction` rather than retrieving all
templates and then filtering them on the coordinating node.
The 'verbose' option to /_segments returns memory information
for each segment. However, lucene 9 has stopped tracking this memory
information as it is largely held off-heap and so is no longer significant.
This commit deprecates the 'verbose' parameter and makes it a no-op.
Fixes#75955
* [DOCS] Update remote cluster docs
* Add files, rename files, write new stuff
* Plethora of changes
* Add test and update snippets
* Redirects, moved files, and test updates
* Moved file to x-pack for tests
* Remove older CCS page and add redirects
* Cleanup, link updates, and some rewrites
* Update image
* Incorporating user feedback and rewriting much of the remote clusters page
* More changes from review feedback
* Numerous updates, including request examples for CCS and Kibana
* More changes from review feedback
* Minor clarifications on security for remote clusters
* Incorporate review feedback
Co-authored-by: Yang Wang <ywangd@gmail.com>
* Some review feedback and some editorial changes
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Yang Wang <ywangd@gmail.com>
Related to issue #77823
This does the following:
- Updates several asciidoc files that contained code snippets with
invalid JSON, most involving unnecessary trailing commas.
- Makes the switch from the Groovy JSON parser to the Jackson parser,
pursuant to the general goal of eliminating Groovy dependence.
- Makes testing of JSON validity at build time more strict.
Note that this update still allows backslash escaping for any
character. Currently that matters because of the file
"docs/reference/ml/anomaly-detection/apis/get-datafeed-stats.asciidoc",
specifically this part:
"attributes" : {
"ml.machine_memory" :
"$body.datafeeds.0.node.attributes.ml\.machine_memory",
"ml.max_open_jobs" : "512"
}
It's not clear to me what change, if any, is appropriate there. So,
I've left in the escaped period and configured the parser to ignore
it for the time being.
This commit changes the es merge policy to apply the maximum segment size
on force merges that only expunge deletes (forceMergeDeletes).
This option is useful for read-write use cases that wants to reclaim deleted docs
more aggressively than the `index.merge.policy.deletes_pct_allowed`.
Closes#61764
Relates #77270
Changes:
* Notes the delete index API can delete multiple indices at once.
* Notes deleting an index deletes its docs, shards, and metadata but does not delete any related Kibana components.
* Relocates a note about deleting a data stream's write index to the description.
* Corrects the default `expand_wildcards` value.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This updates the default for the `is_write_index` parameter of the aliases API and create alias API.
The default behavior for `is_write_index` can vary based on:
1. Whether the alias is used for indices or data streams.
2. If an index alias, whether the alias points to multiple indices.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: 诗心客 <ishixinke@qq.com>
PRs #73062 and #73043 repurposed the `alias` anchor for a new guide for index
and data stream aliases. Previously, this anchor was used for our field alias
documentation.
Repurposing the anchor has caused continuity errors for users selecting
different versions of the ES docs. It could also cause confusion for users with
a `/current/` link to the `alias` page.
This updates the anchor for the alias guide and adds a redirect page to
disambiguate the `alias` anchor.
It also fixes a bread crumb issue for redirects following the 'Modifying your
Data' redirect page.
Closes#77034.
Adds new field to recovery API to keep track of amount of data
recovered from snapshots.
The normal recovered_bytes field remains and is also increased for
recovery from snapshot but can go backwards in the unlikely case
that recovery from snapshot fails to download a file.
Relates #73496
