This PR updates the Elasticsearch ILM tutorial for the newer screenshot
and the primary shard update in the text.
Co-authored-by: Satyam Mishra <satyyam.misshra1121@gmail.com>
* Add new Previous Step Info field to LifecycleExecutionState
* Add new field to IndexLifecycleExplainResponse
* Add new field to TransportExplainLifecycleAction
* Add logic to IndexLifecycleTransition to keep previous setp info
* Switch tests to use Java standard Clock class
for any time based testing, this is the recommended method
* Fix tests for new field
Also refactor tests to newer style
* Add test to ensure step info is preserved
Across auto retries
* Add docs for new field
* Changelog Entry
* Update docs/changelog/113187.yaml
* Revert "Switch tests to use Java standard Clock class"
This reverts commit 241074c735.
* PR Changes
* PR Changes - Improve docs wording
* Integration test for new ILM explain field
* Use ROOT locale instead of default toLowerCase
* PR Changes - Switch to block strings
* Remove forbidden API usage
---------
Co-authored-by: Mary Gouseti <mgouseti@gmail.com>
Allows setting index total_shards_per_node in the SearchableSnapshot action of ILM to remediate hot spot in shard allocation for searchable snapshot index.
Closes#112261
* Remove zstd feature flag for index codec best compression. (#112665)
ZStandard was added via #103374 a few months ago to snapshot builds of Elasticsearch only and benchmark results have shown that using zstd is a better trade off compared to deflate for when index.codec is set to best_compression.
This change removes the feature flag for ZStandard stored field compression for indices with index.codec set to best_compression.
* Update docs/changelog/112857.yaml
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* (Doc+) Link API to parent Doc part1
---------
Co-authored-by: shainaraskas <shaina.raskas@elastic.co>
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
Spells out some cases in which ILM doesn't delete the underlying
searchable snapshot and instructs users to delete them manually instead.
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
* (Doc+) Delineate Bootstrapping Data Stream from Alias
👋 howdy, team!
This is follow-up to [elasticsearch#107327](https://github.com/elastic/elasticsearch/pull/107327). I realized my mistake was that we had duplicate sentences in different sections so I edited the wrong area. However, it seemed like a good opportunity to consider clarifying the page better by fixing header links so that the sub-sections reflect as sub-headers instead of all being equal. Thoughts?
* Apply suggestions from code review
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
---------
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
The source index of a shrink action is made read-only to perform the shrink. During shrink, settings are copied from source index to target index, causing the target index to also be read-only. This change adds a shrink policy argument making the target index writable after shrinking. The default is to keep the read-only setting to preserve current behavior.
closes#106599
👋 @dakrone mentioned baby-update for docs.
Sometimes users believe they should point client-side (e.g. Logstash's Elasticsearch output `index`) to the bootstrapped index. This highlights they point ingest towards the alias as we're expecting.
* Clarify data stream recommendations and best practices
Our documentation around data streams versus aliases could be interpreted in a way where someone doing *any* updates thinks they need to use an alias with indices instead of a data stream. This commit enhances the documentation around these areas to determine the correct abstraction in a more concrete way. It also tries to clarify that data streams still allow updates to the backing indices, and that a difference is last-write-wins versus first-write-wins.
These statements come off a little too strongly towards "don't use data streams if you *ever* have updates", but they do support updates when necessary, as long as the backing indices are used.
The ILM `_explain` API displays the user-configured rollover action with the user-defined rollover conditions. However, the implicit conditions were not visible anywhere (except in the documentation and some debug-level logging).
The ILM `_explain` API example responses included human-readable (time) fields, but the example URL didn't actually return those human-readable fields (only the raw fields).
In this PR we enable ILM to handle the following scenarios:
- An ILM policy with the a searchable snapshot action in hot or cold is added on a partially mounted searchable snapshot.
- An ILM policy with the a searchable snapshot action in frozen is added on a fully mounted searchable snapshot.
The searchable snapshot could have had a previous ILM policy that has been removed via POST <index>/_ilm/remove or it might not have been managed at all.
- Creates a new StackTemplateRegistry that uses the new names
- The new registry only respects stack.templates.enabled for index templates
- Renames the old registry to LegacyStackTemplateRegistry
- Component templates are not duplicated but registered under two different names
- Documents the new naming convention
- Index templates are not renamed, at least for now, as there are some challenges with it
See 7fd0423 for more details.
* [DOC+] ILM min_age interpretation
👋 hiya, team!
From the time y'all helped me write this [ILM Troubleshooting blog](https://www.elastic.co/blog/troubleshooting-elasticsearch-ilm-common-issues-and-fixes) in 2021 and we later ported errors to [this doc](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-error-handling.html) via https://github.com/elastic/elasticsearch/issues/75849, the remaining top-gotcha user's raise is "Common issue 3" that ILM's `min_age` calculates off rollover time fallback index creation time.
This PR cross-pollinates the blog quote into the docs so that Support can link it to users and so it becomes Google-able.
> Common issue 3: min_age calculation clarification
> When working with customers, I have seen confusion about how min_age works. The min_age must increase between subsequent phases. If rollover is used, min_age is calculated off the rollover date. This is because rollover generates a new index and the new index’s creation date is used in the calculation. Otherwise, min_age is calculated off the original index’s creation date.
* Apply suggestions from code review
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
When ILM is mounting a searchable snapshot in the frozen tier, we chose to ignore the `total_shards_per_node` setting because it's very likely to block shards from being assigned. Usually the `total_shards_per_node` is configured for one of the previous tiers that have usually more nodes than the frozen tier.
The migrate action (although no allowed in the frozen phase) would seem
to convert `frozen` to `data_frozen,data_cold,data_warm,data_hot` tier
configuration. As the migrate action is not allowed in the frozen phase
this would never happen, however the code is confusing as it seems like
it could.
The migrate to data tiers routing service shared the code used by the
`migrate` action that converted `frozen` to
`data_frozen,data_cold,data_warm,data_hot` if it would encounter an
index without any `_tier_preference` setting but with a custom node
attribute configured to `frozen` e.g. `include.data: frozen`
As part of https://github.com/elastic/elasticsearch/issues/84758 we have
seen frozen indices with the `data_frozen,data_cold,data_warm,data_hot`
tier preference however we could never reproduce it.
Relates to https://github.com/elastic/elasticsearch/issues/84758
The `data-streams/downsampling.asciidoc` test was missing a teardown clean of the ILM policies created. Due to this tests *do not have* the string `ilm` in its name, the automatic teardown process that cleans up the resources (check `ESRestTestCase.java#L815` & `DocsClientYamlTestSuiteIT.java` lines 177 & 195) is not executed for this specific test. In the case this test runs right before the `get-lifecycle` test, the policy won't be automatically deleted hence the test checking the version will fail. Finally, the order of execution of the test is not guaranteed by the suite.
* [+DOC] Restore policies in restoring ILM indices
👋 howdy! This may need Asciidoc reformatting. Will you kindly add in express commentary on [Restore a managed Datastream or Index](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-and-snapshots.html?edit) to also restore ILM policies as needed (via `include_global_state`). Otherwise, you induce ILM errors once ILM starts (and have to do a form of repeating the entire outlined procedure to get indices going through correctly.)
* Apply suggestions from code review
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
