Initially the index.look_head_time was both used to define the index.time_series.start_time and index.time_series.end_time.
The former is now controlled by index.look_back_time and the maximum value of 7 days for index.look_ahead_time is too generous. As it also delays data being indexed to new index after rollover by up to 7 days.
This PR changes the index.look_ahead_time setting's maximum allowed value from 7 days to 2 hours, which is equal to the index.look_ahead_time setting's default. A look ahead time of 2 hours is high enough to accept data that is ahead of the current time, but avoids configuring the index.look_ahead_time setting to a too high value that causes rolled over indices to not receive writes for a very long period.
This is a breaking change, but configuring the index.look_ahead_time setting to a higher value than 2 hours will not fail. Instead 2 hours will be used a look ahead time.
This change adds a `index.look_back_time` index setting that sets the `index.time_series.start_time` setting for the first backing index when a data stream is created.
This allows accepting data that is older for initial indexing without changing the `index.look_ahead_time` setting. This setting also controls the `index.time_series.end_time` setting and would affect rollovers as well.
The default for the `index.look_back_time` is `2h`, which means documents with `@timestamp` up to 2 hours after creation of the data stream are allowed to be indexed. This is the same as is without this change, because `index.look_ahead_time` is used to set `index.time_series.start_time` of the first backing index.
Closes#98463
Added a requirement that index.look_ahead_time index setting
can't be lower than time_series.poll_interval setting.
Additional changes:
* Fixed a mistake in the docs that referenced indices.lifecycle.poll_interval
instead of time_series.poll_interval.
* Moved index.look_ahead_time setting to data stream module.
* Revert "Revert "[DOCS] Add TSDS docs (#86905)" (#87702)"
This reverts commit 0c86d7b9b2.
* First fix to tests
* Add data_stream object to index template
* small rewording
* Add enable data stream object in gradle example setup
* Add bullet about data stream must be enabled in template
* [DOCS] Add TSDB docs
* Update docs/build.gradle
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Address Nik's comments, part 1
* Address Nik's comments, part deux
* Reword write index
* Add feature flags
* Wrap one more section in feature flag
* Small fixes
* set index.routing_path to optional
* Update storage reduction value
* Update create index template code example
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
