elasticsearch/docs/reference
Martijn van Groningen 52afaf2060
Protect replicated data streams against local rollovers (#64710)
When a data stream is being auto followed then a rollover in a local cluster can break auto following,
if the local cluster performs a rollover then it creates a new write index and if then later the remote
cluster rolls over as well then that new write index can't be replicated, because it has the same name
as in the write index in the local cluster, which was created earlier.

If a data stream is managed by ccr, then the local cluster should not do a rollover for those data streams.
The data stream should be rolled over in the remote cluster and that change should replicate to the local
cluster. Performing a rollover in the local cluster is an operation that the data stream support in ccr should
perform.

To protect against rolling over a replicated data stream, this PR adds a replicate field to DataStream class.
The rollover api will fail with an error in case a data stream is being rolled over and the targeted data stream is
a replicated data stream. When the put follow api creates a data stream in the local cluster then the replicate flag
is set to true. There should be a way to turn a replicated data stream into a regular data stream when for example
during disaster recovery. The newly added api in this pr (promote data stream api) is doing that. After a replicated
data stream is promoted to a regular data stream then the local data stream can be rolled over, so that the new
write index is no longer a follower index. Also if the put follow api is attempting to update this data stream
(for example to attempt to resume auto following) then that with fail, because the data stream is no longer a
replicated data stream.

Today with time based indices behind an alias, the is_write_index property isn't replicated from remote cluster
to the local cluster, so when attempting to rollover the alias in the local cluster the rollover fails, because the
alias doesn't have a write index. The added replicated field in the DataStream class and added validation
achieve the same kind of protection, but in a more robust way.

A followup from #61993.
2020-12-08 08:34:24 +01:00
..
aggregations Update inference-bucket-aggregation.asciidoc 2020-12-03 11:48:45 -05:00
analysis [DOCS] Fix typo (#65912) 2020-12-05 10:05:13 -05:00
autoscaling Autoscaling delete policy by simple pattern (#64739) 2020-11-09 15:44:33 +01:00
cat Clarify field data cache behavior in docs (#64375) 2020-11-20 13:53:23 -08:00
ccr Protect replicated data streams against local rollovers (#64710) 2020-12-08 08:34:24 +01:00
cluster [DOCS] Correct the default value of wait_for_completion query param (#65800) 2020-12-04 15:52:35 -05:00
commands Deprecate cert gen without a CA and add self-signed option (#64037) 2020-11-30 08:46:02 +11:00
data-streams Protect replicated data streams against local rollovers (#64710) 2020-12-08 08:34:24 +01:00
docs [DOCS] Correct the default value of wait_for_completion query param (#65800) 2020-12-04 15:52:35 -05:00
eql [DOCS] EQL: Add diagrams for sequence matching (#65898) 2020-12-07 07:55:38 -05:00
graph [DOCS] Document xpack.graph.enabled setting (#60073) 2020-08-21 12:37:05 -04:00
high-availability [DOCS] Fix "the the" typos (#64344) 2020-10-29 10:11:58 -04:00
how-to [DOCS] Fix redirects and anchors (#65472) (#65474) 2020-11-24 19:59:52 -05:00
ilm [DOCS] Fix redirects and anchors (#65472) (#65474) 2020-11-24 19:59:52 -05:00
images [DOCS] EQL: Add diagrams for sequence matching (#65898) 2020-12-07 07:55:38 -05:00
index-modules Fix Allocation include Filter Docs (#65202) 2020-11-18 17:59:59 +01:00
indices [DOCS] Remove inert component template file (#65749) 2020-12-02 09:46:02 -05:00
ingest [DOCS] URI parts processor (#65695) 2020-12-02 07:22:40 -06:00
licensing [DOCS] Fix case for 'Boolean' (#64299) 2020-10-29 09:04:43 -04:00
mapping [DOCS] Fix _doc_count field title (#65704) 2020-12-02 08:13:43 -05:00
migration Remove escape hatch permitting incompatible builds (#65753) 2020-12-02 19:30:55 +00:00
ml [ML] Adding assignment_memory_basis to model_size_stats (#65561) 2020-12-03 17:18:08 +00:00
modules [DOCS] Fix wording for HTTP settings (#65964) 2020-12-07 12:18:55 -05:00
monitoring [DOCS] Add monitoring note (#61793) 2020-10-16 13:56:54 -07:00
query-dsl Fix range query on date fields for number inputs (#63692) 2020-12-01 18:49:50 +01:00
release-notes Fix range query on date fields for number inputs (#63692) 2020-12-01 18:49:50 +01:00
repositories-metering-api [DOCS] Fix case for 'Boolean' (#64299) 2020-10-29 09:04:43 -04:00
rest-api [Transform] use ISO dates in output instead of epoch millis (#65584) 2020-12-07 15:34:28 +01:00
rollup [DOCS] Fix docs integ tests for release builds (#65761) 2020-12-02 12:09:53 -05:00
scripting [DOCS] Fix ingest script compilation rate and cache size (#61468) 2020-08-24 10:26:38 -04:00
search Support unmapped fields in search 'fields' option (#65386) 2020-12-01 21:40:27 +01:00
searchable-snapshots [DOCS] Correct the default value of wait_for_completion query param (#65800) 2020-12-04 15:52:35 -05:00
settings Introduce an additional hasher (PBKDF2_STRETCH) (#65328) 2020-11-26 13:29:19 +01:00
setup Add snapshots to important config list (#65338) 2020-11-23 14:26:07 +00:00
slm Document Recommended Maximum Repository Size in SLM Docs (#64485) 2020-11-02 17:20:20 +01:00
snapshot-restore [DOCS] Correct restore snapshot API request example (#65525) 2020-11-30 13:55:37 -05:00
sql Abort sorting in case of local agg sort queue overflow (#65687) 2020-12-03 19:19:15 +01:00
tab-widgets [DOCS] Improve docs for Windows DOS/UNC paths in path.* settings (#64668) 2020-11-09 10:36:48 -05:00
transform [DOCS] Adds Working with transforms at scale to docs (#65726) 2020-12-07 18:00:42 +01:00
upgrade [DOCS] Add missing "with" in remote reindex doc (#65532) 2020-11-30 08:32:18 -05:00
vectors [DOCS] Update my-index examples (#60132) 2020-07-27 14:46:39 -04:00
aggregations.asciidoc [DOCS] Rewrite aggs overview (#64318) 2020-10-30 08:39:38 -04:00
analysis.asciidoc [DOCS] Swap [float] for [discrete] (#60124) 2020-07-23 11:48:22 -04:00
api-conventions.asciidoc [DOCS] Make system & hidden index info more prominent. (#64839) 2020-11-11 10:35:02 -08:00
cat.asciidoc AwaitsFix for #51619 2020-08-06 09:59:56 +01:00
cluster.asciidoc Password-protected Keystore Feature Branch PR (#51123) 2020-01-27 19:51:39 -05:00
data-management.asciidoc [DOCS] Move Kibana index mgmt docs to ES (#64380) 2020-10-30 09:14:52 -04:00
data-rollup-transform.asciidoc [DOCS] Changes level offset of transform pages (#60066) 2020-07-22 10:55:48 -07:00
datatiers.asciidoc Cold tier time-range should not be specified (#65546) 2020-11-30 15:04:41 +01:00
docs.asciidoc [DOCS] Remove heading offsets for REST APIs (#44568) 2019-07-19 14:35:36 -04:00
frozen-indices.asciidoc [DOCS] Replace twitter dataset in docs (#60604) 2020-08-03 12:49:56 -04:00
getting-started.asciidoc [DOCS] Swap [float] for [discrete] (#60124) 2020-07-23 11:48:22 -04:00
glossary.asciidoc [DOCS] Update rollup glossary item (#65519) 2020-12-01 08:57:07 -05:00
gs-index.asciidoc [DOCS] Adding index file for GS "mini book". 2017-07-18 13:44:08 -07:00
high-availability.asciidoc [DOCS] Add docs for designing resilient clusters (#47233) 2020-06-05 11:48:44 -04:00
how-to.asciidoc [DOCS] Document shard sizing guide (#61942) 2020-09-28 09:24:40 -04:00
index-extra-title-page.html [DOCS] Add index-extra-title-page.html for direct HTML migration (#50189) 2019-12-13 12:44:12 -05:00
index-modules.asciidoc [DOCS] Update CCR docs to focus on Kibana (#60555) 2020-08-17 15:36:54 -04:00
index.asciidoc [DOCS] Add top-level Data management section. (#64185) 2020-10-28 15:38:22 -07:00
index.x.asciidoc [DOCS] Removes redundant index.asciidoc files (#30707) 2018-05-18 11:05:40 -07:00
indices.asciidoc [DOCS] Split delete index template API docs (#62074) (#62168) 2020-09-09 10:13:13 -04:00
ingest.asciidoc [DOCS] Update my-index examples (#60132) 2020-07-27 14:46:39 -04:00
intro.asciidoc [DOCS] Fix hyphenation for "time series" (#61472) 2020-08-24 10:34:41 -04:00
links.asciidoc [DOCS] Fix security links in machine learning APIs (#60098) 2020-07-23 12:14:56 -07:00
mapping.asciidoc Remove a redundant section on field data types. (#61821) 2020-09-02 15:00:01 -07:00
query-dsl.asciidoc [DOCS] Add redirects for wildcard and constant keyword (#61815) 2020-09-01 15:32:35 -04:00
redirects.asciidoc Searchable snapshot terminology (#65549) 2020-11-30 17:14:47 +01:00
release-notes.asciidoc [DOCS] Adds placeholders for v8 highlights, breaking changes, release notes (#38641) 2019-02-08 15:48:42 -08:00
scripting.asciidoc [DOCS] Swap [float] for [discrete] (#60124) 2020-07-23 11:48:22 -04:00
search.asciidoc [DOCS] Display point in time API docs (#61527) 2020-08-25 11:03:41 -04:00
setup.asciidoc [DOCS] http -> https, remove outdated plugin docs (#60380) 2020-07-31 15:58:38 -04:00
upgrade.asciidoc [DOCS] Add tip for upgrade assistant (#55027) 2020-05-05 09:49:22 -04:00