mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-24 23:27:25 -04:00
52afaf2060
When a data stream is being auto followed then a rollover in a local cluster can break auto following, if the local cluster performs a rollover then it creates a new write index and if then later the remote cluster rolls over as well then that new write index can't be replicated, because it has the same name as in the write index in the local cluster, which was created earlier. If a data stream is managed by ccr, then the local cluster should not do a rollover for those data streams. The data stream should be rolled over in the remote cluster and that change should replicate to the local cluster. Performing a rollover in the local cluster is an operation that the data stream support in ccr should perform. To protect against rolling over a replicated data stream, this PR adds a replicate field to DataStream class. The rollover api will fail with an error in case a data stream is being rolled over and the targeted data stream is a replicated data stream. When the put follow api creates a data stream in the local cluster then the replicate flag is set to true. There should be a way to turn a replicated data stream into a regular data stream when for example during disaster recovery. The newly added api in this pr (promote data stream api) is doing that. After a replicated data stream is promoted to a regular data stream then the local data stream can be rolled over, so that the new write index is no longer a follower index. Also if the put follow api is attempting to update this data stream (for example to attempt to resume auto following) then that with fail, because the data stream is no longer a replicated data stream. Today with time based indices behind an alias, the is_write_index property isn't replicated from remote cluster to the local cluster, so when attempting to rollover the alias in the local cluster the rollover fails, because the alias doesn't have a write index. The added replicated field in the DataStream class and added validation achieve the same kind of protection, but in a more robust way. A followup from #61993.
92 lines
3.5 KiB
Text
92 lines
3.5 KiB
Text
[role="xpack"]
|
|
[testenv="platinum"]
|
|
[[ccr-auto-follow]]
|
|
=== Manage auto-follow patterns
|
|
To replicate time series indices, you configure an auto-follow pattern so that
|
|
each new index in the series is replicated automatically. Whenever the name of
|
|
a new index on the remote cluster matches the auto-follow pattern, a
|
|
corresponding follower index is added to the local cluster.
|
|
|
|
You can also create auto-follow patterns for data streams. When a new backing
|
|
index is generated on a remote cluster, that index and its data stream are
|
|
automatically followed if the data stream name matches an auto-follow
|
|
pattern. If you create a data stream after creating the auto-follow pattern,
|
|
all backing indices are followed automatically.
|
|
|
|
The data streams replicated from a remote cluster by CCR are protected from
|
|
local rollovers. The <<promote-data-stream-api,promote data stream API>>
|
|
can be used to turn these data streams into regular data streams.
|
|
|
|
Auto-follow patterns are especially useful with
|
|
<<index-lifecycle-management,{ilm-cap}>>, which might continually create
|
|
new indices on the cluster containing the leader index.
|
|
|
|
[[ccr-access-ccr-auto-follow]]
|
|
To start using {ccr} auto-follow patterns, access {kib} and go to
|
|
*Management > Stack Management*. In the side navigation, select
|
|
*Cross-Cluster Replication* and choose the *Auto-follow patterns* tab
|
|
|
|
[[ccr-auto-follow-create]]
|
|
==== Create auto-follow patterns
|
|
When you <<ccr-getting-started-auto-follow,create an auto-follow pattern>>,
|
|
you are configuring a collection of patterns against a single remote cluster.
|
|
When an index is created in the remote cluster with a name that matches one of
|
|
the patterns in the collection, a follower index is configured in the local
|
|
cluster. The follower index uses the new index as its leader index.
|
|
|
|
[%collapsible]
|
|
.Use the API
|
|
====
|
|
Use the <<ccr-put-auto-follow-pattern,create auto-follow pattern API>> to add a
|
|
new auto-follow pattern configuration.
|
|
====
|
|
|
|
[[ccr-auto-follow-retrieve]]
|
|
==== Retrieve auto-follow patterns
|
|
To view existing auto-follow patterns and make changes to the backing
|
|
patterns, <<ccr-access-ccr-auto-follow,access {kib}>> on your _remote_ cluster.
|
|
|
|
Select the auto-follow pattern that you want to view details about. From there,
|
|
you can make changes to the auto-follow pattern. You can also view your
|
|
follower indices included in the auto-follow pattern.
|
|
|
|
[%collapsible]
|
|
.Use the API
|
|
====
|
|
Use the <<ccr-get-auto-follow-pattern,get auto-follow pattern API>> to inspect
|
|
all configured auto-follow pattern collections.
|
|
====
|
|
|
|
[[ccr-auto-follow-pause]]
|
|
==== Pause and resume auto-follow patterns
|
|
To pause and resume replication of auto-follow pattern collections,
|
|
<<ccr-access-ccr-auto-follow,access {kib}>>, select the auto-follow pattern,
|
|
and pause replication.
|
|
|
|
To resume replication, select the pattern and choose
|
|
*Manage pattern > Resume replication*.
|
|
|
|
[%collapsible]
|
|
.Use the API
|
|
====
|
|
Use the <<ccr-pause-auto-follow-pattern,pause auto-follow pattern API>> to
|
|
pause auto-follow patterns.
|
|
Use the <<ccr-resume-auto-follow-pattern,resume auto-follow pattern API>> to
|
|
resume auto-follow patterns.
|
|
====
|
|
|
|
[[ccr-auto-follow-delete]]
|
|
==== Delete auto-follow patterns
|
|
To delete an auto-follow pattern collection,
|
|
<<ccr-access-ccr-auto-follow,access {kib}>>, select the auto-follow pattern,
|
|
and pause replication.
|
|
|
|
When the pattern status changes to Paused, choose
|
|
*Manage pattern > Delete pattern*.
|
|
|
|
[%collapsible]
|
|
.Use the API
|
|
====
|
|
Use the <<ccr-delete-auto-follow-pattern,delete auto-follow pattern API>> to
|
|
delete a configured auto-follow pattern collection.
|
|
====
|