github-mirrors/elasticsearch
1
0
Fork 0
mirror of https://github.com/elastic/elasticsearch.git synced 2025-04-24 15:17:30 -04:00
Code Issues Projects Releases Packages Wiki Activity

Add data stream support to CCR (#61993)

Browse source 
This commit adds support data stream support to CCR's auto following by making the following changes:
* When the auto follow coordinator iterates over the candidate indices to follow,
  the auto follow coordinator also checks whether the index is part of a data stream and
  if the name of data stream also matches with the auto follow pattern then the index
  will be auto followed.
* When following an index, the put follow api also checks whether that index is part
  of a data stream and if so then also replicates the data stream definition to the
  local cluster.
* In order for the follow index api to determine whether an index is part of a data
  stream, the cluster state api was modified to also fetch the data stream definition
  of the cluster state if only the state is queried for specific indices.

When a data stream is auto followed, only new backing indices are auto followed.
This is in line with how time based indices patterns are replicated today. This
means that the data stream isn't copied 1 to 1 into the local cluster. The local
cluster's data stream definition contains the same name, timestamp field and
generation, but the list of backing indices may be different (depending on when
a data stream was auto followed).

Closes #56259
This commit is contained in:
Martijn van Groningen 2020-11-03 14:01:14 +01:00 • committed by GitHub
parent 08c0a8703f
commit c4c3c8b422
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 621 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/reference/ccr/auto-follow.asciidoc
View file

@ -7,6 +7,13 @@ each new index in the series is replicated automatically. Whenever the name of
a new index on the remote cluster matches the auto-follow pattern, a
corresponding follower index is added to the local cluster.
You can also create auto-follow patterns for data streams. When a new backing
index is generated on a remote cluster, that index and its data stream are
automatically followed if the data stream name matches an auto-follow
pattern. If you create a data stream after creating the auto-follow pattern,
all backing indices are followed automatically.
Auto-follow patterns are especially useful with
<<index-lifecycle-management,{ilm-cap}>>, which might continually create
new indices on the cluster containing the leader index.