github-mirrors/elasticsearch
1
0
Fork 0
mirror of https://github.com/elastic/elasticsearch.git synced 2025-04-24 15:17:30 -04:00
Code Issues Projects Releases Packages Wiki Activity

Fix renaming data streams with CCR replication (#88875)

Browse source 
This commit fixes the situation where a user wants to use CCR to replicate indices that are part of
a data stream while renaming the data stream. For example, assume a user has an auto-follow request
that looks like this:

```
PUT /_ccr/auto_follow/my-auto-follow-pattern
{
  "remote_cluster" : "other-cluster",
  "leader_index_patterns" : ["logs-*"],
  "follow_index_pattern" : "{{leader_index}}_copy"
}
```

And then the data stream `logs-mysql-error` was created, creating the backing index
`.ds-logs-mysql-error-2022-07-29-000001`.

Prior to this commit, replicating this data stream means that the backing index would be renamed to
`.ds-logs-mysql-error-2022-07-29-000001_copy` and the data stream would *not* be renamed. This
caused a check to trip in `TransportPutLifecycleAction` asserting that a backing index was not
renamed for a data stream during following.

After this commit, there are a couple of changes:

First, the data stream will also be renamed. This means that the `logs-mysql-error` becomes
`logs-mysql-error_copy` when created on the follower cluster. Because of the way that CCR works,
this means we need to support renaming a data stream for a regular "create follower" request, so a
new parameter has been added: `data_stream_name`. It works like this:

```
PUT /mynewindex/_ccr/follow
{
  "remote_cluster": "other-cluster",
  "leader_index": "myotherindex",
  "data_stream_name": "new_ds"
}
```

Second, the backing index for a data stream must be renamed in a way that does not break the parsing
of a data stream backing pattern, whereas previously the index
`.ds-logs-mysql-error-2022-07-29-000001` would be renamed to
`.ds-logs-mysql-error-2022-07-29-000001_copy` (an illegal name since it doesn't end with the
rollover digit), after this commit it will be renamed to
`.ds-logs-mysql-error_copy-2022-07-29-000001` to match the renamed data stream. This means that for
the given `follow_index_pattern` of `{{leader_index}}_copy` the index changes look like:

| Leader Cluster | Follower Cluster |
|--------------|-----------|
| `logs-mysql-error` (data stream) | `logs-mysql-error_copy` (data stream) |
| `.ds-logs-mysql-error-2022-07-29-000001`      | `.ds-logs-mysql-error_copy-2022-07-29-000001` |

Which internally means the auto-follow request turned into the create follower request of:

```
PUT /.ds-logs-mysql-error_copy-2022-07-29-000001/_ccr/follow
{
  "remote_cluster": "other-cluster",
  "leader_index": ".ds-logs-mysql-error-2022-07-29-000001",
  "data_stream_name": "logs-mysql-error_copy"
}
```

Relates to https://github.com/elastic/elasticsearch/pull/84940 (cherry-picked the commit for a test)
Relates to https://github.com/elastic/elasticsearch/pull/61993 (where data stream support was first introduced for CCR)
Resolves https://github.com/elastic/elasticsearch/issues/81751
This commit is contained in:
Lee Hinman 2022-08-01 09:17:50 -06:00 • committed by GitHub
parent 579692d5a3
commit 3420be0ca5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 753 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/reference/ccr/apis/auto-follow/put-auto-follow-pattern.asciidoc
View file

@ -85,11 +85,14 @@ the new patterns.
more `leader_index_patterns` and one or more `leader_index_exclusion_patterns` won't be followed.
`follow_index_pattern`::
(Optional, string) The name of follower index. The template `{{leader_index}}`
can be used to derive the name of the follower index from the name of the
leader index. When following a data stream, use `{{leader_index}}`; {ccr-init}
does not support changes to the names of a follower data stream's backing
indices.
(Optional, string) The name of follower index. The template `{{leader_index}}` can be used to
derive the name of the follower index from the name of the leader index. When following a data
stream, the `follow_index_pattern` will be used for renaming not only the leader index, but also
the data stream containing the leader index. For example, a data stream called
`logs-mysql-default` with a backing index of `.ds-logs-mysql-default-2022-01-01-000001` and a
`follow_index_pattern` of `{{leader_index}}_copy` will replicate the data stream as
`logs-mysql-default_copy` and the backing index as
`.ds-logs-mysql-default_copy-2022-01-01-000001`.
include::../follow-request-body.asciidoc[]

20
docs/reference/ccr/apis/follow/put-follow.asciidoc
View file

@ -76,6 +76,26 @@ referenced leader index. When this API returns, the follower index exists, and
(Required, string) The <<remote-clusters,remote cluster>> containing
the leader index.
[[ccr-put-follow-request-body-data_stream_name]]`data_stream_name`::
(Optional, string) If the leader index is part of a <<data-streams,data stream>>, the name to
which the local data stream for the followed index should be renamed. For example, A request like:
[source,console]
--------------------------------------------------
PUT /.ds-logs-mysql-default_copy-2022-01-01-000001/_ccr/follow
{
"remote_cluster" : "remote_cluster",
"leader_index" : ".ds-logs-mysql-default-2022-01-01-000001",
"data_stream_name": "logs-mysql-default_copy"
}
--------------------------------------------------
// TEST[skip:no setup]
Replicates the leader index `.ds-logs-mysql-default-2022-01-01-000001` into the follower index
`.ds-logs-mysql-default_copy-2022-01-01-000001` and will do so using the data stream
`logs-mysql-default_copy`, as opposed to the original leader data stream name of
`logs-mysql-default`.
include::../follow-request-body.asciidoc[]
[[ccr-put-follow-examples]]