mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-29 18:03:32 -04:00
3420be0ca5
This commit fixes the situation where a user wants to use CCR to replicate indices that are part of
a data stream while renaming the data stream. For example, assume a user has an auto-follow request
that looks like this:
```
PUT /_ccr/auto_follow/my-auto-follow-pattern
{
"remote_cluster" : "other-cluster",
"leader_index_patterns" : ["logs-*"],
"follow_index_pattern" : "{{leader_index}}_copy"
}
```
And then the data stream `logs-mysql-error` was created, creating the backing index
`.ds-logs-mysql-error-2022-07-29-000001`.
Prior to this commit, replicating this data stream means that the backing index would be renamed to
`.ds-logs-mysql-error-2022-07-29-000001_copy` and the data stream would *not* be renamed. This
caused a check to trip in `TransportPutLifecycleAction` asserting that a backing index was not
renamed for a data stream during following.
After this commit, there are a couple of changes:
First, the data stream will also be renamed. This means that the `logs-mysql-error` becomes
`logs-mysql-error_copy` when created on the follower cluster. Because of the way that CCR works,
this means we need to support renaming a data stream for a regular "create follower" request, so a
new parameter has been added: `data_stream_name`. It works like this:
```
PUT /mynewindex/_ccr/follow
{
"remote_cluster": "other-cluster",
"leader_index": "myotherindex",
"data_stream_name": "new_ds"
}
```
Second, the backing index for a data stream must be renamed in a way that does not break the parsing
of a data stream backing pattern, whereas previously the index
`.ds-logs-mysql-error-2022-07-29-000001` would be renamed to
`.ds-logs-mysql-error-2022-07-29-000001_copy` (an illegal name since it doesn't end with the
rollover digit), after this commit it will be renamed to
`.ds-logs-mysql-error_copy-2022-07-29-000001` to match the renamed data stream. This means that for
the given `follow_index_pattern` of `{{leader_index}}_copy` the index changes look like:
| Leader Cluster | Follower Cluster |
|--------------|-----------|
| `logs-mysql-error` (data stream) | `logs-mysql-error_copy` (data stream) |
| `.ds-logs-mysql-error-2022-07-29-000001` | `.ds-logs-mysql-error_copy-2022-07-29-000001` |
Which internally means the auto-follow request turned into the create follower request of:
```
PUT /.ds-logs-mysql-error_copy-2022-07-29-000001/_ccr/follow
{
"remote_cluster": "other-cluster",
"leader_index": ".ds-logs-mysql-error-2022-07-29-000001",
"data_stream_name": "logs-mysql-error_copy"
}
```
Relates to https://github.com/elastic/elasticsearch/pull/84940 (cherry-picked the commit for a test)
Relates to https://github.com/elastic/elasticsearch/pull/61993 (where data stream support was first introduced for CCR)
Resolves https://github.com/elastic/elasticsearch/issues/81751
148 lines
5 KiB
Text
148 lines
5 KiB
Text
[role="xpack"]
|
|
[[ccr-put-auto-follow-pattern]]
|
|
=== Create auto-follow pattern API
|
|
++++
|
|
<titleabbrev>Create auto-follow pattern</titleabbrev>
|
|
++++
|
|
|
|
Creates an auto-follow pattern.
|
|
|
|
[[ccr-put-auto-follow-pattern-request]]
|
|
==== {api-request-title}
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_ccr/auto_follow/<auto_follow_pattern_name>
|
|
{
|
|
"remote_cluster" : "<remote_cluster>",
|
|
"leader_index_patterns" :
|
|
[
|
|
"<leader_index_pattern>"
|
|
],
|
|
"leader_index_exclusion_patterns":
|
|
[
|
|
"<leader_index_exclusion_pattern>"
|
|
],
|
|
"follow_index_pattern" : "<follow_index_pattern>"
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:remote_cluster]
|
|
// TEST[s/<auto_follow_pattern_name>/auto_follow_pattern_name/]
|
|
// TEST[s/<remote_cluster>/remote_cluster/]
|
|
// TEST[s/<leader_index_patterns>/leader_index*/]
|
|
// TEST[s/<leader_index_exclusion_pattern>//]
|
|
// TEST[s/<follow_index_pattern>/{{leader_index}}-follower/]
|
|
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE /_ccr/auto_follow/auto_follow_pattern_name
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|
|
|
|
[[ccr-put-auto-follow-pattern-prereqs]]
|
|
==== {api-prereq-title}
|
|
|
|
* If the {es} {security-features} are enabled, you must have `read` and `monitor`
|
|
index privileges for the leader index patterns. You must also have `manage_ccr`
|
|
cluster privileges on the cluster that contains the follower index. For more
|
|
information, see <<security-privileges>>.
|
|
|
|
[[ccr-put-auto-follow-pattern-desc]]
|
|
==== {api-description-title}
|
|
|
|
This API creates a new named collection of
|
|
<<ccr-auto-follow,auto-follow patterns>> against the remote cluster
|
|
specified in the request body. Newly created indices on the remote cluster
|
|
matching any of the specified patterns will be automatically configured as follower
|
|
indices. Additionally, this API can be used to update existing
|
|
<<ccr-auto-follow,auto-follow patterns>>. Note that follower indices that were configured automatically
|
|
before updating an auto-follow pattern will remain unchanged even if they don't match against
|
|
the new patterns.
|
|
|
|
[[ccr-put-auto-follow-pattern-path-parms]]
|
|
==== {api-path-parms-title}
|
|
`<auto_follow_pattern_name>`::
|
|
(Required, string) The name of the collection of auto-follow patterns.
|
|
|
|
[[ccr-put-auto-follow-pattern-request-body]]
|
|
==== {api-request-body-title}
|
|
|
|
`remote_cluster`::
|
|
(Required, string) The <<remote-clusters,remote cluster>> containing
|
|
the leader indices to match against.
|
|
|
|
`leader_index_patterns`::
|
|
(Optional, array) An array of simple index patterns to match against indices
|
|
in the remote cluster specified by the `remote_cluster` field.
|
|
|
|
`leader_index_exclusion_patterns`::
|
|
(Optional, array) An array of simple index patterns that can be used to exclude indices
|
|
from being auto-followed. Indices in the remote cluster whose names are matching one or
|
|
more `leader_index_patterns` and one or more `leader_index_exclusion_patterns` won't be followed.
|
|
|
|
`follow_index_pattern`::
|
|
(Optional, string) The name of follower index. The template `{{leader_index}}` can be used to
|
|
derive the name of the follower index from the name of the leader index. When following a data
|
|
stream, the `follow_index_pattern` will be used for renaming not only the leader index, but also
|
|
the data stream containing the leader index. For example, a data stream called
|
|
`logs-mysql-default` with a backing index of `.ds-logs-mysql-default-2022-01-01-000001` and a
|
|
`follow_index_pattern` of `{{leader_index}}_copy` will replicate the data stream as
|
|
`logs-mysql-default_copy` and the backing index as
|
|
`.ds-logs-mysql-default_copy-2022-01-01-000001`.
|
|
|
|
include::../follow-request-body.asciidoc[]
|
|
|
|
[[ccr-put-auto-follow-pattern-examples]]
|
|
==== {api-examples-title}
|
|
|
|
This example creates an auto-follow pattern named `my_auto_follow_pattern`:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_ccr/auto_follow/my_auto_follow_pattern
|
|
{
|
|
"remote_cluster" : "remote_cluster",
|
|
"leader_index_patterns" :
|
|
[
|
|
"leader_index*"
|
|
],
|
|
"follow_index_pattern" : "{{leader_index}}-follower",
|
|
"settings": {
|
|
"index.number_of_replicas": 0
|
|
},
|
|
"max_read_request_operation_count" : 1024,
|
|
"max_outstanding_read_requests" : 16,
|
|
"max_read_request_size" : "1024k",
|
|
"max_write_request_operation_count" : 32768,
|
|
"max_write_request_size" : "16k",
|
|
"max_outstanding_write_requests" : 8,
|
|
"max_write_buffer_count" : 512,
|
|
"max_write_buffer_size" : "512k",
|
|
"max_retry_delay" : "10s",
|
|
"read_poll_timeout" : "30s"
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[setup:remote_cluster]
|
|
|
|
The API returns the following result:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"acknowledged" : true
|
|
}
|
|
--------------------------------------------------
|
|
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE /_ccr/auto_follow/my_auto_follow_pattern
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|