[role="xpack"] [[ccr-getting-started-tutorial]] === Tutorial: Set up {ccr} ++++ Set up {ccr} ++++ //// [source,console] ---- PUT /server-metrics { "settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 0 } }, "mappings" : { "properties" : { "@timestamp" : { "type" : "date" }, "accept" : { "type" : "long" }, "deny" : { "type" : "long" }, "host" : { "type" : "keyword" }, "response" : { "type" : "float" }, "service" : { "type" : "keyword" }, "total" : { "type" : "long" } } } } ---- // TESTSETUP //// Use this guide to set up {ccr} (CCR) between clusters in two datacenters. Replicating your data across datacenters provides several benefits: * Brings data closer to your users or application server to reduce latency and response time * Provides your mission-critical applications with the tolerance to withstand datacenter or region outages In this guide, you'll learn how to: * Configure a <> with a leader index * Create a follower index on a local cluster * Create an auto-follow pattern to automatically follow time series indices that are periodically created in a remote cluster You can manually create follower indices to replicate specific indices on a remote cluster, or configure auto-follow patterns to replicate rolling time series indices. TIP: If you want to replicate data across clusters in the cloud, you can link:{cloud}/ec-enable-ccs.html[configure remote clusters on {ess}]. Then, you can <> and set up {ccr}. [[ccr-getting-started-prerequisites]] ==== Prerequisites To complete this tutorial, you need: * The `manage` cluster privilege on the local cluster. * A license on both clusters that includes {ccr}. {kibana-ref}/managing-licenses.html[Activate a free 30-day trial]. * An index on the remote cluster that contains the data you want to replicate. This tutorial uses the sample eCommerce orders data set. {kibana-ref}/get-started.html#gs-get-data-into-kibana[Load sample data]. * In the local cluster, all nodes with the `master` <> must also have the <> role. The local cluster must also have at least one node with both a data role and the <> role. Individual tasks for coordinating replication scale based on the number of data nodes with the `remote_cluster_client` role in the local cluster. ==== Connect to a remote cluster To replicate an index on a remote cluster (Cluster A) to a local cluster (Cluster B), you configure Cluster A as a remote on Cluster B. image::images/ccr-tutorial-clusters.png[ClusterA contains the leader index and ClusterB contains the follower index] To configure a remote cluster from Stack Management in {kib}: . Select *Remote Clusters* from the side navigation. . Specify the {es} endpoint URL, or the IP address or host name of the remote cluster (`ClusterA`) followed by the transport port (defaults to `9300`). For example, `cluster.es.eastus2.staging.azure.foundit.no:9400` or `192.168.1.1:9300`. [%collapsible%open] .API example ==== You can also use the <> to add a remote cluster: [source,console] ---- PUT /_cluster/settings { "persistent" : { "cluster" : { "remote" : { "leader" : { "seeds" : [ "127.0.0.1:9300" <1> ] } } } } } ---- // TEST[setup:host] // TEST[s/127.0.0.1:9300/\${transport_host}/] <1> Specifies the hostname and transport port of a seed node in the remote cluster. You can verify that the local cluster is successfully connected to the remote cluster. [source,console] ---- GET /_remote/info ---- // TEST[continued] The API response indicates that the local cluster is connected to the remote cluster with cluster alias `leader`. [source,console-result] ---- { "leader" : { "seeds" : [ "127.0.0.1:9300" ], "connected" : true, "num_nodes_connected" : 1, <1> "max_connections_per_cluster" : 3, "initial_connect_timeout" : "30s", "skip_unavailable" : true, "mode" : "sniff" } } ---- // TESTRESPONSE[s/127.0.0.1:9300/$body.leader.seeds.0/] // TEST[s/"connected" : true/"connected" : $body.leader.connected/] // TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/] <1> The number of nodes in the remote cluster the local cluster is connected to. ==== include::{es-ref-dir}/security/authentication/remote-clusters-privileges-cert.asciidoc[tag=configure-ccr-privileges] [[ccr-getting-started-follower-index]] ==== Create a follower index to replicate a specific index When you create a follower index, you reference the remote cluster and the leader index in your remote cluster. To create a follower index from Stack Management in {kib}: . Select *Cross-Cluster Replication* in the side navigation and choose the *Follower Indices* tab. . Choose the cluster (ClusterA) containing the leader index you want to replicate. . Enter the name of the leader index, which is `kibana_sample_data_ecommerce` if you are following the tutorial. . Enter a name for your follower index, such as `follower-kibana-sample-data`. {es} initializes the follower using the <> process, which transfers the existing Lucene segment files from the leader index to the follower index. The index status changes to *Paused*. When the remote recovery process is complete, the index following begins and the status changes to *Active*. When you index documents into your leader index, {es} replicates the documents in the follower index. [role="screenshot"] image::images/ccr-follower-index.png["The Cross-Cluster Replication page in {kib}"] [%collapsible%open] .API example ==== You can also use the <> to create follower indices. When you create a follower index, you must reference the remote cluster and the leader index that you created in the remote cluster. When initiating the follower request, the response returns before the <> process completes. To wait for the process to complete, add the `wait_for_active_shards` parameter to your request. [source,console] ---- PUT /server-metrics-follower/_ccr/follow?wait_for_active_shards=1 { "remote_cluster" : "leader", "leader_index" : "server-metrics" } ---- // TEST[continued] ////////////////////////// [source,console-result] -------------------------------------------------- { "follow_index_created" : true, "follow_index_shards_acked" : true, "index_following_started" : true } -------------------------------------------------- ////////////////////////// Use the <> to inspect the status of replication. ////////////////////////// [source,console] -------------------------------------------------- POST /server-metrics-follower/_ccr/pause_follow POST /server-metrics-follower/_close POST /server-metrics-follower/_ccr/unfollow -------------------------------------------------- // TEST[continued] ////////////////////////// ==== [[ccr-getting-started-auto-follow]] ==== Create an auto-follow pattern to replicate time series indices You use <> to automatically create new followers for rolling time series indices. Whenever the name of a new index on the remote cluster matches the auto-follow pattern, a corresponding follower index is added to the local cluster. Note that only indices created on the remote cluster after the auto-follow pattern is created will be auto-followed: existing indices on the remote cluster are ignored even if they match the pattern. An auto-follow pattern specifies the remote cluster you want to replicate from, and one or more index patterns that specify the rolling time series indices you want to replicate. // tag::ccr-create-auto-follow-pattern-tag[] To create an auto-follow pattern from Stack Management in {kib}: . Select *Cross Cluster Replication* in the side navigation and choose the *Auto-follow patterns* tab. . Enter a name for the auto-follow pattern, such as `beats`. . Choose the remote cluster that contains the index you want to replicate, which in the example scenario is Cluster A. . Enter one or more index patterns that identify the indices you want to replicate from the remote cluster. For example, enter `metricbeat-* packetbeat-*` to automatically create followers for {metricbeat} and {packetbeat} indices. . Enter *follower-* as the prefix to apply to the names of the follower indices so you can more easily identify replicated indices. As new indices matching these patterns are created on the remote, {es} automatically replicates them to local follower indices. [role="screenshot"] image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"] // end::ccr-create-auto-follow-pattern-tag[] [%collapsible%open] .API example ==== Use the <> to configure auto-follow patterns. [source,console] ---- PUT /_ccr/auto_follow/beats { "remote_cluster" : "leader", "leader_index_patterns" : [ "metricbeat-*", <1> "packetbeat-*" <2> ], "follow_index_pattern" : "{{leader_index}}-copy" <3> } ---- // TEST[continued] <1> Automatically follow new {metricbeat} indices. <2> Automatically follow new {packetbeat} indices. <3> The name of the follower index is derived from the name of the leader index by adding the suffix `-copy` to the name of the leader index. ////////////////////////// [source,console-result] -------------------------------------------------- { "acknowledged" : true } -------------------------------------------------- ////////////////////////// ////////////////////////// [source,console] -------------------------------------------------- DELETE /_ccr/auto_follow/beats -------------------------------------------------- // TEST[continued] ////////////////////////// ====