[[remote-clusters-migrate]]
=== Migrate remote clusters from certificate to API key authentication

++++
<titleabbrev>Migrate from certificate to API key authentication</titleabbrev>
++++

The API key based security model for remote clusters offers administrators more
fine-grained access controls compared to the TLS certificate based security
model. For that reason, you may want to migrate from the certificate based
security model to the API key based model.

While it is possible to migrate by defining a new remote cluster connection,
using a new alias, this has several downsides:

- For {ccr}, it's not possible to change the leader cluster alias for existing
tasks. As a result, with a new remote cluster, follower indices would need to be
re-created from scratch.
- For {ccs}, transform and anomaly detection jobs do allow updating the remote
cluster alias. However, if the job was created with wildcards, for example
`*:source_index`, and `superuser`, adding a new remote cluster will cause the
job to do double the amount of work and potentially skew results with
duplications.

For these reasons, you may prefer to migrate a remote cluster in-place, by
following these steps:

. <<remote-clusters-migration-prerequisites,Review the prerequisites>>
. <<remote-clusters-migration-remote-cluster>>
. <<remote-clusters-migration-stop>>
. <<remote-clusters-migration-reconnect>>
. <<remote-clusters-migration-resume>>
. <<remote-clusters-migration-disable-cert>>

If you run into any issues, refer to <<remote-clusters-troubleshooting>>.

[[remote-clusters-migration-prerequisites]]
==== Prerequisites

* The nodes of the local and remote clusters must be on version 8.10 or later.
* The local and remote clusters must have an appropriate license. For more
information, refer to https://www.elastic.co/subscriptions.

[[remote-clusters-migration-remote-cluster]]
==== Reconfigure the remote cluster and generate a cross-cluster API key

On the remote cluster:

include::remote-clusters-api-key.asciidoc[tag=remote-cluster-steps]

[[remote-clusters-migration-stop]]
==== Stop cross-cluster operations

On the local cluster, stop any persistent tasks that refer to the remote
cluster:

* Use the <<stop-transform>> API to stop any transforms.
* Use the <<ml-close-job>> API to close any anomaly detection jobs.
* Use the <<ccr-pause-auto-follow-pattern>> API to pause any auto-follow {ccr}.
* Use the <<ccr-post-pause-follow>> API to pause any manual {ccr} or existing
indices that were created from the auto-follow pattern.

[[remote-clusters-migration-reconnect]]
==== Reconnect to the remote cluster

On the local cluster:

. Enhance any roles used by local cluster users with the required
<<roles-remote-indices-priv,remote indices privileges>> or 
<<roles-remote-cluster-priv, remote cluster privileges>> for {ccr} and {ccs}.
Refer to <<remote-clusters-privileges-api-key>>. Note:

** You only need to assign additional `remote_indices` or `remote_cluster` privileges to existing
roles used for cross-cluster operations. You should be able to copy these
privileges from the original roles on the remote cluster, where they are defined
under the certification based security model.
** The roles on the local cluster can't exceed the `access` privilege granted by
the cross-cluster API key. Any extra local privileges will be suppressed by the
cross-cluster API key's privileges.
** No update is needed if the {ccr} or {ccs} tasks have been configured with a
`superuser` role. The `superuser` role is automatically updated to allow access
to all remote indices.
** Tasks that are run as regular users with named roles are immediately updated
with the new privileges. A task will load a new definition the next time it
runs.
** You need to restart tasks that are run using an API key (done in a later
step).

. If you've dynamically configured the remote cluster (via the cluster settings
API):

.. Retrieve the current remote cluster configuration, and store it in a safe
place. You may need it later in case you need to
<<remote-clusters-migration-rollback,roll back>>. Use the cluster settings API:
+
[source,console]
----
GET /_cluster/settings?filter_path=persistent.cluster.remote
----

.. Remove the existing remote cluster definition by setting the remote cluster
settings to `null`.

. If you've statically configured the remote cluster (via `elasticsearch.yml`),
copy the `cluster.remote` settings from `elasticsearch.yml`, and store them
in a safe place. You may need them later in case you need to
<<remote-clusters-migration-rollback,roll back>>.


include::remote-clusters-api-key.asciidoc[tag=local-cluster-steps]

.. Add the cross-cluster API key, created on the remote cluster earlier, to the
keystore:
+
[source,sh]
----
./bin/elasticsearch-keystore add cluster.remote.ALIAS.credentials
----
+
Replace `ALIAS` with the same alias that was used for cross-cluster operations
before the migration. When prompted, enter the encoded cross-cluster API key
created on the remote cluster earlier.

. If you've dynamically configured the remote cluster (via the cluster settings
API):

.. Restart the local cluster to load changes to the keystore and settings.

.. Re-add the remote cluster. Use the same remote cluster alias, and change the
transport port into the remote cluster port. For example:
+
[source,console]
----
PUT /_cluster/settings
{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "my_remote" : { <1>
          "mode": "proxy",
          "proxy_address": "my.remote.cluster.com:9443" <2>
        }
      }
    }
  }
}
----
// TEST[skip:TODO]
<1> The remote cluster alias. Use the same alias that was used before the
migration.
<2> The remote cluster address with the remote cluster port, which defaults to
`9443`.

. If you've statically configured the remote cluster (via `elasticsearch.yml`):

.. Update the `cluster.remote` settings in `elasticsearch.yml` on each node of
the local cluster. Change the port into the remote cluster port, which defaults
to `9443`.

.. Restart the local cluster to load changes to the keystore and settings.

. Use the <<cluster-remote-info,remote cluster info API>> to verify that the
local cluster has successfully connected to the remote cluster:
+
[source,console]
----
GET /_remote/info
----
// TEST[skip:TODO]
+
The API response should indicate that the local cluster has connected to the
remote cluster:
+
[source,console-result]
----
{
  "my_remote": {
    "connected": true, <1>
    "mode": "proxy",
    "proxy_address": "my.remote.cluster.com:9443",
    "server_name": "",
    "num_proxy_sockets_connected": 0,
    "max_proxy_socket_connections": 18,
    "initial_connect_timeout": "30s",
    "skip_unavailable": false,
    "cluster_credentials": "::es_redacted::" <2>
  }
}
----
// TEST[skip:TODO]
<1> The remote cluster is connected.
<2> If present, indicates the remote cluster has connected using API key
authentication.

[[remote-clusters-migration-resume]]
==== Resume cross-cluster operations

Resume any persistent tasks that you stopped earlier. Tasks should be restarted
by the same user or API key that created the task before the migration. Ensure
the roles of this user or API key have been updated with the required
`remote_indices` or `remote_cluster` privileges. For users, tasks capture the caller's credentials
when started and run in that user's security context. For API keys, restarting a
task will update the task with the updated API key.

* Use the <<start-transform>> API to start any transforms.
* Use the <<ml-open-job>> API to open any anomaly detection jobs.
* Use the <<ccr-post-resume-follow>> API to resume any auto-follow {ccr}.
* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or
existing indices that were created from the auto-follow pattern.

[[remote-clusters-migration-disable-cert]]
==== Disable certificate based authentication and authorization

NOTE: Only proceed with this step if the migration has been proved successful on
the local cluster. If the migration is unsuccessful, either
<<remote-clusters-troubleshooting,find out what the problem is and attempt to
fix it>> or <<remote-clusters-migration-rollback,roll back>>.

Next, disable the certification based connection. Optionally, you can also
revoke the authorization.

. There is no particular setting to enable or disable a certificate based cross
cluster connection, because it shares the same transport protocol with the
intra-cluster node-to-node communication.
+
One way a remote cluster administrator can stop an existing local cluster from
connecting, is by changing TLS trust. The exact steps vary, depending on how the
clusters have been configured. A generic solution is to
<<encrypt-internode-communication,recreate the CA and certificate/key used by
the remote transport interface>> so that any existing certificate/key, locally
or distributed, is no longer trusted.
+
Another solution is to apply IP filters to the transport interface, blocking
traffic from outside the cluster.

. Optionally, delete any roles on the remote cluster that were only used for
cross-cluster operations. These roles are no longer used under the API key based
security model.

[[remote-clusters-migration-rollback]]
==== Rollback

If you need to roll back, follow these steps on the local cluster:

. Stop any persistent tasks that refer to the remote cluster.

. Remove the remote cluster definition by setting the remote cluster settings to
`null`.

. Remove the `remote_indices` or 'remote_cluster' privileges from any roles that were updated during
the migration.

. On each node, remove the `remote_cluster_client.ssl.*` settings from
`elasticsearch.yml`.

. Restart the local cluster to apply changes to the keystore and
`elasticsearch.yml`.

. On the local cluster, apply the original remote cluster settings. If the
remote cluster connection has been configured statically (using the
`elasticsearch.yml` file), restart the cluster.

. Use the <<cluster-remote-info,remote cluster info API>> to verify that the
local cluster has connected to the remote cluster. The response should have
`"connected": true` and not have `"cluster_credentials": "::es_redacted::"`.

. Restart any persistent tasks that you've stopped earlier.