mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-24 23:27:25 -04:00
207 lines
8.3 KiB
Text
207 lines
8.3 KiB
Text
[#es-native-connectors]
|
||
== Elastic managed connectors
|
||
|
||
.Naming history
|
||
****
|
||
Elastic managed connectors were initially known as "native connectors".
|
||
You might find this term in older documentation.
|
||
****
|
||
|
||
Managed <<es-connectors,Elastic connectors>> are available directly within your Elastic Cloud deployment.
|
||
No additional infrastructure is required.
|
||
|
||
Managed connectors sync data sources directly to Elasticsearch indices.
|
||
Create these indices using the *Connector* workflow within {kib}.
|
||
|
||
The following connectors are available as Elastic managed connectors.
|
||
Refer to each connector reference for additional information specific to each connector.
|
||
|
||
include::_connectors-list-native.asciidoc[]
|
||
|
||
[discrete#es-native-connectors-prerequisites]
|
||
=== Availability and prerequisites
|
||
|
||
Managed connectors were introduced in Elastic version *8.5.0*.
|
||
|
||
Your Elastic Cloud deployment must include the following Elastic services:
|
||
|
||
* *Elasticsearch*
|
||
* *{kib}*
|
||
* {enterprise-search-ref}/server.html[*Enterprise Search*]
|
||
|
||
Refer to _Native Integrations_ on the https://www.elastic.co/subscriptions/cloud[Elastic subscriptions page], in the *Elastic Search* section for managed connector licensing requirements.
|
||
|
||
.Elastic Cloud IPs
|
||
[sidebar]
|
||
--
|
||
Using our Elastic managed connectors involves outbound data transfer (egress) from your Elastic Cloud deployment.
|
||
If you have IP/firewall rules on your third party service, you'll need to add the Elastic Cloud egress static IP ranges to your service’s configuration.
|
||
|
||
Refer to {cloud}/ec-static-ips.html[static IP ranges] in the Elastic Cloud documentation for the latest list of IP addresses.
|
||
--
|
||
|
||
[discrete#es-native-connectors-usage]
|
||
=== Usage in {kib} UI
|
||
|
||
Follow the *Connector* workflow in {kib} to select the *Connector* ingestion method.
|
||
Choose a data source, create an Elasticsearch index, and configure a managed connector to manage the index.
|
||
|
||
[.screenshot]
|
||
image::images/use-a-connector-workflow.png[]
|
||
|
||
[discrete#es-native-connectors-select-connector]
|
||
==== Select a connector
|
||
|
||
Choose the data source to sync from the available options and select *Continue*.
|
||
|
||
[discrete#es-native-connectors-index]
|
||
==== Create index
|
||
|
||
Create a new index to be managed by the connector:
|
||
|
||
. Name your index and optionally change the language analyzer to match the human language of your data source.
|
||
(The index name will be automatically prefixed with `search-`.)
|
||
. Select *Create index*.
|
||
|
||
The index is created and ready to <<es-native-connectors-configuration,configure>>.
|
||
|
||
This operation requires:
|
||
|
||
. Access to {kib}
|
||
. Permission to create or manage the index
|
||
. `write` {ref}/security-privileges.html[indices privilege^] for the `.elastic-connectors` index
|
||
. `manage_api_key` {ref}/security-privileges.html[cluster privilege^] to allow the API key to be created
|
||
. `write_connector_secrets` {ref}/security-privileges.html[cluster privilege^] to allow the API key to be stored as a secret
|
||
|
||
[discrete#es-native-connectors-configuration]
|
||
==== Configure connector
|
||
|
||
Create a new index to be managed by the connector.
|
||
|
||
Continue from above, or navigate to the following location within the {kib} UI:
|
||
|
||
*Search > Content > Elasticsearch indices*
|
||
|
||
Choose the index to configure, and then choose the *Configuration* tab.
|
||
|
||
Configure the connector:
|
||
|
||
. Edit the name and description for the connector.
|
||
Your team can use this information to differentiate this index from other connector indices.
|
||
(These fields describe the _connector_ and are independent of the Elasticsearch index name.)
|
||
. Save your changes.
|
||
. Edit the data source configuration.
|
||
The fields here vary by connector.
|
||
Refer to the documentation for each connector for details (refer to list of Elastic managed connectors, above).
|
||
Refer to <<es-connectors-security>> for security considerations.
|
||
. Save your changes.
|
||
|
||
Optionally choose *Edit sync schedule* to begin <<es-native-connectors-management,managing the connector>>.
|
||
|
||
This operation requires:
|
||
|
||
. Access to {kib}
|
||
. Permission to create or manage the index
|
||
. `write` {ref}/security-privileges.html[indices privilege^] for the `.elastic-connectors` index
|
||
. `manage_api_key` {ref}/security-privileges.html[cluster privilege^] to allow the API key to be created
|
||
. `write_connector_secrets` {ref}/security-privileges.html[cluster privilege^] to allow the API key to be stored as a secret
|
||
|
||
[discrete#es-native-connectors-management]
|
||
==== Manage connector
|
||
|
||
To manage documents, syncs, sync rules, ingest pipelines, and other connector features, refer to <<es-connectors-usage>>.
|
||
|
||
[discrete#es-native-connectors-manage-API-keys]
|
||
=== Manage API keys
|
||
|
||
[NOTE]
|
||
====
|
||
API keys for Elastic managed connectors were introduced in Elastic version *8.13.0*.
|
||
Managed connectors created in earlier versions will not automatically use API keys upon upgrading to *8.13.0*.
|
||
Refer to <<es-native-connectors-enabling-API-keys-for-upgraded-connectors,enabling API keys>> for more information.
|
||
====
|
||
|
||
Managed connectors communicate with Elasticsearch using API keys.
|
||
When managing a managed connector through the Connectors page in the {kib} UI, API key management is handled automatically.
|
||
API keys for Elastic managed connectors have the `manage` permission for three indices:
|
||
|
||
. The attached index
|
||
. The access control (ACL) index used for document level security
|
||
. The internal `.elastic-connectors` index.
|
||
|
||
Changing the attached index through {kib} will automatically invalidate the existing API key and generate a new one.
|
||
If you want to rotate an existing API key, navigate to the *Configuration* tab.
|
||
Scroll down to *Manage API key* and select *Generate API key*.
|
||
This action will invalidate the previous API key, create a new API key, and update the connector secret.
|
||
|
||
API keys for Elastic managed connectors are stored on an internal system index called `.connector-secrets`.
|
||
This index can only be written to through API requests by users with the `write_connector-secrets` cluster privilege.
|
||
Only the Enterprise Search instance has permission to read from this index.
|
||
|
||
Users managing Elastic managed connectors will need the `write_connector_secrets` cluster privilege assigned to their role.
|
||
Refer to <<es-connectors-security>> for security considerations.
|
||
|
||
[discrete#es-native-connectors-manage-API-keys-programmatically]
|
||
==== Manage API keys programmatically
|
||
|
||
You can also create and store API keys programmatically.
|
||
|
||
.*Expand* the following section for details.
|
||
[%collapsible]
|
||
===================================
|
||
include::_connectors-create-native-api-key.asciidoc[]
|
||
===================================
|
||
|
||
[discrete#es-native-connectors-enabling-API-keys-for-upgraded-connectors]
|
||
==== Enabling API keys for upgraded connectors
|
||
|
||
Managed connectors created before *8.13.0* do not initially have API keys upon upgrading.
|
||
The attached indices of these connectors cannot be changed until the connector has been converted to use an API key.
|
||
|
||
.*Expand* the following section for steps on enabling API keys for upgraded Elastic managed connectors.
|
||
[%collapsible]
|
||
===================================
|
||
. Run the following command in *Dev Tools* to enable API keys for the connector, replacing values where indicated.
|
||
+
|
||
[source, console,subs="+attributes"]
|
||
----
|
||
POST .elastic-connectors/_update/connector_id
|
||
{
|
||
"doc": {
|
||
"features": {
|
||
"native_connector_api_keys": {
|
||
"enabled": true
|
||
}
|
||
}
|
||
}
|
||
}
|
||
----
|
||
// TEST[skip:requires connector_id]
|
||
+
|
||
. Go back to the Connectors page and navigate to the *Configuration* tab.
|
||
. Scroll down to *Manage API key* and select *Generate API key*.
|
||
|
||
Your managed connector is now using API keys to authorize ingestion into Elasticsearch.
|
||
===================================
|
||
|
||
[discrete#es-native-connectors-usage-api]
|
||
=== Usage via API
|
||
|
||
In 8.12 we introduced a set of {ref}/connector-apis.html[Connector APIs] to create and manage Elastic connectors and sync jobs, along with a https://github.com/elastic/connectors/blob/main/docs/CLI.md[CLI tool].
|
||
Use these tools if you'd like to work with connectors and sync jobs programmatically.
|
||
|
||
[discrete#es-native-connectors-example]
|
||
=== End-to-end example
|
||
|
||
The following example demonstrates how to use a managed connector on Elastic Cloud: <<es-mongodb-start>>.
|
||
|
||
[discrete#es-native-connectors-convert]
|
||
=== Convert a managed connector
|
||
|
||
You can convert a managed connector to a self-managed connector to be run on your own infrastructure.
|
||
You'll find instructions in the UI on the connector index's overview page.
|
||
|
||
[WARNING]
|
||
====
|
||
Converting a managed connector to a self-managed connector is an irreversible operation!
|
||
====
|