mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-28 17:34:17 -04:00
225 lines
12 KiB
Text
225 lines
12 KiB
Text
[#es-connectors-usage]
|
|
== Connectors UI in {kib}
|
|
|
|
This document describes operations available to <<es-native-connectors,managed connectors>> and <<es-build-connector,self-managed connectors>>, using the UI.
|
|
|
|
In the Kibana UI, navigate to *Search > Content > Connectors* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]. Here, you can view a summary of all your connectors and sync jobs, and to create new connectors.
|
|
|
|
[TIP]
|
|
====
|
|
In 8.12 we introduced a set of {ref}/connector-apis.html[Connector APIs] to create and manage Elastic connectors and sync jobs, along with a https://github.com/elastic/connectors/blob/main/docs/CLI.md[CLI tool].
|
|
Use these tools if you'd like to work with connectors and sync jobs programmatically, without using the UI.
|
|
====
|
|
|
|
[discrete#es-connectors-usage-index-create]
|
|
=== Create and configure connectors
|
|
|
|
You connector writes data to an {es} index.
|
|
|
|
To create <<es-native-connectors,*managed connectors*>> or self-managed <<es-build-connector,*self-managed connectors*>>, use the buttons under *Search > Content > Connectors*.
|
|
Once you've chosen the data source type you'd like to sync, you'll be prompted to create an {es} index.
|
|
|
|
[discrete#es-connectors-usage-indices]
|
|
=== Manage connector indices
|
|
|
|
View and manage all Elasticsearch indices managed by connectors.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Connectors* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]. Here, you can view a list of connector indices and their attributes, including connector type health and ingestion status.
|
|
|
|
Within this interface, you can choose to view the details for each existing index or delete an index.
|
|
Or, you can <<es-connectors-usage-index-create,create a new connector index>>.
|
|
|
|
These operations require access to Kibana and additional index privileges.
|
|
|
|
[discrete#es-connectors-usage-index-create-configure-existing-index]
|
|
=== Customize connector index mappings and settings
|
|
|
|
{es} stores your data as documents in an index. Each index is made up of a set of fields and each field has a type (such as `keyword`, `boolean`, or `date`).
|
|
|
|
*Mapping* is the process of defining how a document, and the fields it contains, are stored and indexed.
|
|
Connectors use {ref}/dynamic-field-mapping.html[dynamic mapping] to automatically create mappings based on the data fetched from the source.
|
|
|
|
Index *settings* are configurations that can be adjusted on a per-index basis. They control things like the index's performance, the resources it uses, and how it should handle operations.
|
|
|
|
When you create an index with a connector, the index is created with _default_ search-optimized field template mappings and index settings. Mappings for specific fields are then dynamically created based on the data fetched from the source.
|
|
|
|
You can inspect your index mappings in the following ways:
|
|
|
|
* *In the {kib} UI*: Navigate to *Search > Content > Indices > _YOUR-INDEX_ > Index Mappings*
|
|
* *By API*: Use the {ref}/indices-get-mapping.html[Get mapping API]
|
|
|
|
You can manually *edit* the mappings and settings via the {es} APIs:
|
|
|
|
* Use the {ref}/indices-put-mapping.html[Put mapping API] to update index mappings.
|
|
* Use the {ref}/indices-update-settings.html[Update index settings API] to update index settings.
|
|
|
|
It's important to note that these updates are more complex when the index already contains data.
|
|
|
|
Refer to the following sections for more information.
|
|
|
|
[discrete#es-connectors-usage-index-create-configure-existing-index-no-data]
|
|
==== Customize mappings and settings before syncing data
|
|
|
|
Updating mappings and settings is simpler when your index has no data.
|
|
If you create and attach a _new_ index while setting up a connector, you can customize the mappings and settings before syncing data, using the APIs mentioned earlier.
|
|
|
|
[discrete#es-connectors-usage-index-create-configure-existing-index-have-data]
|
|
==== Customize mappings and settings after syncing data
|
|
|
|
Once data has been added to {es} using dynamic mappings, you can't directly update existing field mappings.
|
|
If you've already synced data into an index and want to change the mappings, you'll need to {ref}/docs-reindex.html[reindex your data].
|
|
|
|
The workflow for these updates is as follows:
|
|
|
|
. {ref}/indices-create-index.html[Create] a new index with the desired mappings and settings.
|
|
. {ref}/docs-reindex.html[Reindex] your data from the old index into this new index.
|
|
. Delete the old index.
|
|
. (Optional) Use an {ref}/aliases.html[alias], if you want to retain the old index name.
|
|
. Attach your connector to the new index or alias.
|
|
|
|
[discrete#es-connectors-usage-syncs-recurring]
|
|
=== Manage recurring syncs
|
|
|
|
After creating an index to be managed by a connector, you can configure automatic, recurring syncs.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Connectors* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Choose the index to configure, and then choose the *Scheduling* tab.
|
|
|
|
Within this interface, you can enable or disable scheduled:
|
|
|
|
. Full content syncs
|
|
. Incremental content syncs (if supported)
|
|
. Access control syncs (if supported)
|
|
|
|
When enabled, you can additionally manage the sync schedule.
|
|
|
|
This operation requires access to Kibana and the `write` {ref}/security-privileges.html[indices privilege^] for the `.elastic-connectors` index.
|
|
|
|
Alternatively, you can <<es-connectors-usage-syncs-manual,sync once>>.
|
|
|
|
After you enable recurring syncs or sync once, the first sync will begin.
|
|
(There may be a short delay before the connector service begins the first sync.)
|
|
You may want to <<es-connectors-usage-index-view,view the index details>> to see the status or errors, or <<es-connectors-usage-documents,view the synced documents>>.
|
|
|
|
[discrete#es-connectors-usage-syncs-manual]
|
|
=== Sync once
|
|
|
|
After creating the index to be managed by a connector, you can request a single sync at any time.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Then choose the index to sync.
|
|
|
|
Regardless of which tab is active, the *Sync* button is always visible in the top right.
|
|
Choose this button to reveal sync options:
|
|
|
|
. Full content
|
|
. Incremental content (if supported)
|
|
. Access control (if supported)
|
|
|
|
Choose one of the options to request a sync.
|
|
(There may be a short delay before the connector service begins the sync.)
|
|
|
|
This operation requires access to Kibana and the `write` {ref}/security-privileges.html[indices privilege^] for the `.elastic-connectors` index.
|
|
|
|
[discrete#es-connectors-usage-syncs-cancel]
|
|
=== Cancel sync
|
|
|
|
After a sync has started, you can cancel the sync before it completes.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Then choose the index with the running sync.
|
|
|
|
Regardless of which tab is active, the *Sync* button is always visible in the top right.
|
|
Choose this button to reveal sync options, and choose *Cancel Syncs* to cancel active syncs.
|
|
This will cancel the running job, and marks all _pending_ and _suspended_ jobs as canceled as well.
|
|
(There may be a short delay before the connector service cancels the syncs.)
|
|
|
|
This operation requires access to Kibana and the `write` {ref}/security-privileges.html[indices privilege^] for the `.elastic-connectors` and `.elastic-connectors-sync-jobs` index.
|
|
|
|
[discrete#es-connectors-usage-index-view]
|
|
=== View status
|
|
|
|
View the index details to see a variety of information that communicate the status of the index and connector.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Then choose the index to view.
|
|
|
|
The *Overview* tab presents a variety of information, including:
|
|
|
|
* General information about the connector index, for example: name, description, ingestion type, connector type, and language analyzer.
|
|
* Any errors affecting the connector or sync process.
|
|
* The current ingestion status (see below for possible values).
|
|
* The current document count.
|
|
|
|
Possible values of ingestion status:
|
|
|
|
* Incomplete - A connector that is not configured yet.
|
|
* Configured - A connector that is configured.
|
|
* Connected - A connector that can successfully connect to a data source.
|
|
* Error - A connector that failed to connect to the data source.
|
|
* Connector failure - A connector that has not seen any update for more than 30 minutes.
|
|
* Sync failure - A connector that failed in the last sync job.
|
|
|
|
This tab also displays the recent sync history, including sync status (see below for possible values).
|
|
|
|
Possible values of sync status:
|
|
|
|
* Sync pending - The initial job status, the job is pending to be picked up.
|
|
* Sync in progress - The job is running.
|
|
* Canceling sync - Cancelation of the job has been requested.
|
|
* Sync canceled - The job was canceled
|
|
* Sync suspended - The job was suspended due to service shutdown, and it can be resumed when the service restarts.
|
|
* Sync complete - The job completed successfully.
|
|
* Sync failure - The job failed.
|
|
|
|
For each sync, choose the `view` button to display the job details, including:
|
|
|
|
* The job ID
|
|
* Document stats, including: number of documents added/deleted, total number of documents, and volume of documented added
|
|
* Event logs
|
|
* Sync rules that were active when the sync was requested
|
|
* Pipelines that were active when the sync was requested
|
|
|
|
This operation requires access to Kibana and the `read` {ref}/security-privileges.html[indices privilege^] for the `.elastic-connectors` index.
|
|
|
|
[discrete#es-connectors-usage-documents]
|
|
=== View documents
|
|
|
|
View the documents the connector has synced from the data.
|
|
Additionally view the index mappings to determine the current document schema.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Then choose the index to view.
|
|
|
|
Choose the *Documents* tab to view the synced documents.
|
|
Choose the *Index Mappings* tab to view the index mappings that were created by the connector.
|
|
|
|
When setting up a new connector, ensure you are getting the documents and fields you were expecting from the data source.
|
|
If not, see <<es-connectors-troubleshooting>> for help.
|
|
|
|
These operations require access to Kibana and the `read` and `manage` {ref}/security-privileges.html[indices privileges^] for the index containing the documents.
|
|
|
|
See <<es-connectors-security>> for security details.
|
|
|
|
[discrete#es-connectors-usage-sync-rules]
|
|
=== Manage sync rules
|
|
|
|
Use <<es-sync-rules,sync rules>> to limit which documents are fetched from the data source, or limit which fetched documents are stored in Elastic.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Then choose the index to manage and choose the *Sync rules* tab.
|
|
|
|
[discrete#es-connectors-usage-pipelines]
|
|
=== Manage ingest pipelines
|
|
|
|
Use {ref}/ingest-pipeline-search.html[ingest pipelines] to transform fetched data before it is stored in Elastic.
|
|
|
|
In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
|
|
|
|
Then choose the index to manage and choose the *Pipelines* tab.
|