mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-30 10:23:41 -04:00
248 lines
8 KiB
Text
248 lines
8 KiB
Text
[#es-connectors-graphql]
|
|
=== Elastic GraphQL connector reference
|
|
++++
|
|
<titleabbrev>GraphQL</titleabbrev>
|
|
++++
|
|
|
|
// Attributes used in this file
|
|
:service-name: GraphQL
|
|
:service-name-stub: graphql
|
|
|
|
The Elastic GraphQL connector is written in Python using the https://github.com/elastic/connectors/tree/main[Elastic connector framework]. View the https://github.com/elastic/connectors/blob/main/connectors/sources/graphql.py[source code for this connector].
|
|
|
|
[discrete#es-connectors-graphql-connector-availability-and-prerequisites]
|
|
==== Availability and prerequisites
|
|
|
|
This connector was introduced in Elastic *8.14.0*, available as a *self-managed* self-managed connector.
|
|
|
|
To use this connector, satisfy all <<es-build-connector, self-managed connector prerequisites>>.
|
|
Importantly, you must deploy the connectors service on your own infrastructure.
|
|
You have two deployment options:
|
|
|
|
* <<es-connectors-run-from-source, Run connectors service from source>>. Use this option if you're comfortable working with Python and want to iterate quickly locally.
|
|
* <<es-connectors-run-from-docker, Run connectors service in Docker>>. Use this option if you want to deploy the connectors to a server, or use a container orchestration platform.
|
|
|
|
[NOTE]
|
|
====
|
|
This connector is in *technical preview* and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Technical preview features are not subject to the support SLA of official GA features.
|
|
====
|
|
|
|
[discrete#es-connectors-graphql-connector-usage]
|
|
==== Usage
|
|
|
|
To set up this connector in the UI, select the *GraphQL* tile when creating a new connector under *Search -> Connectors*.
|
|
|
|
If you're already familiar with how connectors work, you can also use the {ref}/connector-apis.html[Connector APIs].
|
|
|
|
For additional operations, see <<es-connectors-usage>>.
|
|
|
|
[discrete#es-connectors-graphql-connector-docker]
|
|
==== Deploy with Docker
|
|
|
|
include::_connectors-docker-instructions.asciidoc[]
|
|
|
|
[discrete#es-connectors-graphql-connector-configuration]
|
|
==== Configuration
|
|
|
|
[discrete#es-connectors-graphql-connector-configure-graphql-connector]
|
|
===== Configure GraphQL connector
|
|
|
|
Note the following configuration fields:
|
|
|
|
`http_endpoint` (required)::
|
|
Base URL of the GraphQL endpoint.
|
|
*Example*: `https://api.xyz.com/graphql`
|
|
|
|
`http_method` (required)::
|
|
`GET` or `POST`.
|
|
|
|
`authentication_method`(required)::
|
|
Select from `No Auth`, `Basic Auth`, and `Bearer Token`.
|
|
|
|
`username`::
|
|
Required when using basic authentication.
|
|
|
|
`password`::
|
|
Required when using basic authentication.
|
|
|
|
`token`::
|
|
Required when using bearer token authentication.
|
|
|
|
`graphql_query` (required)::
|
|
Query used to fetch data from the source.
|
|
Can contain variables provided in the `graphql_variables` field.
|
|
The connector will substitute the variables in the query with values from `graphql_variables` and make a GraphQL query to the source.
|
|
+
|
|
*Example*:
|
|
+
|
|
[source,js]
|
|
----
|
|
query getUser($id: ID!) {
|
|
user(id: $id) {
|
|
name
|
|
email
|
|
}
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
`graphql_variables`::
|
|
A JSON object of key/value pairs containing variables used in the GraphQL query.
|
|
The connector will substitute the variables in the query with the values provided here and make a GraphQL query to the source.
|
|
+
|
|
*Example*:
|
|
+
|
|
For the GraphQL query `query getUser($id: ID!) { user(id: $id) { name } }`
|
|
+
|
|
* Where the value of `graphql_variables` is `{"id": "123"}`
|
|
* The connector will execute `query getUser { user(id: "123") { name } }` to fetch data from the source
|
|
|
|
`graphql_object_to_id_map` (required)::
|
|
A JSON mapping between GraphQL response objects to index and their ID fields.
|
|
The connector will fetch data for each object (JSON key) and use the provided ID field (JSON value) to index the object into Elasticsearch.
|
|
The connector will index all fields for each object specified in the mapping.
|
|
Use dot `(.)` notation to specify the full path from the root of the GraphQL response to the desired object.
|
|
+
|
|
*Example*:
|
|
+
|
|
The GraphQL query `query getUser { organization { users{ user_id name email} } }` fetches all available users from the source.
|
|
To index every user as a separate document configure this field as below.
|
|
+
|
|
[source,js]
|
|
----
|
|
{
|
|
"organization.users": "user_id"
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
+
|
|
In this example `user_id` is unique in every user document. Therefore, we set `user_id` as the value for `organization.users`.
|
|
+
|
|
[NOTE]
|
|
====
|
|
The path provided in this field should only contain JSON objects and not lists.
|
|
====
|
|
|
|
`headers`::
|
|
JSON object containing custom headers to be sent with each GraphQL request:
|
|
+
|
|
[source,js]
|
|
----
|
|
{
|
|
"content-type": "Application/json"
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
`pagination_model` (required)::
|
|
This field specifies the pagination model to be used by the connector.
|
|
The connector supports `No pagination` and `Cursor-based pagination` pagination models.
|
|
+
|
|
For cursor-based pagination, add `pageInfo {endCursor hasNextPage}` and an `after` argument variable in your query at the desired node (`Pagination key`).
|
|
Use the `after` query argument with a variable to iterate through pages.
|
|
The default value for this field is `No pagination`. Example:
|
|
+
|
|
For `Cursor-based pagination`, the query should look like this example:
|
|
+
|
|
[source,js]
|
|
----
|
|
query getUsers($cursor: String!) {
|
|
sampleData {
|
|
users(after: $cursor) {
|
|
pageInfo {
|
|
endCursor
|
|
hasNextPage
|
|
}
|
|
nodes {
|
|
first_name
|
|
last_name
|
|
address
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
+
|
|
The value of `pagination_key` is `sampleData.users` so it must contain:
|
|
+
|
|
* `pageInfo {endCursor hasNextPage}`
|
|
* the `after` argument with a variable when using cursor-based pagination
|
|
|
|
`pagination_key` (required)::
|
|
Specifies which GraphQL object is used for pagination.
|
|
Use `.` to provide the full path of the object from the root of the response.
|
|
+
|
|
*Example*:
|
|
+
|
|
* `organization.users`
|
|
|
|
`connection_timeout`::
|
|
Specifies the maximum time in seconds to wait for a response from the GraphQL source.
|
|
Default value is *30 seconds*.
|
|
|
|
[discrete#es-connectors-graphql-connector-documents-and-syncs]
|
|
==== Documents and syncs
|
|
|
|
The connector syncs the objects and entities based on GraphQL Query and GraphQL Object List.
|
|
|
|
[discrete#es-connectors-graphql-connector-sync-types]
|
|
==== Sync types
|
|
|
|
<<es-connectors-sync-types-full,Full syncs>> are supported by default for all connectors.
|
|
|
|
This connector currently does not support <<es-connectors-sync-types-incremental,incremental syncs>>.
|
|
|
|
[discrete#es-connectors-graphql-connector-sync-rules]
|
|
==== Sync rules
|
|
|
|
<<es-sync-rules-basic,Basic sync rules>> are identical for all connectors and are available by default.
|
|
|
|
[discrete#es-connectors-graphql-connector-advanced-sync-rules]
|
|
==== Advanced Sync Rules
|
|
|
|
Advanced sync rules are not available for this connector in the present version.
|
|
|
|
[discrete#es-connectors-graphql-connector-connector-client-operations]
|
|
==== Connector Client operations
|
|
|
|
[discrete#es-connectors-graphql-connector-end-to-end-testing]
|
|
===== End-to-end Testing
|
|
|
|
The connector framework enables operators to run functional tests against a real data source, using Docker Compose.
|
|
You don't need a running Elasticsearch instance or GraphQL source to run this test.
|
|
|
|
Refer to <<es-build-connector-testing>> for more details.
|
|
|
|
To perform E2E testing for the GraphQL connector, run the following command:
|
|
|
|
```shell
|
|
$ make ftest NAME=graphql
|
|
```
|
|
For faster tests, add the `DATA_SIZE=small` flag:
|
|
|
|
[source,shell]
|
|
----
|
|
make ftest NAME=graphql DATA_SIZE=small
|
|
----
|
|
|
|
By default, `DATA_SIZE=MEDIUM`.
|
|
|
|
[discrete#es-connectors-graphql-connector-known-issues]
|
|
==== Known issues
|
|
|
|
* Every document will be updated in every sync.
|
|
* If the same field name exists with different types across different objects, the connector might raise a mapping parser exception.
|
|
|
|
Refer to <<es-connectors-known-issues>> for a list of known issues for all connectors.
|
|
|
|
[discrete#es-connectors-graphql-connector-troubleshooting]
|
|
==== Troubleshooting
|
|
|
|
See <<es-connectors-troubleshooting>>.
|
|
|
|
[discrete#es-connectors-graphql-connector-security]
|
|
==== Security
|
|
|
|
See <<es-connectors-security>>.
|
|
|