mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-04-24 23:27:25 -04:00
Ingest download databases docs (#111688)
Co-authored-by: Joe Gallo <joegallo@gmail.com>
This commit is contained in:
Keith Massey
GitHub
parent
f02be0779f
commit
c6a7537df7
8 changed files with 233 additions and 8 deletions
55
docs/reference/ingest/apis/delete-geoip-database.asciidoc
Normal file
55
docs/reference/ingest/apis/delete-geoip-database.asciidoc
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
[[delete-geoip-database-api]]
|
||||
=== Delete geoip database configuration API
|
||||
++++
|
||||
<titleabbrev>Delete geoip database configuration</titleabbrev>
|
||||
++++
|
||||
|
||||
Deletes a geoip database configuration.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
DELETE /_ingest/geoip/database/my-database-id
|
||||
----
|
||||
// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
|
||||
|
||||
|
||||
[[delete-geoip-database-api-request]]
|
||||
==== {api-request-title}
|
||||
|
||||
`DELETE /_ingest/geoip/database/<database>`
|
||||
|
||||
[[delete-geoip-database-api-prereqs]]
|
||||
==== {api-prereq-title}
|
||||
|
||||
* If the {es} {security-features} are enabled, you must have the
|
||||
`manage` <<privileges-list-cluster,cluster privilege>> to use this API.
|
||||
|
||||
[[delete-geoip-database-api-path-params]]
|
||||
==== {api-path-parms-title}
|
||||
|
||||
`<database>`::
|
||||
+
|
||||
--
|
||||
(Required, string) Database configuration ID used to limit the request.
|
||||
|
||||
--
|
||||
|
||||
|
||||
[[delete-geoip-database-api-query-params]]
|
||||
==== {api-query-parms-title}
|
||||
|
||||
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
|
||||
|
||||
|
||||
[[delete-geoip-database-api-example]]
|
||||
==== {api-examples-title}
|
||||
|
||||
|
||||
[[delete-geoip-database-api-specific-ex]]
|
||||
===== Delete a specific geoip database configuration
|
||||
|
||||
[source,console]
|
||||
----
|
||||
DELETE /_ingest/geoip/database/example-database-id
|
||||
----
|
||||
// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
|
||||
|
|
@ -62,7 +62,7 @@ use a value of `*`.
|
|||
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
|
||||
|
||||
|
||||
[[delete-pipeline-api-api-example]]
|
||||
[[delete-pipeline-api-example]]
|
||||
==== {api-examples-title}
|
||||
|
||||
|
||||
|
|
|
|||
80
docs/reference/ingest/apis/get-geoip-database.asciidoc
Normal file
80
docs/reference/ingest/apis/get-geoip-database.asciidoc
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
[[get-geoip-database-api]]
|
||||
=== Get geoip database configuration API
|
||||
++++
|
||||
<titleabbrev>Get geoip database configuration</titleabbrev>
|
||||
++++
|
||||
|
||||
Returns information about one or more geoip database configurations.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
GET /_ingest/geoip/database/my-database-id
|
||||
----
|
||||
// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
|
||||
|
||||
|
||||
|
||||
[[get-geoip-database-api-request]]
|
||||
==== {api-request-title}
|
||||
|
||||
`GET /_ingest/geoip/database/<database>`
|
||||
|
||||
`GET /_ingest/geoip/database`
|
||||
|
||||
[[get-geoip-database-api-prereqs]]
|
||||
==== {api-prereq-title}
|
||||
|
||||
* If the {es} {security-features} are enabled, you must have the
|
||||
`manage` <<privileges-list-cluster,cluster privilege>> to use this API.
|
||||
|
||||
[[get-geoip-database-api-path-params]]
|
||||
==== {api-path-parms-title}
|
||||
|
||||
`<database>`::
|
||||
(Optional, string)
|
||||
Comma-separated list of database configuration IDs to retrieve. Wildcard (`*`) expressions are
|
||||
supported.
|
||||
+
|
||||
To get all database configurations, omit this parameter or use `*`.
|
||||
|
||||
|
||||
[[get-geoip-database-api-query-params]]
|
||||
==== {api-query-parms-title}
|
||||
|
||||
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
|
||||
|
||||
|
||||
[[get-geoip-database-api-example]]
|
||||
==== {api-examples-title}
|
||||
|
||||
|
||||
[[get-geoip-database-api-specific-ex]]
|
||||
===== Get information for a specific geoip database configuration
|
||||
|
||||
[source,console]
|
||||
----
|
||||
GET /_ingest/geoip/database/my-database-id
|
||||
----
|
||||
// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
|
||||
|
||||
The API returns the following response:
|
||||
|
||||
[source,console-result]
|
||||
----
|
||||
{
|
||||
"databases" : [
|
||||
{
|
||||
"id" : "my-database-id",
|
||||
"version" : 1,
|
||||
"modified_date_millis" : 1723040276114,
|
||||
"database" : {
|
||||
"name" : "GeoIP2-Domain",
|
||||
"maxmind" : {
|
||||
"account_id" : "1234567"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
----
|
||||
// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
|
||||
|
|
@ -65,7 +65,7 @@ To get all ingest pipelines, omit this parameter or use `*`.
|
|||
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
|
||||
|
||||
|
||||
[[get-pipeline-api-api-example]]
|
||||
[[get-pipeline-api-example]]
|
||||
==== {api-examples-title}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ Use the following APIs to create, manage, and test ingest pipelines:
|
|||
* <<put-pipeline-api>> to create or update a pipeline
|
||||
* <<get-pipeline-api>> to retrieve a pipeline configuration
|
||||
* <<delete-pipeline-api>> to delete a pipeline
|
||||
* <<simulate-pipeline-api>> to test a pipeline
|
||||
* <<simulate-pipeline-api>> and <<simulate-ingest-api>> to test ingest pipelines
|
||||
|
||||
[discrete]
|
||||
[[ingest-stat-apis]]
|
||||
|
|
@ -21,12 +21,27 @@ Use the following APIs to create, manage, and test ingest pipelines:
|
|||
|
||||
Use the following APIs to get statistics about ingest processing:
|
||||
|
||||
* <<geoip-stats-api>> to get download statistics for GeoIP2 databases used with
|
||||
* <<geoip-stats-api>> to get download statistics for IP geolocation databases used with
|
||||
the <<geoip-processor,`geoip` processor>>.
|
||||
|
||||
[discrete]
|
||||
[[ingest-geoip-database-apis]]
|
||||
=== Ingest GeoIP Database APIs
|
||||
|
||||
preview::["The commercial IP geolocation database download management APIs are in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."]
|
||||
|
||||
Use the following APIs to configure and manage commercial IP geolocation database downloads:
|
||||
|
||||
* <<put-geoip-database-api>> to create or update a database configuration
|
||||
* <<get-geoip-database-api>> to retrieve a database configuration
|
||||
* <<delete-geoip-database-api>> to delete a database configuration
|
||||
|
||||
include::put-pipeline.asciidoc[]
|
||||
include::delete-pipeline.asciidoc[]
|
||||
include::geoip-stats-api.asciidoc[]
|
||||
include::get-pipeline.asciidoc[]
|
||||
include::delete-pipeline.asciidoc[]
|
||||
include::simulate-pipeline.asciidoc[]
|
||||
include::simulate-ingest.asciidoc[]
|
||||
include::geoip-stats.asciidoc[]
|
||||
include::put-geoip-database.asciidoc[]
|
||||
include::get-geoip-database.asciidoc[]
|
||||
include::delete-geoip-database.asciidoc[]
|
||||
|
|
|
|||
72
docs/reference/ingest/apis/put-geoip-database.asciidoc
Normal file
72
docs/reference/ingest/apis/put-geoip-database.asciidoc
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
[[put-geoip-database-api]]
|
||||
=== Create or update geoip database configuration API
|
||||
++++
|
||||
<titleabbrev>Create or update geoip database configuration</titleabbrev>
|
||||
++++
|
||||
|
||||
Creates or updates an IP geolocation database configuration.
|
||||
|
||||
IMPORTANT: The Maxmind `account_id` shown below requires a license key. Because the license key is sensitive information,
|
||||
it is stored as a <<secure-settings,secure setting>> in {es} named `ingest.geoip.downloader.maxmind.license_key`. Only
|
||||
one Maxmind license key is currently allowed per {es} cluster. A valid license key must be in the secure settings in order
|
||||
to download from Maxmind. The license key setting does not take effect until all nodes are restarted.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT _ingest/geoip/database/my-database-id
|
||||
{
|
||||
"name": "GeoIP2-Domain",
|
||||
"maxmind": {
|
||||
"account_id": "1025402"
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
|
||||
|
||||
[[put-geoip-database-api-request]]
|
||||
==== {api-request-title}
|
||||
|
||||
`PUT /_ingest/geoip/database/<database>`
|
||||
|
||||
[[put-geoip-database-api-prereqs]]
|
||||
==== {api-prereq-title}
|
||||
|
||||
* If the {es} {security-features} are enabled, you must have the
|
||||
`manage` <<privileges-list-cluster,cluster privilege>> to use this API.
|
||||
|
||||
|
||||
[[put-geoip-database-api-path-params]]
|
||||
==== {api-path-parms-title}
|
||||
|
||||
`<database>`::
|
||||
+
|
||||
__
|
||||
(Required, string) ID of the database configuration to create or update.
|
||||
|
||||
[[put-geoip-database-api-query-params]]
|
||||
==== {api-query-parms-title}
|
||||
|
||||
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
|
||||
|
||||
[[put-geoip-database-api-request-body]]
|
||||
==== {api-request-body-title}
|
||||
|
||||
// tag::geoip-database-object[]
|
||||
`name`::
|
||||
(Required, string)
|
||||
The provider-assigned name of the IP geolocation database to download.
|
||||
|
||||
`<provider>`::
|
||||
(Required, a provider object and its associated configuration)
|
||||
The configuration necessary to identify which IP geolocation provider to use to download
|
||||
the database, as well as any provider-specific configuration necessary for such downloading.
|
||||
+
|
||||
At present, the only supported provider is `maxmind`, and the maxmind provider
|
||||
requires that an `account_id` (string) is configured.
|
||||
// end::geoip-database-object[]
|
||||
|
||||
[[geoip-database-configuration-licensing]]
|
||||
==== Licensing
|
||||
|
||||
Downloading databases from third party providers is a commercial feature that requires an
|
||||
appropriate license. For more information, refer to https://www.elastic.co/subscriptions.
|
||||
|
|
@ -24,6 +24,9 @@ stats API>>.
|
|||
If your cluster can't connect to the Elastic GeoIP endpoint or you want to
|
||||
manage your own updates, see <<manage-geoip-database-updates>>.
|
||||
|
||||
If you would like to have {es} download database files directly from Maxmind using your own provided
|
||||
license key, see <<put-geoip-database-api>>.
|
||||
|
||||
If {es} can't connect to the endpoint for 30 days all updated databases will become
|
||||
invalid. {es} will stop enriching documents with geoip data and will add `tags: ["_geoip_expired_database"]`
|
||||
field instead.
|
||||
|
|
@ -36,9 +39,9 @@ field instead.
|
|||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `field` | yes | - | The field to get the ip address from for the geographical lookup.
|
||||
| `field` | yes | - | The field to get the IP address from for the geographical lookup.
|
||||
| `target_field` | no | geoip | The field that will hold the geographical information looked up from the MaxMind database.
|
||||
| `database_file` | no | GeoLite2-City.mmdb | The database filename referring to one of the automatically downloaded GeoLite2 databases (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or the name of a supported database file in the `ingest-geoip` config directory.
|
||||
| `database_file` | no | GeoLite2-City.mmdb | The database filename referring to one of the automatically downloaded GeoLite2 databases (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb), or the name of a supported database file in the `ingest-geoip` config directory, or the name of a <<get-geoip-database-api, configured database>> (with the `.mmdb` suffix appended).
|
||||
| `properties` | no | [`continent_name`, `country_iso_code`, `country_name`, `region_iso_code`, `region_name`, `city_name`, `location`] * | Controls what properties are added to the `target_field` based on the geoip lookup.
|
||||
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|
||||
| `first_only` | no | `true` | If `true` only first found geoip data will be returned, even if `field` contains array
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue