Fix broken cross-repo links, versions in search connectors docker instructions (#123700) (#125341)

(cherry picked from commit 91c2654570)

# Conflicts:
#	docs/reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md
#	docs/reference/elasticsearch-plugins/cloud/ec-adding-plugins.md
#	docs/reference/elasticsearch/configuration-reference/elastic-cloud-serverless-elasticsearch-settings.md
#	docs/reference/query-languages/esql/esql-enrich-data.md
This commit is contained in:
Liam Thompson 2025-03-20 20:38:57 +01:00 committed by GitHub
parent c36b4db253
commit cebc46f401
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
53 changed files with 292 additions and 76 deletions

View file

@ -42,7 +42,7 @@ a number of clients that have been contributed by the community for various lang
## Go [go] ## Go [go]
Also see the [official Elasticsearch Go client](go-elasticsearch://docs/reference/index.md). Also see the [official Elasticsearch Go client](go-elasticsearch://reference/index.md).
* [elastigo](https://github.com/mattbaird/elastigo): Go client. **Last commit more than a year ago** * [elastigo](https://github.com/mattbaird/elastigo): Go client. **Last commit more than a year ago**
@ -56,7 +56,7 @@ Also see the [official Elasticsearch Go client](go-elasticsearch://docs/referenc
## Java [java] ## Java [java]
Also see the [official Elasticsearch Java client](elasticsearch-java://docs/reference/index.md). Also see the [official Elasticsearch Java client](elasticsearch-java://reference/index.md).
* [Flummi](https://github.com/otto-de/flummi): Java Rest client with comprehensive Query DSL API. * [Flummi](https://github.com/otto-de/flummi): Java Rest client with comprehensive Query DSL API.
@ -64,15 +64,15 @@ Also see the [official Elasticsearch Java client](elasticsearch-java://docs/refe
## JavaScript [javascript] ## JavaScript [javascript]
See the [official Elasticsearch JavaScript client](elasticsearch-js://docs/reference/index.md). See the [official Elasticsearch JavaScript client](elasticsearch-js://reference/index.md).
## Julia [julia] ## Julia [julia]
* [ElasticsearchClient.jl](https://github.com/OpenSesame/ElasticsearchClient.jl): Elasticsearch client inspired by the [official Elasticsearch Ruby client](elasticsearch-ruby://docs/reference/index.md). * [ElasticsearchClient.jl](https://github.com/OpenSesame/ElasticsearchClient.jl): Elasticsearch client inspired by the [official Elasticsearch Ruby client](elasticsearch-ruby://reference/index.md).
## Kotlin [kotlin] ## Kotlin [kotlin]
* [ES Kotlin](https://github.com/mbuhot/eskotlin): Elasticsearch Query DSL for kotlin based on the [official Elasticsearch Java client](elasticsearch-java://docs/reference/index.md). **Last commit more than a year ago** * [ES Kotlin](https://github.com/mbuhot/eskotlin): Elasticsearch Query DSL for kotlin based on the [official Elasticsearch Java client](elasticsearch-java://reference/index.md). **Last commit more than a year ago**
* [ES Kotlin Wrapper Client](https://github.com/jillesvangurp/es-kotlin-wrapper-client): Kotlin extension functions and abstractions for the [official Elasticsearch high-level client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html). Aims to reduce the amount of boilerplate needed to do searches, bulk indexing and other common things users do with the client. **No longer maintained** * [ES Kotlin Wrapper Client](https://github.com/jillesvangurp/es-kotlin-wrapper-client): Kotlin extension functions and abstractions for the [official Elasticsearch high-level client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html). Aims to reduce the amount of boilerplate needed to do searches, bulk indexing and other common things users do with the client. **No longer maintained**
@ -82,7 +82,7 @@ See the [official Elasticsearch JavaScript client](elasticsearch-js://docs/refer
## .NET [dotnet] ## .NET [dotnet]
See the [official Elasticsearch .NET client](elasticsearch-net://docs/reference/index.md). See the [official Elasticsearch .NET client](elasticsearch-net://reference/index.md).
## Perl [perl] ## Perl [perl]
@ -92,7 +92,7 @@ Also see the [official Elasticsearch Perl client](https://www.elastic.co/guide/e
## PHP [php] ## PHP [php]
Also see the [official Elasticsearch PHP client](elasticsearch-php://docs/reference/index.md). Also see the [official Elasticsearch PHP client](elasticsearch-php://reference/index.md).
* [Elastica](https://github.com/ruflin/Elastica): PHP client. * [Elastica](https://github.com/ruflin/Elastica): PHP client.
@ -102,7 +102,7 @@ Also see the [official Elasticsearch PHP client](elasticsearch-php://docs/refere
## Python [python] ## Python [python]
See the [official Elasticsearch Python client](elasticsearch-py://docs/reference/index.md). See the [official Elasticsearch Python client](elasticsearch-py://reference/index.md).
## R [r] ## R [r]
@ -114,7 +114,7 @@ See the [official Elasticsearch Python client](elasticsearch-py://docs/reference
## Ruby [ruby] ## Ruby [ruby]
Also see the [official Elasticsearch Ruby client](elasticsearch-ruby://docs/reference/index.md). Also see the [official Elasticsearch Ruby client](elasticsearch-ruby://reference/index.md).
* [chewy](https://github.com/toptal/chewy): An ODM and wrapper for the official Elasticsearch client. * [chewy](https://github.com/toptal/chewy): An ODM and wrapper for the official Elasticsearch client.
@ -128,7 +128,7 @@ Also see the [official Elasticsearch Ruby client](elasticsearch-ruby://docs/refe
## Rust [rust] ## Rust [rust]
Also see the [official Elasticsearch Rust client](elasticsearch-rs://docs/reference/index.md). Also see the [official Elasticsearch Rust client](elasticsearch-rs://reference/index.md).
* [rs-es](https://github.com/benashford/rs-es): A REST API client with a strongly-typed Query DSL. **Last commit more than a year ago** * [rs-es](https://github.com/benashford/rs-es): A REST API client with a strongly-typed Query DSL. **Last commit more than a year ago**

View file

@ -0,0 +1,30 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-adding-elastic-plugins.html
---
# Add plugins provided with Elasticsearch Service [ec-adding-elastic-plugins]
You can use a variety of official plugins that are compatible with your version of {{es}}. When you upgrade to a new {{es}} version, these plugins are simply upgraded with the rest of your deployment.
## Before you begin [ec_before_you_begin_6]
Some restrictions apply when adding plugins. To learn more, check [Restrictions for {{es}} and {{kib}} plugins](cloud://release-notes/cloud-hosted/known-issues.md#ec-restrictions-plugins).
Only Gold, Platinum, Enterprise and Private subscriptions, running version 2.4.6 or later, have access to uploading custom plugins. All subscription levels, including Standard, can upload scripts and dictionaries.
To enable a plugin for a deployment:
1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body).
2. Find your deployment on the home page in the Elasticsearch Service card and select **Manage** to access it directly. Or, select **Hosted deployments** to go to the deployments page to view all of your deployments.
On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.
3. From the **Actions** dropdown, select **Edit deployment**.
4. Select **Manage user settings and extensions**.
5. Select the **Extensions** tab.
6. Select the plugins that you want to enable.
7. Select **Back**.
8. Select **Save**. The {{es}} cluster is then updated with new nodes that have the plugin installed.

View file

@ -0,0 +1,31 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-adding-plugins.html
---
# Add plugins and extensions [ec-adding-plugins]
Plugins extend the core functionality of {{es}}. There are many suitable plugins, including:
* Discovery plugins, such as the cloud AWS plugin that allows discovering nodes on EC2 instances.
* Analysis plugins, to provide analyzers targeted at languages other than English.
* Scripting plugins, to provide additional scripting languages.
Plugins can come from different sources: the official ones created or at least maintained by Elastic, community-sourced plugins from other users, and plugins that you provide. Some of the official plugins are always provided with our service, and can be [enabled per deployment](/reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md\).
There are two ways to add plugins to a deployment in Elasticsearch Service:
* [Enable one of the official plugins already available in Elasticsearch Service](/reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md\).
* [Upload a custom plugin and then enable it per deployment](/reference/elasticsearch-plugins/cloud/ec-custom-bundles.md\).
Custom plugins can include the official {{es}} plugins not provided with Elasticsearch Service, any of the community-sourced plugins, or [plugins that you write yourself](/extend/index.md). Uploading custom plugins is available only to Gold, Platinum, and Enterprise subscriptions. For more information, check [Upload custom plugins and bundles](/reference/elasticsearch-plugins/cloud/ec-custom-bundles.md\).
To learn more about the official and community-sourced plugins, refer to [{{es}} Plugins and Integrations](/reference/elasticsearch-plugins/index.md).
For a detailed guide with examples of using the Elasticsearch Service API to create, get information about, update, and delete extensions and plugins, check [Managing plugins and extensions through the API](/reference/elasticsearch-plugins/cloud/ec-plugins-guide.md\).
Plugins are not supported for {{kib}}. To learn more, check [Restrictions for {{es}} and {{kib}} plugins](cloud://release-notes/cloud-hosted/known-issues.md#ec-restrictions-plugins).

View file

@ -20,10 +20,10 @@ Integrations are not plugins, but are external tools or modules that make it eas
### Supported by Elastic: [_supported_by_elastic] ### Supported by Elastic: [_supported_by_elastic]
* [Logstash output to Elasticsearch](logstash://docs/reference/plugins-outputs-elasticsearch.md): The Logstash `elasticsearch` output plugin. * [Logstash output to Elasticsearch](logstash://reference/plugins-outputs-elasticsearch.md): The Logstash `elasticsearch` output plugin.
* [Elasticsearch input to Logstash](logstash://docs/reference/plugins-inputs-elasticsearch.md) The Logstash `elasticsearch` input plugin. * [Elasticsearch input to Logstash](logstash://reference/plugins-inputs-elasticsearch.md) The Logstash `elasticsearch` input plugin.
* [Elasticsearch event filtering in Logstash](logstash://docs/reference/plugins-filters-elasticsearch.md) The Logstash `elasticsearch` filter plugin. * [Elasticsearch event filtering in Logstash](logstash://reference/plugins-filters-elasticsearch.md) The Logstash `elasticsearch` filter plugin.
* [Elasticsearch bulk codec](logstash://docs/reference/plugins-codecs-es_bulk.md) The Logstash `es_bulk` plugin decodes the Elasticsearch bulk format into individual events. * [Elasticsearch bulk codec](logstash://reference/plugins-codecs-es_bulk.md) The Logstash `es_bulk` plugin decodes the Elasticsearch bulk format into individual events.
### Supported by the community: [_supported_by_the_community_2] ### Supported by the community: [_supported_by_the_community_2]
@ -71,7 +71,7 @@ Integrations are not plugins, but are external tools or modules that make it eas
### Supported by Elastic: [_supported_by_elastic_2] ### Supported by Elastic: [_supported_by_elastic_2]
* [es-hadoop](elasticsearch-hadoop://docs/reference/preface.md): Elasticsearch real-time search and analytics natively integrated with Hadoop. Supports Map/Reduce, Cascading, Apache Hive, Apache Pig, Apache Spark and Apache Storm. * [es-hadoop](https://www.elastic.co/elasticsearch/hadoop): Elasticsearch real-time search and analytics natively integrated with Hadoop. Supports Map/Reduce, Cascading, Apache Hive, Apache Pig, Apache Spark and Apache Storm.
### Supported by the community: [_supported_by_the_community_5] ### Supported by the community: [_supported_by_the_community_5]

View file

@ -0,0 +1,157 @@
---
navigation_title: "Serverless differences"
mapped_pages:
- https://www.elastic.co/guide/en/serverless/current/elasticsearch-differences.html
---
# Differences from other {{es}} offerings [elasticsearch-differences]
[{{es-serverless}}](docs-content://solutions/search.md) handles all the infrastructure management for you, providing a fully managed {{es}} service.
If youve used {{es}} before, youll notice some differences in how you work with the service on {{serverless-full}}, because a number of APIs and settings are not required for serverless projects.
This guide helps you understand whats different, whats available, and how to work effectively when running {{es}} on {{serverless-full}}.
## Fully managed infrastructure [elasticsearch-differences-serverless-infrastructure-management]
{{es-serverless}} manages all infrastructure automatically, including:
* Cluster scaling and optimization
* Node management and allocation
* Shard distribution and replication
* Resource utilization and monitoring
This fully managed approach means many traditional {{es}} infrastructure APIs and settings are not available to end users, as detailed in the following sections.
## Index size guidelines [elasticsearch-differences-serverless-index-size]
To ensure optimal performance, follow these recommendations for sizing individual indices on {{es-serverless}}:
| Use case | Maximum index size | Project configuration |
| --- | --- | --- |
| Vector search | 150GB | Vector optimized |
| General search (non data-stream) | 300GB | General purpose |
| Other uses (non data-stream) | 600GB | General purpose |
For large datasets that exceed the recommended maximum size for a single index, consider splitting your data across smaller indices and using an alias to search them collectively.
These recommendations do not apply to indices using better binary quantization (BBQ). Refer to [vector quantization](/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) in the core {{es}} docs for more information.
## API availability [elasticsearch-differences-serverless-apis-availability]
Because {{es-serverless}} manages infrastructure automatically, certain APIs are not available, while others remain fully accessible.
::::{tip}
Refer to the [{{es-serverless}} API reference](https://www.elastic.co/docs/api/doc/elasticsearch-serverless) for a complete list of available APIs.
::::
The following categories of operations are unavailable:
Infrastructure operations
: * All `_nodes/*` operations
* All `_cluster/*` operations
* Most `_cat/*` operations, except for index-related operations such as `/_cat/indices` and `/_cat/aliases`
Storage and backup
: * All `_snapshot/*` operations
* Repository management operations
Index management
: * `indices/close` operations
* `indices/open` operations
* Recovery and stats operations
* Force merge operations
When attempting to use an unavailable API, youll receive a clear error message:
```json
{
"error": {
"root_cause": [
{
"type": "api_not_available_exception",
"reason": "Request for uri [/<API_ENDPOINT>] with method [<METHOD>] exists but is not available when running in serverless mode"
}
],
"status": 410
}
}
```
## Settings availability [elasticsearch-differences-serverless-settings-availability]
In {{es-serverless}}, you can only configure [index-level settings](/reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic.
Available settings
: **Index-level settings**: Settings that control how {{es}} documents are processed, stored, and searched are available to end users. These include:
* Analysis configuration
* Mapping parameters
* Search/query settings
* Indexing settings such as `refresh_interval`
Managed settings
: **Infrastructure-related settings**: Settings that affect cluster resources or data distribution are not available to end users. These include:
* Node configurations
* Cluster topology
* Shard allocation
* Resource management
## Feature availability [elasticsearch-differences-serverless-feature-categories]
Some features that are available in Elastic Cloud Hosted and self-managed offerings are not available in {{es-serverless}}. These features have either been replaced by a new feature, are planned to be released in future, or are not applicable in the new serverless architecture.
### Replaced features [elasticsearch-differences-serverless-features-replaced]
These features have been replaced by a new feature and are therefore not available on {{es-serverless}}:
* **Index lifecycle management ({{ilm-init}})** is not available, in favor of [**data stream lifecycle**](docs-content://manage-data/data-store/index-basics.md).
In an Elastic Cloud Hosted or self-managed environment, {{ilm-init}} lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{es-serverless}} eliminates this complexity by optimizing your cluster performance for you.
Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers.
* **Watcher** is not available, in favor of [**Alerts**](docs-content://explore-analyze/alerts-cases/alerts.md#rules-alerts).
Kibana Alerts allows rich integrations across use cases like APM, metrics, security, and uptime. Prepackaged rule types simplify setup and hide the details of complex, domain-specific detections, while providing a consistent interface across Kibana.
### Planned features [elasticsearch-differences-serverless-feature-planned]
The following features are planned for future support in all {{serverless-full}} projects:
* Reindexing from remote clusters
* Cross-project search and replication
* Snapshot and restore
* Migrations from non-serverless deployments
* Audit logging
* Clone index API
* Traffic filtering and VPCs
### Unplanned features [elasticsearch-differences-serverless-feature-unavailable]
The following features are not available in {{es-serverless}} and are not planned for future support:
* [Custom plugins and bundles](docs-content://deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md)
* {{es}} for Apache Hadoop
* [Scripted metric aggregations](/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md)
* Managed web crawler: You can use the [self-managed web crawler](https://github.com/elastic/crawler) instead.
* Managed Search connectors: You can use [self-managed Search connectors](/reference/ingestion-tools/search-connectors/self-managed-connectors.md) instead.

View file

@ -22,7 +22,7 @@ By default, {{es}} {{monitor-features}} are enabled but data collection is disab
Except where noted otherwise, these settings can be dynamically updated on a live cluster with the [cluster-update-settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) API. Except where noted otherwise, these settings can be dynamically updated on a live cluster with the [cluster-update-settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) API.
To adjust how monitoring data is displayed in the monitoring UI, configure [`xpack.monitoring` settings](kibana://docs/reference/configuration-reference/monitoring-settings.md) in `kibana.yml`. To control how monitoring data is collected from {{ls}}, configure monitoring settings in `logstash.yml`. To adjust how monitoring data is displayed in the monitoring UI, configure [`xpack.monitoring` settings](kibana://reference/configuration-reference/monitoring-settings.md) in `kibana.yml`. To control how monitoring data is collected from {{ls}}, configure monitoring settings in `logstash.yml`.
For more information, see [Monitor a cluster](docs-content://deploy-manage/monitor.md). For more information, see [Monitor a cluster](docs-content://deploy-manage/monitor.md).

View file

@ -21,7 +21,7 @@ All of these settings can be added to the `elasticsearch.yml` configuration file
`xpack.security.enabled` `xpack.security.enabled`
: ([Static](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md#static-cluster-setting)) Defaults to `true`, which enables {{es}} {{security-features}} on the node. This setting must be enabled to use Elasticsearchs authentication, authorization and audit features.<br> : ([Static](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md#static-cluster-setting)) Defaults to `true`, which enables {{es}} {{security-features}} on the node. This setting must be enabled to use Elasticsearchs authentication, authorization and audit features.<br>
If set to `false`, {{security-features}} are disabled, which is not recommended. It also affects all {{kib}} instances that connect to this {{es}} instance; you do not need to disable {{security-features}} in those `kibana.yml` files. For more information about disabling {{security-features}} in specific {{kib}} instances, see [{{kib}} security settings](kibana://docs/reference/configuration-reference/security-settings.md). If set to `false`, {{security-features}} are disabled, which is not recommended. It also affects all {{kib}} instances that connect to this {{es}} instance; you do not need to disable {{security-features}} in those `kibana.yml` files. For more information about disabling {{security-features}} in specific {{kib}} instances, see [{{kib}} security settings](kibana://reference/configuration-reference/security-settings.md).
`xpack.security.autoconfiguration.enabled` `xpack.security.autoconfiguration.enabled`

View file

@ -319,7 +319,7 @@ $$$email-account-attributes$$$
: ([Static](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md#static-cluster-setting)) Set to `false` to completely disable HTML sanitation. Not recommended. Defaults to `true`. : ([Static](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md#static-cluster-setting)) Set to `false` to completely disable HTML sanitation. Not recommended. Defaults to `true`.
`xpack.notification.reporting.warning.kbn-csv-contains-formulas.text` `xpack.notification.reporting.warning.kbn-csv-contains-formulas.text`
: ([Dynamic](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md#dynamic-cluster-setting)) Specifies a custom message, which is sent if the formula verification criteria for CSV files from {{kib}}'s [`xpack.reporting.csv.checkForFormulas`](kibana://docs/reference/configuration-reference/reporting-settings.md#reporting-csv-settings) is `true`. Use `%s` in the message as a placeholder for the filename. Defaults to `Warning: The attachment [%s] contains characters which spreadsheet applications may interpret as formulas. Please ensure that the attachment is safe prior to opening.` : ([Dynamic](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md#dynamic-cluster-setting)) Specifies a custom message, which is sent if the formula verification criteria for CSV files from {{kib}}'s [`xpack.reporting.csv.checkForFormulas`](kibana://reference/configuration-reference/reporting-settings.md#reporting-csv-settings) is `true`. Use `%s` in the message as a placeholder for the filename. Defaults to `Warning: The attachment [%s] contains characters which spreadsheet applications may interpret as formulas. Please ensure that the attachment is safe prior to opening.`
## {{watcher}} Email TLS/SSL settings [ssl-notification-smtp-settings] ## {{watcher}} Email TLS/SSL settings [ssl-notification-smtp-settings]

View file

@ -290,7 +290,7 @@ Python
: See [elasticsearch.helpers.*](https://elasticsearch-py.readthedocs.io/en/stable/helpers.html) : See [elasticsearch.helpers.*](https://elasticsearch-py.readthedocs.io/en/stable/helpers.html)
JavaScript JavaScript
: See [client.helpers.*](elasticsearch-js://docs/reference/client-helpers.md) : See [client.helpers.*](elasticsearch-js://reference/client-helpers.md)
:::: ::::

View file

@ -567,7 +567,7 @@ You have the following options:
* Use the the built-in [Elastic Rerank](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) cross-encoder model via the inference APIs {{es}} service. * Use the the built-in [Elastic Rerank](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) cross-encoder model via the inference APIs {{es}} service.
* Use the [Cohere Rerank inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `rerank` task type. * Use the [Cohere Rerank inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `rerank` task type.
* Use the [Google Vertex AI inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `rerank` task type. * Use the [Google Vertex AI inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `rerank` task type.
* Upload a model to {{es}} with [Eland](eland://docs/reference/machine-learning.md#ml-nlp-pytorch) using the `text_similarity` NLP task type. * Upload a model to {{es}} with [Eland](eland://reference/machine-learning.md#ml-nlp-pytorch) using the `text_similarity` NLP task type.
* Then set up an [{{es}} service inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `rerank` task type. * Then set up an [{{es}} service inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the `rerank` task type.
* Refer to the [example](#text-similarity-reranker-retriever-example-eland) on this page for a step-by-step guide. * Refer to the [example](#text-similarity-reranker-retriever-example-eland) on this page for a step-by-step guide.
@ -727,7 +727,7 @@ GET /index/_search
### Example: Semantic re-ranking with a Hugging Face model [text-similarity-reranker-retriever-example-eland] ### Example: Semantic re-ranking with a Hugging Face model [text-similarity-reranker-retriever-example-eland]
The following example uses the `cross-encoder/ms-marco-MiniLM-L-6-v2` model from Hugging Face to rerank search results based on semantic similarity. The model must be uploaded to {{es}} using [Eland](eland://docs/reference/machine-learning.md#ml-nlp-pytorch). The following example uses the `cross-encoder/ms-marco-MiniLM-L-6-v2` model from Hugging Face to rerank search results based on semantic similarity. The model must be uploaded to {{es}} using [Eland](eland://reference/machine-learning.md#ml-nlp-pytorch).
::::{tip} ::::{tip}
Refer to [the Elastic NLP model reference](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md#ml-nlp-model-ref-text-similarity) for a list of third party text similarity models supported by {{es}}. Refer to [the Elastic NLP model reference](docs-content://explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md#ml-nlp-model-ref-text-similarity) for a list of third party text similarity models supported by {{es}}.
@ -743,7 +743,7 @@ Follow these steps to load the model and create a semantic re-ranker.
python -m pip install eland[pytorch] python -m pip install eland[pytorch]
``` ```
2. Upload the model to {{es}} using Eland. This example assumes you have an Elastic Cloud deployment and an API key. Refer to the [Eland documentation](eland://docs/reference/machine-learning.md#ml-nlp-pytorch-auth) for more authentication options. 2. Upload the model to {{es}} using Eland. This example assumes you have an Elastic Cloud deployment and an API key. Refer to the [Eland documentation](eland://reference/machine-learning.md#ml-nlp-pytorch-auth) for more authentication options.
```sh ```sh
eland_import_hub_model \ eland_import_hub_model \

View file

@ -80,7 +80,7 @@ $$$built-in-roles-logstash-admin$$$ `logstash_admin`
: Grants access to the `.logstash*` indices for managing configurations, and grants necessary access for logstash-specific APIs exposed by the logstash x-pack plugin. : Grants access to the `.logstash*` indices for managing configurations, and grants necessary access for logstash-specific APIs exposed by the logstash x-pack plugin.
$$$built-in-roles-logstash-system$$$ `logstash_system` $$$built-in-roles-logstash-system$$$ `logstash_system`
: Grants access necessary for the Logstash system user to send system-level data (such as monitoring) to {{es}}. For more information, see [Configuring Security in Logstash](logstash://docs/reference/secure-connection.md). : Grants access necessary for the Logstash system user to send system-level data (such as monitoring) to {{es}}. For more information, see [Configuring Security in Logstash](logstash://reference/secure-connection.md).
::::{note} ::::{note}
* This role should not be assigned to users as the granted permissions may change between releases. * This role should not be assigned to users as the granted permissions may change between releases.

View file

@ -9,7 +9,7 @@ mapped_pages:
Computes the Community ID for network flow data as defined in the [Community ID Specification](https://github.com/corelight/community-id-spec). You can use a community ID to correlate network events related to a single flow. Computes the Community ID for network flow data as defined in the [Community ID Specification](https://github.com/corelight/community-id-spec). You can use a community ID to correlate network events related to a single flow.
The community ID processor reads network flow data from related [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://docs/reference/index.md)) fields by default. If you use the ECS, no configuration is required. The community ID processor reads network flow data from related [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://reference/index.md)) fields by default. If you use the ECS, no configuration is required.
$$$community-id-options$$$ $$$community-id-options$$$

View file

@ -22,7 +22,7 @@ $$$grok-options$$$
| `field` | yes | - | The field to use for grok expression parsing | | `field` | yes | - | The field to use for grok expression parsing |
| `patterns` | yes | - | An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches. | | `patterns` | yes | - | An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches. |
| `pattern_definitions` | no | - | A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition. | | `pattern_definitions` | no | - | A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition. |
| `ecs_compatibility` | no | `disabled` | Must be `disabled` or `v1`. If `v1`, the processor uses patterns with [Elastic Common Schema (ECS)](ecs://docs/reference/ecs-field-reference.md) field names. | | `ecs_compatibility` | no | `disabled` | Must be `disabled` or `v1`. If `v1`, the processor uses patterns with [Elastic Common Schema (ECS)](ecs://reference/ecs-field-reference.md) field names. |
| `trace_match` | no | false | when true, `_ingest._grok_match_index` will be inserted into your matched documents metadata with the index into the pattern found in `patterns` that matched. | | `trace_match` | no | false | when true, `_ingest._grok_match_index` will be inserted into your matched documents metadata with the index into the pattern found in `patterns` that matched. |
| `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document | | `ignore_missing` | no | false | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document |
| `description` | no | - | Description of the processor. Useful for describing the purpose of the processor or its configuration. | | `description` | no | - | Description of the processor. Useful for describing the purpose of the processor or its configuration. |
@ -215,7 +215,7 @@ The above request will return a response body containing a key-value representat
} }
``` ```
By default, the API returns a list of legacy Grok patterns. These legacy patterns predate the [Elastic Common Schema (ECS)](ecs://docs/reference/ecs-field-reference.md) and dont use ECS field names. To return patterns that extract ECS field names, specify `v1` in the optional `ecs_compatibility` query parameter. By default, the API returns a list of legacy Grok patterns. These legacy patterns predate the [Elastic Common Schema (ECS)](ecs://reference/ecs-field-reference.md) and dont use ECS field names. To return patterns that extract ECS field names, specify `v1` in the optional `ecs_compatibility` query parameter.
```console ```console
GET _ingest/processor/grok?ecs_compatibility=v1 GET _ingest/processor/grok?ecs_compatibility=v1

View file

@ -9,7 +9,7 @@ mapped_pages:
Calculates the network direction given a source IP address, destination IP address, and a list of internal networks. Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.
The network direction processor reads IP addresses from [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://docs/reference/index.md)) fields by default. If you use the ECS, only the `internal_networks` option must be specified. The network direction processor reads IP addresses from [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://reference/index.md)) fields by default. If you use the ECS, only the `internal_networks` option must be specified.
$$$network-direction-options$$$ $$$network-direction-options$$$

View file

@ -11,7 +11,7 @@ The Redact processor uses the Grok rules engine to obscure text in the input doc
{{es}} comes packaged with a number of useful predefined [patterns](https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/ecs-v1) that can be conveniently referenced by the Redact processor. If one of those does not suit your needs, create a new pattern with a custom pattern definition. The Redact processor replaces every occurrence of a match. If there are multiple matches all will be replaced with the pattern name. {{es}} comes packaged with a number of useful predefined [patterns](https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/ecs-v1) that can be conveniently referenced by the Redact processor. If one of those does not suit your needs, create a new pattern with a custom pattern definition. The Redact processor replaces every occurrence of a match. If there are multiple matches all will be replaced with the pattern name.
The Redact processor is compatible with [Elastic Common Schema (ECS)](ecs://docs/reference/ecs-field-reference.md) patterns. Legacy Grok patterns are not supported. The Redact processor is compatible with [Elastic Common Schema (ECS)](ecs://reference/ecs-field-reference.md) patterns. Legacy Grok patterns are not supported.
## Using the Redact processor in a pipeline [using-redact] ## Using the Redact processor in a pipeline [using-redact]

View file

@ -44,7 +44,7 @@ docker run -p 9200:9200 -d --name elasticsearch \
-e "xpack.security.enabled=false" \ -e "xpack.security.enabled=false" \
-e "xpack.security.http.ssl.enabled=false" \ -e "xpack.security.http.ssl.enabled=false" \
-e "xpack.license.self_generated.type=trial" \ -e "xpack.license.self_generated.type=trial" \
docker.elastic.co/elasticsearch/elasticsearch:9.0.0-beta1 docker.elastic.co/elasticsearch/elasticsearch:9.0.0
``` ```
::::{warning} ::::{warning}
@ -262,7 +262,7 @@ docker run \
--rm \ --rm \
--tty -i \ --tty -i \
--network host \ --network host \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -189,7 +189,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -237,7 +237,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -247,7 +247,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -100,7 +100,7 @@ Local content extraction is available for the following self-managed connectors:
Self-hosted content extraction is handled by a **separate** extraction service. Self-hosted content extraction is handled by a **separate** extraction service.
The versions for the extraction service do not align with the Elastic stack. For versions after `8.11.x` (including 9.0.0-beta1), you should use extraction service version `0.3.x`. The versions for the extraction service do not align with the Elastic stack. For versions after `8.11.x` (including 9.0.0), you should use extraction service version `0.3.x`.
You can run the service with the following command: You can run the service with the following command:

View file

@ -257,7 +257,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -282,7 +282,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -217,7 +217,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -103,7 +103,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -170,7 +170,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -88,7 +88,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -247,7 +247,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -37,7 +37,7 @@ You can filter by `service.type`:
## Logs reference [es-connectors-logs-reference] ## Logs reference [es-connectors-logs-reference]
Logs use Elastic Common Schema (ECS), without extensions. See [the ECS Reference^](ecs://docs/reference/index.md) for more information. Logs use Elastic Common Schema (ECS), without extensions. See [the ECS Reference^](ecs://reference/index.md) for more information.
The fields logged are: The fields logged are:

View file

@ -315,7 +315,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -243,7 +243,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -317,7 +317,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -150,7 +150,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -172,7 +172,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -179,7 +179,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -238,7 +238,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -235,7 +235,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -256,7 +256,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -86,7 +86,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -7,10 +7,8 @@ mapped_pages:
::::{tip} ::::{tip}
Use our [Docker Compose quickstart](/reference/ingestion-tools/search-connectors/es-connectors-docker-compose-quickstart.md) to quickly get started with a full Elastic Stack deployment using Connectors. Use our [Docker Compose quickstart](/reference/ingestion-tools/search-connectors/es-connectors-docker-compose-quickstart.md) to quickly get started with a full Elastic Stack deployment using Connectors.
:::: ::::
Instead of running the Connectors Service from source, you can use the official Docker image to run the service in a container. Instead of running the Connectors Service from source, you can use the official Docker image to run the service in a container.
As a prerequisite, you need to have an Elasticsearch and Kibana instance running. From inside your Kibana UI, You will need to [follow the initial setup](/reference/ingestion-tools/search-connectors/es-connectors-run-from-source.md#es-connectors-run-from-source-setup-kibana) in the same manner as if you are running the service from source. As a prerequisite, you need to have an Elasticsearch and Kibana instance running. From inside your Kibana UI, You will need to [follow the initial setup](/reference/ingestion-tools/search-connectors/es-connectors-run-from-source.md#es-connectors-run-from-source-setup-kibana) in the same manner as if you are running the service from source.
@ -59,13 +57,13 @@ docker run \
--rm \ --rm \
--tty -i \ --tty -i \
--network host \ --network host \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```
::::{tip} ::::{tip}
For unreleased versions, append the `-SNAPSHOT` suffix to the version number. For example, `docker.elastic.co/integrations/elastic-connectors:8.14.0.0-SNAPSHOT`. For unreleased versions, append the `-SNAPSHOT` suffix to the version number. For example, `docker.elastic.co/integrations/elastic-connectors:9.0.0-SNAPSHOT`.
:::: ::::

View file

@ -211,7 +211,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -235,7 +235,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -268,7 +268,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -326,7 +326,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -237,7 +237,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -198,7 +198,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -228,7 +228,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -227,7 +227,7 @@ docker run \
--network "elastic" \ --network "elastic" \
--tty \ --tty \
--rm \ --rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \ docker.elastic.co/integrations/elastic-connectors:9.0.0 \
/app/bin/elastic-ingest \ /app/bin/elastic-ingest \
-c /config/config.yml -c /config/config.yml
``` ```

View file

@ -25,14 +25,14 @@ Event Query Language (EQL) is a query language for event-based time series data,
## Required fields [eql-required-fields] ## Required fields [eql-required-fields]
With the exception of sample queries, EQL searches require that the searched data stream or index contains a *timestamp* field. By default, EQL uses the `@timestamp` field from the [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://docs/reference/index.md)). With the exception of sample queries, EQL searches require that the searched data stream or index contains a *timestamp* field. By default, EQL uses the `@timestamp` field from the [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://reference/index.md)).
EQL searches also require an *event category* field, unless you use the [`any` keyword](/reference/query-languages/eql/eql-syntax.md#eql-syntax-match-any-event-category) to search for documents without an event category field. By default, EQL uses the ECS `event.category` field. EQL searches also require an *event category* field, unless you use the [`any` keyword](/reference/query-languages/eql/eql-syntax.md#eql-syntax-match-any-event-category) to search for documents without an event category field. By default, EQL uses the ECS `event.category` field.
To use a different timestamp or event category field, see [Specify a timestamp or event category field](#specify-a-timestamp-or-event-category-field). To use a different timestamp or event category field, see [Specify a timestamp or event category field](#specify-a-timestamp-or-event-category-field).
::::{tip} ::::{tip}
While no schema is required to use EQL, we recommend using the [ECS](ecs://docs/reference/index.md). EQL searches are designed to work with core ECS fields by default. While no schema is required to use EQL, we recommend using the [ECS](ecs://reference/index.md). EQL searches are designed to work with core ECS fields by default.
:::: ::::
@ -1043,7 +1043,7 @@ The API returns:
## Specify a timestamp or event category field [specify-a-timestamp-or-event-category-field] ## Specify a timestamp or event category field [specify-a-timestamp-or-event-category-field]
The EQL search API uses the `@timestamp` and `event.category` fields from the [ECS](ecs://docs/reference/index.md) by default. To specify different fields, use the `timestamp_field` and `event_category_field` parameters: The EQL search API uses the `@timestamp` and `event.category` fields from the [ECS](ecs://reference/index.md) by default. To specify different fields, use the `timestamp_field` and `event_category_field` parameters:
```console ```console
GET /my-data-stream/_eql/search GET /my-data-stream/_eql/search
@ -1065,7 +1065,7 @@ By default, the EQL search API returns matching hits by timestamp. If two or mor
If you dont specify a tiebreaker field or the events also share the same tiebreaker value, {{es}} considers the events concurrent and may not return them in a consistent sort order. If you dont specify a tiebreaker field or the events also share the same tiebreaker value, {{es}} considers the events concurrent and may not return them in a consistent sort order.
To specify a tiebreaker field, use the `tiebreaker_field` parameter. If you use the [ECS](ecs://docs/reference/index.md), we recommend using `event.sequence` as the tiebreaker field. To specify a tiebreaker field, use the `tiebreaker_field` parameter. If you use the [ECS](ecs://reference/index.md), we recommend using `event.sequence` as the tiebreaker field.
```console ```console
GET /my-data-stream/_eql/search GET /my-data-stream/_eql/search

View file

@ -18,7 +18,7 @@ One common variant of regsvr32 misuse is a [Squiblydoo attack](https://attack.mi
## Setup [eql-ex-threat-detection-setup] ## Setup [eql-ex-threat-detection-setup]
This tutorial uses a test dataset from [Atomic Red Team](https://github.com/redcanaryco/atomic-red-team) that includes events imitating a Squiblydoo attack. The data has been mapped to [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://docs/reference/index.md)) fields. This tutorial uses a test dataset from [Atomic Red Team](https://github.com/redcanaryco/atomic-red-team) that includes events imitating a Squiblydoo attack. The data has been mapped to [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://reference/index.md)) fields.
To get started: To get started:

View file

@ -16,7 +16,7 @@ EQL queries require an event category and a matching condition. The `where` keyw
event_category where condition event_category where condition
``` ```
An event category is an indexed value of the [event category field](/reference/query-languages/eql.md#eql-required-fields). By default, the [EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) uses the `event.category` field from the [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://docs/reference/index.md)). You can specify another event category field using the APIs [`event_category_field`](/reference/query-languages/eql.md#specify-a-timestamp-or-event-category-field) parameter. An event category is an indexed value of the [event category field](/reference/query-languages/eql.md#eql-required-fields). By default, the [EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) uses the `event.category` field from the [Elastic Common Schema (ECS)][Elastic Common Schema (ECS)](ecs://reference/index.md)). You can specify another event category field using the APIs [`event_category_field`](/reference/query-languages/eql.md#specify-a-timestamp-or-event-category-field) parameter.
For example, the following EQL query matches events with an event category of `process` and a `process.name` of `svchost.exe`: For example, the following EQL query matches events with an event category of `process` and a `process.name` of `svchost.exe`:

View file

@ -93,7 +93,7 @@ To begin, add documents to one or more source indices. These documents should co
You can manage source indices just like regular {{es}} indices using the [document](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document) and [index](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices) APIs. You can manage source indices just like regular {{es}} indices using the [document](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document) and [index](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices) APIs.
You also can set up [{{beats}}](beats://reference/index.md), such as a [{{filebeat}}](beats://reference/filebeat/filebeat-installation-configuration.md), to automatically send and index documents to your source indices. See [Getting started with {{beats}}](beats://reference/index.md). You also can set up [{{beats}}](beats://docs/reference/index.md), such as a [{{filebeat}}](beats://docs/reference/filebeat/filebeat-installation-configuration.md), to automatically send and index documents to your source indices. See [Getting started with {{beats}}](beats://reference/index.md).
### Create an enrich policy [esql-create-enrich-policy] ### Create an enrich policy [esql-create-enrich-policy]
@ -197,13 +197,13 @@ Once created, you cant update or change an enrich policy. Instead, you can:
The {{esql}} `ENRICH` command supports all three enrich policy types: The {{esql}} `ENRICH` command supports all three enrich policy types:
`geo_match` `geo_match`
: Matches enrich data to incoming documents based on a [`geo_shape` query](/reference/query-languages/query-dsl/query-dsl-geo-shape-query.md). For an example, see [Example: Enrich your data based on geolocation](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md). : Matches enrich data to incoming documents based on a [`geo_shape` query](/reference/query-languages/query-dsl-geo-shape-query.md). For an example, see [Example: Enrich your data based on geolocation](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md).
`match` `match`
: Matches enrich data to incoming documents based on a [`term` query](/reference/query-languages/query-dsl/query-dsl-term-query.md). For an example, see [Example: Enrich your data based on exact values](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md). : Matches enrich data to incoming documents based on a [`term` query](/reference/query-languages/query-dsl-term-query.md). For an example, see [Example: Enrich your data based on exact values](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md).
`range` `range`
: Matches a number, date, or IP address in incoming documents to a range in the enrich index based on a [`term` query](/reference/query-languages/query-dsl/query-dsl-term-query.md). For an example, see [Example: Enrich your data by matching a value to a range](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md). : Matches a number, date, or IP address in incoming documents to a range in the enrich index based on a [`term` query](/reference/query-languages/query-dsl-term-query.md). For an example, see [Example: Enrich your data by matching a value to a range](docs-content://manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md).
While all three enrich policy types are supported, there are some limitations to be aware of: While all three enrich policy types are supported, there are some limitations to be aware of:

View file

@ -13,7 +13,7 @@ Your data may contain unstructured strings that you want to structure. This make
:alt: unstructured data :alt: unstructured data
::: :::
{{es}} can structure your data at index time or query time. At index time, you can use the [Dissect](/reference/ingestion-tools/enrich-processor/dissect-processor.md) and [Grok](/reference/ingestion-tools/enrich-processor/grok-processor.md) ingest processors, or the {{ls}} [Dissect](logstash://docs/reference/plugins-filters-dissect.md) and [Grok](logstash://docs/reference/plugins-filters-grok.md) filters. At query time, you can use the {{esql}} [`DISSECT`](/reference/query-languages/esql/esql-commands.md#esql-dissect) and [`GROK`](/reference/query-languages/esql/esql-commands.md#esql-grok) commands. {{es}} can structure your data at index time or query time. At index time, you can use the [Dissect](/reference/ingestion-tools/enrich-processor/dissect-processor.md) and [Grok](/reference/ingestion-tools/enrich-processor/grok-processor.md) ingest processors, or the {{ls}} [Dissect](logstash://reference/plugins-filters-dissect.md) and [Grok](logstash://reference/plugins-filters-grok.md) filters. At query time, you can use the {{esql}} [`DISSECT`](/reference/query-languages/esql/esql-commands.md#esql-dissect) and [`GROK`](/reference/query-languages/esql/esql-commands.md#esql-grok) commands.
## `DISSECT` or `GROK`? Or both? [esql-grok-or-dissect] ## `DISSECT` or `GROK`? Or both? [esql-grok-or-dissect]

View file

@ -102,7 +102,7 @@ To search for documents matching a pattern, use the wildcard syntax. For example
http.response.status_code: 4* http.response.status_code: 4*
``` ```
By default, leading wildcards are not allowed for performance reasons. You can modify this with the [`query:allowLeadingWildcards`](kibana://docs/reference/advanced-settings.md#query-allowleadingwildcards) advanced setting. By default, leading wildcards are not allowed for performance reasons. You can modify this with the [`query:allowLeadingWildcards`](kibana://reference/advanced-settings.md#query-allowleadingwildcards) advanced setting.
::::{note} ::::{note}
Only `*` is currently supported. This matches zero or more characters. Only `*` is currently supported. This matches zero or more characters.