diff --git a/.ci/dockerOnLinuxExclusions b/.ci/dockerOnLinuxExclusions index c150cca590f7..715ed86188dd 100644 --- a/.ci/dockerOnLinuxExclusions +++ b/.ci/dockerOnLinuxExclusions @@ -15,6 +15,7 @@ sles-15.2 sles-15.3 sles-15.4 sles-15.5 +sles-15.6 # These OSes are deprecated and filtered starting with 8.0.0, but need to be excluded # for PR checks diff --git a/README.asciidoc b/README.asciidoc index bac6d0ed7175..df6208a8f422 100644 --- a/README.asciidoc +++ b/README.asciidoc @@ -4,7 +4,7 @@ Elasticsearch is a distributed search and analytics engine, scalable data store Use cases enabled by Elasticsearch include: -* https://www.elastic.co/search-labs/blog/articles/retrieval-augmented-generation-rag[Retrieval Augmented Generation (RAG)] +* https://www.elastic.co/search-labs/blog/articles/retrieval-augmented-generation-rag[Retrieval Augmented Generation (RAG)] * https://www.elastic.co/search-labs/blog/categories/vector-search[Vector search] * Full-text search * Logs @@ -17,7 +17,7 @@ Use cases enabled by Elasticsearch include: To learn more about Elasticsearch's features and capabilities, see our https://www.elastic.co/products/elasticsearch[product page]. -To access information on https://www.elastic.co/search-labs/blog/categories/ml-research[machine learning innovations] and the latest https://www.elastic.co/search-labs/blog/categories/lucene[Lucene contributions from Elastic], more information can be found in https://www.elastic.co/search-labs[Search Labs]. +To access information on https://www.elastic.co/search-labs/blog/categories/ml-research[machine learning innovations] and the latest https://www.elastic.co/search-labs/blog/categories/lucene[Lucene contributions from Elastic], more information can be found in https://www.elastic.co/search-labs[Search Labs]. [[get-started]] == Get started @@ -27,20 +27,20 @@ https://www.elastic.co/cloud/as-a-service[Elasticsearch Service on Elastic Cloud]. If you prefer to install and manage Elasticsearch yourself, you can download -the latest version from +the latest version from https://www.elastic.co/downloads/elasticsearch[elastic.co/downloads/elasticsearch]. === Run Elasticsearch locally -//// +//// IMPORTANT: This content is replicated in the Elasticsearch repo. See `run-elasticsearch-locally.asciidoc`. Ensure both files are in sync. https://github.com/elastic/start-local is the source of truth. -//// +//// [WARNING] -==== +==== DO NOT USE THESE INSTRUCTIONS FOR PRODUCTION DEPLOYMENTS. This setup is intended for local development and testing only. @@ -93,7 +93,7 @@ Use this key to connect to Elasticsearch with a https://www.elastic.co/guide/en/ From the `elastic-start-local` folder, check the connection to Elasticsearch using `curl`: [source,sh] ----- +---- source .env curl $ES_LOCAL_URL -H "Authorization: ApiKey ${ES_LOCAL_API_KEY}" ---- @@ -101,12 +101,12 @@ curl $ES_LOCAL_URL -H "Authorization: ApiKey ${ES_LOCAL_API_KEY}" === Send requests to Elasticsearch -You send data and other requests to Elasticsearch through REST APIs. -You can interact with Elasticsearch using any client that sends HTTP requests, +You send data and other requests to Elasticsearch through REST APIs. +You can interact with Elasticsearch using any client that sends HTTP requests, such as the https://www.elastic.co/guide/en/elasticsearch/client/index.html[Elasticsearch -language clients] and https://curl.se[curl]. +language clients] and https://curl.se[curl]. -==== Using curl +==== Using curl Here's an example curl command to create a new Elasticsearch index, using basic auth: @@ -149,19 +149,19 @@ print(client.info()) ==== Using the Dev Tools Console -Kibana's developer console provides an easy way to experiment and test requests. +Kibana's developer console provides an easy way to experiment and test requests. To access the console, open Kibana, then go to **Management** > **Dev Tools**. **Add data** -You index data into Elasticsearch by sending JSON objects (documents) through the REST APIs. -Whether you have structured or unstructured text, numerical data, or geospatial data, -Elasticsearch efficiently stores and indexes it in a way that supports fast searches. +You index data into Elasticsearch by sending JSON objects (documents) through the REST APIs. +Whether you have structured or unstructured text, numerical data, or geospatial data, +Elasticsearch efficiently stores and indexes it in a way that supports fast searches. For timestamped data such as logs and metrics, you typically add documents to a data stream made up of multiple auto-generated backing indices. -To add a single document to an index, submit an HTTP post request that targets the index. +To add a single document to an index, submit an HTTP post request that targets the index. ---- POST /customer/_doc/1 @@ -171,11 +171,11 @@ POST /customer/_doc/1 } ---- -This request automatically creates the `customer` index if it doesn't exist, -adds a new document that has an ID of 1, and +This request automatically creates the `customer` index if it doesn't exist, +adds a new document that has an ID of 1, and stores and indexes the `firstname` and `lastname` fields. -The new document is available immediately from any node in the cluster. +The new document is available immediately from any node in the cluster. You can retrieve it with a GET request that specifies its document ID: ---- @@ -183,7 +183,7 @@ GET /customer/_doc/1 ---- To add multiple documents in one request, use the `_bulk` API. -Bulk data must be newline-delimited JSON (NDJSON). +Bulk data must be newline-delimited JSON (NDJSON). Each line must end in a newline character (`\n`), including the last line. ---- @@ -200,15 +200,15 @@ PUT customer/_bulk **Search** -Indexed documents are available for search in near real-time. -The following search matches all customers with a first name of _Jennifer_ +Indexed documents are available for search in near real-time. +The following search matches all customers with a first name of _Jennifer_ in the `customer` index. ---- GET customer/_search { "query" : { - "match" : { "firstname": "Jennifer" } + "match" : { "firstname": "Jennifer" } } } ---- @@ -223,9 +223,9 @@ data streams, or index aliases. . Go to **Management > Stack Management > Kibana > Data Views**. . Select **Create data view**. -. Enter a name for the data view and a pattern that matches one or more indices, -such as _customer_. -. Select **Save data view to Kibana**. +. Enter a name for the data view and a pattern that matches one or more indices, +such as _customer_. +. Select **Save data view to Kibana**. To start exploring, go to **Analytics > Discover**. @@ -254,11 +254,6 @@ To build a distribution for another platform, run the related command: ./gradlew :distribution:archives:windows-zip:assemble ---- -To build distributions for all supported platforms, run: ----- -./gradlew assemble ----- - Distributions are output to `distribution/archives`. To run the test suite, see xref:TESTING.asciidoc[TESTING]. @@ -281,7 +276,7 @@ The https://github.com/elastic/elasticsearch-labs[`elasticsearch-labs`] repo con [[contribute]] == Contribute -For contribution guidelines, see xref:CONTRIBUTING.md[CONTRIBUTING]. +For contribution guidelines, see xref:CONTRIBUTING.md[CONTRIBUTING]. [[questions]] == Questions? Problems? Suggestions? diff --git a/benchmarks/src/main/java/org/elasticsearch/benchmark/indices/resolution/IndexNameExpressionResolverBenchmark.java b/benchmarks/src/main/java/org/elasticsearch/benchmark/indices/resolution/IndexNameExpressionResolverBenchmark.java new file mode 100644 index 000000000000..976108cd57f5 --- /dev/null +++ b/benchmarks/src/main/java/org/elasticsearch/benchmark/indices/resolution/IndexNameExpressionResolverBenchmark.java @@ -0,0 +1,138 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the "Elastic License + * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side + * Public License v 1"; you may not use this file except in compliance with, at + * your election, the "Elastic License 2.0", the "GNU Affero General Public + * License v3.0 only", or the "Server Side Public License, v 1". + */ + +package org.elasticsearch.benchmark.indices.resolution; + +import org.elasticsearch.action.IndicesRequest; +import org.elasticsearch.action.support.IndicesOptions; +import org.elasticsearch.cluster.ClusterName; +import org.elasticsearch.cluster.ClusterState; +import org.elasticsearch.cluster.metadata.DataStream; +import org.elasticsearch.cluster.metadata.IndexMetadata; +import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver; +import org.elasticsearch.cluster.metadata.Metadata; +import org.elasticsearch.cluster.project.DefaultProjectResolver; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.util.concurrent.ThreadContext; +import org.elasticsearch.index.Index; +import org.elasticsearch.index.IndexVersion; +import org.elasticsearch.indices.SystemIndices; +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; + +@State(Scope.Benchmark) +@Fork(3) +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.MILLISECONDS) +@SuppressWarnings("unused") // invoked by benchmarking framework +public class IndexNameExpressionResolverBenchmark { + + private static final String DATA_STREAM_PREFIX = "my-ds-"; + private static final String INDEX_PREFIX = "my-index-"; + + @Param( + { + // # data streams | # indices + " 1000| 100", + " 5000| 500", + " 10000| 1000" } + ) + public String resourceMix = "100|10"; + + @Setup + public void setUp() { + final String[] params = resourceMix.split("\\|"); + + int numDataStreams = toInt(params[0]); + int numIndices = toInt(params[1]); + + Metadata.Builder mb = Metadata.builder(); + String[] indices = new String[numIndices + numDataStreams * (numIndices + 1)]; + int position = 0; + for (int i = 1; i <= numIndices; i++) { + String indexName = INDEX_PREFIX + i; + createIndexMetadata(indexName, mb); + indices[position++] = indexName; + } + + for (int i = 1; i <= numDataStreams; i++) { + String dataStreamName = DATA_STREAM_PREFIX + i; + List backingIndices = new ArrayList<>(); + for (int j = 1; j <= numIndices; j++) { + String backingIndexName = DataStream.getDefaultBackingIndexName(dataStreamName, j); + backingIndices.add(createIndexMetadata(backingIndexName, mb).getIndex()); + indices[position++] = backingIndexName; + } + indices[position++] = dataStreamName; + mb.put(DataStream.builder(dataStreamName, backingIndices).build()); + } + int mid = indices.length / 2; + clusterState = ClusterState.builder(ClusterName.DEFAULT).metadata(mb).build(); + resolver = new IndexNameExpressionResolver( + new ThreadContext(Settings.EMPTY), + new SystemIndices(List.of()), + DefaultProjectResolver.INSTANCE + ); + indexListRequest = new Request(IndicesOptions.lenientExpandOpenHidden(), indices); + starRequest = new Request(IndicesOptions.lenientExpandOpenHidden(), "*"); + String[] mixed = indices.clone(); + mixed[mid] = "my-*"; + mixedRequest = new Request(IndicesOptions.lenientExpandOpenHidden(), mixed); + } + + private IndexMetadata createIndexMetadata(String indexName, Metadata.Builder mb) { + IndexMetadata indexMetadata = IndexMetadata.builder(indexName) + .settings(Settings.builder().put(IndexMetadata.SETTING_VERSION_CREATED, IndexVersion.current())) + .numberOfShards(1) + .numberOfReplicas(0) + .build(); + mb.put(indexMetadata, false); + return indexMetadata; + } + + private IndexNameExpressionResolver resolver; + private ClusterState clusterState; + private Request starRequest; + private Request indexListRequest; + private Request mixedRequest; + + @Benchmark + public String[] resolveResourcesListToConcreteIndices() { + return resolver.concreteIndexNames(clusterState, indexListRequest); + } + + @Benchmark + public String[] resolveAllStarToConcreteIndices() { + return resolver.concreteIndexNames(clusterState, starRequest); + } + + @Benchmark + public String[] resolveMixedConcreteIndices() { + return resolver.concreteIndexNames(clusterState, mixedRequest); + } + + private int toInt(String v) { + return Integer.parseInt(v.trim()); + } + + record Request(IndicesOptions indicesOptions, String... indices) implements IndicesRequest { + + } +} diff --git a/build.gradle b/build.gradle index 2ef0511b2be8..a91347ca6e19 100644 --- a/build.gradle +++ b/build.gradle @@ -13,14 +13,13 @@ import com.avast.gradle.dockercompose.tasks.ComposePull import com.fasterxml.jackson.databind.JsonNode import com.fasterxml.jackson.databind.ObjectMapper +import org.elasticsearch.gradle.DistributionDownloadPlugin import org.elasticsearch.gradle.Version import org.elasticsearch.gradle.internal.BaseInternalPluginBuildPlugin import org.elasticsearch.gradle.internal.ResolveAllDependencies import org.elasticsearch.gradle.internal.info.BuildParams import org.elasticsearch.gradle.util.GradleUtils import org.gradle.plugins.ide.eclipse.model.AccessRule -import org.gradle.plugins.ide.eclipse.model.ProjectDependency -import org.elasticsearch.gradle.DistributionDownloadPlugin import java.nio.file.Files @@ -89,7 +88,7 @@ class ListExpansion { // Filters out intermediate patch releases to reduce the load of CI testing def filterIntermediatePatches = { List versions -> - versions.groupBy {"${it.major}.${it.minor}"}.values().collect {it.max()} + versions.groupBy { "${it.major}.${it.minor}" }.values().collect { it.max() } } tasks.register("updateCIBwcVersions") { @@ -101,7 +100,10 @@ tasks.register("updateCIBwcVersions") { } } - def writeBuildkitePipeline = { String outputFilePath, String pipelineTemplatePath, List listExpansions, List stepExpansions = [] -> + def writeBuildkitePipeline = { String outputFilePath, + String pipelineTemplatePath, + List listExpansions, + List stepExpansions = [] -> def outputFile = file(outputFilePath) def pipelineTemplate = file(pipelineTemplatePath) @@ -132,7 +134,12 @@ tasks.register("updateCIBwcVersions") { // Writes a Buildkite pipeline from a template, and replaces $BWC_STEPS with a list of steps, one for each version // Useful when you need to configure more versions than are allowed in a matrix configuration def expandBwcSteps = { String outputFilePath, String pipelineTemplatePath, String stepTemplatePath, List versions -> - writeBuildkitePipeline(outputFilePath, pipelineTemplatePath, [], [new StepExpansion(templatePath: stepTemplatePath, versions: versions, variable: "BWC_STEPS")]) + writeBuildkitePipeline( + outputFilePath, + pipelineTemplatePath, + [], + [new StepExpansion(templatePath: stepTemplatePath, versions: versions, variable: "BWC_STEPS")] + ) } doLast { @@ -150,7 +157,11 @@ tasks.register("updateCIBwcVersions") { new ListExpansion(versions: filterIntermediatePatches(BuildParams.bwcVersions.unreleasedIndexCompatible), variable: "BWC_LIST"), ], [ - new StepExpansion(templatePath: ".buildkite/pipelines/periodic.bwc.template.yml", versions: filterIntermediatePatches(BuildParams.bwcVersions.indexCompatible), variable: "BWC_STEPS"), + new StepExpansion( + templatePath: ".buildkite/pipelines/periodic.bwc.template.yml", + versions: filterIntermediatePatches(BuildParams.bwcVersions.indexCompatible), + variable: "BWC_STEPS" + ), ] ) @@ -302,7 +313,7 @@ allprojects { if (project.path.startsWith(":x-pack:")) { if (project.path.contains("security") || project.path.contains(":ml")) { tasks.register('checkPart4') { dependsOn 'check' } - } else if (project.path == ":x-pack:plugin" || project.path.contains("ql") || project.path.contains("smoke-test")) { + } else if (project.path == ":x-pack:plugin" || project.path.contains("ql") || project.path.contains("smoke-test")) { tasks.register('checkPart3') { dependsOn 'check' } } else if (project.path.contains("multi-node")) { tasks.register('checkPart5') { dependsOn 'check' } diff --git a/docs/changelog/114964.yaml b/docs/changelog/114964.yaml new file mode 100644 index 000000000000..8274aeb76a93 --- /dev/null +++ b/docs/changelog/114964.yaml @@ -0,0 +1,6 @@ +pr: 114964 +summary: Add a `monitor_stats` privilege and allow that privilege for remote cluster + privileges +area: Authorization +type: enhancement +issues: [] diff --git a/docs/changelog/115744.yaml b/docs/changelog/115744.yaml new file mode 100644 index 000000000000..9b8c91e59f45 --- /dev/null +++ b/docs/changelog/115744.yaml @@ -0,0 +1,6 @@ +pr: 115744 +summary: Use `SearchStats` instead of field.isAggregatable in data node planning +area: ES|QL +type: bug +issues: + - 115737 diff --git a/docs/changelog/116325.yaml b/docs/changelog/116325.yaml new file mode 100644 index 000000000000..b8cd16dc8577 --- /dev/null +++ b/docs/changelog/116325.yaml @@ -0,0 +1,5 @@ +pr: 116325 +summary: Adjust analyze limit exception to be a `bad_request` +area: Analysis +type: bug +issues: [] diff --git a/docs/changelog/116382.yaml b/docs/changelog/116382.yaml new file mode 100644 index 000000000000..c941fb6eaa1e --- /dev/null +++ b/docs/changelog/116382.yaml @@ -0,0 +1,5 @@ +pr: 116382 +summary: Validate missing shards after the coordinator rewrite +area: Search +type: bug +issues: [] diff --git a/docs/changelog/116478.yaml b/docs/changelog/116478.yaml new file mode 100644 index 000000000000..ec50799eb201 --- /dev/null +++ b/docs/changelog/116478.yaml @@ -0,0 +1,5 @@ +pr: 116478 +summary: Semantic text simple partial update +area: Search +type: bug +issues: [] diff --git a/docs/reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc b/docs/reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc index 658470c8d5a4..d5bd86825808 100644 --- a/docs/reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc +++ b/docs/reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc @@ -127,10 +127,11 @@ And the following may be the response: ==== Percentiles_bucket implementation -The Percentile Bucket returns the nearest input data point that is not greater than the requested percentile; it does not -interpolate between data points. - The percentiles are calculated exactly and is not an approximation (unlike the Percentiles Metric). This means the implementation maintains an in-memory, sorted list of your data to compute the percentiles, before discarding the data. You may run into memory pressure issues if you attempt to calculate percentiles over many millions of data-points in a single `percentiles_bucket`. + +The Percentile Bucket returns the nearest input data point to the requested percentile, rounding indices toward +positive infinity; it does not interpolate between data points. For example, if there are eight data points and +you request the `50%th` percentile, it will return the `4th` item because `ROUND_UP(.50 * (8-1))` is `4`. diff --git a/docs/reference/esql/esql-kibana.asciidoc b/docs/reference/esql/esql-kibana.asciidoc index 9850e012fc04..85969e19957a 100644 --- a/docs/reference/esql/esql-kibana.asciidoc +++ b/docs/reference/esql/esql-kibana.asciidoc @@ -9,9 +9,9 @@ You can use {esql} in {kib} to query and aggregate your data, create visualizations, and set up alerts. This guide shows you how to use {esql} in Kibana. To follow along with the -queries, load the "Sample web logs" sample data set by clicking *Try sample -data* from the {kib} Home, selecting *Other sample data sets*, and clicking *Add -data* on the *Sample web logs* card. +queries, load the "Sample web logs" sample data set by selecting **Sample Data** +from the **Integrations** page in {kib}, selecting *Other sample data sets*, +and clicking *Add data* on the *Sample web logs* card. [discrete] [[esql-kibana-enable]] @@ -30,9 +30,7 @@ However, users will be able to access existing {esql} artifacts like saved searc // tag::esql-mode[] To get started with {esql} in Discover, open the main menu and select -*Discover*. Next, from the Data views menu, select *Language: ES|QL*. - -image::images/esql/esql-data-view-menu.png[align="center",width=33%] +*Discover*. Next, select *Try ES|QL* from the application menu bar. // end::esql-mode[] [discrete] @@ -54,8 +52,9 @@ A source command can be followed by one or more <>. In this query, the processing command is <>. `LIMIT` limits the number of rows that are retrieved. -TIP: Click the help icon (image:images/esql/esql-icon-help.svg[Static,20]) to open the -in-product reference documentation for all commands and functions. +TIP: Click the **ES|QL help** button to open the +in-product reference documentation for all commands and functions or to get +recommended queries that will help you get started. // tag::autocomplete[] To make it easier to write queries, auto-complete offers suggestions with @@ -76,7 +75,7 @@ FROM kibana_sample_data_logs | LIMIT 10 ==== [discrete] -==== Expand the query bar +==== Make your query readable For readability, you can put each processing command on a new line. The following query is identical to the previous one: @@ -87,15 +86,12 @@ FROM kibana_sample_data_logs | LIMIT 10 ---- +You can do that using the **Add line breaks on pipes** button from the query editor's footer. + +image::https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd5554518309e10f6/672d153cfeb8f9d479ebcc6e/esql-line-breakdown.gif[Automatic line breaks for ES|QL queries] + // tag::compact[] -To make it easier to write multi-line queries, click the double-headed arrow -button (image:images/esql/esql-icon-expand-query-bar.svg[]) to expand the query -bar: - -image::images/esql/esql-expanded-query-bar.png[align="center"] - -To return to a compact query bar, click the minimize editor button -(image:images/esql/esql-icon-minimize-query-bar.svg[]). +You can adjust the editor's height by dragging its bottom border to your liking. // end::compact[] [discrete] @@ -110,9 +106,7 @@ detailed warning, expand the query bar, and click *warnings*. ==== Query history You can reuse your recent {esql} queries in the query bar. -In the query bar click *Show recent queries*: - -image::images/esql/esql-discover-show-recent-query.png[align="center",size="50%"] +In the query bar click *Show recent queries*. You can then scroll through your recent queries: @@ -220,8 +214,9 @@ FROM kibana_sample_data_logs === Analyze and visualize data Between the query bar and the results table, Discover shows a date histogram -visualization. If the indices you're querying do not contain a `@timestamp` -field, the histogram is not shown. +visualization. By default, if the indices you're querying do not contain a `@timestamp` +field, the histogram is not shown. But you can use a custom time field with the `?_tstart` +and `?_tend` parameters to enable it. The visualization adapts to the query. A query's nature determines the type of visualization. For example, this query aggregates the total number of bytes per @@ -250,7 +245,7 @@ save button (image:images/esql/esql-icon-save-visualization.svg[]). Once saved to a dashboard, you'll be taken to the Dashboards page. You can continue to make changes to the visualization. Click the options button in the top-right (image:images/esql/esql-icon-options.svg[]) and -select *Edit ESQL visualization* to open the in-line editor: +select *Edit ES|QL visualization* to open the in-line editor: image::images/esql/esql-kibana-edit-on-dashboard.png[align="center",width=66%] diff --git a/docs/reference/how-to/knn-search.asciidoc b/docs/reference/how-to/knn-search.asciidoc index 83614b0d9902..e884c01dd350 100644 --- a/docs/reference/how-to/knn-search.asciidoc +++ b/docs/reference/how-to/knn-search.asciidoc @@ -72,15 +72,13 @@ least enough RAM to hold the vector data and index structures. To check the size of the vector data, you can use the <> API. Here are estimates for different element types and quantization levels: -+ --- -`element_type: float`: `num_vectors * num_dimensions * 4` -`element_type: float` with `quantization: int8`: `num_vectors * (num_dimensions + 4)` -`element_type: float` with `quantization: int4`: `num_vectors * (num_dimensions/2 + 4)` -`element_type: float` with `quantization: bbq`: `num_vectors * (num_dimensions/8 + 12)` -`element_type: byte`: `num_vectors * num_dimensions` -`element_type: bit`: `num_vectors * (num_dimensions/8)` --- + +* `element_type: float`: `num_vectors * num_dimensions * 4` +* `element_type: float` with `quantization: int8`: `num_vectors * (num_dimensions + 4)` +* `element_type: float` with `quantization: int4`: `num_vectors * (num_dimensions/2 + 4)` +* `element_type: float` with `quantization: bbq`: `num_vectors * (num_dimensions/8 + 12)` +* `element_type: byte`: `num_vectors * num_dimensions` +* `element_type: bit`: `num_vectors * (num_dimensions/8)` If utilizing HNSW, the graph must also be in memory, to estimate the required bytes use `num_vectors * 4 * HNSW.m`. The default value for `HNSW.m` is 16, so by default `num_vectors * 4 * 16`. diff --git a/docs/reference/images/esql/esql-dashboard-panel.png b/docs/reference/images/esql/esql-dashboard-panel.png index d621d1170edc..61b44f7c9f85 100644 Binary files a/docs/reference/images/esql/esql-dashboard-panel.png and b/docs/reference/images/esql/esql-dashboard-panel.png differ diff --git a/docs/reference/images/esql/esql-discover-query-history.png b/docs/reference/images/esql/esql-discover-query-history.png index da31e4a6acce..ff1d2ffa8b28 100644 Binary files a/docs/reference/images/esql/esql-discover-query-history.png and b/docs/reference/images/esql/esql-discover-query-history.png differ diff --git a/docs/reference/images/esql/esql-discover-show-recent-query.png b/docs/reference/images/esql/esql-discover-show-recent-query.png deleted file mode 100644 index 13c8df9965ea..000000000000 Binary files a/docs/reference/images/esql/esql-discover-show-recent-query.png and /dev/null differ diff --git a/docs/reference/images/esql/esql-kibana-auto-complete.png b/docs/reference/images/esql/esql-kibana-auto-complete.png index d50d6b133442..155df2447dd6 100644 Binary files a/docs/reference/images/esql/esql-kibana-auto-complete.png and b/docs/reference/images/esql/esql-kibana-auto-complete.png differ diff --git a/docs/reference/images/esql/esql-kibana-bar-chart.png b/docs/reference/images/esql/esql-kibana-bar-chart.png index a760d3d69920..b74b33710d90 100644 Binary files a/docs/reference/images/esql/esql-kibana-bar-chart.png and b/docs/reference/images/esql/esql-kibana-bar-chart.png differ diff --git a/docs/reference/images/esql/esql-kibana-create-rule.png b/docs/reference/images/esql/esql-kibana-create-rule.png index c9fb14b0d2ee..a763c6f366df 100644 Binary files a/docs/reference/images/esql/esql-kibana-create-rule.png and b/docs/reference/images/esql/esql-kibana-create-rule.png differ diff --git a/docs/reference/images/esql/esql-kibana-edit-on-dashboard.png b/docs/reference/images/esql/esql-kibana-edit-on-dashboard.png index 14f6be81af7d..348b77150ed8 100644 Binary files a/docs/reference/images/esql/esql-kibana-edit-on-dashboard.png and b/docs/reference/images/esql/esql-kibana-edit-on-dashboard.png differ diff --git a/docs/reference/images/esql/esql-kibana-enrich-autocomplete.png b/docs/reference/images/esql/esql-kibana-enrich-autocomplete.png index 95a997ca2ac3..f2a0779be348 100644 Binary files a/docs/reference/images/esql/esql-kibana-enrich-autocomplete.png and b/docs/reference/images/esql/esql-kibana-enrich-autocomplete.png differ diff --git a/docs/reference/images/esql/esql-kibana-in-line-editor.png b/docs/reference/images/esql/esql-kibana-in-line-editor.png index 7b7a11e53222..85631896e833 100644 Binary files a/docs/reference/images/esql/esql-kibana-in-line-editor.png and b/docs/reference/images/esql/esql-kibana-in-line-editor.png differ diff --git a/docs/reference/rest-api/security/bulk-create-roles.asciidoc b/docs/reference/rest-api/security/bulk-create-roles.asciidoc index a1fe998c0814..a198f4938390 100644 --- a/docs/reference/rest-api/security/bulk-create-roles.asciidoc +++ b/docs/reference/rest-api/security/bulk-create-roles.asciidoc @@ -327,7 +327,7 @@ The result would then have the `errors` field set to `true` and hold the error f "details": { "my_admin_role": { <4> "type": "action_request_validation_exception", - "reason": "Validation Failed: 1: unknown cluster privilege [bad_cluster_privilege]. a privilege must be either one of the predefined cluster privilege names [manage_own_api_key,manage_data_stream_global_retention,monitor_data_stream_global_retention,none,cancel_task,cross_cluster_replication,cross_cluster_search,delegate_pki,grant_api_key,manage_autoscaling,manage_index_templates,manage_logstash_pipelines,manage_oidc,manage_saml,manage_search_application,manage_search_query_rules,manage_search_synonyms,manage_service_account,manage_token,manage_user_profile,monitor_connector,monitor_enrich,monitor_inference,monitor_ml,monitor_rollup,monitor_snapshot,monitor_text_structure,monitor_watcher,post_behavioral_analytics_event,read_ccr,read_connector_secrets,read_fleet_secrets,read_ilm,read_pipeline,read_security,read_slm,transport_client,write_connector_secrets,write_fleet_secrets,create_snapshot,manage_behavioral_analytics,manage_ccr,manage_connector,manage_enrich,manage_ilm,manage_inference,manage_ml,manage_rollup,manage_slm,manage_watcher,monitor_data_frame_transforms,monitor_transform,manage_api_key,manage_ingest_pipelines,manage_pipeline,manage_data_frame_transforms,manage_transform,manage_security,monitor,manage,all] or a pattern over one of the available cluster actions;" + "reason": "Validation Failed: 1: unknown cluster privilege [bad_cluster_privilege]. a privilege must be either one of the predefined cluster privilege names [manage_own_api_key,manage_data_stream_global_retention,monitor_data_stream_global_retention,none,cancel_task,cross_cluster_replication,cross_cluster_search,delegate_pki,grant_api_key,manage_autoscaling,manage_index_templates,manage_logstash_pipelines,manage_oidc,manage_saml,manage_search_application,manage_search_query_rules,manage_search_synonyms,manage_service_account,manage_token,manage_user_profile,monitor_connector,monitor_enrich,monitor_inference,monitor_ml,monitor_rollup,monitor_snapshot,monitor_stats,monitor_text_structure,monitor_watcher,post_behavioral_analytics_event,read_ccr,read_connector_secrets,read_fleet_secrets,read_ilm,read_pipeline,read_security,read_slm,transport_client,write_connector_secrets,write_fleet_secrets,create_snapshot,manage_behavioral_analytics,manage_ccr,manage_connector,manage_enrich,manage_ilm,manage_inference,manage_ml,manage_rollup,manage_slm,manage_watcher,monitor_data_frame_transforms,monitor_transform,manage_api_key,manage_ingest_pipelines,manage_pipeline,manage_data_frame_transforms,manage_transform,manage_security,monitor,manage,all] or a pattern over one of the available cluster actions;" } } } diff --git a/docs/reference/rest-api/security/get-builtin-privileges.asciidoc b/docs/reference/rest-api/security/get-builtin-privileges.asciidoc index 8435f5539ab9..7f3d75b92678 100644 --- a/docs/reference/rest-api/security/get-builtin-privileges.asciidoc +++ b/docs/reference/rest-api/security/get-builtin-privileges.asciidoc @@ -111,6 +111,7 @@ A successful call returns an object with "cluster", "index", and "remote_cluster "monitor_ml", "monitor_rollup", "monitor_snapshot", + "monitor_stats", "monitor_text_structure", "monitor_transform", "monitor_watcher", @@ -152,7 +153,8 @@ A successful call returns an object with "cluster", "index", and "remote_cluster "write" ], "remote_cluster" : [ - "monitor_enrich" + "monitor_enrich", + "monitor_stats" ] } -------------------------------------------------- diff --git a/docs/reference/setup/install/docker.asciidoc b/docs/reference/setup/install/docker.asciidoc index 370fc5c4ccf7..58feb55f32e2 100644 --- a/docs/reference/setup/install/docker.asciidoc +++ b/docs/reference/setup/install/docker.asciidoc @@ -86,6 +86,15 @@ docker run --name es01 --net elastic -p 9200:9200 -it -m 1GB {docker-image} TIP: Use the `-m` flag to set a memory limit for the container. This removes the need to <>. + + +{ml-cap} features such as <> +require a larger container with more than 1GB of memory. +If you intend to use the {ml} capabilities, then start the container with this command: ++ +[source,sh,subs="attributes"] +---- +docker run --name es01 --net elastic -p 9200:9200 -it -m 6GB -e "xpack.ml.use_auto_machine_memory_percent=true" {docker-image} +---- The command prints the `elastic` user password and an enrollment token for {kib}. . Copy the generated `elastic` password and enrollment token. These credentials diff --git a/docs/reference/setup/install/docker/docker-compose.yml b/docs/reference/setup/install/docker/docker-compose.yml index 15d8c11e2f12..db5b6e6c91b4 100644 --- a/docs/reference/setup/install/docker/docker-compose.yml +++ b/docs/reference/setup/install/docker/docker-compose.yml @@ -90,6 +90,7 @@ services: - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} + - xpack.ml.use_auto_machine_memory_percent=true mem_limit: ${MEM_LIMIT} ulimits: memlock: @@ -130,6 +131,7 @@ services: - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} + - xpack.ml.use_auto_machine_memory_percent=true mem_limit: ${MEM_LIMIT} ulimits: memlock: @@ -170,6 +172,7 @@ services: - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} + - xpack.ml.use_auto_machine_memory_percent=true mem_limit: ${MEM_LIMIT} ulimits: memlock: diff --git a/libs/entitlement/README.md b/libs/entitlement/README.md index 76e4db0505d3..2ab76cf1c222 100644 --- a/libs/entitlement/README.md +++ b/libs/entitlement/README.md @@ -5,3 +5,7 @@ This module implements mechanisms to grant and check permissions under the _enti The entitlements system provides an alternative to the legacy `SecurityManager` system, which is deprecated for removal. The `entitlement-agent` instruments sensitive class library methods with calls to this module, in order to enforce the controls. +This feature is currently under development, and it is completely disabled by default (the agent is not loaded). To enable it, run Elasticsearch with +```shell +./gradlew run --entitlements +``` diff --git a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/api/ElasticsearchEntitlementChecker.java b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/api/ElasticsearchEntitlementChecker.java index 330205997d21..6d5dbd4098aa 100644 --- a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/api/ElasticsearchEntitlementChecker.java +++ b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/api/ElasticsearchEntitlementChecker.java @@ -15,8 +15,6 @@ import org.elasticsearch.logging.Logger; import java.util.Optional; -import static org.elasticsearch.entitlement.runtime.internals.EntitlementInternals.isActive; - /** * Implementation of the {@link EntitlementChecker} interface, providing additional * API methods for managing the checks. @@ -25,13 +23,6 @@ import static org.elasticsearch.entitlement.runtime.internals.EntitlementInterna public class ElasticsearchEntitlementChecker implements EntitlementChecker { private static final Logger logger = LogManager.getLogger(ElasticsearchEntitlementChecker.class); - /** - * Causes entitlements to be enforced. - */ - public void activate() { - isActive = true; - } - @Override public void checkSystemExit(Class callerClass, int status) { var requestingModule = requestingModule(callerClass); @@ -66,10 +57,6 @@ public class ElasticsearchEntitlementChecker implements EntitlementChecker { } private static boolean isTriviallyAllowed(Module requestingModule) { - if (isActive == false) { - logger.debug("Trivially allowed: entitlements are inactive"); - return true; - } if (requestingModule == null) { logger.debug("Trivially allowed: Entire call stack is in the boot module layer"); return true; @@ -81,5 +68,4 @@ public class ElasticsearchEntitlementChecker implements EntitlementChecker { logger.trace("Not trivially allowed"); return false; } - } diff --git a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/internals/EntitlementInternals.java b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/internals/EntitlementInternals.java deleted file mode 100644 index ea83caf198b0..000000000000 --- a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/internals/EntitlementInternals.java +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the "Elastic License - * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side - * Public License v 1"; you may not use this file except in compliance with, at - * your election, the "Elastic License 2.0", the "GNU Affero General Public - * License v3.0 only", or the "Server Side Public License, v 1". - */ - -package org.elasticsearch.entitlement.runtime.internals; - -/** - * Don't export this from the module. Just don't. - */ -public class EntitlementInternals { - /** - * When false, entitlement rules are not enforced; all operations are allowed. - */ - public static volatile boolean isActive = false; - - public static void reset() { - isActive = false; - } -} diff --git a/muted-tests.yml b/muted-tests.yml index c36b71edad3f..1da1e370bfd1 100644 --- a/muted-tests.yml +++ b/muted-tests.yml @@ -112,9 +112,6 @@ tests: - class: org.elasticsearch.xpack.remotecluster.RemoteClusterSecurityWithApmTracingRestIT method: testTracingCrossCluster issue: https://github.com/elastic/elasticsearch/issues/112731 -- class: org.elasticsearch.xpack.test.rest.XPackRestIT - method: test {p0=esql/60_usage/Basic ESQL usage output (telemetry)} - issue: https://github.com/elastic/elasticsearch/issues/115231 - class: org.elasticsearch.xpack.inference.DefaultEndPointsIT method: testInferDeploysDefaultE5 issue: https://github.com/elastic/elasticsearch/issues/115361 @@ -279,9 +276,32 @@ tests: - class: org.elasticsearch.smoketest.MlWithSecurityIT method: test {yaml=ml/inference_crud/Test force delete given model with alias referenced by pipeline} issue: https://github.com/elastic/elasticsearch/issues/116443 +- class: org.elasticsearch.xpack.downsample.ILMDownsampleDisruptionIT + method: testILMDownsampleRollingRestart + issue: https://github.com/elastic/elasticsearch/issues/114233 - class: org.elasticsearch.xpack.test.rest.XPackRestIT - method: test {p0=esql/60_usage/Basic ESQL usage output (telemetry) non-snapshot version} - issue: https://github.com/elastic/elasticsearch/issues/116448 + method: test {p0=ml/data_frame_analytics_crud/Test put config with unknown field in outlier detection analysis} + issue: https://github.com/elastic/elasticsearch/issues/116458 +- class: org.elasticsearch.xpack.test.rest.XPackRestIT + method: test {p0=ml/evaluate_data_frame/Test outlier_detection with query} + issue: https://github.com/elastic/elasticsearch/issues/116484 +- class: org.elasticsearch.xpack.kql.query.KqlQueryBuilderTests + issue: https://github.com/elastic/elasticsearch/issues/116487 +- class: org.elasticsearch.reservedstate.service.FileSettingsServiceTests + method: testInvalidJSON + issue: https://github.com/elastic/elasticsearch/issues/116521 +- class: org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsCanMatchOnCoordinatorIntegTests + method: testSearchableSnapshotShardsAreSkippedBySearchRequestWithoutQueryingAnyNodeWhenTheyAreOutsideOfTheQueryRange + issue: https://github.com/elastic/elasticsearch/issues/116523 +- class: org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissionsTests + method: testCollapseAndRemoveUnsupportedPrivileges + issue: https://github.com/elastic/elasticsearch/issues/116520 +- class: org.elasticsearch.xpack.logsdb.qa.StandardVersusLogsIndexModeRandomDataDynamicMappingChallengeRestIT + method: testMatchAllQuery + issue: https://github.com/elastic/elasticsearch/issues/116536 +- class: org.elasticsearch.xpack.test.rest.XPackRestIT + method: test {p0=ml/inference_crud/Test force delete given model referenced by pipeline} + issue: https://github.com/elastic/elasticsearch/issues/116555 # Examples: # diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/30_multi_dense_vector.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/30_multi_dense_vector.yml new file mode 100644 index 000000000000..80d1d25dfcbd --- /dev/null +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/30_multi_dense_vector.yml @@ -0,0 +1,141 @@ +setup: + - requires: + capabilities: + - method: POST + path: /_search + capabilities: [ multi_dense_vector_field_mapper ] + test_runner_features: capabilities + reason: "Support for multi dense vector field mapper capability required" +--- +"Test create multi-vector field": + - do: + indices.create: + index: test + body: + mappings: + properties: + vector1: + type: multi_dense_vector + dims: 3 + - do: + index: + index: test + id: "1" + body: + vector1: [[2, -1, 1]] + - do: + index: + index: test + id: "2" + body: + vector1: [[2, -1, 1], [3, 4, 5]] + - do: + index: + index: test + id: "3" + body: + vector1: [[2, -1, 1], [3, 4, 5], [6, 7, 8]] + - do: + indices.refresh: {} +--- +"Test create dynamic dim multi-vector field": + - do: + indices.create: + index: test + body: + mappings: + properties: + name: + type: keyword + vector1: + type: multi_dense_vector + - do: + index: + index: test + id: "1" + body: + vector1: [[2, -1, 1]] + - do: + index: + index: test + id: "2" + body: + vector1: [[2, -1, 1], [3, 4, 5]] + - do: + index: + index: test + id: "3" + body: + vector1: [[2, -1, 1], [3, 4, 5], [6, 7, 8]] + - do: + cluster.health: + wait_for_events: languid + + # verify some other dimension will fail + - do: + catch: bad_request + index: + index: test + id: "4" + body: + vector1: [[2, -1, 1], [3, 4, 5], [6, 7, 8, 9]] +--- +"Test dynamic dim mismatch fails multi-vector field": + - do: + indices.create: + index: test + body: + mappings: + properties: + vector1: + type: multi_dense_vector + - do: + catch: bad_request + index: + index: test + id: "1" + body: + vector1: [[2, -1, 1], [2]] +--- +"Test static dim mismatch fails multi-vector field": + - do: + indices.create: + index: test + body: + mappings: + properties: + vector1: + type: multi_dense_vector + dims: 3 + - do: + catch: bad_request + index: + index: test + id: "1" + body: + vector1: [[2, -1, 1], [2]] +--- +"Test poorly formatted multi-vector field": + - do: + indices.create: + index: poorly_formatted_vector + body: + mappings: + properties: + vector1: + type: multi_dense_vector + dims: 3 + - do: + catch: bad_request + index: + index: poorly_formatted_vector + id: "1" + body: + vector1: [[[2, -1, 1]]] + - do: + catch: bad_request + index: + index: poorly_formatted_vector + id: "1" + body: + vector1: [[2, -1, 1], [[2, -1, 1]]] diff --git a/server/src/internalClusterTest/java/org/elasticsearch/action/support/AutoCreateIndexIT.java b/server/src/internalClusterTest/java/org/elasticsearch/action/support/AutoCreateIndexIT.java index bcaca0766e53..0aedfed037a6 100644 --- a/server/src/internalClusterTest/java/org/elasticsearch/action/support/AutoCreateIndexIT.java +++ b/server/src/internalClusterTest/java/org/elasticsearch/action/support/AutoCreateIndexIT.java @@ -10,15 +10,14 @@ package org.elasticsearch.action.support; import org.elasticsearch.action.ActionListener; -import org.elasticsearch.action.DocWriteResponse; import org.elasticsearch.cluster.service.ClusterService; import org.elasticsearch.common.Priority; import org.elasticsearch.test.ESIntegTestCase; +import org.elasticsearch.test.ESTestCase; import org.elasticsearch.xcontent.XContentType; import java.util.concurrent.CountDownLatch; import java.util.concurrent.CyclicBarrier; -import java.util.concurrent.TimeUnit; import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.hasItems; @@ -29,65 +28,39 @@ public class AutoCreateIndexIT extends ESIntegTestCase { final var masterNodeClusterService = internalCluster().getCurrentMasterNodeInstance(ClusterService.class); final var barrier = new CyclicBarrier(2); masterNodeClusterService.createTaskQueue("block", Priority.NORMAL, batchExecutionContext -> { - barrier.await(10, TimeUnit.SECONDS); - barrier.await(10, TimeUnit.SECONDS); + safeAwait(barrier); + safeAwait(barrier); batchExecutionContext.taskContexts().forEach(c -> c.success(() -> {})); return batchExecutionContext.initialState(); - }).submitTask("block", e -> { assert false : e; }, null); + }).submitTask("block", ESTestCase::fail, null); - barrier.await(10, TimeUnit.SECONDS); + safeAwait(barrier); final var countDownLatch = new CountDownLatch(2); final var client = client(); - client.prepareIndex("no-dot").setSource("{}", XContentType.JSON).execute(new ActionListener<>() { - @Override - public void onResponse(DocWriteResponse indexResponse) { - try { - final var warningHeaders = client.threadPool().getThreadContext().getResponseHeaders().get("Warning"); - if (warningHeaders != null) { - assertThat( - warningHeaders, - not( - hasItems( - containsString("index names starting with a dot are reserved for hidden indices and system indices") - ) - ) - ); - } - } finally { - countDownLatch.countDown(); - } - } - - @Override - public void onFailure(Exception e) { - countDownLatch.countDown(); - assert false : e; - } - }); - - client.prepareIndex(".has-dot").setSource("{}", XContentType.JSON).execute(new ActionListener<>() { - @Override - public void onResponse(DocWriteResponse indexResponse) { - try { - final var warningHeaders = client.threadPool().getThreadContext().getResponseHeaders().get("Warning"); - assertNotNull(warningHeaders); + client.prepareIndex("no-dot") + .setSource("{}", XContentType.JSON) + .execute(ActionListener.releaseAfter(ActionTestUtils.assertNoFailureListener(indexResponse -> { + final var warningHeaders = client.threadPool().getThreadContext().getResponseHeaders().get("Warning"); + if (warningHeaders != null) { assertThat( warningHeaders, - hasItems(containsString("index names starting with a dot are reserved for hidden indices and system indices")) + not(hasItems(containsString("index names starting with a dot are reserved for hidden indices and system indices"))) ); - } finally { - countDownLatch.countDown(); } - } + }), countDownLatch::countDown)); - @Override - public void onFailure(Exception e) { - countDownLatch.countDown(); - assert false : e; - } - }); + client.prepareIndex(".has-dot") + .setSource("{}", XContentType.JSON) + .execute(ActionListener.releaseAfter(ActionTestUtils.assertNoFailureListener(indexResponse -> { + final var warningHeaders = client.threadPool().getThreadContext().getResponseHeaders().get("Warning"); + assertNotNull(warningHeaders); + assertThat( + warningHeaders, + hasItems(containsString("index names starting with a dot are reserved for hidden indices and system indices")) + ); + }), countDownLatch::countDown)); assertBusy( () -> assertThat( @@ -100,7 +73,7 @@ public class AutoCreateIndexIT extends ESIntegTestCase { ) ); - barrier.await(10, TimeUnit.SECONDS); - assertTrue(countDownLatch.await(10, TimeUnit.SECONDS)); + safeAwait(barrier); + safeAwait(countDownLatch); } } diff --git a/server/src/internalClusterTest/java/org/elasticsearch/cluster/ClusterStateDiffIT.java b/server/src/internalClusterTest/java/org/elasticsearch/cluster/ClusterStateDiffIT.java index cab952994e6f..9ab6f3b6dddf 100644 --- a/server/src/internalClusterTest/java/org/elasticsearch/cluster/ClusterStateDiffIT.java +++ b/server/src/internalClusterTest/java/org/elasticsearch/cluster/ClusterStateDiffIT.java @@ -150,7 +150,7 @@ public class ClusterStateDiffIT extends ESIntegTestCase { for (Map.Entry node : clusterStateFromDiffs.nodes().getNodes().entrySet()) { DiscoveryNode node1 = clusterState.nodes().get(node.getKey()); DiscoveryNode node2 = clusterStateFromDiffs.nodes().get(node.getKey()); - assertThat(node1.getVersion(), equalTo(node2.getVersion())); + assertThat(node1.getBuildVersion(), equalTo(node2.getBuildVersion())); assertThat(node1.getAddress(), equalTo(node2.getAddress())); assertThat(node1.getAttributes(), equalTo(node2.getAttributes())); } diff --git a/server/src/main/java/org/elasticsearch/TransportVersions.java b/server/src/main/java/org/elasticsearch/TransportVersions.java index 8a1f026c13c7..94c436f01e19 100644 --- a/server/src/main/java/org/elasticsearch/TransportVersions.java +++ b/server/src/main/java/org/elasticsearch/TransportVersions.java @@ -189,6 +189,8 @@ public class TransportVersions { public static final TransportVersion ESQL_CCS_EXEC_INFO_WITH_FAILURES = def(8_783_00_0); public static final TransportVersion LOGSDB_TELEMETRY = def(8_784_00_0); public static final TransportVersion LOGSDB_TELEMETRY_STATS = def(8_785_00_0); + public static final TransportVersion KQL_QUERY_ADDED = def(8_786_00_0); + public static final TransportVersion ROLE_MONITOR_STATS = def(8_787_00_0); /* * WARNING: DO NOT MERGE INTO MAIN! diff --git a/server/src/main/java/org/elasticsearch/action/admin/indices/analyze/TransportAnalyzeAction.java b/server/src/main/java/org/elasticsearch/action/admin/indices/analyze/TransportAnalyzeAction.java index d48a723b2d67..3283400059d8 100644 --- a/server/src/main/java/org/elasticsearch/action/admin/indices/analyze/TransportAnalyzeAction.java +++ b/server/src/main/java/org/elasticsearch/action/admin/indices/analyze/TransportAnalyzeAction.java @@ -18,6 +18,7 @@ import org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute; import org.apache.lucene.analysis.tokenattributes.TypeAttribute; import org.apache.lucene.util.BytesRef; import org.elasticsearch.ElasticsearchException; +import org.elasticsearch.ElasticsearchStatusException; import org.elasticsearch.action.support.ActionFilters; import org.elasticsearch.action.support.single.shard.TransportSingleShardAction; import org.elasticsearch.cluster.ProjectState; @@ -45,6 +46,7 @@ import org.elasticsearch.index.mapper.StringFieldType; import org.elasticsearch.index.shard.ShardId; import org.elasticsearch.indices.IndicesService; import org.elasticsearch.injection.guice.Inject; +import org.elasticsearch.rest.RestStatus; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.TransportService; @@ -458,11 +460,12 @@ public class TransportAnalyzeAction extends TransportSingleShardAction maxTokenCount) { - throw new IllegalStateException( + throw new ElasticsearchStatusException( "The number of tokens produced by calling _analyze has exceeded the allowed maximum of [" + maxTokenCount + "]." - + " This limit can be set by changing the [index.analyze.max_token_count] index level setting." + + " This limit can be set by changing the [index.analyze.max_token_count] index level setting.", + RestStatus.BAD_REQUEST ); } } diff --git a/server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java b/server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java index 13c085c9875d..c051f0ca7a6f 100644 --- a/server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java +++ b/server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java @@ -68,7 +68,7 @@ import static org.elasticsearch.core.Strings.format; * The fan out and collect algorithm is traditionally used as the initial phase which can either be a query execution or collection of * distributed frequencies */ -abstract class AbstractSearchAsyncAction extends SearchPhase implements SearchPhaseContext { +abstract class AbstractSearchAsyncAction extends SearchPhase { private static final float DEFAULT_INDEX_BOOST = 1.0f; private final Logger logger; private final NamedWriteableRegistry namedWriteableRegistry; @@ -106,7 +106,8 @@ abstract class AbstractSearchAsyncAction exten private final boolean throttleConcurrentRequests; private final AtomicBoolean requestCancelled = new AtomicBoolean(); - private final List releasables = new ArrayList<>(); + // protected for tests + protected final List releasables = new ArrayList<>(); AbstractSearchAsyncAction( String name, @@ -194,7 +195,9 @@ abstract class AbstractSearchAsyncAction exten ); } - @Override + /** + * Registers a {@link Releasable} that will be closed when the search request finishes or fails. + */ public void addReleasable(Releasable releasable) { releasables.add(releasable); } @@ -333,8 +336,12 @@ abstract class AbstractSearchAsyncAction exten SearchActionListener listener ); - @Override - public final void executeNextPhase(SearchPhase currentPhase, Supplier nextPhaseSupplier) { + /** + * Processes the phase transition from on phase to another. This method handles all errors that happen during the initial run execution + * of the next phase. If there are no successful operations in the context when this method is executed the search is aborted and + * a response is returned to the user indicating that all shards have failed. + */ + protected void executeNextPhase(SearchPhase currentPhase, Supplier nextPhaseSupplier) { /* This is the main search phase transition where we move to the next phase. If all shards * failed or if there was a failure and partial results are not allowed, then we immediately * fail. Otherwise we continue to the next phase. @@ -470,8 +477,7 @@ abstract class AbstractSearchAsyncAction exten * @param shardTarget the shard target for this failure * @param e the failure reason */ - @Override - public final void onShardFailure(final int shardIndex, SearchShardTarget shardTarget, Exception e) { + void onShardFailure(final int shardIndex, SearchShardTarget shardTarget, Exception e) { if (TransportActions.isShardNotAvailableException(e)) { // Groups shard not available exceptions under a generic exception that returns a SERVICE_UNAVAILABLE(503) // temporary error. @@ -568,32 +574,45 @@ abstract class AbstractSearchAsyncAction exten } } - @Override + /** + * Returns the total number of shards to the current search across all indices + */ public final int getNumShards() { return results.getNumShards(); } - @Override + /** + * Returns a logger for this context to prevent each individual phase to create their own logger. + */ public final Logger getLogger() { return logger; } - @Override + /** + * Returns the currently executing search task + */ public final SearchTask getTask() { return task; } - @Override + /** + * Returns the currently executing search request + */ public final SearchRequest getRequest() { return request; } - @Override + /** + * Returns the targeted {@link OriginalIndices} for the provided {@code shardIndex}. + */ public OriginalIndices getOriginalIndices(int shardIndex) { return shardIterators[shardIndex].getOriginalIndices(); } - @Override + /** + * Checks if the given context id is part of the point in time of this search (if exists). + * We should not release search contexts that belong to the point in time during or after searches. + */ public boolean isPartOfPointInTime(ShardSearchContextId contextId) { final PointInTimeBuilder pointInTimeBuilder = request.pointInTimeBuilder(); if (pointInTimeBuilder != null) { @@ -630,7 +649,12 @@ abstract class AbstractSearchAsyncAction exten return false; } - @Override + /** + * Builds and sends the final search response back to the user. + * + * @param internalSearchResponse the internal search response + * @param queryResults the results of the query phase + */ public void sendSearchResponse(SearchResponseSections internalSearchResponse, AtomicArray queryResults) { ShardSearchFailure[] failures = buildShardFailures(); Boolean allowPartialResults = request.allowPartialSearchResults(); @@ -655,8 +679,14 @@ abstract class AbstractSearchAsyncAction exten } } - @Override - public final void onPhaseFailure(SearchPhase phase, String msg, Throwable cause) { + /** + * This method will communicate a fatal phase failure back to the user. In contrast to a shard failure + * will this method immediately fail the search request and return the failure to the issuer of the request + * @param phase the phase that failed + * @param msg an optional message + * @param cause the cause of the phase failure + */ + public void onPhaseFailure(SearchPhase phase, String msg, Throwable cause) { raisePhaseFailure(new SearchPhaseExecutionException(phase.getName(), msg, cause, buildShardFailures())); } @@ -683,6 +713,19 @@ abstract class AbstractSearchAsyncAction exten listener.onFailure(exception); } + /** + * Releases a search context with the given context ID on the node the given connection is connected to. + * @see org.elasticsearch.search.query.QuerySearchResult#getContextId() + * @see org.elasticsearch.search.fetch.FetchSearchResult#getContextId() + * + */ + void sendReleaseSearchContext(ShardSearchContextId contextId, Transport.Connection connection, OriginalIndices originalIndices) { + assert isPartOfPointInTime(contextId) == false : "Must not release point in time context [" + contextId + "]"; + if (connection != null) { + searchTransportService.sendFreeContext(connection, contextId, originalIndices); + } + } + /** * Executed once all shard results have been received and processed * @see #onShardFailure(int, SearchShardTarget, Exception) @@ -692,23 +735,29 @@ abstract class AbstractSearchAsyncAction exten executeNextPhase(this, this::getNextPhase); } - @Override + /** + * Returns a connection to the node if connected otherwise and {@link org.elasticsearch.transport.ConnectTransportException} will be + * thrown. + */ public final Transport.Connection getConnection(String clusterAlias, String nodeId) { return nodeIdToConnection.apply(clusterAlias, nodeId); } - @Override - public final SearchTransportService getSearchTransport() { + /** + * Returns the {@link SearchTransportService} to send shard request to other nodes + */ + public SearchTransportService getSearchTransport() { return searchTransportService; } - @Override public final void execute(Runnable command) { executor.execute(command); } - @Override - public final void onFailure(Exception e) { + /** + * Notifies the top-level listener of the provided exception + */ + public void onFailure(Exception e) { listener.onFailure(e); } diff --git a/server/src/main/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhase.java b/server/src/main/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhase.java index 8dcfbf5f070a..c4aea73cc614 100644 --- a/server/src/main/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhase.java @@ -131,7 +131,6 @@ final class CanMatchPreFilterSearchPhase extends SearchPhase { @Override public void run() { assert assertSearchCoordinationThread(); - checkNoMissingShards(); runCoordinatorRewritePhase(); } @@ -175,7 +174,10 @@ final class CanMatchPreFilterSearchPhase extends SearchPhase { if (matchedShardLevelRequests.isEmpty()) { finishPhase(); } else { - new Round(new GroupShardsIterator<>(matchedShardLevelRequests)).run(); + GroupShardsIterator matchingShards = new GroupShardsIterator<>(matchedShardLevelRequests); + // verify missing shards only for the shards that we hit for the query + checkNoMissingShards(matchingShards); + new Round(matchingShards).run(); } } @@ -185,9 +187,9 @@ final class CanMatchPreFilterSearchPhase extends SearchPhase { results.consumeResult(result, () -> {}); } - private void checkNoMissingShards() { + private void checkNoMissingShards(GroupShardsIterator shards) { assert assertSearchCoordinationThread(); - doCheckNoMissingShards(getName(), request, shardsIts); + doCheckNoMissingShards(getName(), request, shards); } private Map> groupByNode(GroupShardsIterator shards) { diff --git a/server/src/main/java/org/elasticsearch/action/search/CountedCollector.java b/server/src/main/java/org/elasticsearch/action/search/CountedCollector.java index 2603e3a5a51b..3d15e11a19d3 100644 --- a/server/src/main/java/org/elasticsearch/action/search/CountedCollector.java +++ b/server/src/main/java/org/elasticsearch/action/search/CountedCollector.java @@ -22,9 +22,9 @@ final class CountedCollector { private final SearchPhaseResults resultConsumer; private final CountDown counter; private final Runnable onFinish; - private final SearchPhaseContext context; + private final AbstractSearchAsyncAction context; - CountedCollector(SearchPhaseResults resultConsumer, int expectedOps, Runnable onFinish, SearchPhaseContext context) { + CountedCollector(SearchPhaseResults resultConsumer, int expectedOps, Runnable onFinish, AbstractSearchAsyncAction context) { this.resultConsumer = resultConsumer; this.counter = new CountDown(expectedOps); this.onFinish = onFinish; @@ -50,7 +50,7 @@ final class CountedCollector { } /** - * Escalates the failure via {@link SearchPhaseContext#onShardFailure(int, SearchShardTarget, Exception)} + * Escalates the failure via {@link AbstractSearchAsyncAction#onShardFailure(int, SearchShardTarget, Exception)} * and then runs {@link #countDown()} */ void onFailure(final int shardIndex, @Nullable SearchShardTarget shardTarget, Exception e) { diff --git a/server/src/main/java/org/elasticsearch/action/search/DfsQueryPhase.java b/server/src/main/java/org/elasticsearch/action/search/DfsQueryPhase.java index 36d73c0db166..285dd0a22fd7 100644 --- a/server/src/main/java/org/elasticsearch/action/search/DfsQueryPhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/DfsQueryPhase.java @@ -44,7 +44,7 @@ final class DfsQueryPhase extends SearchPhase { private final AggregatedDfs dfs; private final List knnResults; private final Function, SearchPhase> nextPhaseFactory; - private final SearchPhaseContext context; + private final AbstractSearchAsyncAction context; private final SearchTransportService searchTransportService; private final SearchProgressListener progressListener; @@ -54,7 +54,7 @@ final class DfsQueryPhase extends SearchPhase { List knnResults, SearchPhaseResults queryResult, Function, SearchPhase> nextPhaseFactory, - SearchPhaseContext context + AbstractSearchAsyncAction context ) { super("dfs_query"); this.progressListener = context.getTask().getProgressListener(); diff --git a/server/src/main/java/org/elasticsearch/action/search/ExpandSearchPhase.java b/server/src/main/java/org/elasticsearch/action/search/ExpandSearchPhase.java index 968d9dac958f..8feed2aea00b 100644 --- a/server/src/main/java/org/elasticsearch/action/search/ExpandSearchPhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/ExpandSearchPhase.java @@ -31,11 +31,11 @@ import java.util.function.Supplier; * forwards to the next phase immediately. */ final class ExpandSearchPhase extends SearchPhase { - private final SearchPhaseContext context; + private final AbstractSearchAsyncAction context; private final SearchHits searchHits; private final Supplier nextPhase; - ExpandSearchPhase(SearchPhaseContext context, SearchHits searchHits, Supplier nextPhase) { + ExpandSearchPhase(AbstractSearchAsyncAction context, SearchHits searchHits, Supplier nextPhase) { super("expand"); this.context = context; this.searchHits = searchHits; diff --git a/server/src/main/java/org/elasticsearch/action/search/FetchLookupFieldsPhase.java b/server/src/main/java/org/elasticsearch/action/search/FetchLookupFieldsPhase.java index e73ec5cb14e3..d8671bcadf86 100644 --- a/server/src/main/java/org/elasticsearch/action/search/FetchLookupFieldsPhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/FetchLookupFieldsPhase.java @@ -33,11 +33,15 @@ import java.util.stream.Collectors; * @see org.elasticsearch.index.mapper.LookupRuntimeFieldType */ final class FetchLookupFieldsPhase extends SearchPhase { - private final SearchPhaseContext context; + private final AbstractSearchAsyncAction context; private final SearchResponseSections searchResponse; private final AtomicArray queryResults; - FetchLookupFieldsPhase(SearchPhaseContext context, SearchResponseSections searchResponse, AtomicArray queryResults) { + FetchLookupFieldsPhase( + AbstractSearchAsyncAction context, + SearchResponseSections searchResponse, + AtomicArray queryResults + ) { super("fetch_lookup_fields"); this.context = context; this.searchResponse = searchResponse; diff --git a/server/src/main/java/org/elasticsearch/action/search/FetchSearchPhase.java b/server/src/main/java/org/elasticsearch/action/search/FetchSearchPhase.java index d7b847d835b8..0fbface3793a 100644 --- a/server/src/main/java/org/elasticsearch/action/search/FetchSearchPhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/FetchSearchPhase.java @@ -36,7 +36,7 @@ import java.util.function.BiFunction; final class FetchSearchPhase extends SearchPhase { private final AtomicArray searchPhaseShardResults; private final BiFunction, SearchPhase> nextPhaseFactory; - private final SearchPhaseContext context; + private final AbstractSearchAsyncAction context; private final Logger logger; private final SearchProgressListener progressListener; private final AggregatedDfs aggregatedDfs; @@ -47,7 +47,7 @@ final class FetchSearchPhase extends SearchPhase { FetchSearchPhase( SearchPhaseResults resultConsumer, AggregatedDfs aggregatedDfs, - SearchPhaseContext context, + AbstractSearchAsyncAction context, @Nullable SearchPhaseController.ReducedQueryPhase reducedQueryPhase ) { this( @@ -66,7 +66,7 @@ final class FetchSearchPhase extends SearchPhase { FetchSearchPhase( SearchPhaseResults resultConsumer, AggregatedDfs aggregatedDfs, - SearchPhaseContext context, + AbstractSearchAsyncAction context, @Nullable SearchPhaseController.ReducedQueryPhase reducedQueryPhase, BiFunction, SearchPhase> nextPhaseFactory ) { diff --git a/server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java b/server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java index 05213eb94b75..199228c9f992 100644 --- a/server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java @@ -38,7 +38,7 @@ import java.util.List; public class RankFeaturePhase extends SearchPhase { private static final Logger logger = LogManager.getLogger(RankFeaturePhase.class); - private final SearchPhaseContext context; + private final AbstractSearchAsyncAction context; final SearchPhaseResults queryPhaseResults; final SearchPhaseResults rankPhaseResults; private final AggregatedDfs aggregatedDfs; @@ -48,7 +48,7 @@ public class RankFeaturePhase extends SearchPhase { RankFeaturePhase( SearchPhaseResults queryPhaseResults, AggregatedDfs aggregatedDfs, - SearchPhaseContext context, + AbstractSearchAsyncAction context, RankFeaturePhaseRankCoordinatorContext rankFeaturePhaseRankCoordinatorContext ) { super("rank-feature"); @@ -179,22 +179,25 @@ public class RankFeaturePhase extends SearchPhase { RankFeaturePhaseRankCoordinatorContext rankFeaturePhaseRankCoordinatorContext, SearchPhaseController.ReducedQueryPhase reducedQueryPhase ) { - ThreadedActionListener rankResultListener = new ThreadedActionListener<>(context, new ActionListener<>() { - @Override - public void onResponse(RankFeatureDoc[] docsWithUpdatedScores) { - RankFeatureDoc[] topResults = rankFeaturePhaseRankCoordinatorContext.rankAndPaginate(docsWithUpdatedScores); - SearchPhaseController.ReducedQueryPhase reducedRankFeaturePhase = newReducedQueryPhaseResults( - reducedQueryPhase, - topResults - ); - moveToNextPhase(rankPhaseResults, reducedRankFeaturePhase); - } + ThreadedActionListener rankResultListener = new ThreadedActionListener<>( + context::execute, + new ActionListener<>() { + @Override + public void onResponse(RankFeatureDoc[] docsWithUpdatedScores) { + RankFeatureDoc[] topResults = rankFeaturePhaseRankCoordinatorContext.rankAndPaginate(docsWithUpdatedScores); + SearchPhaseController.ReducedQueryPhase reducedRankFeaturePhase = newReducedQueryPhaseResults( + reducedQueryPhase, + topResults + ); + moveToNextPhase(rankPhaseResults, reducedRankFeaturePhase); + } - @Override - public void onFailure(Exception e) { - context.onPhaseFailure(RankFeaturePhase.this, "Computing updated ranks for results failed", e); + @Override + public void onFailure(Exception e) { + context.onPhaseFailure(RankFeaturePhase.this, "Computing updated ranks for results failed", e); + } } - }); + ); rankFeaturePhaseRankCoordinatorContext.computeRankScoresForGlobalResults( rankPhaseResults.getAtomicArray().asList().stream().map(SearchPhaseResult::rankFeatureResult).toList(), rankResultListener diff --git a/server/src/main/java/org/elasticsearch/action/search/SearchPhase.java b/server/src/main/java/org/elasticsearch/action/search/SearchPhase.java index e312b7399e70..e4fef357cb4e 100644 --- a/server/src/main/java/org/elasticsearch/action/search/SearchPhase.java +++ b/server/src/main/java/org/elasticsearch/action/search/SearchPhase.java @@ -74,7 +74,7 @@ abstract class SearchPhase implements CheckedRunnable { /** * Releases shard targets that are not used in the docsIdsToLoad. */ - protected void releaseIrrelevantSearchContext(SearchPhaseResult searchPhaseResult, SearchPhaseContext context) { + protected void releaseIrrelevantSearchContext(SearchPhaseResult searchPhaseResult, AbstractSearchAsyncAction context) { // we only release search context that we did not fetch from, if we are not scrolling // or using a PIT and if it has at least one hit that didn't make it to the global topDocs if (searchPhaseResult == null) { diff --git a/server/src/main/java/org/elasticsearch/action/search/SearchPhaseContext.java b/server/src/main/java/org/elasticsearch/action/search/SearchPhaseContext.java deleted file mode 100644 index d048887b69c9..000000000000 --- a/server/src/main/java/org/elasticsearch/action/search/SearchPhaseContext.java +++ /dev/null @@ -1,130 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the "Elastic License - * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side - * Public License v 1"; you may not use this file except in compliance with, at - * your election, the "Elastic License 2.0", the "GNU Affero General Public - * License v3.0 only", or the "Server Side Public License, v 1". - */ -package org.elasticsearch.action.search; - -import org.apache.logging.log4j.Logger; -import org.elasticsearch.action.OriginalIndices; -import org.elasticsearch.common.util.concurrent.AtomicArray; -import org.elasticsearch.core.Nullable; -import org.elasticsearch.core.Releasable; -import org.elasticsearch.search.SearchPhaseResult; -import org.elasticsearch.search.SearchShardTarget; -import org.elasticsearch.search.internal.ShardSearchContextId; -import org.elasticsearch.transport.Transport; - -import java.util.concurrent.Executor; -import java.util.function.Supplier; - -/** - * This class provide contextual state and access to resources across multiple search phases. - */ -interface SearchPhaseContext extends Executor { - // TODO maybe we can make this concrete later - for now we just implement this in the base class for all initial phases - - /** - * Returns the total number of shards to the current search across all indices - */ - int getNumShards(); - - /** - * Returns a logger for this context to prevent each individual phase to create their own logger. - */ - Logger getLogger(); - - /** - * Returns the currently executing search task - */ - SearchTask getTask(); - - /** - * Returns the currently executing search request - */ - SearchRequest getRequest(); - - /** - * Returns the targeted {@link OriginalIndices} for the provided {@code shardIndex}. - */ - OriginalIndices getOriginalIndices(int shardIndex); - - /** - * Checks if the given context id is part of the point in time of this search (if exists). - * We should not release search contexts that belong to the point in time during or after searches. - */ - boolean isPartOfPointInTime(ShardSearchContextId contextId); - - /** - * Builds and sends the final search response back to the user. - * - * @param internalSearchResponse the internal search response - * @param queryResults the results of the query phase - */ - void sendSearchResponse(SearchResponseSections internalSearchResponse, AtomicArray queryResults); - - /** - * Notifies the top-level listener of the provided exception - */ - void onFailure(Exception e); - - /** - * This method will communicate a fatal phase failure back to the user. In contrast to a shard failure - * will this method immediately fail the search request and return the failure to the issuer of the request - * @param phase the phase that failed - * @param msg an optional message - * @param cause the cause of the phase failure - */ - void onPhaseFailure(SearchPhase phase, String msg, Throwable cause); - - /** - * This method will record a shard failure for the given shard index. In contrast to a phase failure - * ({@link #onPhaseFailure(SearchPhase, String, Throwable)}) this method will immediately return to the user but will record - * a shard failure for the given shard index. This should be called if a shard failure happens after we successfully retrieved - * a result from that shard in a previous phase. - */ - void onShardFailure(int shardIndex, @Nullable SearchShardTarget shardTarget, Exception e); - - /** - * Returns a connection to the node if connected otherwise and {@link org.elasticsearch.transport.ConnectTransportException} will be - * thrown. - */ - Transport.Connection getConnection(String clusterAlias, String nodeId); - - /** - * Returns the {@link SearchTransportService} to send shard request to other nodes - */ - SearchTransportService getSearchTransport(); - - /** - * Releases a search context with the given context ID on the node the given connection is connected to. - * @see org.elasticsearch.search.query.QuerySearchResult#getContextId() - * @see org.elasticsearch.search.fetch.FetchSearchResult#getContextId() - * - */ - default void sendReleaseSearchContext( - ShardSearchContextId contextId, - Transport.Connection connection, - OriginalIndices originalIndices - ) { - assert isPartOfPointInTime(contextId) == false : "Must not release point in time context [" + contextId + "]"; - if (connection != null) { - getSearchTransport().sendFreeContext(connection, contextId, originalIndices); - } - } - - /** - * Processes the phase transition from on phase to another. This method handles all errors that happen during the initial run execution - * of the next phase. If there are no successful operations in the context when this method is executed the search is aborted and - * a response is returned to the user indicating that all shards have failed. - */ - void executeNextPhase(SearchPhase currentPhase, Supplier nextPhaseSupplier); - - /** - * Registers a {@link Releasable} that will be closed when the search request finishes or fails. - */ - void addReleasable(Releasable releasable); -} diff --git a/server/src/main/java/org/elasticsearch/action/search/SearchQueryThenFetchAsyncAction.java b/server/src/main/java/org/elasticsearch/action/search/SearchQueryThenFetchAsyncAction.java index e92b5bbf4b5e..84e0e2adea61 100644 --- a/server/src/main/java/org/elasticsearch/action/search/SearchQueryThenFetchAsyncAction.java +++ b/server/src/main/java/org/elasticsearch/action/search/SearchQueryThenFetchAsyncAction.java @@ -135,7 +135,7 @@ class SearchQueryThenFetchAsyncAction extends AbstractSearchAsyncAction context, SearchPhaseResults queryResults, AggregatedDfs aggregatedDfs ) { diff --git a/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNode.java b/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNode.java index d3410b9139b4..7bf367f99b92 100644 --- a/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNode.java +++ b/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNode.java @@ -21,8 +21,8 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.transport.TransportAddress; import org.elasticsearch.common.util.StringLiteralDeduplicator; import org.elasticsearch.core.Nullable; +import org.elasticsearch.env.BuildVersion; import org.elasticsearch.index.IndexVersion; -import org.elasticsearch.index.IndexVersions; import org.elasticsearch.node.Node; import org.elasticsearch.xcontent.ToXContentFragment; import org.elasticsearch.xcontent.XContentBuilder; @@ -33,7 +33,6 @@ import java.util.Comparator; import java.util.Map; import java.util.Objects; import java.util.Optional; -import java.util.OptionalInt; import java.util.Set; import java.util.SortedSet; import java.util.TreeSet; @@ -290,18 +289,6 @@ public class DiscoveryNode implements Writeable, ToXContentFragment { return Set.copyOf(NODE_ROLES_SETTING.get(settings)); } - private static VersionInformation inferVersionInformation(Version version) { - if (version.before(Version.V_8_10_0)) { - return new VersionInformation( - version, - IndexVersion.getMinimumCompatibleIndexVersion(version.id), - IndexVersion.fromId(version.id) - ); - } else { - return new VersionInformation(version, IndexVersions.MINIMUM_COMPATIBLE, IndexVersion.current()); - } - } - private static final Writeable.Reader readStringLiteral = s -> nodeStringDeduplicator.deduplicate(s.readString()); /** @@ -338,11 +325,7 @@ public class DiscoveryNode implements Writeable, ToXContentFragment { } } this.roles = Collections.unmodifiableSortedSet(roles); - if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_10_X)) { - versionInfo = new VersionInformation(Version.readVersion(in), IndexVersion.readVersion(in), IndexVersion.readVersion(in)); - } else { - versionInfo = inferVersionInformation(Version.readVersion(in)); - } + versionInfo = new VersionInformation(Version.readVersion(in), IndexVersion.readVersion(in), IndexVersion.readVersion(in)); if (in.getTransportVersion().onOrAfter(EXTERNAL_ID_VERSION)) { this.externalId = readStringLiteral.read(in); } else { @@ -375,13 +358,9 @@ public class DiscoveryNode implements Writeable, ToXContentFragment { o.writeString(role.roleNameAbbreviation()); o.writeBoolean(role.canContainData()); }); - if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_10_X)) { - Version.writeVersion(versionInfo.nodeVersion(), out); - IndexVersion.writeVersion(versionInfo.minIndexVersion(), out); - IndexVersion.writeVersion(versionInfo.maxIndexVersion(), out); - } else { - Version.writeVersion(versionInfo.nodeVersion(), out); - } + Version.writeVersion(versionInfo.nodeVersion(), out); + IndexVersion.writeVersion(versionInfo.minIndexVersion(), out); + IndexVersion.writeVersion(versionInfo.maxIndexVersion(), out); if (out.getTransportVersion().onOrAfter(EXTERNAL_ID_VERSION)) { out.writeString(externalId); } @@ -486,18 +465,13 @@ public class DiscoveryNode implements Writeable, ToXContentFragment { return this.versionInfo; } - public Version getVersion() { - return this.versionInfo.nodeVersion(); + public BuildVersion getBuildVersion() { + return versionInfo.buildVersion(); } - public OptionalInt getPre811VersionId() { - // Even if Version is removed from this class completely it will need to read the version ID - // off the wire for old node versions, so the value of this variable can be obtained from that - int versionId = versionInfo.nodeVersion().id; - if (versionId >= Version.V_8_11_0.id) { - return OptionalInt.empty(); - } - return OptionalInt.of(versionId); + @Deprecated + public Version getVersion() { + return this.versionInfo.nodeVersion(); } public IndexVersion getMinIndexVersion() { @@ -564,7 +538,7 @@ public class DiscoveryNode implements Writeable, ToXContentFragment { appendRoleAbbreviations(stringBuilder, ""); stringBuilder.append('}'); } - stringBuilder.append('{').append(versionInfo.nodeVersion()).append('}'); + stringBuilder.append('{').append(versionInfo.buildVersion()).append('}'); stringBuilder.append('{').append(versionInfo.minIndexVersion()).append('-').append(versionInfo.maxIndexVersion()).append('}'); } @@ -601,7 +575,7 @@ public class DiscoveryNode implements Writeable, ToXContentFragment { builder.value(role.roleName()); } builder.endArray(); - builder.field("version", versionInfo.nodeVersion()); + builder.field("version", versionInfo.buildVersion().toString()); builder.field("min_index_version", versionInfo.minIndexVersion()); builder.field("max_index_version", versionInfo.maxIndexVersion()); builder.endObject(); diff --git a/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodes.java b/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodes.java index 9477f9c6a5cc..12c698a6ed95 100644 --- a/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodes.java +++ b/server/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodes.java @@ -339,6 +339,13 @@ public class DiscoveryNodes implements Iterable, SimpleDiffable "Error processing watched file: " + watchedFile(), e); + onProcessFileChangesException(e); + return; + } + for (var listener : eventListeners) { + listener.watchedFileChanged(); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java index 253f70f4fda4..cf75f1ddf3b9 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java @@ -968,15 +968,27 @@ public final class TextFieldMapper extends FieldMapper { return fielddata; } - public boolean canUseSyntheticSourceDelegateForQuerying() { + /** + * Returns true if the delegate sub-field can be used for loading and querying (ie. either isIndexed or isStored is true) + */ + public boolean canUseSyntheticSourceDelegateForLoading() { return syntheticSourceDelegate != null && syntheticSourceDelegate.ignoreAbove() == Integer.MAX_VALUE && (syntheticSourceDelegate.isIndexed() || syntheticSourceDelegate.isStored()); } + /** + * Returns true if the delegate sub-field can be used for querying only (ie. isIndexed must be true) + */ + public boolean canUseSyntheticSourceDelegateForQuerying() { + return syntheticSourceDelegate != null + && syntheticSourceDelegate.ignoreAbove() == Integer.MAX_VALUE + && syntheticSourceDelegate.isIndexed(); + } + @Override public BlockLoader blockLoader(BlockLoaderContext blContext) { - if (canUseSyntheticSourceDelegateForQuerying()) { + if (canUseSyntheticSourceDelegateForLoading()) { return new BlockLoader.Delegating(syntheticSourceDelegate.blockLoader(blContext)) { @Override protected String delegatingTo() { diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java index 1c61dcec906a..dea9368a9377 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java @@ -416,13 +416,18 @@ public class DenseVectorFieldMapper extends FieldMapper { return VectorUtil.dotProduct(vectorData.asByteVector(), vectorData.asByteVector()); } - private VectorData parseVectorArray(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + private VectorData parseVectorArray( + DocumentParserContext context, + int dims, + IntBooleanConsumer dimChecker, + VectorSimilarity similarity + ) throws IOException { int index = 0; - byte[] vector = new byte[fieldMapper.fieldType().dims]; + byte[] vector = new byte[dims]; float squaredMagnitude = 0; for (XContentParser.Token token = context.parser().nextToken(); token != Token.END_ARRAY; token = context.parser() .nextToken()) { - fieldMapper.checkDimensionExceeded(index, context); + dimChecker.accept(index, false); ensureExpectedToken(Token.VALUE_NUMBER, token, context.parser()); final int value; if (context.parser().numberType() != XContentParser.NumberType.INT) { @@ -460,30 +465,31 @@ public class DenseVectorFieldMapper extends FieldMapper { vector[index++] = (byte) value; squaredMagnitude += value * value; } - fieldMapper.checkDimensionMatches(index, context); - checkVectorMagnitude(fieldMapper.fieldType().similarity, errorByteElementsAppender(vector), squaredMagnitude); + dimChecker.accept(index, true); + checkVectorMagnitude(similarity, errorByteElementsAppender(vector), squaredMagnitude); return VectorData.fromBytes(vector); } - private VectorData parseHexEncodedVector(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + private VectorData parseHexEncodedVector( + DocumentParserContext context, + IntBooleanConsumer dimChecker, + VectorSimilarity similarity + ) throws IOException { byte[] decodedVector = HexFormat.of().parseHex(context.parser().text()); - fieldMapper.checkDimensionMatches(decodedVector.length, context); + dimChecker.accept(decodedVector.length, true); VectorData vectorData = VectorData.fromBytes(decodedVector); double squaredMagnitude = computeSquaredMagnitude(vectorData); - checkVectorMagnitude( - fieldMapper.fieldType().similarity, - errorByteElementsAppender(decodedVector), - (float) squaredMagnitude - ); + checkVectorMagnitude(similarity, errorByteElementsAppender(decodedVector), (float) squaredMagnitude); return vectorData; } @Override - VectorData parseKnnVector(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + VectorData parseKnnVector(DocumentParserContext context, int dims, IntBooleanConsumer dimChecker, VectorSimilarity similarity) + throws IOException { XContentParser.Token token = context.parser().currentToken(); return switch (token) { - case START_ARRAY -> parseVectorArray(context, fieldMapper); - case VALUE_STRING -> parseHexEncodedVector(context, fieldMapper); + case START_ARRAY -> parseVectorArray(context, dims, dimChecker, similarity); + case VALUE_STRING -> parseHexEncodedVector(context, dimChecker, similarity); default -> throw new ParsingException( context.parser().getTokenLocation(), format("Unsupported type [%s] for provided value [%s]", token, context.parser().text()) @@ -493,7 +499,13 @@ public class DenseVectorFieldMapper extends FieldMapper { @Override public void parseKnnVectorAndIndex(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { - VectorData vectorData = parseKnnVector(context, fieldMapper); + VectorData vectorData = parseKnnVector(context, fieldMapper.fieldType().dims, (i, end) -> { + if (end) { + fieldMapper.checkDimensionMatches(i, context); + } else { + fieldMapper.checkDimensionExceeded(i, context); + } + }, fieldMapper.fieldType().similarity); Field field = createKnnVectorField( fieldMapper.fieldType().name(), vectorData.asByteVector(), @@ -677,21 +689,22 @@ public class DenseVectorFieldMapper extends FieldMapper { } @Override - VectorData parseKnnVector(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + VectorData parseKnnVector(DocumentParserContext context, int dims, IntBooleanConsumer dimChecker, VectorSimilarity similarity) + throws IOException { int index = 0; float squaredMagnitude = 0; - float[] vector = new float[fieldMapper.fieldType().dims]; + float[] vector = new float[dims]; for (Token token = context.parser().nextToken(); token != Token.END_ARRAY; token = context.parser().nextToken()) { - fieldMapper.checkDimensionExceeded(index, context); + dimChecker.accept(index, false); ensureExpectedToken(Token.VALUE_NUMBER, token, context.parser()); float value = context.parser().floatValue(true); vector[index] = value; squaredMagnitude += value * value; index++; } - fieldMapper.checkDimensionMatches(index, context); + dimChecker.accept(index, true); checkVectorBounds(vector); - checkVectorMagnitude(fieldMapper.fieldType().similarity, errorFloatElementsAppender(vector), squaredMagnitude); + checkVectorMagnitude(similarity, errorFloatElementsAppender(vector), squaredMagnitude); return VectorData.fromFloats(vector); } @@ -816,12 +829,17 @@ public class DenseVectorFieldMapper extends FieldMapper { return count; } - private VectorData parseVectorArray(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + private VectorData parseVectorArray( + DocumentParserContext context, + int dims, + IntBooleanConsumer dimChecker, + VectorSimilarity similarity + ) throws IOException { int index = 0; - byte[] vector = new byte[fieldMapper.fieldType().dims / Byte.SIZE]; + byte[] vector = new byte[dims / Byte.SIZE]; for (XContentParser.Token token = context.parser().nextToken(); token != Token.END_ARRAY; token = context.parser() .nextToken()) { - fieldMapper.checkDimensionExceeded(index, context); + dimChecker.accept(index * Byte.SIZE, false); ensureExpectedToken(Token.VALUE_NUMBER, token, context.parser()); final int value; if (context.parser().numberType() != XContentParser.NumberType.INT) { @@ -856,35 +874,25 @@ public class DenseVectorFieldMapper extends FieldMapper { + "];" ); } - if (index >= vector.length) { - throw new IllegalArgumentException( - "The number of dimensions for field [" - + fieldMapper.fieldType().name() - + "] should be [" - + fieldMapper.fieldType().dims - + "] but found [" - + (index + 1) * Byte.SIZE - + "]" - ); - } vector[index++] = (byte) value; } - fieldMapper.checkDimensionMatches(index * Byte.SIZE, context); + dimChecker.accept(index * Byte.SIZE, true); return VectorData.fromBytes(vector); } - private VectorData parseHexEncodedVector(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + private VectorData parseHexEncodedVector(DocumentParserContext context, IntBooleanConsumer dimChecker) throws IOException { byte[] decodedVector = HexFormat.of().parseHex(context.parser().text()); - fieldMapper.checkDimensionMatches(decodedVector.length * Byte.SIZE, context); + dimChecker.accept(decodedVector.length * Byte.SIZE, true); return VectorData.fromBytes(decodedVector); } @Override - VectorData parseKnnVector(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { + VectorData parseKnnVector(DocumentParserContext context, int dims, IntBooleanConsumer dimChecker, VectorSimilarity similarity) + throws IOException { XContentParser.Token token = context.parser().currentToken(); return switch (token) { - case START_ARRAY -> parseVectorArray(context, fieldMapper); - case VALUE_STRING -> parseHexEncodedVector(context, fieldMapper); + case START_ARRAY -> parseVectorArray(context, dims, dimChecker, similarity); + case VALUE_STRING -> parseHexEncodedVector(context, dimChecker); default -> throw new ParsingException( context.parser().getTokenLocation(), format("Unsupported type [%s] for provided value [%s]", token, context.parser().text()) @@ -894,7 +902,13 @@ public class DenseVectorFieldMapper extends FieldMapper { @Override public void parseKnnVectorAndIndex(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException { - VectorData vectorData = parseKnnVector(context, fieldMapper); + VectorData vectorData = parseKnnVector(context, fieldMapper.fieldType().dims, (i, end) -> { + if (end) { + fieldMapper.checkDimensionMatches(i, context); + } else { + fieldMapper.checkDimensionExceeded(i, context); + } + }, fieldMapper.fieldType().similarity); Field field = createKnnVectorField( fieldMapper.fieldType().name(), vectorData.asByteVector(), @@ -958,7 +972,12 @@ public class DenseVectorFieldMapper extends FieldMapper { abstract void parseKnnVectorAndIndex(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException; - abstract VectorData parseKnnVector(DocumentParserContext context, DenseVectorFieldMapper fieldMapper) throws IOException; + abstract VectorData parseKnnVector( + DocumentParserContext context, + int dims, + IntBooleanConsumer dimChecker, + VectorSimilarity similarity + ) throws IOException; abstract int getNumBytes(int dimensions); @@ -2180,7 +2199,13 @@ public class DenseVectorFieldMapper extends FieldMapper { : elementType.getNumBytes(dims); ByteBuffer byteBuffer = elementType.createByteBuffer(indexCreatedVersion, numBytes); - VectorData vectorData = elementType.parseKnnVector(context, this); + VectorData vectorData = elementType.parseKnnVector(context, dims, (i, b) -> { + if (b) { + checkDimensionMatches(i, context); + } else { + checkDimensionExceeded(i, context); + } + }, fieldType().similarity); vectorData.addToBuffer(byteBuffer); if (indexCreatedVersion.onOrAfter(MAGNITUDE_STORED_INDEX_VERSION)) { // encode vector magnitude at the end @@ -2433,4 +2458,11 @@ public class DenseVectorFieldMapper extends FieldMapper { return fullPath(); } } + + /** + * @FunctionalInterface for a function that takes a int and boolean + */ + interface IntBooleanConsumer { + void accept(int value, boolean isComplete); + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldMapper.java new file mode 100644 index 000000000000..b23a1f1f6679 --- /dev/null +++ b/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldMapper.java @@ -0,0 +1,431 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the "Elastic License + * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side + * Public License v 1"; you may not use this file except in compliance with, at + * your election, the "Elastic License 2.0", the "GNU Affero General Public + * License v3.0 only", or the "Server Side Public License, v 1". + */ + +package org.elasticsearch.index.mapper.vectors; + +import org.apache.lucene.document.BinaryDocValuesField; +import org.apache.lucene.index.BinaryDocValues; +import org.apache.lucene.index.LeafReader; +import org.apache.lucene.search.FieldExistsQuery; +import org.apache.lucene.search.Query; +import org.apache.lucene.util.BytesRef; +import org.elasticsearch.common.util.FeatureFlag; +import org.elasticsearch.common.xcontent.support.XContentMapValues; +import org.elasticsearch.index.IndexVersion; +import org.elasticsearch.index.fielddata.FieldDataContext; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.mapper.ArraySourceValueFetcher; +import org.elasticsearch.index.mapper.DocumentParserContext; +import org.elasticsearch.index.mapper.FieldMapper; +import org.elasticsearch.index.mapper.MappedFieldType; +import org.elasticsearch.index.mapper.Mapper; +import org.elasticsearch.index.mapper.MapperBuilderContext; +import org.elasticsearch.index.mapper.MapperParsingException; +import org.elasticsearch.index.mapper.SimpleMappedFieldType; +import org.elasticsearch.index.mapper.SourceLoader; +import org.elasticsearch.index.mapper.TextSearchInfo; +import org.elasticsearch.index.mapper.ValueFetcher; +import org.elasticsearch.index.query.SearchExecutionContext; +import org.elasticsearch.search.DocValueFormat; +import org.elasticsearch.search.aggregations.support.CoreValuesSourceType; +import org.elasticsearch.search.vectors.VectorData; +import org.elasticsearch.xcontent.XContentBuilder; +import org.elasticsearch.xcontent.XContentParser; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.time.ZoneId; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +import static org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.MAX_DIMS_COUNT; +import static org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.MAX_DIMS_COUNT_BIT; +import static org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.namesToElementType; + +public class MultiDenseVectorFieldMapper extends FieldMapper { + + public static final String VECTOR_MAGNITUDES_SUFFIX = "._magnitude"; + public static final FeatureFlag FEATURE_FLAG = new FeatureFlag("multi_dense_vector"); + public static final String CONTENT_TYPE = "multi_dense_vector"; + + private static MultiDenseVectorFieldMapper toType(FieldMapper in) { + return (MultiDenseVectorFieldMapper) in; + } + + public static class Builder extends FieldMapper.Builder { + + private final Parameter elementType = new Parameter<>( + "element_type", + false, + () -> DenseVectorFieldMapper.ElementType.FLOAT, + (n, c, o) -> { + DenseVectorFieldMapper.ElementType elementType = namesToElementType.get((String) o); + if (elementType == null) { + throw new MapperParsingException( + "invalid element_type [" + o + "]; available types are " + namesToElementType.keySet() + ); + } + return elementType; + }, + m -> toType(m).fieldType().elementType, + XContentBuilder::field, + Objects::toString + ); + + // This is defined as updatable because it can be updated once, from [null] to a valid dim size, + // by a dynamic mapping update. Once it has been set, however, the value cannot be changed. + private final Parameter dims = new Parameter<>("dims", true, () -> null, (n, c, o) -> { + if (o instanceof Integer == false) { + throw new MapperParsingException("Property [dims] on field [" + n + "] must be an integer but got [" + o + "]"); + } + + return XContentMapValues.nodeIntegerValue(o); + }, m -> toType(m).fieldType().dims, XContentBuilder::field, Object::toString).setSerializerCheck((id, ic, v) -> v != null) + .setMergeValidator((previous, current, c) -> previous == null || Objects.equals(previous, current)) + .addValidator(dims -> { + if (dims == null) { + return; + } + int maxDims = elementType.getValue() == DenseVectorFieldMapper.ElementType.BIT ? MAX_DIMS_COUNT_BIT : MAX_DIMS_COUNT; + int minDims = elementType.getValue() == DenseVectorFieldMapper.ElementType.BIT ? Byte.SIZE : 1; + if (dims < minDims || dims > maxDims) { + throw new MapperParsingException( + "The number of dimensions should be in the range [" + minDims + ", " + maxDims + "] but was [" + dims + "]" + ); + } + if (elementType.getValue() == DenseVectorFieldMapper.ElementType.BIT) { + if (dims % Byte.SIZE != 0) { + throw new MapperParsingException("The number of dimensions for should be a multiple of 8 but was [" + dims + "]"); + } + } + }); + private final Parameter> meta = Parameter.metaParam(); + + private final IndexVersion indexCreatedVersion; + + public Builder(String name, IndexVersion indexCreatedVersion) { + super(name); + this.indexCreatedVersion = indexCreatedVersion; + } + + @Override + protected Parameter[] getParameters() { + return new Parameter[] { elementType, dims, meta }; + } + + public MultiDenseVectorFieldMapper.Builder dimensions(int dimensions) { + this.dims.setValue(dimensions); + return this; + } + + public MultiDenseVectorFieldMapper.Builder elementType(DenseVectorFieldMapper.ElementType elementType) { + this.elementType.setValue(elementType); + return this; + } + + @Override + public MultiDenseVectorFieldMapper build(MapperBuilderContext context) { + // Validate again here because the dimensions or element type could have been set programmatically, + // which affects index option validity + validate(); + return new MultiDenseVectorFieldMapper( + leafName(), + new MultiDenseVectorFieldType( + context.buildFullName(leafName()), + elementType.getValue(), + dims.getValue(), + indexCreatedVersion, + meta.getValue() + ), + builderParams(this, context), + indexCreatedVersion + ); + } + } + + public static final TypeParser PARSER = new TypeParser( + (n, c) -> new MultiDenseVectorFieldMapper.Builder(n, c.indexVersionCreated()), + notInMultiFields(CONTENT_TYPE) + ); + + public static final class MultiDenseVectorFieldType extends SimpleMappedFieldType { + private final DenseVectorFieldMapper.ElementType elementType; + private final Integer dims; + private final IndexVersion indexCreatedVersion; + + public MultiDenseVectorFieldType( + String name, + DenseVectorFieldMapper.ElementType elementType, + Integer dims, + IndexVersion indexCreatedVersion, + Map meta + ) { + super(name, false, false, true, TextSearchInfo.NONE, meta); + this.elementType = elementType; + this.dims = dims; + this.indexCreatedVersion = indexCreatedVersion; + } + + @Override + public String typeName() { + return CONTENT_TYPE; + } + + @Override + public ValueFetcher valueFetcher(SearchExecutionContext context, String format) { + if (format != null) { + throw new IllegalArgumentException("Field [" + name() + "] of type [" + typeName() + "] doesn't support formats."); + } + return new ArraySourceValueFetcher(name(), context) { + @Override + protected Object parseSourceValue(Object value) { + return value; + } + }; + } + + @Override + public DocValueFormat docValueFormat(String format, ZoneId timeZone) { + throw new IllegalArgumentException( + "Field [" + name() + "] of type [" + typeName() + "] doesn't support docvalue_fields or aggregations" + ); + } + + @Override + public boolean isAggregatable() { + return false; + } + + @Override + public IndexFieldData.Builder fielddataBuilder(FieldDataContext fieldDataContext) { + return new MultiVectorIndexFieldData.Builder(name(), CoreValuesSourceType.KEYWORD, indexCreatedVersion, dims, elementType); + } + + @Override + public Query existsQuery(SearchExecutionContext context) { + return new FieldExistsQuery(name()); + } + + @Override + public Query termQuery(Object value, SearchExecutionContext context) { + throw new IllegalArgumentException("Field [" + name() + "] of type [" + typeName() + "] doesn't support term queries"); + } + + int getVectorDimensions() { + return dims; + } + + DenseVectorFieldMapper.ElementType getElementType() { + return elementType; + } + } + + private final IndexVersion indexCreatedVersion; + + private MultiDenseVectorFieldMapper( + String simpleName, + MappedFieldType fieldType, + BuilderParams params, + IndexVersion indexCreatedVersion + ) { + super(simpleName, fieldType, params); + this.indexCreatedVersion = indexCreatedVersion; + } + + @Override + public MultiDenseVectorFieldType fieldType() { + return (MultiDenseVectorFieldType) super.fieldType(); + } + + @Override + public boolean parsesArrayValue() { + return true; + } + + @Override + public void parse(DocumentParserContext context) throws IOException { + if (context.doc().getByKey(fieldType().name()) != null) { + throw new IllegalArgumentException( + "Field [" + + fullPath() + + "] of type [" + + typeName() + + "] doesn't support indexing multiple values for the same field in the same document" + ); + } + if (XContentParser.Token.VALUE_NULL == context.parser().currentToken()) { + return; + } + if (XContentParser.Token.START_ARRAY != context.parser().currentToken()) { + throw new IllegalArgumentException( + "Field [" + fullPath() + "] of type [" + typeName() + "] cannot be indexed with a single value" + ); + } + if (fieldType().dims == null) { + int currentDims = -1; + while (XContentParser.Token.END_ARRAY != context.parser().nextToken()) { + int dims = fieldType().elementType.parseDimensionCount(context); + if (currentDims == -1) { + currentDims = dims; + } else if (currentDims != dims) { + throw new IllegalArgumentException( + "Field [" + fullPath() + "] of type [" + typeName() + "] cannot be indexed with vectors of different dimensions" + ); + } + } + MultiDenseVectorFieldType updatedFieldType = new MultiDenseVectorFieldType( + fieldType().name(), + fieldType().elementType, + currentDims, + indexCreatedVersion, + fieldType().meta() + ); + Mapper update = new MultiDenseVectorFieldMapper(leafName(), updatedFieldType, builderParams, indexCreatedVersion); + context.addDynamicMapper(update); + return; + } + int dims = fieldType().dims; + DenseVectorFieldMapper.ElementType elementType = fieldType().elementType; + List vectors = new ArrayList<>(); + while (XContentParser.Token.END_ARRAY != context.parser().nextToken()) { + VectorData vector = elementType.parseKnnVector(context, dims, (i, b) -> { + if (b) { + checkDimensionMatches(i, context); + } else { + checkDimensionExceeded(i, context); + } + }, null); + vectors.add(vector); + } + int bufferSize = elementType.getNumBytes(dims) * vectors.size(); + ByteBuffer buffer = ByteBuffer.allocate(bufferSize).order(ByteOrder.LITTLE_ENDIAN); + ByteBuffer magnitudeBuffer = ByteBuffer.allocate(vectors.size() * Float.BYTES).order(ByteOrder.LITTLE_ENDIAN); + for (VectorData vector : vectors) { + vector.addToBuffer(buffer); + magnitudeBuffer.putFloat((float) Math.sqrt(elementType.computeSquaredMagnitude(vector))); + } + String vectorFieldName = fieldType().name(); + String vectorMagnitudeFieldName = vectorFieldName + VECTOR_MAGNITUDES_SUFFIX; + context.doc().addWithKey(vectorFieldName, new BinaryDocValuesField(vectorFieldName, new BytesRef(buffer.array()))); + context.doc() + .addWithKey( + vectorMagnitudeFieldName, + new BinaryDocValuesField(vectorMagnitudeFieldName, new BytesRef(magnitudeBuffer.array())) + ); + } + + private void checkDimensionExceeded(int index, DocumentParserContext context) { + if (index >= fieldType().dims) { + throw new IllegalArgumentException( + "The [" + + typeName() + + "] field [" + + fullPath() + + "] in doc [" + + context.documentDescription() + + "] has more dimensions " + + "than defined in the mapping [" + + fieldType().dims + + "]" + ); + } + } + + private void checkDimensionMatches(int index, DocumentParserContext context) { + if (index != fieldType().dims) { + throw new IllegalArgumentException( + "The [" + + typeName() + + "] field [" + + fullPath() + + "] in doc [" + + context.documentDescription() + + "] has a different number of dimensions " + + "[" + + index + + "] than defined in the mapping [" + + fieldType().dims + + "]" + ); + } + } + + @Override + protected void parseCreateField(DocumentParserContext context) { + throw new AssertionError("parse is implemented directly"); + } + + @Override + protected String contentType() { + return CONTENT_TYPE; + } + + @Override + public FieldMapper.Builder getMergeBuilder() { + return new MultiDenseVectorFieldMapper.Builder(leafName(), indexCreatedVersion).init(this); + } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new SyntheticSourceSupport.Native(new MultiDenseVectorFieldMapper.DocValuesSyntheticFieldLoader()); + } + + private class DocValuesSyntheticFieldLoader extends SourceLoader.DocValuesBasedSyntheticFieldLoader { + private BinaryDocValues values; + private boolean hasValue; + + @Override + public DocValuesLoader docValuesLoader(LeafReader leafReader, int[] docIdsInLeaf) throws IOException { + values = leafReader.getBinaryDocValues(fullPath()); + if (values == null) { + return null; + } + return docId -> { + hasValue = docId == values.advance(docId); + return hasValue; + }; + } + + @Override + public boolean hasValue() { + return hasValue; + } + + @Override + public void write(XContentBuilder b) throws IOException { + if (false == hasValue) { + return; + } + b.startArray(leafName()); + BytesRef ref = values.binaryValue(); + ByteBuffer byteBuffer = ByteBuffer.wrap(ref.bytes, ref.offset, ref.length).order(ByteOrder.LITTLE_ENDIAN); + assert ref.length % fieldType().elementType.getNumBytes(fieldType().dims) == 0; + int numVecs = ref.length / fieldType().elementType.getNumBytes(fieldType().dims); + for (int i = 0; i < numVecs; i++) { + b.startArray(); + int dims = fieldType().elementType == DenseVectorFieldMapper.ElementType.BIT + ? fieldType().dims / Byte.SIZE + : fieldType().dims; + for (int dim = 0; dim < dims; dim++) { + fieldType().elementType.readAndWriteValue(byteBuffer, b); + } + b.endArray(); + } + b.endArray(); + } + + @Override + public String fieldName() { + return fullPath(); + } + } +} diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiVectorDVLeafFieldData.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiVectorDVLeafFieldData.java new file mode 100644 index 000000000000..cc6fb3827445 --- /dev/null +++ b/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiVectorDVLeafFieldData.java @@ -0,0 +1,54 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the "Elastic License + * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side + * Public License v 1"; you may not use this file except in compliance with, at + * your election, the "Elastic License 2.0", the "GNU Affero General Public + * License v3.0 only", or the "Server Side Public License, v 1". + */ + +package org.elasticsearch.index.mapper.vectors; + +import org.apache.lucene.index.LeafReader; +import org.elasticsearch.index.IndexVersion; +import org.elasticsearch.index.fielddata.LeafFieldData; +import org.elasticsearch.index.fielddata.SortedBinaryDocValues; +import org.elasticsearch.script.field.DocValuesScriptFieldFactory; + +final class MultiVectorDVLeafFieldData implements LeafFieldData { + private final LeafReader reader; + private final String field; + private final IndexVersion indexVersion; + private final DenseVectorFieldMapper.ElementType elementType; + private final int dims; + + MultiVectorDVLeafFieldData( + LeafReader reader, + String field, + IndexVersion indexVersion, + DenseVectorFieldMapper.ElementType elementType, + int dims + ) { + this.reader = reader; + this.field = field; + this.indexVersion = indexVersion; + this.elementType = elementType; + this.dims = dims; + } + + @Override + public DocValuesScriptFieldFactory getScriptFieldFactory(String name) { + // TODO + return null; + } + + @Override + public SortedBinaryDocValues getBytesValues() { + throw new UnsupportedOperationException("String representation of doc values for multi-vector fields is not supported"); + } + + @Override + public long ramBytesUsed() { + return 0; + } +} diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiVectorIndexFieldData.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiVectorIndexFieldData.java new file mode 100644 index 000000000000..65ef492ce052 --- /dev/null +++ b/server/src/main/java/org/elasticsearch/index/mapper/vectors/MultiVectorIndexFieldData.java @@ -0,0 +1,114 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the "Elastic License + * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side + * Public License v 1"; you may not use this file except in compliance with, at + * your election, the "Elastic License 2.0", the "GNU Affero General Public + * License v3.0 only", or the "Server Side Public License, v 1". + */ + +package org.elasticsearch.index.mapper.vectors; + +import org.apache.lucene.index.LeafReaderContext; +import org.apache.lucene.search.SortField; +import org.elasticsearch.common.util.BigArrays; +import org.elasticsearch.index.IndexVersion; +import org.elasticsearch.index.fielddata.IndexFieldData; +import org.elasticsearch.index.fielddata.IndexFieldDataCache; +import org.elasticsearch.indices.breaker.CircuitBreakerService; +import org.elasticsearch.search.DocValueFormat; +import org.elasticsearch.search.MultiValueMode; +import org.elasticsearch.search.aggregations.support.ValuesSourceType; +import org.elasticsearch.search.sort.BucketedSort; +import org.elasticsearch.search.sort.SortOrder; + +public class MultiVectorIndexFieldData implements IndexFieldData { + protected final String fieldName; + protected final ValuesSourceType valuesSourceType; + private final int dims; + private final IndexVersion indexVersion; + private final DenseVectorFieldMapper.ElementType elementType; + + public MultiVectorIndexFieldData( + String fieldName, + int dims, + ValuesSourceType valuesSourceType, + IndexVersion indexVersion, + DenseVectorFieldMapper.ElementType elementType + ) { + this.fieldName = fieldName; + this.valuesSourceType = valuesSourceType; + this.indexVersion = indexVersion; + this.elementType = elementType; + this.dims = dims; + } + + @Override + public String getFieldName() { + return fieldName; + } + + @Override + public ValuesSourceType getValuesSourceType() { + return valuesSourceType; + } + + @Override + public MultiVectorDVLeafFieldData load(LeafReaderContext context) { + return new MultiVectorDVLeafFieldData(context.reader(), fieldName, indexVersion, elementType, dims); + } + + @Override + public MultiVectorDVLeafFieldData loadDirect(LeafReaderContext context) throws Exception { + return load(context); + } + + @Override + public SortField sortField(Object missingValue, MultiValueMode sortMode, XFieldComparatorSource.Nested nested, boolean reverse) { + throw new IllegalArgumentException( + "Field [" + fieldName + "] of type [" + MultiDenseVectorFieldMapper.CONTENT_TYPE + "] doesn't support sort" + ); + } + + @Override + public BucketedSort newBucketedSort( + BigArrays bigArrays, + Object missingValue, + MultiValueMode sortMode, + XFieldComparatorSource.Nested nested, + SortOrder sortOrder, + DocValueFormat format, + int bucketSize, + BucketedSort.ExtraData extra + ) { + throw new IllegalArgumentException("only supported on numeric fields"); + } + + public static class Builder implements IndexFieldData.Builder { + + private final String name; + private final ValuesSourceType valuesSourceType; + private final IndexVersion indexVersion; + private final int dims; + private final DenseVectorFieldMapper.ElementType elementType; + + public Builder( + String name, + ValuesSourceType valuesSourceType, + IndexVersion indexVersion, + int dims, + DenseVectorFieldMapper.ElementType elementType + ) { + this.name = name; + this.valuesSourceType = valuesSourceType; + this.indexVersion = indexVersion; + this.dims = dims; + this.elementType = elementType; + } + + @Override + public IndexFieldData build(IndexFieldDataCache cache, CircuitBreakerService breakerService) { + return new MultiVectorIndexFieldData(name, dims, valuesSourceType, indexVersion, elementType); + } + } +} diff --git a/server/src/main/java/org/elasticsearch/indices/IndicesModule.java b/server/src/main/java/org/elasticsearch/indices/IndicesModule.java index 09be98630d5c..340bff4e1c85 100644 --- a/server/src/main/java/org/elasticsearch/indices/IndicesModule.java +++ b/server/src/main/java/org/elasticsearch/indices/IndicesModule.java @@ -67,6 +67,7 @@ import org.elasticsearch.index.mapper.TimeSeriesRoutingHashFieldMapper; import org.elasticsearch.index.mapper.VersionFieldMapper; import org.elasticsearch.index.mapper.flattened.FlattenedFieldMapper; import org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper; +import org.elasticsearch.index.mapper.vectors.MultiDenseVectorFieldMapper; import org.elasticsearch.index.mapper.vectors.SparseVectorFieldMapper; import org.elasticsearch.index.seqno.RetentionLeaseBackgroundSyncAction; import org.elasticsearch.index.seqno.RetentionLeaseSyncAction; @@ -210,6 +211,9 @@ public class IndicesModule extends AbstractModule { mappers.put(DenseVectorFieldMapper.CONTENT_TYPE, DenseVectorFieldMapper.PARSER); mappers.put(SparseVectorFieldMapper.CONTENT_TYPE, SparseVectorFieldMapper.PARSER); + if (MultiDenseVectorFieldMapper.FEATURE_FLAG.isEnabled()) { + mappers.put(MultiDenseVectorFieldMapper.CONTENT_TYPE, MultiDenseVectorFieldMapper.PARSER); + } for (MapperPlugin mapperPlugin : mapperPlugins) { for (Map.Entry entry : mapperPlugin.getMappers().entrySet()) { diff --git a/server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java b/server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java index 601fc3c86d98..ae9ae6f8b5bf 100644 --- a/server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java +++ b/server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java @@ -15,12 +15,14 @@ import org.elasticsearch.action.ActionResponse; import org.elasticsearch.action.support.PlainActionFuture; import org.elasticsearch.cluster.ClusterState; import org.elasticsearch.cluster.ClusterStateListener; +import org.elasticsearch.cluster.NotMasterException; import org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException; import org.elasticsearch.cluster.metadata.Metadata; import org.elasticsearch.cluster.metadata.ReservedStateMetadata; import org.elasticsearch.cluster.service.ClusterService; import org.elasticsearch.common.file.MasterNodeFileWatchingService; import org.elasticsearch.env.Environment; +import org.elasticsearch.xcontent.XContentParseException; import org.elasticsearch.xcontent.XContentParserConfiguration; import java.io.BufferedInputStream; @@ -146,11 +148,20 @@ public class FileSettingsService extends MasterNodeFileWatchingService implement @Override protected void onProcessFileChangesException(Exception e) { - if (e instanceof ExecutionException && e.getCause() instanceof FailedToCommitClusterStateException f) { - logger.error("Unable to commit cluster state", e); - } else { - super.onProcessFileChangesException(e); + if (e instanceof ExecutionException) { + var cause = e.getCause(); + if (cause instanceof FailedToCommitClusterStateException) { + logger.error("Unable to commit cluster state", e); + return; + } else if (cause instanceof XContentParseException) { + logger.error("Unable to parse settings", e); + return; + } else if (cause instanceof NotMasterException) { + logger.error("Node is no longer master", e); + return; + } } + super.onProcessFileChangesException(e); } @Override diff --git a/server/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java b/server/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java index 99fde2731644..39e679f2c0ad 100644 --- a/server/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java +++ b/server/src/main/java/org/elasticsearch/rest/action/cat/RestNodesAction.java @@ -367,7 +367,7 @@ public class RestNodesAction extends AbstractCatAction { table.addCell("-"); } - table.addCell(node.getVersion().toString()); + table.addCell(node.getBuildVersion().toString()); table.addCell(info == null ? null : info.getBuild().type().displayName()); table.addCell(info == null ? null : info.getBuild().hash()); table.addCell(jvmInfo == null ? null : jvmInfo.version()); diff --git a/server/src/main/java/org/elasticsearch/rest/action/cat/RestTasksAction.java b/server/src/main/java/org/elasticsearch/rest/action/cat/RestTasksAction.java index fb0814d5746a..b0f94fd1420e 100644 --- a/server/src/main/java/org/elasticsearch/rest/action/cat/RestTasksAction.java +++ b/server/src/main/java/org/elasticsearch/rest/action/cat/RestTasksAction.java @@ -140,7 +140,7 @@ public class RestTasksAction extends AbstractCatAction { table.addCell(node == null ? "-" : node.getHostAddress()); table.addCell(node.getAddress().address().getPort()); table.addCell(node == null ? "-" : node.getName()); - table.addCell(node == null ? "-" : node.getVersion().toString()); + table.addCell(node == null ? "-" : node.getBuildVersion().toString()); table.addCell(taskInfo.headers().getOrDefault(Task.X_OPAQUE_ID_HTTP_HEADER, "-")); if (detailed) { diff --git a/server/src/main/java/org/elasticsearch/rest/action/search/SearchCapabilities.java b/server/src/main/java/org/elasticsearch/rest/action/search/SearchCapabilities.java index 03467fb6cfdd..338dabb23ab4 100644 --- a/server/src/main/java/org/elasticsearch/rest/action/search/SearchCapabilities.java +++ b/server/src/main/java/org/elasticsearch/rest/action/search/SearchCapabilities.java @@ -9,6 +9,10 @@ package org.elasticsearch.rest.action.search; +import org.elasticsearch.Build; +import org.elasticsearch.index.mapper.vectors.MultiDenseVectorFieldMapper; + +import java.util.HashSet; import java.util.Set; /** @@ -28,12 +32,25 @@ public final class SearchCapabilities { private static final String DENSE_VECTOR_DOCVALUE_FIELDS = "dense_vector_docvalue_fields"; /** Support transforming rank rrf queries to the corresponding rrf retriever. */ private static final String TRANSFORM_RANK_RRF_TO_RETRIEVER = "transform_rank_rrf_to_retriever"; + /** Support kql query. */ + private static final String KQL_QUERY_SUPPORTED = "kql_query"; + /** Support multi-dense-vector field mapper. */ + private static final String MULTI_DENSE_VECTOR_FIELD_MAPPER = "multi_dense_vector_field_mapper"; - public static final Set CAPABILITIES = Set.of( - RANGE_REGEX_INTERVAL_QUERY_CAPABILITY, - BIT_DENSE_VECTOR_SYNTHETIC_SOURCE_CAPABILITY, - BYTE_FLOAT_BIT_DOT_PRODUCT_CAPABILITY, - DENSE_VECTOR_DOCVALUE_FIELDS, - TRANSFORM_RANK_RRF_TO_RETRIEVER - ); + public static final Set CAPABILITIES; + static { + HashSet capabilities = new HashSet<>(); + capabilities.add(RANGE_REGEX_INTERVAL_QUERY_CAPABILITY); + capabilities.add(BIT_DENSE_VECTOR_SYNTHETIC_SOURCE_CAPABILITY); + capabilities.add(BYTE_FLOAT_BIT_DOT_PRODUCT_CAPABILITY); + capabilities.add(DENSE_VECTOR_DOCVALUE_FIELDS); + capabilities.add(TRANSFORM_RANK_RRF_TO_RETRIEVER); + if (MultiDenseVectorFieldMapper.FEATURE_FLAG.isEnabled()) { + capabilities.add(MULTI_DENSE_VECTOR_FIELD_MAPPER); + } + if (Build.current().isSnapshot()) { + capabilities.add(KQL_QUERY_SUPPORTED); + } + CAPABILITIES = Set.copyOf(capabilities); + } } diff --git a/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java b/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java index 71de6a9bbb82..38cab1761d40 100644 --- a/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java +++ b/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregation.java @@ -9,6 +9,7 @@ package org.elasticsearch.search.aggregations; import org.elasticsearch.common.Strings; +import org.elasticsearch.common.io.stream.DelayableWriteable; import org.elasticsearch.common.io.stream.NamedWriteable; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; @@ -51,7 +52,12 @@ public abstract class InternalAggregation implements Aggregation, NamedWriteable * Read from a stream. */ protected InternalAggregation(StreamInput in) throws IOException { - name = in.readString(); + final String name = in.readString(); + if (in instanceof DelayableWriteable.Deduplicator d) { + this.name = d.deduplicate(name); + } else { + this.name = name; + } metadata = in.readGenericMap(); } diff --git a/server/src/test/java/org/elasticsearch/action/admin/indices/TransportAnalyzeActionTests.java b/server/src/test/java/org/elasticsearch/action/admin/indices/TransportAnalyzeActionTests.java index 901f0d700054..0b9cba837583 100644 --- a/server/src/test/java/org/elasticsearch/action/admin/indices/TransportAnalyzeActionTests.java +++ b/server/src/test/java/org/elasticsearch/action/admin/indices/TransportAnalyzeActionTests.java @@ -13,6 +13,7 @@ import org.apache.lucene.tests.analysis.MockTokenFilter; import org.apache.lucene.tests.analysis.MockTokenizer; import org.apache.lucene.util.automaton.Automata; import org.apache.lucene.util.automaton.CharacterRunAutomaton; +import org.elasticsearch.ElasticsearchStatusException; import org.elasticsearch.action.admin.indices.analyze.AnalyzeAction; import org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction; import org.elasticsearch.cluster.metadata.IndexMetadata; @@ -460,8 +461,8 @@ public class TransportAnalyzeActionTests extends ESTestCase { AnalyzeAction.Request request = new AnalyzeAction.Request(); request.text(text); request.analyzer("standard"); - IllegalStateException e = expectThrows( - IllegalStateException.class, + ElasticsearchStatusException e = expectThrows( + ElasticsearchStatusException.class, () -> TransportAnalyzeAction.analyze(request, registry, null, maxTokenCount) ); assertEquals( @@ -477,8 +478,8 @@ public class TransportAnalyzeActionTests extends ESTestCase { request2.text(text); request2.analyzer("standard"); request2.explain(true); - IllegalStateException e2 = expectThrows( - IllegalStateException.class, + ElasticsearchStatusException e2 = expectThrows( + ElasticsearchStatusException.class, () -> TransportAnalyzeAction.analyze(request2, registry, null, maxTokenCount) ); assertEquals( @@ -506,8 +507,8 @@ public class TransportAnalyzeActionTests extends ESTestCase { AnalyzeAction.Request request = new AnalyzeAction.Request(); request.text(text); request.analyzer("standard"); - IllegalStateException e = expectThrows( - IllegalStateException.class, + ElasticsearchStatusException e = expectThrows( + ElasticsearchStatusException.class, () -> TransportAnalyzeAction.analyze(request, registry, null, idxMaxTokenCount) ); assertEquals( diff --git a/server/src/test/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhaseTests.java b/server/src/test/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhaseTests.java index d8b40511af67..bd99f6a79eb3 100644 --- a/server/src/test/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhaseTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/CanMatchPreFilterSearchPhaseTests.java @@ -28,6 +28,7 @@ import org.elasticsearch.cluster.routing.allocation.DataTier; import org.elasticsearch.common.Strings; import org.elasticsearch.common.UUIDs; import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.core.Tuple; import org.elasticsearch.index.Index; import org.elasticsearch.index.IndexVersion; import org.elasticsearch.index.mapper.DateFieldMapper; @@ -68,6 +69,7 @@ import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import java.util.function.BiConsumer; @@ -77,6 +79,7 @@ import static org.elasticsearch.action.search.SearchAsyncActionTests.getShardsIt import static org.elasticsearch.core.Types.forciblyCast; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.greaterThan; +import static org.hamcrest.Matchers.instanceOf; import static org.hamcrest.Matchers.lessThanOrEqualTo; import static org.mockito.Mockito.mock; @@ -1087,6 +1090,137 @@ public class CanMatchPreFilterSearchPhaseTests extends ESTestCase { ); } + public void testCanMatchFilteringOnCoordinatorWithMissingShards() throws Exception { + // we'll test that we're executing _tier coordinator rewrite for indices (data stream backing or regular) without any @timestamp + // or event.ingested fields + // for both data stream backing and regular indices we'll have one index in hot and one UNASSIGNED (targeting warm though). + // the warm indices will be skipped as our queries will filter based on _tier: hot and the can match phase will not report error the + // missing index even if allow_partial_search_results is false (because the warm index would've not been part of the search anyway) + + Map indexNameToSettings = new HashMap<>(); + ClusterState state = ClusterState.EMPTY_STATE; + + String dataStreamName = randomAlphaOfLengthBetween(10, 20); + Index warmDataStreamIndex = new Index(DataStream.getDefaultBackingIndexName(dataStreamName, 1), UUIDs.base64UUID()); + indexNameToSettings.put( + warmDataStreamIndex, + settings(IndexVersion.current()).put(IndexMetadata.SETTING_INDEX_UUID, warmDataStreamIndex.getUUID()) + .put(DataTier.TIER_PREFERENCE, "data_warm,data_hot") + ); + Index hotDataStreamIndex = new Index(DataStream.getDefaultBackingIndexName(dataStreamName, 2), UUIDs.base64UUID()); + indexNameToSettings.put( + hotDataStreamIndex, + settings(IndexVersion.current()).put(IndexMetadata.SETTING_INDEX_UUID, hotDataStreamIndex.getUUID()) + .put(DataTier.TIER_PREFERENCE, "data_hot") + ); + DataStream dataStream = DataStreamTestHelper.newInstance(dataStreamName, List.of(warmDataStreamIndex, hotDataStreamIndex)); + + Index warmRegularIndex = new Index("warm-index", UUIDs.base64UUID()); + indexNameToSettings.put( + warmRegularIndex, + settings(IndexVersion.current()).put(IndexMetadata.SETTING_INDEX_UUID, warmRegularIndex.getUUID()) + .put(DataTier.TIER_PREFERENCE, "data_warm,data_hot") + ); + Index hotRegularIndex = new Index("hot-index", UUIDs.base64UUID()); + indexNameToSettings.put( + hotRegularIndex, + settings(IndexVersion.current()).put(IndexMetadata.SETTING_INDEX_UUID, hotRegularIndex.getUUID()) + .put(DataTier.TIER_PREFERENCE, "data_hot") + ); + + List allIndices = new ArrayList<>(4); + allIndices.addAll(dataStream.getIndices()); + allIndices.add(warmRegularIndex); + allIndices.add(hotRegularIndex); + + List hotIndices = List.of(hotRegularIndex, hotDataStreamIndex); + List warmIndices = List.of(warmRegularIndex, warmDataStreamIndex); + + for (Index index : allIndices) { + IndexMetadata.Builder indexMetadataBuilder = IndexMetadata.builder(index.getName()) + .settings(indexNameToSettings.get(index)) + .numberOfShards(1) + .numberOfReplicas(0); + Metadata.Builder metadataBuilder = Metadata.builder(state.metadata()).put(indexMetadataBuilder); + state = ClusterState.builder(state).metadata(metadataBuilder).build(); + } + + ClusterState finalState = state; + CoordinatorRewriteContextProvider coordinatorRewriteContextProvider = new CoordinatorRewriteContextProvider( + parserConfig(), + mock(Client.class), + System::currentTimeMillis, + () -> finalState, + (index) -> null + ); + + BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery() + .filter(QueryBuilders.termQuery(CoordinatorRewriteContext.TIER_FIELD_NAME, "data_hot")); + + { + // test that a search doesn't fail if the query filters out the unassigned shards + // via _tier (coordinator rewrite will eliminate the shards that don't match) + assignShardsAndExecuteCanMatchPhase( + List.of(dataStream), + List.of(hotRegularIndex, warmRegularIndex), + coordinatorRewriteContextProvider, + boolQueryBuilder, + List.of(), + null, + warmIndices, + false, + (updatedSearchShardIterators, requests) -> { + var skippedShards = updatedSearchShardIterators.stream().filter(SearchShardIterator::skip).toList(); + var nonSkippedShards = updatedSearchShardIterators.stream() + .filter(searchShardIterator -> searchShardIterator.skip() == false) + .toList(); + + boolean allSkippedShardAreFromWarmIndices = skippedShards.stream() + .allMatch(shardIterator -> warmIndices.contains(shardIterator.shardId().getIndex())); + assertThat(allSkippedShardAreFromWarmIndices, equalTo(true)); + boolean allNonSkippedShardAreHotIndices = nonSkippedShards.stream() + .allMatch(shardIterator -> hotIndices.contains(shardIterator.shardId().getIndex())); + assertThat(allNonSkippedShardAreHotIndices, equalTo(true)); + boolean allRequestMadeToHotIndices = requests.stream() + .allMatch(request -> hotIndices.contains(request.shardId().getIndex())); + assertThat(allRequestMadeToHotIndices, equalTo(true)); + } + ); + } + + { + // test that a search does fail if the query does NOT filter ALL the + // unassigned shards + CountDownLatch latch = new CountDownLatch(1); + Tuple> canMatchPhaseAndRequests = getCanMatchPhaseAndRequests( + List.of(dataStream), + List.of(hotRegularIndex, warmRegularIndex), + coordinatorRewriteContextProvider, + boolQueryBuilder, + List.of(), + null, + List.of(hotRegularIndex, warmRegularIndex, warmDataStreamIndex), + false, + new ActionListener<>() { + @Override + public void onResponse(GroupShardsIterator searchShardIterators) { + fail(null, "unexpected success with result [%s] while expecting to handle failure with [%s]", searchShardIterators); + latch.countDown(); + } + + @Override + public void onFailure(Exception e) { + assertThat(e, instanceOf(SearchPhaseExecutionException.class)); + latch.countDown(); + } + } + ); + + canMatchPhaseAndRequests.v1().start(); + latch.await(10, TimeUnit.SECONDS); + } + } + private void assertAllShardsAreQueried(List updatedSearchShardIterators, List requests) { int skippedShards = (int) updatedSearchShardIterators.stream().filter(SearchShardIterator::skip).count(); @@ -1111,6 +1245,69 @@ public class CanMatchPreFilterSearchPhaseTests extends ESTestCase { SuggestBuilder suggest, BiConsumer, List> canMatchResultsConsumer ) throws Exception { + assignShardsAndExecuteCanMatchPhase( + dataStreams, + regularIndices, + contextProvider, + query, + aggregations, + suggest, + List.of(), + true, + canMatchResultsConsumer + ); + } + + private void assignShardsAndExecuteCanMatchPhase( + List dataStreams, + List regularIndices, + CoordinatorRewriteContextProvider contextProvider, + QueryBuilder query, + List aggregations, + SuggestBuilder suggest, + List unassignedIndices, + boolean allowPartialResults, + BiConsumer, List> canMatchResultsConsumer + ) throws Exception { + AtomicReference> result = new AtomicReference<>(); + CountDownLatch latch = new CountDownLatch(1); + Tuple> canMatchAndShardRequests = getCanMatchPhaseAndRequests( + dataStreams, + regularIndices, + contextProvider, + query, + aggregations, + suggest, + unassignedIndices, + allowPartialResults, + ActionTestUtils.assertNoFailureListener(iter -> { + result.set(iter); + latch.countDown(); + }) + ); + + canMatchAndShardRequests.v1().start(); + latch.await(); + + List updatedSearchShardIterators = new ArrayList<>(); + for (SearchShardIterator updatedSearchShardIterator : result.get()) { + updatedSearchShardIterators.add(updatedSearchShardIterator); + } + + canMatchResultsConsumer.accept(updatedSearchShardIterators, canMatchAndShardRequests.v2()); + } + + private Tuple> getCanMatchPhaseAndRequests( + List dataStreams, + List regularIndices, + CoordinatorRewriteContextProvider contextProvider, + QueryBuilder query, + List aggregations, + SuggestBuilder suggest, + List unassignedIndices, + boolean allowPartialResults, + ActionListener> canMatchActionListener + ) { Map lookup = new ConcurrentHashMap<>(); DiscoveryNode primaryNode = DiscoveryNodeUtils.create("node_1"); DiscoveryNode replicaNode = DiscoveryNodeUtils.create("node_2"); @@ -1136,23 +1333,31 @@ public class CanMatchPreFilterSearchPhaseTests extends ESTestCase { // and none is assigned, the phase is considered as failed meaning that the next phase won't be executed boolean withAssignedPrimaries = randomBoolean() || atLeastOnePrimaryAssigned == false; int numShards = randomIntBetween(1, 6); - originalShardIters.addAll( - getShardsIter(dataStreamIndex, originalIndices, numShards, false, withAssignedPrimaries ? primaryNode : null, null) - ); - atLeastOnePrimaryAssigned |= withAssignedPrimaries; + if (unassignedIndices.contains(dataStreamIndex)) { + originalShardIters.addAll(getShardsIter(dataStreamIndex, originalIndices, numShards, false, null, null)); + } else { + originalShardIters.addAll( + getShardsIter(dataStreamIndex, originalIndices, numShards, false, withAssignedPrimaries ? primaryNode : null, null) + ); + atLeastOnePrimaryAssigned |= withAssignedPrimaries; + } } } for (Index regularIndex : regularIndices) { - originalShardIters.addAll( - getShardsIter(regularIndex, originalIndices, randomIntBetween(1, 6), randomBoolean(), primaryNode, replicaNode) - ); + if (unassignedIndices.contains(regularIndex)) { + originalShardIters.addAll(getShardsIter(regularIndex, originalIndices, randomIntBetween(1, 6), false, null, null)); + } else { + originalShardIters.addAll( + getShardsIter(regularIndex, originalIndices, randomIntBetween(1, 6), randomBoolean(), primaryNode, replicaNode) + ); + } } GroupShardsIterator shardsIter = GroupShardsIterator.sortAndCreate(originalShardIters); final SearchRequest searchRequest = new SearchRequest(); searchRequest.indices(indices); - searchRequest.allowPartialSearchResults(true); + searchRequest.allowPartialSearchResults(allowPartialResults); final AliasFilter aliasFilter; if (aggregations.isEmpty() == false || randomBoolean()) { @@ -1212,35 +1417,24 @@ public class CanMatchPreFilterSearchPhaseTests extends ESTestCase { ); AtomicReference> result = new AtomicReference<>(); - CountDownLatch latch = new CountDownLatch(1); - CanMatchPreFilterSearchPhase canMatchPhase = new CanMatchPreFilterSearchPhase( - logger, - searchTransportService, - (clusterAlias, node) -> lookup.get(node), - aliasFilters, - Collections.emptyMap(), - threadPool.executor(ThreadPool.Names.SEARCH_COORDINATION), - searchRequest, - shardsIter, - timeProvider, - null, - true, - contextProvider, - ActionTestUtils.assertNoFailureListener(iter -> { - result.set(iter); - latch.countDown(); - }) + return new Tuple<>( + new CanMatchPreFilterSearchPhase( + logger, + searchTransportService, + (clusterAlias, node) -> lookup.get(node), + aliasFilters, + Collections.emptyMap(), + threadPool.executor(ThreadPool.Names.SEARCH_COORDINATION), + searchRequest, + shardsIter, + timeProvider, + null, + true, + contextProvider, + canMatchActionListener + ), + requests ); - - canMatchPhase.start(); - latch.await(); - - List updatedSearchShardIterators = new ArrayList<>(); - for (SearchShardIterator updatedSearchShardIterator : result.get()) { - updatedSearchShardIterators.add(updatedSearchShardIterator); - } - - canMatchResultsConsumer.accept(updatedSearchShardIterators, requests); } static class StaticCoordinatorRewriteContextProviderBuilder { diff --git a/server/src/test/java/org/elasticsearch/action/search/CountedCollectorTests.java b/server/src/test/java/org/elasticsearch/action/search/CountedCollectorTests.java index 40c6a707a87f..4ba03f85aa05 100644 --- a/server/src/test/java/org/elasticsearch/action/search/CountedCollectorTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/CountedCollectorTests.java @@ -93,6 +93,7 @@ public class CountedCollectorTests extends ESTestCase { for (int i = numResultsExpected; i < results.length(); i++) { assertNull("index: " + i, results.get(i)); } + context.results.close(); } } } diff --git a/server/src/test/java/org/elasticsearch/action/search/DfsQueryPhaseTests.java b/server/src/test/java/org/elasticsearch/action/search/DfsQueryPhaseTests.java index 99401e8a8d40..64362daf7f75 100644 --- a/server/src/test/java/org/elasticsearch/action/search/DfsQueryPhaseTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/DfsQueryPhaseTests.java @@ -134,7 +134,7 @@ public class DfsQueryPhaseTests extends ESTestCase { new NoopCircuitBreaker(CircuitBreaker.REQUEST), () -> false, SearchProgressListener.NOOP, - mockSearchPhaseContext.searchRequest, + mockSearchPhaseContext.getRequest(), results.length(), exc -> {} ) @@ -159,6 +159,7 @@ public class DfsQueryPhaseTests extends ESTestCase { assertEquals(84, responseRef.get().get(1).queryResult().topDocs().topDocs.scoreDocs[0].doc); assertTrue(mockSearchPhaseContext.releasedSearchContexts.isEmpty()); assertEquals(2, mockSearchPhaseContext.numSuccess.get()); + mockSearchPhaseContext.results.close(); } } @@ -219,7 +220,7 @@ public class DfsQueryPhaseTests extends ESTestCase { new NoopCircuitBreaker(CircuitBreaker.REQUEST), () -> false, SearchProgressListener.NOOP, - mockSearchPhaseContext.searchRequest, + mockSearchPhaseContext.getRequest(), results.length(), exc -> {} ) @@ -246,6 +247,7 @@ public class DfsQueryPhaseTests extends ESTestCase { assertEquals(1, mockSearchPhaseContext.releasedSearchContexts.size()); assertTrue(mockSearchPhaseContext.releasedSearchContexts.contains(new ShardSearchContextId("", 2L))); assertNull(responseRef.get().get(1)); + mockSearchPhaseContext.results.close(); } } @@ -306,7 +308,7 @@ public class DfsQueryPhaseTests extends ESTestCase { new NoopCircuitBreaker(CircuitBreaker.REQUEST), () -> false, SearchProgressListener.NOOP, - mockSearchPhaseContext.searchRequest, + mockSearchPhaseContext.getRequest(), results.length(), exc -> {} ) @@ -322,6 +324,7 @@ public class DfsQueryPhaseTests extends ESTestCase { assertThat(mockSearchPhaseContext.failures, hasSize(1)); assertThat(mockSearchPhaseContext.failures.get(0).getCause(), instanceOf(UncheckedIOException.class)); assertThat(mockSearchPhaseContext.releasedSearchContexts, hasSize(1)); // phase execution will clean up on the contexts + mockSearchPhaseContext.results.close(); } } @@ -371,6 +374,7 @@ public class DfsQueryPhaseTests extends ESTestCase { ssr.source().subSearches().get(2).getQueryBuilder() ) ); + mspc.results.close(); } private SearchPhaseController searchPhaseController() { diff --git a/server/src/test/java/org/elasticsearch/action/search/ExpandSearchPhaseTests.java b/server/src/test/java/org/elasticsearch/action/search/ExpandSearchPhaseTests.java index 5240d704dea3..23184be02f9c 100644 --- a/server/src/test/java/org/elasticsearch/action/search/ExpandSearchPhaseTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/ExpandSearchPhaseTests.java @@ -229,7 +229,7 @@ public class ExpandSearchPhaseTests extends ESTestCase { assertNotNull(mockSearchPhaseContext.phaseFailure.get()); assertNull(mockSearchPhaseContext.searchResponse.get()); } finally { - mockSearchPhaseContext.execute(() -> {}); + mockSearchPhaseContext.results.close(); hits.decRef(); collapsedHits.decRef(); } @@ -269,7 +269,7 @@ public class ExpandSearchPhaseTests extends ESTestCase { hits.decRef(); } } finally { - mockSearchPhaseContext.execute(() -> {}); + mockSearchPhaseContext.results.close(); var resp = mockSearchPhaseContext.searchResponse.get(); if (resp != null) { resp.decRef(); @@ -356,6 +356,7 @@ public class ExpandSearchPhaseTests extends ESTestCase { hits.decRef(); } } finally { + mockSearchPhaseContext.results.close(); var resp = mockSearchPhaseContext.searchResponse.get(); if (resp != null) { resp.decRef(); @@ -407,6 +408,7 @@ public class ExpandSearchPhaseTests extends ESTestCase { hits.decRef(); } } finally { + mockSearchPhaseContext.results.close(); var resp = mockSearchPhaseContext.searchResponse.get(); if (resp != null) { resp.decRef(); diff --git a/server/src/test/java/org/elasticsearch/action/search/FetchLookupFieldsPhaseTests.java b/server/src/test/java/org/elasticsearch/action/search/FetchLookupFieldsPhaseTests.java index e478ed2d0ccb..1d2daf0cd660 100644 --- a/server/src/test/java/org/elasticsearch/action/search/FetchLookupFieldsPhaseTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/FetchLookupFieldsPhaseTests.java @@ -57,7 +57,6 @@ public class FetchLookupFieldsPhaseTests extends ESTestCase { } searchPhaseContext.assertNoFailure(); assertNotNull(searchPhaseContext.searchResponse.get()); - searchPhaseContext.execute(() -> {}); } finally { var resp = searchPhaseContext.searchResponse.get(); if (resp != null) { @@ -225,8 +224,8 @@ public class FetchLookupFieldsPhaseTests extends ESTestCase { leftHit1.field("lookup_field_3").getValues(), contains(Map.of("field_a", List.of("a2"), "field_b", List.of("b1", "b2"))) ); - searchPhaseContext.execute(() -> {}); } finally { + searchPhaseContext.results.close(); var resp = searchPhaseContext.searchResponse.get(); if (resp != null) { resp.decRef(); diff --git a/server/src/test/java/org/elasticsearch/action/search/FetchSearchPhaseTests.java b/server/src/test/java/org/elasticsearch/action/search/FetchSearchPhaseTests.java index 09dd7821cd12..762a7e0f47ca 100644 --- a/server/src/test/java/org/elasticsearch/action/search/FetchSearchPhaseTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/FetchSearchPhaseTests.java @@ -123,6 +123,7 @@ public class FetchSearchPhaseTests extends ESTestCase { assertProfiles(profiled, 1, searchResponse); assertTrue(mockSearchPhaseContext.releasedSearchContexts.isEmpty()); } finally { + mockSearchPhaseContext.results.close(); var resp = mockSearchPhaseContext.searchResponse.get(); if (resp != null) { resp.decRef(); @@ -252,6 +253,7 @@ public class FetchSearchPhaseTests extends ESTestCase { assertProfiles(profiled, 2, searchResponse); assertTrue(mockSearchPhaseContext.releasedSearchContexts.isEmpty()); } finally { + mockSearchPhaseContext.results.close(); var resp = mockSearchPhaseContext.searchResponse.get(); if (resp != null) { resp.decRef(); diff --git a/server/src/test/java/org/elasticsearch/action/search/MockSearchPhaseContext.java b/server/src/test/java/org/elasticsearch/action/search/MockSearchPhaseContext.java index 5395e4569901..03c5d0a06f6f 100644 --- a/server/src/test/java/org/elasticsearch/action/search/MockSearchPhaseContext.java +++ b/server/src/test/java/org/elasticsearch/action/search/MockSearchPhaseContext.java @@ -10,12 +10,15 @@ package org.elasticsearch.action.search; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.elasticsearch.action.ActionListener; import org.elasticsearch.action.OriginalIndices; +import org.elasticsearch.cluster.ClusterState; +import org.elasticsearch.cluster.routing.GroupShardsIterator; import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.io.stream.NamedWriteableRegistry; import org.elasticsearch.common.util.concurrent.AtomicArray; import org.elasticsearch.core.Nullable; -import org.elasticsearch.core.Releasable; import org.elasticsearch.core.Releasables; import org.elasticsearch.search.SearchPhaseResult; import org.elasticsearch.search.SearchShardTarget; @@ -32,23 +35,41 @@ import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import java.util.function.Supplier; +import static org.mockito.Mockito.mock; + /** * SearchPhaseContext for tests */ -public final class MockSearchPhaseContext implements SearchPhaseContext { +public final class MockSearchPhaseContext extends AbstractSearchAsyncAction { private static final Logger logger = LogManager.getLogger(MockSearchPhaseContext.class); - final AtomicReference phaseFailure = new AtomicReference<>(); + public final AtomicReference phaseFailure = new AtomicReference<>(); final int numShards; final AtomicInteger numSuccess; - final List failures = Collections.synchronizedList(new ArrayList<>()); + public final List failures = Collections.synchronizedList(new ArrayList<>()); SearchTransportService searchTransport; final Set releasedSearchContexts = new HashSet<>(); - final SearchRequest searchRequest = new SearchRequest(); - final AtomicReference searchResponse = new AtomicReference<>(); - - private final List releasables = new ArrayList<>(); + public final AtomicReference searchResponse = new AtomicReference<>(); public MockSearchPhaseContext(int numShards) { + super( + "mock", + logger, + new NamedWriteableRegistry(List.of()), + mock(SearchTransportService.class), + (clusterAlias, nodeId) -> null, + null, + null, + Runnable::run, + new SearchRequest(), + ActionListener.noop(), + new GroupShardsIterator(List.of()), + null, + ClusterState.EMPTY_STATE, + new SearchTask(0, "n/a", "n/a", () -> "test", null, Collections.emptyMap()), + new ArraySearchPhaseResults<>(numShards), + 5, + null + ); this.numShards = numShards; numSuccess = new AtomicInteger(numShards); } @@ -59,28 +80,9 @@ public final class MockSearchPhaseContext implements SearchPhaseContext { } } - @Override - public int getNumShards() { - return numShards; - } - - @Override - public Logger getLogger() { - return logger; - } - - @Override - public SearchTask getTask() { - return new SearchTask(0, "n/a", "n/a", () -> "test", null, Collections.emptyMap()); - } - - @Override - public SearchRequest getRequest() { - return searchRequest; - } - @Override public OriginalIndices getOriginalIndices(int shardIndex) { + var searchRequest = getRequest(); return new OriginalIndices(searchRequest.indices(), searchRequest.indicesOptions()); } @@ -122,8 +124,8 @@ public final class MockSearchPhaseContext implements SearchPhaseContext { } @Override - public Transport.Connection getConnection(String clusterAlias, String nodeId) { - return null; // null is ok here for this test + protected SearchPhase getNextPhase() { + return null; } @Override @@ -143,13 +145,13 @@ public final class MockSearchPhaseContext implements SearchPhaseContext { } @Override - public void addReleasable(Releasable releasable) { - releasables.add(releasable); - } - - @Override - public void execute(Runnable command) { - command.run(); + protected void executePhaseOnShard( + SearchShardIterator shardIt, + SearchShardTarget shard, + SearchActionListener listener + ) { + onShardResult(new SearchPhaseResult() { + }, shardIt); } @Override diff --git a/server/src/test/java/org/elasticsearch/action/search/RankFeaturePhaseTests.java b/server/src/test/java/org/elasticsearch/action/search/RankFeaturePhaseTests.java index a4201716d31e..3d104758b096 100644 --- a/server/src/test/java/org/elasticsearch/action/search/RankFeaturePhaseTests.java +++ b/server/src/test/java/org/elasticsearch/action/search/RankFeaturePhaseTests.java @@ -155,6 +155,7 @@ public class RankFeaturePhaseTests extends ESTestCase { rankFeaturePhase.rankPhaseResults.close(); } } finally { + mockSearchPhaseContext.results.close(); if (mockSearchPhaseContext.searchResponse.get() != null) { mockSearchPhaseContext.searchResponse.get().decRef(); } @@ -281,6 +282,7 @@ public class RankFeaturePhaseTests extends ESTestCase { rankFeaturePhase.rankPhaseResults.close(); } } finally { + mockSearchPhaseContext.results.close(); if (mockSearchPhaseContext.searchResponse.get() != null) { mockSearchPhaseContext.searchResponse.get().decRef(); } @@ -385,6 +387,7 @@ public class RankFeaturePhaseTests extends ESTestCase { rankFeaturePhase.rankPhaseResults.close(); } } finally { + mockSearchPhaseContext.results.close(); if (mockSearchPhaseContext.searchResponse.get() != null) { mockSearchPhaseContext.searchResponse.get().decRef(); } @@ -480,6 +483,7 @@ public class RankFeaturePhaseTests extends ESTestCase { rankFeaturePhase.rankPhaseResults.close(); } } finally { + mockSearchPhaseContext.results.close(); if (mockSearchPhaseContext.searchResponse.get() != null) { mockSearchPhaseContext.searchResponse.get().decRef(); } @@ -626,6 +630,7 @@ public class RankFeaturePhaseTests extends ESTestCase { rankFeaturePhase.rankPhaseResults.close(); } } finally { + mockSearchPhaseContext.results.close(); if (mockSearchPhaseContext.searchResponse.get() != null) { mockSearchPhaseContext.searchResponse.get().decRef(); } @@ -762,6 +767,7 @@ public class RankFeaturePhaseTests extends ESTestCase { rankFeaturePhase.rankPhaseResults.close(); } } finally { + mockSearchPhaseContext.results.close(); if (mockSearchPhaseContext.searchResponse.get() != null) { mockSearchPhaseContext.searchResponse.get().decRef(); } diff --git a/server/src/test/java/org/elasticsearch/action/support/master/TransportMasterNodeActionTests.java b/server/src/test/java/org/elasticsearch/action/support/master/TransportMasterNodeActionTests.java index 1167f9f77983..e11b0749dad4 100644 --- a/server/src/test/java/org/elasticsearch/action/support/master/TransportMasterNodeActionTests.java +++ b/server/src/test/java/org/elasticsearch/action/support/master/TransportMasterNodeActionTests.java @@ -576,7 +576,7 @@ public class TransportMasterNodeActionTests extends ESTestCase { // simulate master restart followed by a state recovery - this will reset the cluster state version final DiscoveryNodes.Builder nodesBuilder = DiscoveryNodes.builder(clusterService.state().nodes()); nodesBuilder.remove(masterNode); - masterNode = DiscoveryNodeUtils.create(masterNode.getId(), masterNode.getAddress(), masterNode.getVersion()); + masterNode = DiscoveryNodeUtils.create(masterNode.getId(), masterNode.getAddress(), masterNode.getVersionInformation()); nodesBuilder.add(masterNode); nodesBuilder.masterNodeId(masterNode.getId()); final ClusterState.Builder builder = ClusterState.builder(clusterService.state()).nodes(nodesBuilder); diff --git a/server/src/test/java/org/elasticsearch/cluster/metadata/AutoExpandReplicasTests.java b/server/src/test/java/org/elasticsearch/cluster/metadata/AutoExpandReplicasTests.java index 8adea7d648ad..0894ca76c018 100644 --- a/server/src/test/java/org/elasticsearch/cluster/metadata/AutoExpandReplicasTests.java +++ b/server/src/test/java/org/elasticsearch/cluster/metadata/AutoExpandReplicasTests.java @@ -9,7 +9,6 @@ package org.elasticsearch.cluster.metadata; import org.elasticsearch.TransportVersion; -import org.elasticsearch.Version; import org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteRequest; import org.elasticsearch.action.admin.indices.create.CreateIndexRequest; import org.elasticsearch.action.support.ActiveShardCount; @@ -98,15 +97,11 @@ public class AutoExpandReplicasTests extends ESTestCase { private static final AtomicInteger nodeIdGenerator = new AtomicInteger(); - protected DiscoveryNode createNode(Version version, DiscoveryNodeRole... mustHaveRoles) { + protected DiscoveryNode createNode(DiscoveryNodeRole... mustHaveRoles) { Set roles = new HashSet<>(randomSubsetOf(DiscoveryNodeRole.roles())); Collections.addAll(roles, mustHaveRoles); final String id = Strings.format("node_%03d", nodeIdGenerator.incrementAndGet()); - return DiscoveryNodeUtils.builder(id).name(id).roles(roles).version(version).build(); - } - - protected DiscoveryNode createNode(DiscoveryNodeRole... mustHaveRoles) { - return createNode(Version.CURRENT, mustHaveRoles); + return DiscoveryNodeUtils.builder(id).name(id).roles(roles).build(); } /** diff --git a/server/src/test/java/org/elasticsearch/cluster/node/DiscoveryNodeTests.java b/server/src/test/java/org/elasticsearch/cluster/node/DiscoveryNodeTests.java index 8924acf892d3..331b5d92ca94 100644 --- a/server/src/test/java/org/elasticsearch/cluster/node/DiscoveryNodeTests.java +++ b/server/src/test/java/org/elasticsearch/cluster/node/DiscoveryNodeTests.java @@ -247,7 +247,7 @@ public class DiscoveryNodeTests extends ESTestCase { assertThat(toString, containsString("{" + node.getEphemeralId() + "}")); assertThat(toString, containsString("{" + node.getAddress() + "}")); assertThat(toString, containsString("{IScdfhilmrstvw}"));// roles - assertThat(toString, containsString("{" + node.getVersion() + "}")); + assertThat(toString, containsString("{" + node.getBuildVersion() + "}")); assertThat(toString, containsString("{test-attr=val}"));// attributes } } diff --git a/server/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedNodeRoutingTests.java b/server/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedNodeRoutingTests.java index d829be2b34d8..32119f1c3dfa 100644 --- a/server/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedNodeRoutingTests.java +++ b/server/src/test/java/org/elasticsearch/cluster/routing/allocation/FailedNodeRoutingTests.java @@ -130,8 +130,7 @@ public class FailedNodeRoutingTests extends ESAllocationTestCase { // Log the node versions (for debugging if necessary) for (DiscoveryNode discoveryNode : state.nodes().getDataNodes().values()) { - Version nodeVer = discoveryNode.getVersion(); - logger.info("--> node [{}] has version [{}]", discoveryNode.getId(), nodeVer); + logger.info("--> node [{}] has version [{}]", discoveryNode.getId(), discoveryNode.getBuildVersion()); } // randomly create some indices diff --git a/server/src/test/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldMapperTests.java new file mode 100644 index 000000000000..6a890328732c --- /dev/null +++ b/server/src/test/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldMapperTests.java @@ -0,0 +1,506 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the "Elastic License + * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side + * Public License v 1"; you may not use this file except in compliance with, at + * your election, the "Elastic License 2.0", the "GNU Affero General Public + * License v3.0 only", or the "Server Side Public License, v 1". + */ + +package org.elasticsearch.index.mapper.vectors; + +import org.apache.lucene.document.BinaryDocValuesField; +import org.apache.lucene.index.IndexableField; +import org.apache.lucene.search.FieldExistsQuery; +import org.apache.lucene.search.Query; +import org.apache.lucene.util.BytesRef; +import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.common.xcontent.XContentHelper; +import org.elasticsearch.index.IndexVersion; +import org.elasticsearch.index.mapper.DocumentMapper; +import org.elasticsearch.index.mapper.DocumentParsingException; +import org.elasticsearch.index.mapper.LuceneDocument; +import org.elasticsearch.index.mapper.MappedFieldType; +import org.elasticsearch.index.mapper.MapperParsingException; +import org.elasticsearch.index.mapper.MapperService; +import org.elasticsearch.index.mapper.MapperTestCase; +import org.elasticsearch.index.mapper.ParsedDocument; +import org.elasticsearch.index.mapper.SourceToParse; +import org.elasticsearch.index.mapper.ValueFetcher; +import org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.ElementType; +import org.elasticsearch.index.query.SearchExecutionContext; +import org.elasticsearch.search.lookup.Source; +import org.elasticsearch.search.lookup.SourceProvider; +import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; +import org.junit.BeforeClass; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.FloatBuffer; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import java.util.stream.Stream; + +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.instanceOf; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class MultiDenseVectorFieldMapperTests extends MapperTestCase { + + @BeforeClass + public static void setup() { + assumeTrue("Requires multi-dense vector support", MultiDenseVectorFieldMapper.FEATURE_FLAG.isEnabled()); + } + + private final ElementType elementType; + private final int dims; + + public MultiDenseVectorFieldMapperTests() { + this.elementType = randomFrom(ElementType.BYTE, ElementType.FLOAT, ElementType.BIT); + this.dims = ElementType.BIT == elementType ? 4 * Byte.SIZE : 4; + } + + @Override + protected void minimalMapping(XContentBuilder b) throws IOException { + indexMapping(b, IndexVersion.current()); + } + + @Override + protected void minimalMapping(XContentBuilder b, IndexVersion indexVersion) throws IOException { + indexMapping(b, indexVersion); + } + + private void indexMapping(XContentBuilder b, IndexVersion indexVersion) throws IOException { + b.field("type", "multi_dense_vector").field("dims", dims); + if (elementType != ElementType.FLOAT) { + b.field("element_type", elementType.toString()); + } + } + + @Override + protected Object getSampleValueForDocument() { + int numVectors = randomIntBetween(1, 16); + return Stream.generate( + () -> elementType == ElementType.FLOAT ? List.of(0.5, 0.5, 0.5, 0.5) : List.of((byte) 1, (byte) 1, (byte) 1, (byte) 1) + ).limit(numVectors).toList(); + } + + @Override + protected void registerParameters(ParameterChecker checker) throws IOException { + checker.registerConflictCheck( + "dims", + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims)), + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims + 8)) + ); + checker.registerConflictCheck( + "element_type", + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims).field("element_type", "byte")), + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims).field("element_type", "float")) + ); + checker.registerConflictCheck( + "element_type", + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims).field("element_type", "float")), + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims * 8).field("element_type", "bit")) + ); + checker.registerConflictCheck( + "element_type", + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims).field("element_type", "byte")), + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", dims * 8).field("element_type", "bit")) + ); + } + + @Override + protected boolean supportsStoredFields() { + return false; + } + + @Override + protected boolean supportsIgnoreMalformed() { + return false; + } + + @Override + protected void assertSearchable(MappedFieldType fieldType) { + assertThat(fieldType, instanceOf(MultiDenseVectorFieldMapper.MultiDenseVectorFieldType.class)); + assertFalse(fieldType.isIndexed()); + assertFalse(fieldType.isSearchable()); + } + + protected void assertExistsQuery(MappedFieldType fieldType, Query query, LuceneDocument fields) { + assertThat(query, instanceOf(FieldExistsQuery.class)); + FieldExistsQuery existsQuery = (FieldExistsQuery) query; + assertEquals("field", existsQuery.getField()); + assertNoFieldNamesField(fields); + } + + // We override this because dense vectors are the only field type that are not aggregatable but + // that do provide fielddata. TODO: resolve this inconsistency! + @Override + public void testAggregatableConsistency() {} + + public void testDims() { + { + Exception e = expectThrows(MapperParsingException.class, () -> createMapperService(fieldMapping(b -> { + b.field("type", "multi_dense_vector"); + b.field("dims", 0); + }))); + assertThat( + e.getMessage(), + equalTo("Failed to parse mapping: " + "The number of dimensions should be in the range [1, 4096] but was [0]") + ); + } + // test max limit for non-indexed vectors + { + Exception e = expectThrows(MapperParsingException.class, () -> createMapperService(fieldMapping(b -> { + b.field("type", "multi_dense_vector"); + b.field("dims", 5000); + }))); + assertThat( + e.getMessage(), + equalTo("Failed to parse mapping: " + "The number of dimensions should be in the range [1, 4096] but was [5000]") + ); + } + } + + public void testMergeDims() throws IOException { + XContentBuilder mapping = mapping(b -> { + b.startObject("field"); + b.field("type", "multi_dense_vector"); + b.endObject(); + }); + MapperService mapperService = createMapperService(mapping); + + mapping = mapping(b -> { + b.startObject("field"); + b.field("type", "multi_dense_vector").field("dims", dims); + b.endObject(); + }); + merge(mapperService, mapping); + assertEquals( + XContentHelper.convertToMap(BytesReference.bytes(mapping), false, mapping.contentType()).v2(), + XContentHelper.convertToMap(mapperService.documentMapper().mappingSource().uncompressed(), false, mapping.contentType()).v2() + ); + } + + public void testLargeDimsBit() throws IOException { + createMapperService(fieldMapping(b -> { + b.field("type", "multi_dense_vector"); + b.field("dims", 1024 * Byte.SIZE); + b.field("element_type", ElementType.BIT.toString()); + })); + } + + public void testNonIndexedVector() throws Exception { + DocumentMapper mapper = createDocumentMapper(fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", 3))); + + float[][] validVectors = { { -12.1f, 100.7f, -4 }, { 42f, .05f, -1f } }; + double[] dotProduct = new double[2]; + int vecId = 0; + for (float[] vector : validVectors) { + for (float value : vector) { + dotProduct[vecId] += value * value; + } + vecId++; + } + ParsedDocument doc1 = mapper.parse(source(b -> { + b.startArray("field"); + for (float[] vector : validVectors) { + b.startArray(); + for (float value : vector) { + b.value(value); + } + b.endArray(); + } + b.endArray(); + })); + + List fields = doc1.rootDoc().getFields("field"); + assertEquals(1, fields.size()); + assertThat(fields.get(0), instanceOf(BinaryDocValuesField.class)); + // assert that after decoding the indexed value is equal to expected + BytesRef vectorBR = fields.get(0).binaryValue(); + assertEquals(ElementType.FLOAT.getNumBytes(validVectors[0].length) * validVectors.length, vectorBR.length); + float[][] decodedValues = new float[validVectors.length][]; + for (int i = 0; i < validVectors.length; i++) { + decodedValues[i] = new float[validVectors[i].length]; + FloatBuffer fb = ByteBuffer.wrap(vectorBR.bytes, i * Float.BYTES * validVectors[i].length, Float.BYTES * validVectors[i].length) + .order(ByteOrder.LITTLE_ENDIAN) + .asFloatBuffer(); + fb.get(decodedValues[i]); + } + List magFields = doc1.rootDoc().getFields("field" + MultiDenseVectorFieldMapper.VECTOR_MAGNITUDES_SUFFIX); + assertEquals(1, magFields.size()); + assertThat(magFields.get(0), instanceOf(BinaryDocValuesField.class)); + BytesRef magBR = magFields.get(0).binaryValue(); + assertEquals(Float.BYTES * validVectors.length, magBR.length); + FloatBuffer fb = ByteBuffer.wrap(magBR.bytes, magBR.offset, magBR.length).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer(); + for (int i = 0; i < validVectors.length; i++) { + assertEquals((float) Math.sqrt(dotProduct[i]), fb.get(), 0.001f); + } + for (int i = 0; i < validVectors.length; i++) { + assertArrayEquals("Decoded dense vector values is not equal to the indexed one.", validVectors[i], decodedValues[i], 0.001f); + } + } + + public void testPoorlyIndexedVector() throws Exception { + DocumentMapper mapper = createDocumentMapper(fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", 3))); + + float[][] validVectors = { { -12.1f, 100.7f, -4 }, { 42f, .05f, -1f } }; + double[] dotProduct = new double[2]; + int vecId = 0; + for (float[] vector : validVectors) { + for (float value : vector) { + dotProduct[vecId] += value * value; + } + vecId++; + } + expectThrows(DocumentParsingException.class, () -> mapper.parse(source(b -> { + b.startArray("field"); + b.startArray(); // double nested array should fail + for (float[] vector : validVectors) { + b.startArray(); + for (float value : vector) { + b.value(value); + } + b.endArray(); + } + b.endArray(); + b.endArray(); + }))); + } + + public void testInvalidParameters() { + + MapperParsingException e = expectThrows( + MapperParsingException.class, + () -> createDocumentMapper( + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", 3).field("element_type", "foo")) + ) + ); + assertThat(e.getMessage(), containsString("invalid element_type [foo]; available types are ")); + e = expectThrows( + MapperParsingException.class, + () -> createDocumentMapper( + fieldMapping(b -> b.field("type", "multi_dense_vector").field("dims", 3).startObject("foo").endObject()) + ) + ); + assertThat( + e.getMessage(), + containsString("Failed to parse mapping: unknown parameter [foo] on mapper [field] of type [multi_dense_vector]") + ); + } + + public void testDocumentsWithIncorrectDims() throws Exception { + int dims = 3; + XContentBuilder fieldMapping = fieldMapping(b -> { + b.field("type", "multi_dense_vector"); + b.field("dims", dims); + }); + + DocumentMapper mapper = createDocumentMapper(fieldMapping); + + // test that error is thrown when a document has number of dims more than defined in the mapping + float[][] invalidVector = new float[4][dims + 1]; + DocumentParsingException e = expectThrows(DocumentParsingException.class, () -> mapper.parse(source(b -> { + b.startArray("field"); + for (float[] vector : invalidVector) { + b.startArray(); + for (float value : vector) { + b.value(value); + } + b.endArray(); + } + b.endArray(); + }))); + assertThat(e.getCause().getMessage(), containsString("has more dimensions than defined in the mapping [3]")); + + // test that error is thrown when a document has number of dims less than defined in the mapping + float[][] invalidVector2 = new float[4][dims - 1]; + DocumentParsingException e2 = expectThrows(DocumentParsingException.class, () -> mapper.parse(source(b -> { + b.startArray("field"); + for (float[] vector : invalidVector2) { + b.startArray(); + for (float value : vector) { + b.value(value); + } + b.endArray(); + } + b.endArray(); + }))); + assertThat(e2.getCause().getMessage(), containsString("has a different number of dimensions [2] than defined in the mapping [3]")); + // test that error is thrown when some of the vectors have correct number of dims, but others do not + DocumentParsingException e3 = expectThrows(DocumentParsingException.class, () -> mapper.parse(source(b -> { + b.startArray("field"); + for (float[] vector : new float[4][dims]) { + b.startArray(); + for (float value : vector) { + b.value(value); + } + b.endArray(); + } + for (float[] vector : invalidVector2) { + b.startArray(); + for (float value : vector) { + b.value(value); + } + b.endArray(); + } + b.endArray(); + }))); + assertThat(e3.getCause().getMessage(), containsString("has a different number of dimensions [2] than defined in the mapping [3]")); + } + + @Override + protected void assertFetchMany(MapperService mapperService, String field, Object value, String format, int count) throws IOException { + assumeFalse("Dense vectors currently don't support multiple values in the same field", false); + } + + /** + * Dense vectors don't support doc values or string representation (for doc value parser/fetching). + * We may eventually support that, but until then, we only verify that the parsing and fields fetching matches the provided value object + */ + @Override + protected void assertFetch(MapperService mapperService, String field, Object value, String format) throws IOException { + MappedFieldType ft = mapperService.fieldType(field); + MappedFieldType.FielddataOperation fdt = MappedFieldType.FielddataOperation.SEARCH; + SourceToParse source = source(b -> b.field(ft.name(), value)); + SearchExecutionContext searchExecutionContext = mock(SearchExecutionContext.class); + when(searchExecutionContext.isSourceEnabled()).thenReturn(true); + when(searchExecutionContext.sourcePath(field)).thenReturn(Set.of(field)); + when(searchExecutionContext.getForField(ft, fdt)).thenAnswer(inv -> fieldDataLookup(mapperService).apply(ft, () -> { + throw new UnsupportedOperationException(); + }, fdt)); + ValueFetcher nativeFetcher = ft.valueFetcher(searchExecutionContext, format); + ParsedDocument doc = mapperService.documentMapper().parse(source); + withLuceneIndex(mapperService, iw -> iw.addDocuments(doc.docs()), ir -> { + Source s = SourceProvider.fromStoredFields().getSource(ir.leaves().get(0), 0); + nativeFetcher.setNextReader(ir.leaves().get(0)); + List fromNative = nativeFetcher.fetchValues(s, 0, new ArrayList<>()); + MultiDenseVectorFieldMapper.MultiDenseVectorFieldType denseVectorFieldType = + (MultiDenseVectorFieldMapper.MultiDenseVectorFieldType) ft; + switch (denseVectorFieldType.getElementType()) { + case BYTE -> assumeFalse("byte element type testing not currently added", false); + case FLOAT -> { + List fetchedFloatsList = new ArrayList<>(); + for (var f : fromNative) { + float[] fetchedFloats = new float[denseVectorFieldType.getVectorDimensions()]; + assert f instanceof List; + List vector = (List) f; + int i = 0; + for (Object v : vector) { + assert v instanceof Number; + fetchedFloats[i++] = ((Number) v).floatValue(); + } + fetchedFloatsList.add(fetchedFloats); + } + float[][] fetchedFloats = fetchedFloatsList.toArray(new float[0][]); + assertThat("fetching " + value, fetchedFloats, equalTo(value)); + } + } + }); + } + + @Override + protected void randomFetchTestFieldConfig(XContentBuilder b) throws IOException { + b.field("type", "multi_dense_vector").field("dims", randomIntBetween(2, 4096)).field("element_type", "float"); + } + + @Override + protected Object generateRandomInputValue(MappedFieldType ft) { + MultiDenseVectorFieldMapper.MultiDenseVectorFieldType vectorFieldType = (MultiDenseVectorFieldMapper.MultiDenseVectorFieldType) ft; + int numVectors = randomIntBetween(1, 16); + return switch (vectorFieldType.getElementType()) { + case BYTE -> { + byte[][] vectors = new byte[numVectors][vectorFieldType.getVectorDimensions()]; + for (int i = 0; i < numVectors; i++) { + vectors[i] = randomByteArrayOfLength(vectorFieldType.getVectorDimensions()); + } + yield vectors; + } + case FLOAT -> { + float[][] vectors = new float[numVectors][vectorFieldType.getVectorDimensions()]; + for (int i = 0; i < numVectors; i++) { + for (int j = 0; j < vectorFieldType.getVectorDimensions(); j++) { + vectors[i][j] = randomFloat(); + } + } + yield vectors; + } + case BIT -> { + byte[][] vectors = new byte[numVectors][vectorFieldType.getVectorDimensions() / 8]; + for (int i = 0; i < numVectors; i++) { + vectors[i] = randomByteArrayOfLength(vectorFieldType.getVectorDimensions() / 8); + } + yield vectors; + } + }; + } + + public void testCannotBeUsedInMultifields() { + Exception e = expectThrows(MapperParsingException.class, () -> createMapperService(fieldMapping(b -> { + b.field("type", "keyword"); + b.startObject("fields"); + b.startObject("vectors"); + minimalMapping(b); + b.endObject(); + b.endObject(); + }))); + assertThat(e.getMessage(), containsString("Field [vectors] of type [multi_dense_vector] can't be used in multifields")); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport(boolean ignoreMalformed) { + return new DenseVectorSyntheticSourceSupport(); + } + + @Override + protected boolean supportsEmptyInputArray() { + return false; + } + + private static class DenseVectorSyntheticSourceSupport implements SyntheticSourceSupport { + private final int dims = between(5, 1000); + private final int numVecs = between(1, 16); + private final ElementType elementType = randomFrom(ElementType.BYTE, ElementType.FLOAT, ElementType.BIT); + + @Override + public SyntheticSourceExample example(int maxValues) { + Object value = switch (elementType) { + case BYTE, BIT: + yield randomList(numVecs, numVecs, () -> randomList(dims, dims, ESTestCase::randomByte)); + case FLOAT: + yield randomList(numVecs, numVecs, () -> randomList(dims, dims, ESTestCase::randomFloat)); + }; + return new SyntheticSourceExample(value, value, this::mapping); + } + + private void mapping(XContentBuilder b) throws IOException { + b.field("type", "multi_dense_vector"); + if (elementType == ElementType.BYTE || elementType == ElementType.BIT || randomBoolean()) { + b.field("element_type", elementType.toString()); + } + b.field("dims", elementType == ElementType.BIT ? dims * Byte.SIZE : dims); + } + + @Override + public List invalidExample() { + return List.of(); + } + } + + @Override + public void testSyntheticSourceKeepArrays() { + // The mapper expects to parse an array of values by default, it's not compatible with array of arrays. + } +} diff --git a/server/src/test/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldTypeTests.java b/server/src/test/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldTypeTests.java new file mode 100644 index 000000000000..14cc63e31fa2 --- /dev/null +++ b/server/src/test/java/org/elasticsearch/index/mapper/vectors/MultiDenseVectorFieldTypeTests.java @@ -0,0 +1,105 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the "Elastic License + * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side + * Public License v 1"; you may not use this file except in compliance with, at + * your election, the "Elastic License 2.0", the "GNU Affero General Public + * License v3.0 only", or the "Server Side Public License, v 1". + */ + +package org.elasticsearch.index.mapper.vectors; + +import org.elasticsearch.index.IndexVersion; +import org.elasticsearch.index.fielddata.FieldDataContext; +import org.elasticsearch.index.mapper.FieldTypeTestCase; +import org.elasticsearch.index.mapper.MappedFieldType; +import org.elasticsearch.index.mapper.vectors.MultiDenseVectorFieldMapper.MultiDenseVectorFieldType; +import org.junit.BeforeClass; + +import java.io.IOException; +import java.util.Collections; +import java.util.List; +import java.util.Set; + +import static org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper.BBQ_MIN_DIMS; + +public class MultiDenseVectorFieldTypeTests extends FieldTypeTestCase { + + @BeforeClass + public static void setup() { + assumeTrue("Requires multi-dense vector support", MultiDenseVectorFieldMapper.FEATURE_FLAG.isEnabled()); + } + + private MultiDenseVectorFieldType createFloatFieldType() { + return new MultiDenseVectorFieldType( + "f", + DenseVectorFieldMapper.ElementType.FLOAT, + BBQ_MIN_DIMS, + IndexVersion.current(), + Collections.emptyMap() + ); + } + + private MultiDenseVectorFieldType createByteFieldType() { + return new MultiDenseVectorFieldType( + "f", + DenseVectorFieldMapper.ElementType.BYTE, + 5, + IndexVersion.current(), + Collections.emptyMap() + ); + } + + public void testHasDocValues() { + MultiDenseVectorFieldType fft = createFloatFieldType(); + assertTrue(fft.hasDocValues()); + MultiDenseVectorFieldType bft = createByteFieldType(); + assertTrue(bft.hasDocValues()); + } + + public void testIsIndexed() { + MultiDenseVectorFieldType fft = createFloatFieldType(); + assertFalse(fft.isIndexed()); + MultiDenseVectorFieldType bft = createByteFieldType(); + assertFalse(bft.isIndexed()); + } + + public void testIsSearchable() { + MultiDenseVectorFieldType fft = createFloatFieldType(); + assertFalse(fft.isSearchable()); + MultiDenseVectorFieldType bft = createByteFieldType(); + assertFalse(bft.isSearchable()); + } + + public void testIsAggregatable() { + MultiDenseVectorFieldType fft = createFloatFieldType(); + assertFalse(fft.isAggregatable()); + MultiDenseVectorFieldType bft = createByteFieldType(); + assertFalse(bft.isAggregatable()); + } + + public void testFielddataBuilder() { + MultiDenseVectorFieldType fft = createFloatFieldType(); + FieldDataContext fdc = new FieldDataContext("test", null, () -> null, Set::of, MappedFieldType.FielddataOperation.SCRIPT); + assertNotNull(fft.fielddataBuilder(fdc)); + + MultiDenseVectorFieldType bft = createByteFieldType(); + FieldDataContext bdc = new FieldDataContext("test", null, () -> null, Set::of, MappedFieldType.FielddataOperation.SCRIPT); + assertNotNull(bft.fielddataBuilder(bdc)); + } + + public void testDocValueFormat() { + MultiDenseVectorFieldType fft = createFloatFieldType(); + expectThrows(IllegalArgumentException.class, () -> fft.docValueFormat(null, null)); + MultiDenseVectorFieldType bft = createByteFieldType(); + expectThrows(IllegalArgumentException.class, () -> bft.docValueFormat(null, null)); + } + + public void testFetchSourceValue() throws IOException { + MultiDenseVectorFieldType fft = createFloatFieldType(); + List> vector = List.of(List.of(0.0, 1.0, 2.0, 3.0, 4.0, 6.0)); + assertEquals(vector, fetchSourceValue(fft, vector)); + MultiDenseVectorFieldType bft = createByteFieldType(); + assertEquals(vector, fetchSourceValue(bft, vector)); + } +} diff --git a/server/src/test/java/org/elasticsearch/reservedstate/service/FileSettingsServiceTests.java b/server/src/test/java/org/elasticsearch/reservedstate/service/FileSettingsServiceTests.java index aa76245c2067..0db29588c429 100644 --- a/server/src/test/java/org/elasticsearch/reservedstate/service/FileSettingsServiceTests.java +++ b/server/src/test/java/org/elasticsearch/reservedstate/service/FileSettingsServiceTests.java @@ -38,6 +38,7 @@ import org.elasticsearch.test.ClusterServiceUtils; import org.elasticsearch.test.ESTestCase; import org.elasticsearch.threadpool.TestThreadPool; import org.elasticsearch.threadpool.ThreadPool; +import org.elasticsearch.xcontent.XContentParseException; import org.elasticsearch.xcontent.XContentParser; import org.junit.After; import org.junit.Before; @@ -55,16 +56,22 @@ import java.time.ZoneOffset; import java.util.List; import java.util.Map; import java.util.Set; +import java.util.concurrent.BrokenBarrierException; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Consumer; import static java.nio.file.StandardCopyOption.ATOMIC_MOVE; +import static java.nio.file.StandardCopyOption.REPLACE_EXISTING; import static org.elasticsearch.node.Node.NODE_NAME_SETTING; import static org.hamcrest.Matchers.anEmptyMap; import static org.hamcrest.Matchers.hasEntry; import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.argThat; import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.mock; @@ -262,6 +269,68 @@ public class FileSettingsServiceTests extends ESTestCase { verify(controller, times(1)).process(any(), any(XContentParser.class), eq(ReservedStateVersionCheck.HIGHER_VERSION_ONLY), any()); } + @SuppressWarnings("unchecked") + public void testInvalidJSON() throws Exception { + doAnswer((Answer) invocation -> { + invocation.getArgument(1, XContentParser.class).map(); // Throw if JSON is invalid + ((Consumer) invocation.getArgument(3)).accept(null); + return null; + }).when(controller).process(any(), any(XContentParser.class), any(), any()); + + CyclicBarrier fileChangeBarrier = new CyclicBarrier(2); + fileSettingsService.addFileChangedListener(() -> awaitOrBust(fileChangeBarrier)); + + Files.createDirectories(fileSettingsService.watchedFileDir()); + // contents of the JSON don't matter, we just need a file to exist + writeTestFile(fileSettingsService.watchedFile(), "{}"); + + doAnswer((Answer) invocation -> { + boolean returnedNormally = false; + try { + var result = invocation.callRealMethod(); + returnedNormally = true; + return result; + } catch (XContentParseException e) { + // We're expecting a parse error. processFileChanges specifies that this is supposed to throw ExecutionException. + throw new ExecutionException(e); + } catch (Throwable e) { + throw new AssertionError("Unexpected exception", e); + } finally { + if (returnedNormally == false) { + // Because of the exception, listeners aren't notified, so we need to activate the barrier ourselves + awaitOrBust(fileChangeBarrier); + } + } + }).when(fileSettingsService).processFileChanges(); + + // Establish the initial valid JSON + fileSettingsService.start(); + fileSettingsService.clusterChanged(new ClusterChangedEvent("test", clusterService.state(), ClusterState.EMPTY_STATE)); + awaitOrBust(fileChangeBarrier); + + // Now break the JSON + writeTestFile(fileSettingsService.watchedFile(), "test_invalid_JSON"); + awaitOrBust(fileChangeBarrier); + + verify(fileSettingsService, times(1)).processFileOnServiceStart(); // The initial state + verify(fileSettingsService, times(1)).processFileChanges(); // The changed state + verify(fileSettingsService, times(1)).onProcessFileChangesException( + argThat(e -> e instanceof ExecutionException && e.getCause() instanceof XContentParseException) + ); + + // Note: the name "processFileOnServiceStart" is a bit misleading because it is not + // referring to fileSettingsService.start(). Rather, it is referring to the initialization + // of the watcher thread itself, which occurs asynchronously when clusterChanged is first called. + } + + private static void awaitOrBust(CyclicBarrier barrier) { + try { + barrier.await(20, TimeUnit.SECONDS); + } catch (InterruptedException | BrokenBarrierException | TimeoutException e) { + throw new AssertionError("Unexpected exception waiting for barrier", e); + } + } + @SuppressWarnings("unchecked") public void testStopWorksInMiddleOfProcessing() throws Exception { CountDownLatch processFileLatch = new CountDownLatch(1); @@ -356,10 +425,10 @@ public class FileSettingsServiceTests extends ESTestCase { Path tempFilePath = createTempFile(); Files.writeString(tempFilePath, contents); try { - Files.move(tempFilePath, path, ATOMIC_MOVE); + Files.move(tempFilePath, path, REPLACE_EXISTING, ATOMIC_MOVE); } catch (AtomicMoveNotSupportedException e) { logger.info("Atomic move not available. Falling back on non-atomic move to write [{}]", path.toAbsolutePath()); - Files.move(tempFilePath, path); + Files.move(tempFilePath, path, REPLACE_EXISTING); } } @@ -374,4 +443,5 @@ public class FileSettingsServiceTests extends ESTestCase { fail(e, "longAwait: interrupted waiting for CountDownLatch to reach zero"); } } + } diff --git a/server/src/test/java/org/elasticsearch/rest/action/document/RestIndexActionTests.java b/server/src/test/java/org/elasticsearch/rest/action/document/RestIndexActionTests.java index 1aa53382666e..c97160427e59 100644 --- a/server/src/test/java/org/elasticsearch/rest/action/document/RestIndexActionTests.java +++ b/server/src/test/java/org/elasticsearch/rest/action/document/RestIndexActionTests.java @@ -10,7 +10,6 @@ package org.elasticsearch.rest.action.document; import org.apache.lucene.util.SetOnce; -import org.elasticsearch.Version; import org.elasticsearch.action.DocWriteRequest; import org.elasticsearch.action.index.IndexRequest; import org.elasticsearch.action.index.IndexResponse; @@ -55,10 +54,10 @@ public final class RestIndexActionTests extends RestActionTestCase { } public void testAutoIdDefaultsToOptypeCreate() { - checkAutoIdOpType(Version.CURRENT, DocWriteRequest.OpType.CREATE); + checkAutoIdOpType(DocWriteRequest.OpType.CREATE); } - private void checkAutoIdOpType(Version minClusterVersion, DocWriteRequest.OpType expectedOpType) { + private void checkAutoIdOpType(DocWriteRequest.OpType expectedOpType) { SetOnce executeCalled = new SetOnce<>(); verifyingClient.setExecuteVerifier((actionType, request) -> { assertThat(request, instanceOf(IndexRequest.class)); @@ -71,9 +70,7 @@ public final class RestIndexActionTests extends RestActionTestCase { .withContent(new BytesArray("{}"), XContentType.JSON) .build(); clusterStateSupplier.set( - ClusterState.builder(ClusterName.DEFAULT) - .nodes(DiscoveryNodes.builder().add(DiscoveryNodeUtils.builder("test").version(minClusterVersion).build()).build()) - .build() + ClusterState.builder(ClusterName.DEFAULT).nodes(DiscoveryNodes.builder().add(DiscoveryNodeUtils.create("test")).build()).build() ); dispatchRequest(autoIdRequest); assertThat(executeCalled.get(), equalTo(true)); diff --git a/test/framework/src/main/java/org/elasticsearch/cluster/ESAllocationTestCase.java b/test/framework/src/main/java/org/elasticsearch/cluster/ESAllocationTestCase.java index c8d66f389dab..a1718e956800 100644 --- a/test/framework/src/main/java/org/elasticsearch/cluster/ESAllocationTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/cluster/ESAllocationTestCase.java @@ -194,7 +194,7 @@ public abstract class ESAllocationTestCase extends ESTestCase { protected static Set MASTER_DATA_ROLES = Set.of(DiscoveryNodeRole.MASTER_ROLE, DiscoveryNodeRole.DATA_ROLE); protected static DiscoveryNode newNode(String nodeId) { - return newNode(nodeId, (Version) null); + return DiscoveryNodeUtils.builder(nodeId).roles(MASTER_DATA_ROLES).build(); } protected static DiscoveryNode newNode(String nodeName, String nodeId, Map attributes) { diff --git a/test/framework/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodeUtils.java b/test/framework/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodeUtils.java index dba7f28db3f4..64f8fa88762b 100644 --- a/test/framework/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodeUtils.java +++ b/test/framework/src/main/java/org/elasticsearch/cluster/node/DiscoveryNodeUtils.java @@ -13,6 +13,7 @@ import org.elasticsearch.Version; import org.elasticsearch.common.UUIDs; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.transport.TransportAddress; +import org.elasticsearch.env.BuildVersion; import org.elasticsearch.index.IndexVersion; import org.elasticsearch.node.Node; @@ -36,10 +37,15 @@ public class DiscoveryNodeUtils { return builder(id).address(address).build(); } + @Deprecated public static DiscoveryNode create(String id, TransportAddress address, Version version) { return builder(id).address(address).version(version).build(); } + public static DiscoveryNode create(String id, TransportAddress address, VersionInformation version) { + return builder(id).address(address).version(version).build(); + } + public static DiscoveryNode create(String id, TransportAddress address, Map attributes, Set roles) { return builder(id).address(address).attributes(attributes).roles(roles).build(); } @@ -67,6 +73,7 @@ public class DiscoveryNodeUtils { private TransportAddress address; private Map attributes = Map.of(); private Set roles = DiscoveryNodeRole.roles(); + private BuildVersion buildVersion; private Version version; private IndexVersion minIndexVersion; private IndexVersion maxIndexVersion; @@ -107,19 +114,33 @@ public class DiscoveryNodeUtils { return this; } + @Deprecated public Builder version(Version version) { this.version = version; return this; } + @Deprecated public Builder version(Version version, IndexVersion minIndexVersion, IndexVersion maxIndexVersion) { + this.buildVersion = BuildVersion.fromVersionId(version.id()); this.version = version; this.minIndexVersion = minIndexVersion; this.maxIndexVersion = maxIndexVersion; return this; } + public Builder version(BuildVersion version, IndexVersion minIndexVersion, IndexVersion maxIndexVersion) { + // see comment in VersionInformation + assert version.equals(BuildVersion.current()); + this.buildVersion = version; + this.version = Version.CURRENT; + this.minIndexVersion = minIndexVersion; + this.maxIndexVersion = maxIndexVersion; + return this; + } + public Builder version(VersionInformation versions) { + this.buildVersion = versions.buildVersion(); this.version = versions.nodeVersion(); this.minIndexVersion = versions.minIndexVersion(); this.maxIndexVersion = versions.maxIndexVersion(); @@ -152,7 +173,7 @@ public class DiscoveryNodeUtils { if (minIndexVersion == null || maxIndexVersion == null) { versionInfo = VersionInformation.inferVersions(version); } else { - versionInfo = new VersionInformation(version, minIndexVersion, maxIndexVersion); + versionInfo = new VersionInformation(buildVersion, version, minIndexVersion, maxIndexVersion); } return new DiscoveryNode(name, id, ephemeralId, hostName, hostAddress, address, attributes, roles, versionInfo, externalId); diff --git a/test/framework/src/main/java/org/elasticsearch/index/mapper/TextFieldFamilySyntheticSourceTestSetup.java b/test/framework/src/main/java/org/elasticsearch/index/mapper/TextFieldFamilySyntheticSourceTestSetup.java index 97ded7f9a06f..475bf9212e1c 100644 --- a/test/framework/src/main/java/org/elasticsearch/index/mapper/TextFieldFamilySyntheticSourceTestSetup.java +++ b/test/framework/src/main/java/org/elasticsearch/index/mapper/TextFieldFamilySyntheticSourceTestSetup.java @@ -39,7 +39,7 @@ public final class TextFieldFamilySyntheticSourceTestSetup { TextFieldMapper.TextFieldType text = (TextFieldMapper.TextFieldType) ft; boolean supportsColumnAtATimeReader = text.syntheticSourceDelegate() != null && text.syntheticSourceDelegate().hasDocValues() - && text.canUseSyntheticSourceDelegateForQuerying(); + && text.canUseSyntheticSourceDelegateForLoading(); return new MapperTestCase.BlockReaderSupport(supportsColumnAtATimeReader, mapper, loaderFieldName); } MappedFieldType parent = mapper.fieldType(parentName); diff --git a/test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java b/test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java index d6709b00b4db..7cd2e6e1cc82 100644 --- a/test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/search/aggregations/AggregatorTestCase.java @@ -111,6 +111,7 @@ import org.elasticsearch.index.mapper.RangeType; import org.elasticsearch.index.mapper.TextFieldMapper; import org.elasticsearch.index.mapper.TimeSeriesIdFieldMapper; import org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapper; +import org.elasticsearch.index.mapper.vectors.MultiDenseVectorFieldMapper; import org.elasticsearch.index.mapper.vectors.SparseVectorFieldMapper; import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.index.shard.IndexShard; @@ -202,6 +203,7 @@ public abstract class AggregatorTestCase extends ESTestCase { private static final List TYPE_TEST_BLACKLIST = List.of( ObjectMapper.CONTENT_TYPE, // Cannot aggregate objects DenseVectorFieldMapper.CONTENT_TYPE, // Cannot aggregate dense vectors + MultiDenseVectorFieldMapper.CONTENT_TYPE, // Cannot aggregate dense vectors SparseVectorFieldMapper.CONTENT_TYPE, // Sparse vectors are no longer supported NestedObjectMapper.CONTENT_TYPE, // TODO support for nested diff --git a/x-pack/plugin/build.gradle b/x-pack/plugin/build.gradle index 8b920ac11cee..1750ccbb8c0c 100644 --- a/x-pack/plugin/build.gradle +++ b/x-pack/plugin/build.gradle @@ -88,5 +88,7 @@ tasks.named("yamlRestCompatTestTransform").configure({ task -> task.skipTest("esql/60_usage/Basic ESQL usage output (telemetry) non-snapshot version", "The number of functions is constantly increasing") task.skipTest("esql/80_text/reverse text", "The output type changed from TEXT to KEYWORD.") task.skipTest("esql/80_text/values function", "The output type changed from TEXT to KEYWORD.") + task.skipTest("privileges/11_builtin/Test get builtin privileges" ,"unnecessary to test compatibility") + task.skipTest("enrich/10_basic/Test using the deprecated elasticsearch_version field results in a warning", "The deprecation message was changed") }) diff --git a/x-pack/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/repository/CcrRepository.java b/x-pack/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/repository/CcrRepository.java index dc1addd1a61e..0bc72c8f7821 100644 --- a/x-pack/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/repository/CcrRepository.java +++ b/x-pack/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/repository/CcrRepository.java @@ -216,8 +216,7 @@ public class CcrRepository extends AbstractLifecycleComponent implements Reposit if (IndexVersion.current().equals(maxIndexVersion)) { for (var node : response.nodes()) { if (node.canContainData() && node.getMaxIndexVersion().equals(maxIndexVersion)) { - // TODO: Revisit when looking into removing release version from DiscoveryNode - BuildVersion remoteVersion = BuildVersion.fromVersionId(node.getVersion().id); + BuildVersion remoteVersion = node.getBuildVersion(); if (remoteVersion.isFutureVersion()) { throw new SnapshotException( snapshot, diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/enrich/EnrichPolicy.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/enrich/EnrichPolicy.java index 916bd3c62a59..9bbe41b4797f 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/enrich/EnrichPolicy.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/enrich/EnrichPolicy.java @@ -36,7 +36,7 @@ import java.util.Objects; public final class EnrichPolicy implements Writeable, ToXContentFragment { private static final String ELASTICEARCH_VERSION_DEPRECATION_MESSAGE = - "the [elasticsearch_version] field of an enrich policy has no effect and will be removed in Elasticsearch 9.0"; + "the [elasticsearch_version] field of an enrich policy has no effect and will be removed in a future version of Elasticsearch"; private static final DeprecationLogger deprecationLogger = DeprecationLogger.getLogger(EnrichPolicy.class); diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/user/GetUserPrivilegesResponse.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/user/GetUserPrivilegesResponse.java index de351cd59c69..763ab6ccb988 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/user/GetUserPrivilegesResponse.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/user/GetUserPrivilegesResponse.java @@ -115,7 +115,7 @@ public final class GetUserPrivilegesResponse extends ActionResponse { } public boolean hasRemoteClusterPrivileges() { - return remoteClusterPermissions.hasPrivileges(); + return remoteClusterPermissions.hasAnyPrivileges(); } @Override diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authc/Authentication.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authc/Authentication.java index 04dda7569220..c2f40a3e393b 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authc/Authentication.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authc/Authentication.java @@ -36,6 +36,7 @@ import org.elasticsearch.xpack.core.security.authc.file.FileRealmSettings; import org.elasticsearch.xpack.core.security.authc.service.ServiceAccountSettings; import org.elasticsearch.xpack.core.security.authc.support.AuthenticationContextSerializer; import org.elasticsearch.xpack.core.security.authz.RoleDescriptor; +import org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions; import org.elasticsearch.xpack.core.security.user.AnonymousUser; import org.elasticsearch.xpack.core.security.user.InternalUser; import org.elasticsearch.xpack.core.security.user.InternalUsers; @@ -76,6 +77,7 @@ import static org.elasticsearch.xpack.core.security.authc.AuthenticationField.CR import static org.elasticsearch.xpack.core.security.authc.AuthenticationField.FALLBACK_REALM_NAME; import static org.elasticsearch.xpack.core.security.authc.AuthenticationField.FALLBACK_REALM_TYPE; import static org.elasticsearch.xpack.core.security.authc.RealmDomain.REALM_DOMAIN_PARSER; +import static org.elasticsearch.xpack.core.security.authz.RoleDescriptor.Fields.REMOTE_CLUSTER; import static org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions.ROLE_REMOTE_CLUSTER_PRIVS; /** @@ -233,8 +235,8 @@ public final class Authentication implements ToXContentObject { + "]" ); } - final Map newMetadata = maybeRewriteMetadata(olderVersion, this); + final Authentication newAuthentication; if (isRunAs()) { // The lookup user for run-as currently doesn't have authentication metadata associated with them because @@ -272,12 +274,23 @@ public final class Authentication implements ToXContentObject { } private static Map maybeRewriteMetadata(TransportVersion olderVersion, Authentication authentication) { - if (authentication.isAuthenticatedAsApiKey()) { - return maybeRewriteMetadataForApiKeyRoleDescriptors(olderVersion, authentication); - } else if (authentication.isCrossClusterAccess()) { - return maybeRewriteMetadataForCrossClusterAccessAuthentication(olderVersion, authentication); - } else { - return authentication.getAuthenticatingSubject().getMetadata(); + try { + if (authentication.isAuthenticatedAsApiKey()) { + return maybeRewriteMetadataForApiKeyRoleDescriptors(olderVersion, authentication); + } else if (authentication.isCrossClusterAccess()) { + return maybeRewriteMetadataForCrossClusterAccessAuthentication(olderVersion, authentication); + } else { + return authentication.getAuthenticatingSubject().getMetadata(); + } + } catch (Exception e) { + // CCS workflows may swallow the exception message making this difficult to troubleshoot, so we explicitly log and re-throw + // here. It may result in duplicate logs, so we only log the message at warn level. + if (logger.isDebugEnabled()) { + logger.debug("Un-expected exception thrown while rewriting metadata. This is likely a bug.", e); + } else { + logger.warn("Un-expected exception thrown while rewriting metadata. This is likely a bug [" + e.getMessage() + "]"); + } + throw e; } } @@ -1323,6 +1336,7 @@ public final class Authentication implements ToXContentObject { if (authentication.getEffectiveSubject().getTransportVersion().onOrAfter(ROLE_REMOTE_CLUSTER_PRIVS) && streamVersion.before(ROLE_REMOTE_CLUSTER_PRIVS)) { + // the authentication understands the remote_cluster field but the stream does not metadata = new HashMap<>(metadata); metadata.put( AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY, @@ -1336,7 +1350,26 @@ public final class Authentication implements ToXContentObject { (BytesReference) metadata.get(AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY) ) ); - } + } else if (authentication.getEffectiveSubject().getTransportVersion().onOrAfter(ROLE_REMOTE_CLUSTER_PRIVS) + && streamVersion.onOrAfter(ROLE_REMOTE_CLUSTER_PRIVS)) { + // both the authentication object and the stream understand the remote_cluster field + // check each individual permission and remove as needed + metadata = new HashMap<>(metadata); + metadata.put( + AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY, + maybeRemoveRemoteClusterPrivilegesFromRoleDescriptors( + (BytesReference) metadata.get(AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY), + streamVersion + ) + ); + metadata.put( + AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY, + maybeRemoveRemoteClusterPrivilegesFromRoleDescriptors( + (BytesReference) metadata.get(AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY), + streamVersion + ) + ); + } if (authentication.getEffectiveSubject().getTransportVersion().onOrAfter(VERSION_API_KEY_ROLES_AS_BYTES) && streamVersion.before(VERSION_API_KEY_ROLES_AS_BYTES)) { @@ -1417,7 +1450,7 @@ public final class Authentication implements ToXContentObject { } static BytesReference maybeRemoveRemoteClusterFromRoleDescriptors(BytesReference roleDescriptorsBytes) { - return maybeRemoveTopLevelFromRoleDescriptors(roleDescriptorsBytes, RoleDescriptor.Fields.REMOTE_CLUSTER.getPreferredName()); + return maybeRemoveTopLevelFromRoleDescriptors(roleDescriptorsBytes, REMOTE_CLUSTER.getPreferredName()); } static BytesReference maybeRemoveRemoteIndicesFromRoleDescriptors(BytesReference roleDescriptorsBytes) { @@ -1450,6 +1483,66 @@ public final class Authentication implements ToXContentObject { } } + /** + * Before we send the role descriptors to the remote cluster, we need to remove the remote cluster privileges that the other cluster + * will not understand. If all privileges are removed, then the entire "remote_cluster" is removed to avoid sending empty privileges. + * @param roleDescriptorsBytes The role descriptors to be sent to the remote cluster, represented as bytes. + * @return The role descriptors with the privileges that unsupported by version removed, represented as bytes. + */ + @SuppressWarnings("unchecked") + static BytesReference maybeRemoveRemoteClusterPrivilegesFromRoleDescriptors( + BytesReference roleDescriptorsBytes, + TransportVersion outboundVersion + ) { + if (roleDescriptorsBytes == null || roleDescriptorsBytes.length() == 0) { + return roleDescriptorsBytes; + } + final Map roleDescriptorsMap = convertRoleDescriptorsBytesToMap(roleDescriptorsBytes); + final Map roleDescriptorsMapMutated = new HashMap<>(roleDescriptorsMap); + final AtomicBoolean modified = new AtomicBoolean(false); + roleDescriptorsMap.forEach((key, value) -> { + if (value instanceof Map) { + Map roleDescriptor = (Map) value; + roleDescriptor.forEach((innerKey, innerValue) -> { + // example: remote_cluster=[{privileges=[monitor_enrich, monitor_stats] + if (REMOTE_CLUSTER.getPreferredName().equals(innerKey)) { + assert innerValue instanceof List; + RemoteClusterPermissions discoveredRemoteClusterPermission = new RemoteClusterPermissions( + (List>>) innerValue + ); + RemoteClusterPermissions mutated = discoveredRemoteClusterPermission.removeUnsupportedPrivileges(outboundVersion); + if (mutated.equals(discoveredRemoteClusterPermission) == false) { + // swap out the old value with the new value + modified.set(true); + Map remoteClusterMap = new HashMap<>((Map) roleDescriptorsMapMutated.get(key)); + if (mutated.hasAnyPrivileges()) { + // has at least one group with privileges + remoteClusterMap.put(innerKey, mutated.toMap()); + } else { + // has no groups with privileges + remoteClusterMap.remove(innerKey); + } + roleDescriptorsMapMutated.put(key, remoteClusterMap); + } + } + }); + } + }); + if (modified.get()) { + logger.debug( + "mutated role descriptors. Changed from {} to {} for outbound version {}", + roleDescriptorsMap, + roleDescriptorsMapMutated, + outboundVersion + ); + return convertRoleDescriptorsMapToBytes(roleDescriptorsMapMutated); + } else { + // No need to serialize if we did not change anything. + logger.trace("no change to role descriptors {} for outbound version {}", roleDescriptorsMap, outboundVersion); + return roleDescriptorsBytes; + } + } + static boolean equivalentRealms(String name1, String type1, String name2, String type2) { if (false == type1.equals(type2)) { return false; diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java index 8d069caf0496..9f5aaa8562a8 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java @@ -6,6 +6,8 @@ */ package org.elasticsearch.xpack.core.security.authz; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; import org.elasticsearch.ElasticsearchParseException; import org.elasticsearch.ElasticsearchSecurityException; import org.elasticsearch.TransportVersion; @@ -62,6 +64,7 @@ public class RoleDescriptor implements ToXContentObject, Writeable { public static final TransportVersion SECURITY_ROLE_DESCRIPTION = TransportVersions.V_8_15_0; public static final String ROLE_TYPE = "role"; + private static final Logger logger = LogManager.getLogger(RoleDescriptor.class); private final String name; private final String[] clusterPrivileges; @@ -191,7 +194,7 @@ public class RoleDescriptor implements ToXContentObject, Writeable { ? Collections.unmodifiableMap(transientMetadata) : Collections.singletonMap("enabled", true); this.remoteIndicesPrivileges = remoteIndicesPrivileges != null ? remoteIndicesPrivileges : RemoteIndicesPrivileges.NONE; - this.remoteClusterPermissions = remoteClusterPermissions != null && remoteClusterPermissions.hasPrivileges() + this.remoteClusterPermissions = remoteClusterPermissions != null && remoteClusterPermissions.hasAnyPrivileges() ? remoteClusterPermissions : RemoteClusterPermissions.NONE; this.restriction = restriction != null ? restriction : Restriction.NONE; @@ -263,7 +266,7 @@ public class RoleDescriptor implements ToXContentObject, Writeable { } public boolean hasRemoteClusterPermissions() { - return remoteClusterPermissions.hasPrivileges(); + return remoteClusterPermissions.hasAnyPrivileges(); } public RemoteClusterPermissions getRemoteClusterPermissions() { @@ -830,25 +833,32 @@ public class RoleDescriptor implements ToXContentObject, Writeable { currentFieldName = parser.currentName(); } else if (Fields.PRIVILEGES.match(currentFieldName, parser.getDeprecationHandler())) { privileges = readStringArray(roleName, parser, false); - if (privileges.length != 1 - || RemoteClusterPermissions.getSupportedRemoteClusterPermissions() - .contains(privileges[0].trim().toLowerCase(Locale.ROOT)) == false) { - throw new ElasticsearchParseException( - "failed to parse remote_cluster for role [{}]. " - + RemoteClusterPermissions.getSupportedRemoteClusterPermissions() - + " is the only value allowed for [{}] within [remote_cluster]", + if (Arrays.stream(privileges) + .map(s -> s.toLowerCase(Locale.ROOT).trim()) + .allMatch(RemoteClusterPermissions.getSupportedRemoteClusterPermissions()::contains) == false) { + final String message = String.format( + Locale.ROOT, + "failed to parse remote_cluster for role [%s]. " + + "%s are the only values allowed for [%s] within [remote_cluster]. Found %s", roleName, - currentFieldName + RemoteClusterPermissions.getSupportedRemoteClusterPermissions(), + currentFieldName, + Arrays.toString(privileges) ); + logger.info(message); + throw new ElasticsearchParseException(message); } } else if (Fields.CLUSTERS.match(currentFieldName, parser.getDeprecationHandler())) { clusters = readStringArray(roleName, parser, false); } else { - throw new ElasticsearchParseException( - "failed to parse remote_cluster for role [{}]. unexpected field [{}]", + final String message = String.format( + Locale.ROOT, + "failed to parse remote_cluster for role [%s]. unexpected field [%s]", roleName, currentFieldName ); + logger.info(message); + throw new ElasticsearchParseException(message); } } if (privileges != null && clusters == null) { diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroup.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroup.java index 1c34a7829fcb..ec245fae2861 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroup.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroup.java @@ -13,11 +13,15 @@ import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; import org.elasticsearch.xcontent.ToXContentObject; import org.elasticsearch.xcontent.XContentBuilder; -import org.elasticsearch.xpack.core.security.authz.RoleDescriptor; import org.elasticsearch.xpack.core.security.support.StringMatcher; import java.io.IOException; import java.util.Arrays; +import java.util.List; +import java.util.Map; + +import static org.elasticsearch.xpack.core.security.authz.RoleDescriptor.Fields.CLUSTERS; +import static org.elasticsearch.xpack.core.security.authz.RoleDescriptor.Fields.PRIVILEGES; /** * Represents a group of permissions for a remote cluster. For example: @@ -41,6 +45,14 @@ public class RemoteClusterPermissionGroup implements NamedWriteable, ToXContentO remoteClusterAliasMatcher = StringMatcher.of(remoteClusterAliases); } + public RemoteClusterPermissionGroup(Map> remoteClusterGroup) { + assert remoteClusterGroup.get(PRIVILEGES.getPreferredName()) != null : "privileges must be non-null"; + assert remoteClusterGroup.get(CLUSTERS.getPreferredName()) != null : "clusters must be non-null"; + clusterPrivileges = remoteClusterGroup.get(PRIVILEGES.getPreferredName()).toArray(new String[0]); + remoteClusterAliases = remoteClusterGroup.get(CLUSTERS.getPreferredName()).toArray(new String[0]); + remoteClusterAliasMatcher = StringMatcher.of(remoteClusterAliases); + } + /** * @param clusterPrivileges The list of cluster privileges that are allowed for the remote cluster. must not be null or empty. * @param remoteClusterAliases The list of remote clusters that the privileges apply to. must not be null or empty. @@ -53,10 +65,14 @@ public class RemoteClusterPermissionGroup implements NamedWriteable, ToXContentO throw new IllegalArgumentException("remote cluster groups must not be null or empty"); } if (Arrays.stream(clusterPrivileges).anyMatch(s -> Strings.hasText(s) == false)) { - throw new IllegalArgumentException("remote_cluster privileges must contain valid non-empty, non-null values"); + throw new IllegalArgumentException( + "remote_cluster privileges must contain valid non-empty, non-null values " + Arrays.toString(clusterPrivileges) + ); } if (Arrays.stream(remoteClusterAliases).anyMatch(s -> Strings.hasText(s) == false)) { - throw new IllegalArgumentException("remote_cluster clusters aliases must contain valid non-empty, non-null values"); + throw new IllegalArgumentException( + "remote_cluster clusters aliases must contain valid non-empty, non-null values " + Arrays.toString(remoteClusterAliases) + ); } this.clusterPrivileges = clusterPrivileges; @@ -86,11 +102,24 @@ public class RemoteClusterPermissionGroup implements NamedWriteable, ToXContentO return Arrays.copyOf(remoteClusterAliases, remoteClusterAliases.length); } + /** + * Converts the group to a map representation. + * @return A map representation of the group. + */ + public Map> toMap() { + return Map.of( + PRIVILEGES.getPreferredName(), + Arrays.asList(clusterPrivileges), + CLUSTERS.getPreferredName(), + Arrays.asList(remoteClusterAliases) + ); + } + @Override public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { builder.startObject(); - builder.array(RoleDescriptor.Fields.PRIVILEGES.getPreferredName(), clusterPrivileges); - builder.array(RoleDescriptor.Fields.CLUSTERS.getPreferredName(), remoteClusterAliases); + builder.array(PRIVILEGES.getPreferredName(), clusterPrivileges); + builder.array(CLUSTERS.getPreferredName(), remoteClusterAliases); builder.endObject(); return builder; } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissions.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissions.java index 0d8880c33720..1928cf117dde 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissions.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissions.java @@ -29,13 +29,19 @@ import java.util.Locale; import java.util.Map; import java.util.Objects; import java.util.Set; +import java.util.TreeSet; import java.util.stream.Collectors; +import static org.elasticsearch.TransportVersions.ROLE_MONITOR_STATS; + /** * Represents the set of permissions for remote clusters. This is intended to be the model for both the {@link RoleDescriptor} - * and {@link Role}. This model is not intended to be sent to a remote cluster, but can be (wire) serialized within a single cluster - * as well as the Xcontent serialization for the REST API and persistence of the role in the security index. The privileges modeled here - * will be converted to the appropriate cluster privileges when sent to a remote cluster. + * and {@link Role}. This model is intended to be converted to local cluster permissions + * {@link #collapseAndRemoveUnsupportedPrivileges(String, TransportVersion)} before sent to the remote cluster. This model also be included + * in the role descriptors for (normal) API keys sent between nodes/clusters. In both cases the outbound transport version can be used to + * remove permissions that are not available to older nodes or clusters. The methods {@link #removeUnsupportedPrivileges(TransportVersion)} + * and {@link #collapseAndRemoveUnsupportedPrivileges(String, TransportVersion)} are used to aid in ensuring correct privileges per + * transport version. * For example, on the local/querying cluster this model represents the following: * * "remote_cluster" : [ @@ -49,15 +55,18 @@ import java.util.stream.Collectors; * } * ] * - * when sent to the remote cluster "clusterA", the privileges will be converted to the appropriate cluster privileges. For example: + * (RCS 2.0) when sent to the remote cluster "clusterA", the privileges will be converted to the appropriate cluster privileges. + * For example: * * "cluster": ["foo"] * - * and when sent to the remote cluster "clusterB", the privileges will be converted to the appropriate cluster privileges. For example: + * and (RCS 2.0) when sent to the remote cluster "clusterB", the privileges will be converted to the appropriate cluster privileges. + * For example: * * "cluster": ["bar"] * - * If the remote cluster does not support the privilege, as determined by the remote cluster version, the privilege will be not be sent. + * For normal API keys and their role descriptors :If the remote cluster does not support the privilege, the privilege will be not be sent. + * Upstream code performs the removal, but this class owns the business logic for how to remove per outbound version. */ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObject { @@ -70,19 +79,33 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec // package private non-final for testing static Map> allowedRemoteClusterPermissions = Map.of( ROLE_REMOTE_CLUSTER_PRIVS, - Set.of(ClusterPrivilegeResolver.MONITOR_ENRICH.name()) + Set.of(ClusterPrivilegeResolver.MONITOR_ENRICH.name()), + ROLE_MONITOR_STATS, + Set.of(ClusterPrivilegeResolver.MONITOR_STATS.name()) ); + static final TransportVersion lastTransportVersionPermission = allowedRemoteClusterPermissions.keySet() + .stream() + .max(TransportVersion::compareTo) + .orElseThrow(); public static final RemoteClusterPermissions NONE = new RemoteClusterPermissions(); public static Set getSupportedRemoteClusterPermissions() { - return allowedRemoteClusterPermissions.values().stream().flatMap(Set::stream).collect(Collectors.toSet()); + return allowedRemoteClusterPermissions.values().stream().flatMap(Set::stream).collect(Collectors.toCollection(TreeSet::new)); } public RemoteClusterPermissions(StreamInput in) throws IOException { remoteClusterPermissionGroups = in.readNamedWriteableCollectionAsList(RemoteClusterPermissionGroup.class); } + public RemoteClusterPermissions(List>> remoteClusters) { + remoteClusterPermissionGroups = new ArrayList<>(); + for (Map> remoteCluster : remoteClusters) { + RemoteClusterPermissionGroup remoteClusterPermissionGroup = new RemoteClusterPermissionGroup(remoteCluster); + remoteClusterPermissionGroups.add(remoteClusterPermissionGroup); + } + } + public RemoteClusterPermissions() { remoteClusterPermissionGroups = new ArrayList<>(); } @@ -97,10 +120,64 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec } /** - * Gets the privilege names for the remote cluster. This method will collapse all groups to single String[] all lowercase - * and will only return the appropriate privileges for the provided remote cluster version. + * Will remove any unsupported privileges for the provided outbound version. This method will not modify the current instance. + * This is useful for (normal) API keys role descriptors to help ensure that we don't send unsupported privileges. The result of + * this method may result in no groups if all privileges are removed. {@link #hasAnyPrivileges()} can be used to check if there are + * any privileges left. + * @param outboundVersion The version by which to remove unsupported privileges, this is typically the version of the remote cluster + * @return a new instance of RemoteClusterPermissions with the unsupported privileges removed */ - public String[] privilegeNames(final String remoteClusterAlias, TransportVersion remoteClusterVersion) { + public RemoteClusterPermissions removeUnsupportedPrivileges(TransportVersion outboundVersion) { + Objects.requireNonNull(outboundVersion, "outboundVersion must not be null"); + if (outboundVersion.onOrAfter(lastTransportVersionPermission)) { + return this; + } + RemoteClusterPermissions copyForOutboundVersion = new RemoteClusterPermissions(); + Set allowedPermissionsPerVersion = getAllowedPermissionsPerVersion(outboundVersion); + for (RemoteClusterPermissionGroup group : remoteClusterPermissionGroups) { + String[] privileges = group.clusterPrivileges(); + List outboundPrivileges = new ArrayList<>(privileges.length); + for (String privilege : privileges) { + if (allowedPermissionsPerVersion.contains(privilege.toLowerCase(Locale.ROOT))) { + outboundPrivileges.add(privilege); + } + } + if (outboundPrivileges.isEmpty() == false) { + RemoteClusterPermissionGroup outboundGroup = new RemoteClusterPermissionGroup( + outboundPrivileges.toArray(new String[0]), + group.remoteClusterAliases() + ); + copyForOutboundVersion.addGroup(outboundGroup); + if (logger.isDebugEnabled()) { + if (group.equals(outboundGroup) == false) { + logger.debug( + "Removed unsupported remote cluster permissions. Remaining {} for remote cluster [{}] for version [{}]." + + "Due to the remote cluster version, only the following permissions are allowed: {}", + outboundPrivileges, + group.remoteClusterAliases(), + outboundVersion, + allowedPermissionsPerVersion + ); + } + } + } else { + logger.debug( + "Removed all remote cluster permissions for remote cluster [{}]. " + + "Due to the remote cluster version, only the following permissions are allowed: {}", + group.remoteClusterAliases(), + allowedPermissionsPerVersion + ); + } + } + return copyForOutboundVersion; + } + + /** + * Gets all the privilege names for the remote cluster. This method will collapse all groups to single String[] all lowercase + * and will only return the appropriate privileges for the provided remote cluster version. This is useful for RCS 2.0 to ensure + * that we properly convert all the remote_cluster -> cluster privileges per remote cluster. + */ + public String[] collapseAndRemoveUnsupportedPrivileges(final String remoteClusterAlias, TransportVersion outboundVersion) { // get all privileges for the remote cluster Set groupPrivileges = remoteClusterPermissionGroups.stream() @@ -111,13 +188,7 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec .collect(Collectors.toSet()); // find all the privileges that are allowed for the remote cluster version - Set allowedPermissionsPerVersion = allowedRemoteClusterPermissions.entrySet() - .stream() - .filter((entry) -> entry.getKey().onOrBefore(remoteClusterVersion)) - .map(Map.Entry::getValue) - .flatMap(Set::stream) - .map(s -> s.toLowerCase(Locale.ROOT)) - .collect(Collectors.toSet()); + Set allowedPermissionsPerVersion = getAllowedPermissionsPerVersion(outboundVersion); // intersect the two sets to get the allowed privileges for the remote cluster version Set allowedPrivileges = new HashSet<>(groupPrivileges); @@ -137,13 +208,21 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec return allowedPrivileges.stream().sorted().toArray(String[]::new); } + /** + * Converts this object to it's {@link Map} representation. + * @return a list of maps representing the remote cluster permissions + */ + public List>> toMap() { + return remoteClusterPermissionGroups.stream().map(RemoteClusterPermissionGroup::toMap).toList(); + } + /** * Validates the remote cluster permissions (regardless of remote cluster version). * This method will throw an {@link IllegalArgumentException} if the permissions are invalid. * Generally, this method is just a safety check and validity should be checked before adding the permissions to this class. */ public void validate() { - assert hasPrivileges(); + assert hasAnyPrivileges(); Set invalid = getUnsupportedPrivileges(); if (invalid.isEmpty() == false) { throw new IllegalArgumentException( @@ -173,11 +252,11 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec return invalid; } - public boolean hasPrivileges(final String remoteClusterAlias) { + public boolean hasAnyPrivileges(final String remoteClusterAlias) { return remoteClusterPermissionGroups.stream().anyMatch(remoteIndicesGroup -> remoteIndicesGroup.hasPrivileges(remoteClusterAlias)); } - public boolean hasPrivileges() { + public boolean hasAnyPrivileges() { return remoteClusterPermissionGroups.isEmpty() == false; } @@ -185,6 +264,16 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec return Collections.unmodifiableList(remoteClusterPermissionGroups); } + private Set getAllowedPermissionsPerVersion(TransportVersion outboundVersion) { + return allowedRemoteClusterPermissions.entrySet() + .stream() + .filter((entry) -> entry.getKey().onOrBefore(outboundVersion)) + .map(Map.Entry::getValue) + .flatMap(Set::stream) + .map(s -> s.toLowerCase(Locale.ROOT)) + .collect(Collectors.toSet()); + } + @Override public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { for (RemoteClusterPermissionGroup remoteClusterPermissionGroup : remoteClusterPermissionGroups) { @@ -220,4 +309,5 @@ public class RemoteClusterPermissions implements NamedWriteable, ToXContentObjec public String getWriteableName() { return NAME; } + } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/Role.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/Role.java index d8d56a4fbb24..f52f8f85f006 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/Role.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/Role.java @@ -283,7 +283,7 @@ public interface Role { public Builder addRemoteClusterPermissions(RemoteClusterPermissions remoteClusterPermissions) { Objects.requireNonNull(remoteClusterPermissions, "remoteClusterPermissions must not be null"); assert this.remoteClusterPermissions == null : "addRemoteClusterPermissions should only be called once"; - if (remoteClusterPermissions.hasPrivileges()) { + if (remoteClusterPermissions.hasAnyPrivileges()) { remoteClusterPermissions.validate(); } this.remoteClusterPermissions = remoteClusterPermissions; diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/SimpleRole.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/SimpleRole.java index 08c86c5f71f4..0ec9d2a48316 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/SimpleRole.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/SimpleRole.java @@ -210,7 +210,7 @@ public class SimpleRole implements Role { final RemoteIndicesPermission remoteIndicesPermission = this.remoteIndicesPermission.forCluster(remoteClusterAlias); if (remoteIndicesPermission.remoteIndicesGroups().isEmpty() - && remoteClusterPermissions.hasPrivileges(remoteClusterAlias) == false) { + && remoteClusterPermissions.hasAnyPrivileges(remoteClusterAlias) == false) { return RoleDescriptorsIntersection.EMPTY; } @@ -224,7 +224,7 @@ public class SimpleRole implements Role { return new RoleDescriptorsIntersection( new RoleDescriptor( REMOTE_USER_ROLE_NAME, - remoteClusterPermissions.privilegeNames(remoteClusterAlias, remoteClusterVersion), + remoteClusterPermissions.collapseAndRemoveUnsupportedPrivileges(remoteClusterAlias, remoteClusterVersion), // The role descriptors constructed here may be cached in raw byte form, using a hash of their content as a // cache key; we therefore need deterministic order when constructing them here, to ensure cache hits for // equivalent role descriptors diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java index 3d1b378f4f51..00d45fb135fb 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java @@ -110,6 +110,8 @@ public class ClusterPrivilegeResolver { private static final Set MONITOR_WATCHER_PATTERN = Set.of("cluster:monitor/xpack/watcher/*"); private static final Set MONITOR_ROLLUP_PATTERN = Set.of("cluster:monitor/xpack/rollup/*"); private static final Set MONITOR_ENRICH_PATTERN = Set.of("cluster:monitor/xpack/enrich/*", "cluster:admin/xpack/enrich/get"); + // intentionally cluster:monitor/stats* to match cluster:monitor/stats, cluster:monitor/stats[n] and cluster:monitor/stats/remote + private static final Set MONITOR_STATS_PATTERN = Set.of("cluster:monitor/stats*"); private static final Set ALL_CLUSTER_PATTERN = Set.of( "cluster:*", @@ -208,7 +210,11 @@ public class ClusterPrivilegeResolver { // esql enrich "cluster:monitor/xpack/enrich/esql/resolve_policy", "cluster:internal:data/read/esql/open_exchange", - "cluster:internal:data/read/esql/exchange" + "cluster:internal:data/read/esql/exchange", + // cluster stats for remote clusters + "cluster:monitor/stats/remote", + "cluster:monitor/stats", + "cluster:monitor/stats[n]" ); private static final Set CROSS_CLUSTER_REPLICATION_PATTERN = Set.of( RemoteClusterService.REMOTE_CLUSTER_HANDSHAKE_ACTION_NAME, @@ -243,6 +249,7 @@ public class ClusterPrivilegeResolver { public static final NamedClusterPrivilege MONITOR_WATCHER = new ActionClusterPrivilege("monitor_watcher", MONITOR_WATCHER_PATTERN); public static final NamedClusterPrivilege MONITOR_ROLLUP = new ActionClusterPrivilege("monitor_rollup", MONITOR_ROLLUP_PATTERN); public static final NamedClusterPrivilege MONITOR_ENRICH = new ActionClusterPrivilege("monitor_enrich", MONITOR_ENRICH_PATTERN); + public static final NamedClusterPrivilege MONITOR_STATS = new ActionClusterPrivilege("monitor_stats", MONITOR_STATS_PATTERN); public static final NamedClusterPrivilege MANAGE = new ActionClusterPrivilege("manage", ALL_CLUSTER_PATTERN, ALL_SECURITY_PATTERN); public static final NamedClusterPrivilege MANAGE_INFERENCE = new ActionClusterPrivilege("manage_inference", MANAGE_INFERENCE_PATTERN); public static final NamedClusterPrivilege MANAGE_ML = new ActionClusterPrivilege("manage_ml", MANAGE_ML_PATTERN); @@ -424,6 +431,7 @@ public class ClusterPrivilegeResolver { MONITOR_WATCHER, MONITOR_ROLLUP, MONITOR_ENRICH, + MONITOR_STATS, MANAGE, MANAGE_CONNECTOR, MANAGE_INFERENCE, @@ -499,7 +507,7 @@ public class ClusterPrivilegeResolver { + Strings.collectionToCommaDelimitedString(VALUES.keySet()) + "] or a pattern over one of the available " + "cluster actions"; - logger.debug(errorMessage); + logger.warn(errorMessage); throw new IllegalArgumentException(errorMessage); } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java index 5fb753ab55aa..259e66f633ba 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java @@ -20,6 +20,9 @@ import org.elasticsearch.xpack.core.security.action.profile.GetProfilesAction; import org.elasticsearch.xpack.core.security.action.profile.SuggestProfilesAction; import org.elasticsearch.xpack.core.security.action.user.ProfileHasPrivilegesAction; import org.elasticsearch.xpack.core.security.authz.RoleDescriptor; +import org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissionGroup; +import org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions; +import org.elasticsearch.xpack.core.security.authz.privilege.ClusterPrivilegeResolver; import org.elasticsearch.xpack.core.security.authz.privilege.ConfigurableClusterPrivilege; import org.elasticsearch.xpack.core.security.authz.privilege.ConfigurableClusterPrivileges; import org.elasticsearch.xpack.core.security.support.MetadataUtils; @@ -497,7 +500,15 @@ class KibanaOwnedReservedRoleDescriptors { getRemoteIndicesReadPrivileges("metrics-apm.*"), getRemoteIndicesReadPrivileges("traces-apm.*"), getRemoteIndicesReadPrivileges("traces-apm-*") }, - null, + new RemoteClusterPermissions().addGroup( + new RemoteClusterPermissionGroup( + RemoteClusterPermissions.getSupportedRemoteClusterPermissions() + .stream() + .filter(s -> s.equals(ClusterPrivilegeResolver.MONITOR_STATS.name())) + .toArray(String[]::new), + new String[] { "*" } + ) + ), null, "Grants access necessary for the Kibana system user to read from and write to the Kibana indices, " + "manage index templates and tokens, and check the availability of the Elasticsearch cluster. " diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/license/AbstractClusterStateLicenseServiceTestCase.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/license/AbstractClusterStateLicenseServiceTestCase.java index 27e5c1213f1f..a3a12792df4a 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/license/AbstractClusterStateLicenseServiceTestCase.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/license/AbstractClusterStateLicenseServiceTestCase.java @@ -85,7 +85,6 @@ public abstract class AbstractClusterStateLicenseServiceTestCase extends ESTestC when(discoveryNodes.stream()).thenAnswer(i -> Stream.of(mockNode)); when(discoveryNodes.iterator()).thenAnswer(i -> Iterators.single(mockNode)); when(discoveryNodes.isLocalNodeElectedMaster()).thenReturn(false); - when(discoveryNodes.getMinNodeVersion()).thenReturn(mockNode.getVersion()); when(state.nodes()).thenReturn(discoveryNodes); when(state.getNodes()).thenReturn(discoveryNodes); // it is really ridiculous we have nodes() and getNodes()... when(clusterService.state()).thenReturn(state); diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/CrossClusterApiKeyRoleDescriptorBuilderTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/CrossClusterApiKeyRoleDescriptorBuilderTests.java index 22590e155e64..1dfd68ea9548 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/CrossClusterApiKeyRoleDescriptorBuilderTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/CrossClusterApiKeyRoleDescriptorBuilderTests.java @@ -10,11 +10,16 @@ package org.elasticsearch.xpack.core.security.action.apikey; import org.elasticsearch.ElasticsearchParseException; import org.elasticsearch.core.Strings; import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.transport.TransportRequest; import org.elasticsearch.xcontent.XContentParseException; import org.elasticsearch.xcontent.XContentParserConfiguration; import org.elasticsearch.xcontent.json.JsonXContent; +import org.elasticsearch.xpack.core.security.authc.AuthenticationTestHelper; import org.elasticsearch.xpack.core.security.authz.RoleDescriptor; +import org.elasticsearch.xpack.core.security.authz.permission.ClusterPermission; import org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions; +import org.elasticsearch.xpack.core.security.authz.privilege.ClusterPrivilege; +import org.elasticsearch.xpack.core.security.authz.privilege.ClusterPrivilegeResolver; import java.io.IOException; import java.util.List; @@ -27,6 +32,7 @@ import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.instanceOf; import static org.hamcrest.Matchers.is; +import static org.mockito.Mockito.mock; public class CrossClusterApiKeyRoleDescriptorBuilderTests extends ESTestCase { @@ -356,9 +362,42 @@ public class CrossClusterApiKeyRoleDescriptorBuilderTests extends ESTestCase { } public void testAPIKeyAllowsAllRemoteClusterPrivilegesForCCS() { - // if users can add remote cluster permissions to a role, then the APIKey should also allow that for that permission - // the inverse however, is not guaranteed. cross_cluster_search exists largely for internal use and is not exposed to the users role - assertTrue(Set.of(CCS_CLUSTER_PRIVILEGE_NAMES).containsAll(RemoteClusterPermissions.getSupportedRemoteClusterPermissions())); + // test to help ensure that at least 1 action that is allowed by the remote cluster permissions are supported by CCS + List actionsToTest = List.of("cluster:monitor/xpack/enrich/esql/resolve_policy", "cluster:monitor/stats/remote"); + // if you add new remote cluster permissions, please define an action we can test to help ensure it is supported by RCS 2.0 + assertThat(actionsToTest.size(), equalTo(RemoteClusterPermissions.getSupportedRemoteClusterPermissions().size())); + + for (String privilege : RemoteClusterPermissions.getSupportedRemoteClusterPermissions()) { + boolean actionPassesRemoteClusterPermissionCheck = false; + ClusterPrivilege clusterPrivilege = ClusterPrivilegeResolver.resolve(privilege); + // each remote cluster privilege has an action to test + for (String action : actionsToTest) { + if (clusterPrivilege.buildPermission(ClusterPermission.builder()) + .build() + .check(action, mock(TransportRequest.class), AuthenticationTestHelper.builder().build())) { + actionPassesRemoteClusterPermissionCheck = true; + break; + } + } + assertTrue( + "privilege [" + privilege + "] does not cover any actions among [" + actionsToTest + "]", + actionPassesRemoteClusterPermissionCheck + ); + } + // test that the actions pass the privilege check for CCS + for (String privilege : Set.of(CCS_CLUSTER_PRIVILEGE_NAMES)) { + boolean actionPassesRemoteCCSCheck = false; + ClusterPrivilege clusterPrivilege = ClusterPrivilegeResolver.resolve(privilege); + for (String action : actionsToTest) { + if (clusterPrivilege.buildPermission(ClusterPermission.builder()) + .build() + .check(action, mock(TransportRequest.class), AuthenticationTestHelper.builder().build())) { + actionPassesRemoteCCSCheck = true; + break; + } + } + assertTrue(actionPassesRemoteCCSCheck); + } } private static void assertRoleDescriptor( diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestTests.java index 97255502bc7b..239d48ca9c2e 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestTests.java @@ -104,7 +104,7 @@ public class PutRoleRequestTests extends ESTestCase { } request.putRemoteCluster(remoteClusterPermissions); assertValidationError("Invalid remote_cluster permissions found. Please remove the following: [", request); - assertValidationError("Only [monitor_enrich] are allowed", request); + assertValidationError("Only [monitor_enrich, monitor_stats] are allowed", request); } public void testValidationErrorWithEmptyClustersInRemoteIndices() { diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authc/AuthenticationTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authc/AuthenticationTests.java index 66e246d1c8a5..c999c970a76d 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authc/AuthenticationTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authc/AuthenticationTests.java @@ -21,6 +21,7 @@ import org.elasticsearch.core.Tuple; import org.elasticsearch.test.ESTestCase; import org.elasticsearch.test.TransportVersionUtils; import org.elasticsearch.transport.RemoteClusterPortSettings; +import org.elasticsearch.xcontent.ObjectPath; import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentType; @@ -32,6 +33,7 @@ import org.elasticsearch.xpack.core.security.authc.support.AuthenticationContext import org.elasticsearch.xpack.core.security.authz.RoleDescriptorsIntersection; import org.elasticsearch.xpack.core.security.user.AnonymousUser; import org.elasticsearch.xpack.core.security.user.User; +import org.hamcrest.Matchers; import java.io.IOException; import java.util.Arrays; @@ -42,6 +44,8 @@ import java.util.function.Consumer; import java.util.stream.Collectors; import static java.util.Map.entry; +import static org.elasticsearch.TransportVersions.ROLE_MONITOR_STATS; +import static org.elasticsearch.xpack.core.security.authc.Authentication.VERSION_API_KEY_ROLES_AS_BYTES; import static org.elasticsearch.xpack.core.security.authc.AuthenticationTestHelper.randomCrossClusterAccessSubjectInfo; import static org.elasticsearch.xpack.core.security.authc.CrossClusterAccessSubjectInfoTests.randomRoleDescriptorsIntersection; import static org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions.ROLE_REMOTE_CLUSTER_PRIVS; @@ -1070,7 +1074,7 @@ public class AuthenticationTests extends ESTestCase { // pick a version before that of the authentication instance to force a rewrite final TransportVersion olderVersion = TransportVersionUtils.randomVersionBetween( random(), - Authentication.VERSION_API_KEY_ROLES_AS_BYTES, + VERSION_API_KEY_ROLES_AS_BYTES, TransportVersionUtils.getPreviousVersion(original.getEffectiveSubject().getTransportVersion()) ); @@ -1115,7 +1119,7 @@ public class AuthenticationTests extends ESTestCase { // pick a version before that of the authentication instance to force a rewrite final TransportVersion olderVersion = TransportVersionUtils.randomVersionBetween( random(), - Authentication.VERSION_API_KEY_ROLES_AS_BYTES, + VERSION_API_KEY_ROLES_AS_BYTES, TransportVersionUtils.getPreviousVersion(original.getEffectiveSubject().getTransportVersion()) ); @@ -1135,6 +1139,84 @@ public class AuthenticationTests extends ESTestCase { ); } + public void testMaybeRewriteMetadataForApiKeyRoleDescriptorsWithRemoteClusterRemovePrivs() throws IOException { + final String apiKeyId = randomAlphaOfLengthBetween(1, 10); + final String apiKeyName = randomAlphaOfLengthBetween(1, 10); + Map metadata = Map.ofEntries( + entry(AuthenticationField.API_KEY_ID_KEY, apiKeyId), + entry(AuthenticationField.API_KEY_NAME_KEY, apiKeyName), + entry(AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY, new BytesArray(""" + {"base_role":{"cluster":["all"], + "remote_cluster":[{"privileges":["monitor_enrich", "monitor_stats"],"clusters":["*"]}] + }}""")), + entry(AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY, new BytesArray(""" + {"limited_by_role":{"cluster":["*"], + "remote_cluster":[{"privileges":["monitor_enrich", "monitor_stats"],"clusters":["*"]}] + }}""")) + ); + + final Authentication with2privs = AuthenticationTestHelper.builder() + .apiKey() + .metadata(metadata) + .transportVersion(TransportVersion.current()) + .build(); + + // pick a version that will only remove one of the two privileges + final TransportVersion olderVersion = TransportVersionUtils.randomVersionBetween( + random(), + ROLE_REMOTE_CLUSTER_PRIVS, + TransportVersionUtils.getPreviousVersion(ROLE_MONITOR_STATS) + ); + + Map rewrittenMetadata = with2privs.maybeRewriteForOlderVersion(olderVersion).getEffectiveSubject().getMetadata(); + assertThat(rewrittenMetadata.keySet(), equalTo(with2privs.getAuthenticatingSubject().getMetadata().keySet())); + + // only one of the two privileges are left after the rewrite + BytesReference baseRoleBytes = (BytesReference) rewrittenMetadata.get(AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY); + Map baseRoleAsMap = XContentHelper.convertToMap(baseRoleBytes, false, XContentType.JSON).v2(); + assertThat(ObjectPath.eval("base_role.remote_cluster.0.privileges", baseRoleAsMap), Matchers.contains("monitor_enrich")); + assertThat(ObjectPath.eval("base_role.remote_cluster.0.clusters", baseRoleAsMap), notNullValue()); + BytesReference limitedByRoleBytes = (BytesReference) rewrittenMetadata.get( + AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY + ); + Map limitedByRoleAsMap = XContentHelper.convertToMap(limitedByRoleBytes, false, XContentType.JSON).v2(); + assertThat(ObjectPath.eval("limited_by_role.remote_cluster.0.privileges", limitedByRoleAsMap), Matchers.contains("monitor_enrich")); + assertThat(ObjectPath.eval("limited_by_role.remote_cluster.0.clusters", limitedByRoleAsMap), notNullValue()); + + // same version, but it removes the only defined privilege + metadata = Map.ofEntries( + entry(AuthenticationField.API_KEY_ID_KEY, apiKeyId), + entry(AuthenticationField.API_KEY_NAME_KEY, apiKeyName), + entry(AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY, new BytesArray(""" + {"base_role":{"cluster":["all"], + "remote_cluster":[{"privileges":["monitor_stats"],"clusters":["*"]}] + }}""")), + entry(AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY, new BytesArray(""" + {"limited_by_role":{"cluster":["*"], + "remote_cluster":[{"privileges":["monitor_stats"],"clusters":["*"]}] + }}""")) + ); + + final Authentication with1priv = AuthenticationTestHelper.builder() + .apiKey() + .metadata(metadata) + .transportVersion(TransportVersion.current()) + .build(); + + rewrittenMetadata = with1priv.maybeRewriteForOlderVersion(olderVersion).getEffectiveSubject().getMetadata(); + assertThat(rewrittenMetadata.keySet(), equalTo(with1priv.getAuthenticatingSubject().getMetadata().keySet())); + + // the one privileges is removed after the rewrite, which removes the full "remote_cluster" object + baseRoleBytes = (BytesReference) rewrittenMetadata.get(AuthenticationField.API_KEY_ROLE_DESCRIPTORS_KEY); + baseRoleAsMap = XContentHelper.convertToMap(baseRoleBytes, false, XContentType.JSON).v2(); + assertThat(ObjectPath.eval("base_role.remote_cluster", baseRoleAsMap), nullValue()); + assertThat(ObjectPath.eval("base_role.cluster", baseRoleAsMap), notNullValue()); + limitedByRoleBytes = (BytesReference) rewrittenMetadata.get(AuthenticationField.API_KEY_LIMITED_ROLE_DESCRIPTORS_KEY); + limitedByRoleAsMap = XContentHelper.convertToMap(limitedByRoleBytes, false, XContentType.JSON).v2(); + assertThat(ObjectPath.eval("limited_by_role.remote_cluster", limitedByRoleAsMap), nullValue()); + assertThat(ObjectPath.eval("limited_by_role.cluster", limitedByRoleAsMap), notNullValue()); + } + public void testMaybeRemoveRemoteIndicesFromRoleDescriptors() { final boolean includeClusterPrivileges = randomBoolean(); final BytesReference roleWithoutRemoteIndices = new BytesArray(Strings.format(""" diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptorTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptorTests.java index 94430a4ed5bb..218876c7d40e 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptorTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptorTests.java @@ -542,6 +542,34 @@ public class RoleDescriptorTests extends ESTestCase { () -> RoleDescriptor.parserBuilder().build().parse("test", new BytesArray(q4), XContentType.JSON) ); assertThat(illegalArgumentException.getMessage(), containsString("remote cluster groups must not be null or empty")); + + // one invalid privilege + String q5 = """ + { + "remote_cluster": [ + { + "privileges": [ + "monitor_stats", "read_pipeline" + ], + "clusters": [ + "*" + ] + } + ] + }"""; + + ElasticsearchParseException parseException = expectThrows( + ElasticsearchParseException.class, + () -> RoleDescriptor.parserBuilder().build().parse("test", new BytesArray(q5), XContentType.JSON) + ); + assertThat( + parseException.getMessage(), + containsString( + "failed to parse remote_cluster for role [test]. " + + "[monitor_enrich, monitor_stats] are the only values allowed for [privileges] within [remote_cluster]. " + + "Found [monitor_stats, read_pipeline]" + ) + ); } public void testParsingFieldPermissionsUsesCache() throws IOException { diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroupTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroupTests.java index cd269bd1a97b..0b99db826d54 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroupTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionGroupTests.java @@ -16,6 +16,7 @@ import org.elasticsearch.xcontent.XContentParser; import java.io.IOException; import java.util.Arrays; import java.util.Locale; +import java.util.Map; import static org.hamcrest.Matchers.containsString; @@ -90,7 +91,7 @@ public class RemoteClusterPermissionGroupTests extends AbstractXContentSerializi ); IllegalArgumentException e = expectThrows(IllegalArgumentException.class, invalidClusterAlias); - assertEquals("remote_cluster clusters aliases must contain valid non-empty, non-null values", e.getMessage()); + assertThat(e.getMessage(), containsString("remote_cluster clusters aliases must contain valid non-empty, non-null values")); final ThrowingRunnable invalidPermission = randomFrom( () -> new RemoteClusterPermissionGroup(new String[] { null }, new String[] { "bar" }), @@ -100,7 +101,17 @@ public class RemoteClusterPermissionGroupTests extends AbstractXContentSerializi ); IllegalArgumentException e2 = expectThrows(IllegalArgumentException.class, invalidPermission); - assertEquals("remote_cluster privileges must contain valid non-empty, non-null values", e2.getMessage()); + assertThat(e2.getMessage(), containsString("remote_cluster privileges must contain valid non-empty, non-null values")); + } + + public void testToMap() { + String[] privileges = generateRandomStringArray(5, 5, false, false); + String[] clusters = generateRandomStringArray(5, 5, false, false); + RemoteClusterPermissionGroup remoteClusterPermissionGroup = new RemoteClusterPermissionGroup(privileges, clusters); + assertEquals( + Map.of("privileges", Arrays.asList(privileges), "clusters", Arrays.asList(clusters)), + remoteClusterPermissionGroup.toMap() + ); } @Override diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionsTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionsTests.java index 5b5a895f12ae..2c3196500927 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionsTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/permission/RemoteClusterPermissionsTests.java @@ -15,6 +15,8 @@ import org.elasticsearch.common.xcontent.XContentHelper; import org.elasticsearch.test.AbstractXContentSerializingTestCase; import org.elasticsearch.test.TransportVersionUtils; import org.elasticsearch.xcontent.XContentParser; +import org.elasticsearch.xpack.core.security.authz.RoleDescriptor; +import org.elasticsearch.xpack.core.security.xcontent.XContentUtils; import org.junit.Before; import java.io.IOException; @@ -27,8 +29,11 @@ import java.util.List; import java.util.Locale; import java.util.Map; import java.util.Set; +import java.util.stream.Collectors; +import static org.elasticsearch.TransportVersions.ROLE_MONITOR_STATS; import static org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions.ROLE_REMOTE_CLUSTER_PRIVS; +import static org.elasticsearch.xpack.core.security.authz.permission.RemoteClusterPermissions.lastTransportVersionPermission; import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; @@ -85,13 +90,13 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe for (int i = 0; i < generateRandomGroups(true).size(); i++) { String[] clusters = groupClusters.get(i); for (String cluster : clusters) { - assertTrue(remoteClusterPermission.hasPrivileges(cluster)); - assertFalse(remoteClusterPermission.hasPrivileges(randomAlphaOfLength(20))); + assertTrue(remoteClusterPermission.hasAnyPrivileges(cluster)); + assertFalse(remoteClusterPermission.hasAnyPrivileges(randomAlphaOfLength(20))); } } } - public void testPrivilegeNames() { + public void testCollapseAndRemoveUnsupportedPrivileges() { Map> original = RemoteClusterPermissions.allowedRemoteClusterPermissions; try { // create random groups with random privileges for random clusters @@ -108,7 +113,7 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe String[] privileges = groupPrivileges.get(i); String[] clusters = groupClusters.get(i); for (String cluster : clusters) { - String[] found = remoteClusterPermission.privilegeNames(cluster, TransportVersion.current()); + String[] found = remoteClusterPermission.collapseAndRemoveUnsupportedPrivileges(cluster, TransportVersion.current()); Arrays.sort(found); // ensure all lowercase since the privilege names are case insensitive and the method will result in lowercase for (int j = 0; j < privileges.length; j++) { @@ -126,13 +131,14 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe // create random groups with random privileges for random clusters List randomGroups = generateRandomGroups(true); // replace a random value with one that is allowed - groupPrivileges.get(0)[0] = "monitor_enrich"; + String singleValidPrivilege = randomFrom(RemoteClusterPermissions.allowedRemoteClusterPermissions.get(TransportVersion.current())); + groupPrivileges.get(0)[0] = singleValidPrivilege; for (int i = 0; i < randomGroups.size(); i++) { String[] privileges = groupPrivileges.get(i); String[] clusters = groupClusters.get(i); for (String cluster : clusters) { - String[] found = remoteClusterPermission.privilegeNames(cluster, TransportVersion.current()); + String[] found = remoteClusterPermission.collapseAndRemoveUnsupportedPrivileges(cluster, TransportVersion.current()); Arrays.sort(found); // ensure all lowercase since the privilege names are case insensitive and the method will result in lowercase for (int j = 0; j < privileges.length; j++) { @@ -149,7 +155,7 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe assertFalse(Arrays.equals(privileges, found)); if (i == 0) { // ensure that for the current version we only find the valid "monitor_enrich" - assertThat(Set.of(found), equalTo(Set.of("monitor_enrich"))); + assertThat(Set.of(found), equalTo(Set.of(singleValidPrivilege))); } else { // all other groups should be found to not have any privileges assertTrue(found.length == 0); @@ -159,21 +165,26 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe } } - public void testMonitorEnrichPerVersion() { - // test monitor_enrich before, after and on monitor enrich version - String[] privileges = randomBoolean() ? new String[] { "monitor_enrich" } : new String[] { "monitor_enrich", "foo", "bar" }; + public void testPermissionsPerVersion() { + testPermissionPerVersion("monitor_enrich", ROLE_REMOTE_CLUSTER_PRIVS); + testPermissionPerVersion("monitor_stats", ROLE_MONITOR_STATS); + } + + private void testPermissionPerVersion(String permission, TransportVersion version) { + // test permission before, after and on the version + String[] privileges = randomBoolean() ? new String[] { permission } : new String[] { permission, "foo", "bar" }; String[] before = new RemoteClusterPermissions().addGroup(new RemoteClusterPermissionGroup(privileges, new String[] { "*" })) - .privilegeNames("*", TransportVersionUtils.getPreviousVersion(ROLE_REMOTE_CLUSTER_PRIVS)); - // empty set since monitor_enrich is not allowed in the before version + .collapseAndRemoveUnsupportedPrivileges("*", TransportVersionUtils.getPreviousVersion(version)); + // empty set since permissions is not allowed in the before version assertThat(Set.of(before), equalTo(Collections.emptySet())); String[] on = new RemoteClusterPermissions().addGroup(new RemoteClusterPermissionGroup(privileges, new String[] { "*" })) - .privilegeNames("*", ROLE_REMOTE_CLUSTER_PRIVS); - // only monitor_enrich since the other values are not allowed - assertThat(Set.of(on), equalTo(Set.of("monitor_enrich"))); + .collapseAndRemoveUnsupportedPrivileges("*", version); + // the permission is found on that provided version + assertThat(Set.of(on), equalTo(Set.of(permission))); String[] after = new RemoteClusterPermissions().addGroup(new RemoteClusterPermissionGroup(privileges, new String[] { "*" })) - .privilegeNames("*", TransportVersion.current()); - // only monitor_enrich since the other values are not allowed - assertThat(Set.of(after), equalTo(Set.of("monitor_enrich"))); + .collapseAndRemoveUnsupportedPrivileges("*", TransportVersion.current()); + // current version (after the version) has the permission + assertThat(Set.of(after), equalTo(Set.of(permission))); } public void testValidate() { @@ -181,12 +192,70 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe // random values not allowed IllegalArgumentException error = expectThrows(IllegalArgumentException.class, () -> remoteClusterPermission.validate()); assertTrue(error.getMessage().contains("Invalid remote_cluster permissions found. Please remove the following:")); - assertTrue(error.getMessage().contains("Only [monitor_enrich] are allowed")); + assertTrue(error.getMessage().contains("Only [monitor_enrich, monitor_stats] are allowed")); new RemoteClusterPermissions().addGroup(new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "*" })) .validate(); // no error } + public void testToMap() { + RemoteClusterPermissions remoteClusterPermissions = new RemoteClusterPermissions(); + List groups = generateRandomGroups(randomBoolean()); + for (int i = 0; i < groups.size(); i++) { + remoteClusterPermissions.addGroup(groups.get(i)); + } + List>> asAsMap = remoteClusterPermissions.toMap(); + RemoteClusterPermissions remoteClusterPermissionsAsMap = new RemoteClusterPermissions(asAsMap); + assertEquals(remoteClusterPermissions, remoteClusterPermissionsAsMap); + } + + public void testRemoveUnsupportedPrivileges() { + RemoteClusterPermissions remoteClusterPermissions = new RemoteClusterPermissions(); + RemoteClusterPermissionGroup group = new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "*" }); + remoteClusterPermissions.addGroup(group); + // this privilege is allowed by versions, so nothing should be removed + assertEquals(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_REMOTE_CLUSTER_PRIVS)); + assertEquals(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_MONITOR_STATS)); + + remoteClusterPermissions = new RemoteClusterPermissions(); + if (randomBoolean()) { + group = new RemoteClusterPermissionGroup(new String[] { "monitor_stats" }, new String[] { "*" }); + } else { + // if somehow duplicates end up here, they should not influence removal + group = new RemoteClusterPermissionGroup(new String[] { "monitor_stats", "monitor_stats" }, new String[] { "*" }); + } + remoteClusterPermissions.addGroup(group); + // this single newer privilege is not allowed in the older version, so it should result in an object with no groups + assertNotEquals(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_REMOTE_CLUSTER_PRIVS)); + assertFalse(remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_REMOTE_CLUSTER_PRIVS).hasAnyPrivileges()); + assertEquals(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_MONITOR_STATS)); + + int groupCount = randomIntBetween(1, 5); + remoteClusterPermissions = new RemoteClusterPermissions(); + group = new RemoteClusterPermissionGroup(new String[] { "monitor_enrich", "monitor_stats" }, new String[] { "*" }); + for (int i = 0; i < groupCount; i++) { + remoteClusterPermissions.addGroup(group); + } + // one of the newer privilege is not allowed in the older version, so it should result in a group with only the allowed privilege + RemoteClusterPermissions expected = new RemoteClusterPermissions(); + for (int i = 0; i < groupCount; i++) { + expected.addGroup(new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "*" })); + } + assertEquals(expected, remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_REMOTE_CLUSTER_PRIVS)); + // both privileges allowed in the newer version, so it should not change the permission + assertEquals(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(ROLE_MONITOR_STATS)); + } + + public void testShortCircuitRemoveUnsupportedPrivileges() { + RemoteClusterPermissions remoteClusterPermissions = new RemoteClusterPermissions(); + assertSame(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(TransportVersion.current())); + assertSame(remoteClusterPermissions, remoteClusterPermissions.removeUnsupportedPrivileges(lastTransportVersionPermission)); + assertNotSame( + remoteClusterPermissions, + remoteClusterPermissions.removeUnsupportedPrivileges(TransportVersionUtils.getPreviousVersion(lastTransportVersionPermission)) + ); + } + private List generateRandomGroups(boolean fuzzyCluster) { clean(); List groups = new ArrayList<>(); @@ -216,22 +285,48 @@ public class RemoteClusterPermissionsTests extends AbstractXContentSerializingTe @Override protected RemoteClusterPermissions createTestInstance() { + Set all = RemoteClusterPermissions.allowedRemoteClusterPermissions.values() + .stream() + .flatMap(Set::stream) + .collect(Collectors.toSet()); + List randomPermission = randomList(1, all.size(), () -> randomFrom(all)); return new RemoteClusterPermissions().addGroup( - new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "*" }) + new RemoteClusterPermissionGroup(randomPermission.toArray(new String[0]), new String[] { "*" }) ); } @Override protected RemoteClusterPermissions mutateInstance(RemoteClusterPermissions instance) throws IOException { return new RemoteClusterPermissions().addGroup( - new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "*" }) + new RemoteClusterPermissionGroup(new String[] { "monitor_enrich", "monitor_stats" }, new String[] { "*" }) ).addGroup(new RemoteClusterPermissionGroup(new String[] { "foobar" }, new String[] { "*" })); } @Override protected RemoteClusterPermissions doParseInstance(XContentParser parser) throws IOException { - // fromXContent/parsing isn't supported since we still do old school manual parsing of the role descriptor - return createTestInstance(); + // fromXContent/object parsing isn't supported since we still do old school manual parsing of the role descriptor + // so this test is silly because it only tests we know how to manually parse the test instance in this test + // this is needed since we want the other parts from the AbstractXContentSerializingTestCase suite + RemoteClusterPermissions remoteClusterPermissions = new RemoteClusterPermissions(); + String[] privileges = null; + String[] clusters = null; + XContentParser.Token token; + String currentFieldName = null; + while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) { + if (token == XContentParser.Token.START_OBJECT) { + continue; + } + if (token == XContentParser.Token.FIELD_NAME) { + currentFieldName = parser.currentName(); + } else if (RoleDescriptor.Fields.PRIVILEGES.match(currentFieldName, parser.getDeprecationHandler())) { + privileges = XContentUtils.readStringArray(parser, false); + + } else if (RoleDescriptor.Fields.CLUSTERS.match(currentFieldName, parser.getDeprecationHandler())) { + clusters = XContentUtils.readStringArray(parser, false); + } + } + remoteClusterPermissions.addGroup(new RemoteClusterPermissionGroup(privileges, clusters)); + return remoteClusterPermissions; } @Override diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStoreTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStoreTests.java index 8b0efc84f972..6de1d425e97f 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStoreTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStoreTests.java @@ -2833,7 +2833,7 @@ public class ReservedRolesStoreTests extends ESTestCase { is(false) ); assertThat( - superuserRole.remoteCluster().privilegeNames("*", TransportVersion.current()), + superuserRole.remoteCluster().collapseAndRemoveUnsupportedPrivileges("*", TransportVersion.current()), equalTo(RemoteClusterPermissions.getSupportedRemoteClusterPermissions().toArray(new String[0])) ); } diff --git a/x-pack/plugin/enrich/qa/rest/build.gradle b/x-pack/plugin/enrich/qa/rest/build.gradle index fdaddbc1f929..064e362c77e6 100644 --- a/x-pack/plugin/enrich/qa/rest/build.gradle +++ b/x-pack/plugin/enrich/qa/rest/build.gradle @@ -32,3 +32,8 @@ testClusters.configureEach { setting 'xpack.security.enabled', 'false' requiresFeature 'es.index_mode_feature_flag_registered', Version.fromString("8.4.0") } + +tasks.named("yamlRestCompatTestTransform").configure({ task -> + task.skipTest("enrich/10_basic/Test using the deprecated elasticsearch_version field results in a warning", "The deprecation message was changed") +}) + diff --git a/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/FieldAttribute.java b/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/FieldAttribute.java index 4076acdb7e7b..d7ae438bc318 100644 --- a/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/FieldAttribute.java +++ b/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/FieldAttribute.java @@ -174,18 +174,7 @@ public class FieldAttribute extends TypedAttribute { @Override protected NodeInfo info() { - return NodeInfo.create( - this, - FieldAttribute::new, - parentName, - name(), - dataType(), - field, - (String) null, - nullable(), - id(), - synthetic() - ); + return NodeInfo.create(this, FieldAttribute::new, parentName, name(), field, nullable(), id(), synthetic()); } public String parentName() { diff --git a/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/MetadataAttribute.java b/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/MetadataAttribute.java index 3641812cd6ca..6e4e9292bfc9 100644 --- a/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/MetadataAttribute.java +++ b/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/MetadataAttribute.java @@ -147,26 +147,7 @@ public class MetadataAttribute extends TypedAttribute { @Override protected NodeInfo info() { - return NodeInfo.create( - this, - (source, name, dataType, qualifier, nullability, id, synthetic, searchable1) -> new MetadataAttribute( - source, - name, - dataType, - qualifier, - nullability, - id, - synthetic, - searchable1 - ), - name(), - dataType(), - (String) null, - nullable(), - id(), - synthetic(), - searchable - ); + return NodeInfo.create(this, MetadataAttribute::new, name(), dataType(), nullable(), id(), synthetic(), searchable); } public boolean searchable() { diff --git a/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/ReferenceAttribute.java b/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/ReferenceAttribute.java index 3626c5d26f23..404cd75edd5e 100644 --- a/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/ReferenceAttribute.java +++ b/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/ReferenceAttribute.java @@ -110,24 +110,7 @@ public class ReferenceAttribute extends TypedAttribute { @Override protected NodeInfo info() { - return NodeInfo.create( - this, - (source, name, dataType, qualifier, nullability, id, synthetic) -> new ReferenceAttribute( - source, - name, - dataType, - qualifier, - nullability, - id, - synthetic - ), - name(), - dataType(), - (String) null, - nullable(), - id(), - synthetic() - ); + return NodeInfo.create(this, ReferenceAttribute::new, name(), dataType(), nullable(), id(), synthetic()); } @Override diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java b/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java index cf9d66727a90..2bd7ecc37b03 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java @@ -72,6 +72,10 @@ public class CsvTestsDataLoader { private static final TestsDataset DECADES = new TestsDataset("decades"); private static final TestsDataset AIRPORTS = new TestsDataset("airports"); private static final TestsDataset AIRPORTS_MP = AIRPORTS.withIndex("airports_mp").withData("airports_mp.csv"); + private static final TestsDataset AIRPORTS_NO_DOC_VALUES = new TestsDataset("airports_no_doc_values").withData("airports.csv"); + private static final TestsDataset AIRPORTS_NOT_INDEXED = new TestsDataset("airports_not_indexed").withData("airports.csv"); + private static final TestsDataset AIRPORTS_NOT_INDEXED_NOR_DOC_VALUES = new TestsDataset("airports_not_indexed_nor_doc_values") + .withData("airports.csv"); private static final TestsDataset AIRPORTS_WEB = new TestsDataset("airports_web"); private static final TestsDataset DATE_NANOS = new TestsDataset("date_nanos"); private static final TestsDataset COUNTRIES_BBOX = new TestsDataset("countries_bbox"); @@ -105,6 +109,9 @@ public class CsvTestsDataLoader { Map.entry(DECADES.indexName, DECADES), Map.entry(AIRPORTS.indexName, AIRPORTS), Map.entry(AIRPORTS_MP.indexName, AIRPORTS_MP), + Map.entry(AIRPORTS_NO_DOC_VALUES.indexName, AIRPORTS_NO_DOC_VALUES), + Map.entry(AIRPORTS_NOT_INDEXED.indexName, AIRPORTS_NOT_INDEXED), + Map.entry(AIRPORTS_NOT_INDEXED_NOR_DOC_VALUES.indexName, AIRPORTS_NOT_INDEXED_NOR_DOC_VALUES), Map.entry(AIRPORTS_WEB.indexName, AIRPORTS_WEB), Map.entry(COUNTRIES_BBOX.indexName, COUNTRIES_BBOX), Map.entry(COUNTRIES_BBOX_WEB.indexName, COUNTRIES_BBOX_WEB), diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java b/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java index bc465e7e9b64..2913401d8aab 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java @@ -89,6 +89,8 @@ import java.time.Duration; import java.time.Period; import java.util.ArrayList; import java.util.EnumSet; +import java.util.HashMap; +import java.util.HashSet; import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; @@ -206,9 +208,30 @@ public final class EsqlTestUtils { return new EsRelation(EMPTY, new EsIndex(randomAlphaOfLength(8), emptyMap()), IndexMode.STANDARD, randomBoolean()); } - public static class TestSearchStats extends SearchStats { - public TestSearchStats() { - super(emptyList()); + /** + * This version of SearchStats always returns true for all fields for all boolean methods. + * For custom behaviour either use {@link TestConfigurableSearchStats} or override the specific methods. + */ + public static class TestSearchStats implements SearchStats { + + @Override + public boolean exists(String field) { + return true; + } + + @Override + public boolean isIndexed(String field) { + return exists(field); + } + + @Override + public boolean hasDocValues(String field) { + return exists(field); + } + + @Override + public boolean hasExactSubfield(String field) { + return exists(field); } @Override @@ -226,11 +249,6 @@ public final class EsqlTestUtils { return exists(field) ? -1 : 0; } - @Override - public boolean exists(String field) { - return true; - } - @Override public byte[] min(String field, DataType dataType) { return null; @@ -245,10 +263,76 @@ public final class EsqlTestUtils { public boolean isSingleValue(String field) { return false; } + } + + /** + * This version of SearchStats can be preconfigured to return true/false for various combinations of the four field settings: + *
    + *
  1. exists
    2. + *
  2. isIndexed
    3. + *
  3. hasDocValues
    4. + *
  4. hasExactSubfield
    5. + *
+ * The default will return true for all fields. The include/exclude methods can be used to configure the settings for specific fields. + * If you call 'include' with no fields, it will switch to return false for all fields. + */ + public static class TestConfigurableSearchStats extends TestSearchStats { + public enum Config { + EXISTS, + INDEXED, + DOC_VALUES, + EXACT_SUBFIELD + } + + private final Map> includes = new HashMap<>(); + private final Map> excludes = new HashMap<>(); + + public TestConfigurableSearchStats include(Config key, String... fields) { + // If this method is called with no fields, it is interpreted to mean include none, so we include a dummy field + for (String field : fields.length == 0 ? new String[] { "-" } : fields) { + includes.computeIfAbsent(key, k -> new HashSet<>()).add(field); + excludes.computeIfAbsent(key, k -> new HashSet<>()).remove(field); + } + return this; + } + + public TestConfigurableSearchStats exclude(Config key, String... fields) { + for (String field : fields) { + includes.computeIfAbsent(key, k -> new HashSet<>()).remove(field); + excludes.computeIfAbsent(key, k -> new HashSet<>()).add(field); + } + return this; + } + + private boolean isConfigationSet(Config config, String field) { + Set in = includes.getOrDefault(config, Set.of()); + Set ex = excludes.getOrDefault(config, Set.of()); + return (in.isEmpty() || in.contains(field)) && ex.contains(field) == false; + } + + @Override + public boolean exists(String field) { + return isConfigationSet(Config.EXISTS, field); + } @Override public boolean isIndexed(String field) { - return exists(field); + return isConfigationSet(Config.INDEXED, field); + } + + @Override + public boolean hasDocValues(String field) { + return isConfigationSet(Config.DOC_VALUES, field); + } + + @Override + public boolean hasExactSubfield(String field) { + return isConfigationSet(Config.EXACT_SUBFIELD, field); + } + + @Override + public String toString() { + return "TestConfigurableSearchStats{" + "includes=" + includes + ", excludes=" + excludes + '}'; } } diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_no_doc_values.json b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_no_doc_values.json new file mode 100644 index 000000000000..d7097f89a17d --- /dev/null +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_no_doc_values.json @@ -0,0 +1,30 @@ +{ + "properties": { + "abbrev": { + "type": "keyword" + }, + "name": { + "type": "text" + }, + "scalerank": { + "type": "integer" + }, + "type": { + "type": "keyword" + }, + "location": { + "type": "geo_point", + "index": true, + "doc_values": false + }, + "country": { + "type": "keyword" + }, + "city": { + "type": "keyword" + }, + "city_location": { + "type": "geo_point" + } + } +} diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_not_indexed.json b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_not_indexed.json new file mode 100644 index 000000000000..1c72cf1f3e1a --- /dev/null +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_not_indexed.json @@ -0,0 +1,30 @@ +{ + "properties": { + "abbrev": { + "type": "keyword" + }, + "name": { + "type": "text" + }, + "scalerank": { + "type": "integer" + }, + "type": { + "type": "keyword" + }, + "location": { + "type": "geo_point", + "index": false, + "doc_values": true + }, + "country": { + "type": "keyword" + }, + "city": { + "type": "keyword" + }, + "city_location": { + "type": "geo_point" + } + } +} diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports-no-doc-values.json b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_not_indexed_nor_doc_values.json similarity index 100% rename from x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports-no-doc-values.json rename to x-pack/plugin/esql/qa/testFixtures/src/main/resources/mapping-airports_not_indexed_nor_doc_values.json diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-function.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-function.csv-spec index b0578aa1a4ed..c35f4c19cc34 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-function.csv-spec +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-function.csv-spec @@ -6,11 +6,11 @@ matchWithField required_capability: match_function // tag::match-with-field[] -from books -| where match(author, "Faulkner") -| keep book_no, author -| sort book_no -| limit 5; +FROM books +| WHERE MATCH(author, "Faulkner") +| KEEP book_no, author +| SORT book_no +| LIMIT 5; // end::match-with-field[] // tag::match-with-field-result[] diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-operator.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-operator.csv-spec index 18bb7cdf866c..7b55ece964b8 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-operator.csv-spec +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/match-operator.csv-spec @@ -6,11 +6,11 @@ matchWithField required_capability: match_operator_colon // tag::match-with-field[] -from books -| where author:"Faulkner" -| keep book_no, author -| sort book_no -| limit 5; +FROM books +| WHERE author:"Faulkner" +| KEEP book_no, author +| SORT book_no +| LIMIT 5; // end::match-with-field[] // tag::match-with-field-result[] diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/qstr-function.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/qstr-function.csv-spec index 6dc03d0debcf..3e92e55928d6 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/qstr-function.csv-spec +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/qstr-function.csv-spec @@ -6,11 +6,11 @@ qstrWithField required_capability: qstr_function // tag::qstr-with-field[] -from books -| where qstr("author: Faulkner") -| keep book_no, author -| sort book_no -| limit 5; +FROM books +| WHERE QSTR("author: Faulkner") +| KEEP book_no, author +| SORT book_no +| LIMIT 5; // end::qstr-with-field[] // tag::qstr-with-field-result[] diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial.csv-spec index c1c4538c7393..01e7258e8a6e 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial.csv-spec +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial.csv-spec @@ -484,6 +484,42 @@ centroid:geo_point | count:long POINT (42.97109629958868 14.7552534006536) | 1 ; +centroidFromAirportsAfterIntersectsCompoundPredicateNoDocValues +required_capability: st_intersects + +FROM airports_no_doc_values +| WHERE scalerank == 9 AND ST_INTERSECTS(location, TO_GEOSHAPE("POLYGON((42 14, 43 14, 43 15, 42 15, 42 14))")) AND country == "Yemen" +| STATS centroid=ST_CENTROID_AGG(location), count=COUNT() +; + +centroid:geo_point | count:long +POINT (42.97109629958868 14.7552534006536) | 1 +; + +centroidFromAirportsAfterIntersectsCompoundPredicateNotIndexedNorDocValues +required_capability: st_intersects + +FROM airports_not_indexed_nor_doc_values +| WHERE scalerank == 9 AND ST_INTERSECTS(location, TO_GEOSHAPE("POLYGON((42 14, 43 14, 43 15, 42 15, 42 14))")) AND country == "Yemen" +| STATS centroid=ST_CENTROID_AGG(location), count=COUNT() +; + +centroid:geo_point | count:long +POINT (42.97109629958868 14.7552534006536) | 1 +; + +centroidFromAirportsAfterIntersectsCompoundPredicateNotIndexed +required_capability: st_intersects + +FROM airports_not_indexed +| WHERE scalerank == 9 AND ST_INTERSECTS(location, TO_GEOSHAPE("POLYGON((42 14, 43 14, 43 15, 42 15, 42 14))")) AND country == "Yemen" +| STATS centroid=ST_CENTROID_AGG(location), count=COUNT() +; + +centroid:geo_point | count:long +POINT (42.97109629958868 14.7552534006536) | 1 +; + ############################################### # Tests for ST_INTERSECTS on GEO_POINT type diff --git a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/AbstractEsqlIntegTestCase.java b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/AbstractEsqlIntegTestCase.java index 669723abe70d..7ae45497f729 100644 --- a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/AbstractEsqlIntegTestCase.java +++ b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/AbstractEsqlIntegTestCase.java @@ -32,6 +32,7 @@ import org.elasticsearch.xpack.esql.plugin.QueryPragmas; import org.elasticsearch.xpack.esql.plugin.TransportEsqlQueryAction; import org.junit.After; +import java.util.ArrayList; import java.util.Collection; import java.util.Iterator; import java.util.List; @@ -39,6 +40,7 @@ import java.util.concurrent.TimeUnit; import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; import static org.elasticsearch.xpack.esql.EsqlTestUtils.getValuesList; +import static org.hamcrest.Matchers.containsInAnyOrder; import static org.hamcrest.Matchers.equalTo; @TestLogging(value = "org.elasticsearch.xpack.esql.session:DEBUG", reason = "to better understand planning") @@ -219,4 +221,16 @@ public abstract class AbstractEsqlIntegTestCase extends ESIntegTestCase { protected static void assertValues(Iterator> actualValues, Iterable> expectedValues) { assertThat(getValuesList(actualValues), equalTo(getValuesList(expectedValues))); } + + protected static void assertValuesInAnyOrder(Iterator> actualValues, Iterable> expectedValues) { + List> items = new ArrayList<>(); + for (Iterable outter : expectedValues) { + var item = new ArrayList<>(); + for (var inner : outter) { + item.add(inner); + } + items.add(item); + } + assertThat(getValuesList(actualValues), containsInAnyOrder(items.toArray())); + } } diff --git a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/QueryStringIT.java b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/QueryStringIT.java index e7da83a40fb2..03af16d29e9b 100644 --- a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/QueryStringIT.java +++ b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/QueryStringIT.java @@ -13,18 +13,12 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.index.query.QueryShardException; import org.elasticsearch.xpack.esql.VerificationException; import org.elasticsearch.xpack.esql.action.AbstractEsqlIntegTestCase; -import org.elasticsearch.xpack.esql.action.ColumnInfoImpl; -import org.elasticsearch.xpack.esql.core.type.DataType; import org.junit.Before; import java.util.List; -import static org.elasticsearch.test.ListMatcher.matchesList; -import static org.elasticsearch.test.MapMatcher.assertMap; import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked; -import static org.elasticsearch.xpack.esql.EsqlTestUtils.getValuesList; import static org.hamcrest.CoreMatchers.containsString; -import static org.hamcrest.Matchers.equalTo; public class QueryStringIT extends AbstractEsqlIntegTestCase { @@ -42,11 +36,9 @@ public class QueryStringIT extends AbstractEsqlIntegTestCase { """; try (var resp = run(query)) { - assertThat(resp.columns().stream().map(ColumnInfoImpl::name).toList(), equalTo(List.of("id"))); - assertThat(resp.columns().stream().map(ColumnInfoImpl::type).map(DataType::toString).toList(), equalTo(List.of("INTEGER"))); - // values - List> values = getValuesList(resp); - assertMap(values, matchesList().item(List.of(1)).item(List.of(3)).item(List.of(4)).item(List.of(5))); + assertColumnNames(resp.columns(), List.of("id")); + assertColumnTypes(resp.columns(), List.of("integer")); + assertValues(resp.values(), List.of(List.of(1), List.of(3), List.of(4), List.of(5))); } } @@ -58,11 +50,9 @@ public class QueryStringIT extends AbstractEsqlIntegTestCase { """; try (var resp = run(query)) { - assertThat(resp.columns().stream().map(ColumnInfoImpl::name).toList(), equalTo(List.of("id"))); - assertThat(resp.columns().stream().map(ColumnInfoImpl::type).map(DataType::toString).toList(), equalTo(List.of("INTEGER"))); - // values - List> values = getValuesList(resp); - assertThat(values.size(), equalTo(5)); + assertColumnNames(resp.columns(), List.of("id")); + assertColumnTypes(resp.columns(), List.of("integer")); + assertValuesInAnyOrder(resp.values(), List.of(List.of(1), List.of(2), List.of(3), List.of(4), List.of(5))); } } diff --git a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownPointsTestCase.java b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownPointsTestCase.java index 0acbe98022f0..b9b003b8255e 100644 --- a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownPointsTestCase.java +++ b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownPointsTestCase.java @@ -12,8 +12,6 @@ import org.elasticsearch.geometry.Point; import org.elasticsearch.geometry.utils.GeometryValidator; import org.elasticsearch.geometry.utils.WellKnownText; import org.elasticsearch.lucene.spatial.CentroidCalculator; -import org.elasticsearch.xpack.core.esql.action.EsqlQueryRequestBuilder; -import org.elasticsearch.xpack.core.esql.action.EsqlQueryResponse; import org.hamcrest.Description; import org.hamcrest.Matcher; import org.hamcrest.TypeSafeMatcher; @@ -22,6 +20,7 @@ import java.io.IOException; import java.text.ParseException; import java.util.ArrayList; import java.util.Arrays; +import java.util.List; import java.util.Locale; import static org.hamcrest.Matchers.closeTo; @@ -62,8 +61,7 @@ public abstract class SpatialPushDownPointsTestCase extends SpatialPushDownTestC CentroidCalculator withinCentroid = new CentroidCalculator(); CentroidCalculator disjointCentroid = new CentroidCalculator(); for (int i = 0; i < data.size(); i++) { - index("indexed", i + "", "{\"location\" : " + data.get(i).data + " }"); - index("not-indexed", i + "", "{\"location\" : " + data.get(i).data + " }"); + addToIndexes(i, data.get(i).data, "indexed", "not-indexed", "not-indexed-nor-doc-values", "no-doc-values"); if (data.get(i).intersects) { expectedIntersects++; data.get(i).centroid.addTo(intersectsCentroid); @@ -76,7 +74,7 @@ public abstract class SpatialPushDownPointsTestCase extends SpatialPushDownTestC data.get(i).centroid.addTo(withinCentroid); } } - refresh("indexed", "not-indexed"); + refresh("indexed", "not-indexed", "not-indexed-nor-doc-values", "no-doc-values"); for (String polygon : new String[] { "POLYGON ((-10 -10, -10 10, 10 10, 10 -10, -10 -10))", @@ -89,24 +87,28 @@ public abstract class SpatialPushDownPointsTestCase extends SpatialPushDownTestC protected void assertFunction(String spatialFunction, String wkt, long expected, CentroidCalculator centroid) throws IOException, ParseException { - final String query1 = String.format(Locale.ROOT, """ + List queries = getQueries(String.format(Locale.ROOT, """ FROM indexed | WHERE %s(location, %s("%s")) | STATS COUNT(*), ST_CENTROID_AGG(location) - """, spatialFunction, castingFunction(), wkt); - final String query2 = String.format(Locale.ROOT, """ - FROM not-indexed | WHERE %s(location, %s("%s")) | STATS COUNT(*), ST_CENTROID_AGG(location) - """, spatialFunction, castingFunction(), wkt); - try ( - EsqlQueryResponse response1 = EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query1).get(); - EsqlQueryResponse response2 = EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query2).get(); - ) { - Object indexedCount = response1.response().column(0).iterator().next(); - Object notIndexedCount = response2.response().column(0).iterator().next(); - assertEquals(spatialFunction + "[expected=" + expected + "]", expected, indexedCount); - assertEquals(spatialFunction + "[expected=" + expected + "]", expected, notIndexedCount); - Object indexedCentroid = response1.response().column(1).iterator().next(); - Object notIndexedCentroid = response2.response().column(1).iterator().next(); - assertThat(spatialFunction + "[expected=" + toString(centroid) + "]", centroid, matchesCentroid(indexedCentroid)); - assertThat(spatialFunction + "[expected=" + toString(centroid) + "]", centroid, matchesCentroid(notIndexedCentroid)); + """, spatialFunction, castingFunction(), wkt)); + try (TestQueryResponseCollection responses = new TestQueryResponseCollection(queries)) { + for (int i = 0; i < ALL_INDEXES.length; i++) { + Object resultCount = responses.getResponse(i, 0); + Object resultCentroid = responses.getResponse(i, 1); + assertEquals(spatialFunction + "[expected=" + expected + "] for " + ALL_INDEXES[i], expected, resultCount); + assertThat( + spatialFunction + "[expected=" + toString(centroid) + "] for " + ALL_INDEXES[i], + centroid, + matchesCentroid(resultCentroid) + ); + } + long allIndexesCount = (long) responses.getResponse(ALL_INDEXES.length, 0); + assertEquals(spatialFunction + "[expected=" + expected + "] for all indexes", expected * 4, allIndexesCount); + Object allIndexesCentroid = responses.getResponse(ALL_INDEXES.length, 1); + assertThat( + spatialFunction + "[expected=" + toString(centroid) + "] for all indexes", + centroid, + matchesCentroid(allIndexesCentroid) + ); } } @@ -126,16 +128,14 @@ public abstract class SpatialPushDownPointsTestCase extends SpatialPushDownTestC for (int j = 0; j < values.length; j++) { values[j] = "\"" + WellKnownText.toWKT(getIndexGeometry()) + "\""; } - index("indexed", i + "", "{\"location\" : " + Arrays.toString(values) + " }"); - index("not-indexed", i + "", "{\"location\" : " + Arrays.toString(values) + " }"); + addToIndexes(i, Arrays.toString(values), "indexed", "not-indexed", "not-indexed-nor-doc-values", "no-doc-values"); } else { final String value = WellKnownText.toWKT(getIndexGeometry()); - index("indexed", i + "", "{\"location\" : \"" + value + "\" }"); - index("not-indexed", i + "", "{\"location\" : \"" + value + "\" }"); + addToIndexes(i, "\"" + value + "\"", "indexed", "not-indexed", "not-indexed-nor-doc-values", "no-doc-values"); } } - refresh("indexed", "not-indexed"); + refresh("indexed", "not-indexed", "not-indexed-nor-doc-values", "no-doc-values"); for (int i = 0; i < 10; i++) { final Geometry geometry = getIndexGeometry(); @@ -149,19 +149,17 @@ public abstract class SpatialPushDownPointsTestCase extends SpatialPushDownTestC protected void assertDistanceFunction(String wkt) { String spatialFunction = "ST_DISTANCE"; String castingFunction = castingFunction().replaceAll("SHAPE", "POINT"); - final String query1 = String.format(Locale.ROOT, """ - FROM indexed | WHERE %s(location, %s("%s")) < %.1f | STATS COUNT(*) - """, spatialFunction, castingFunction, wkt, searchDistance()); - final String query2 = String.format(Locale.ROOT, """ - FROM not-indexed | WHERE %s(location, %s("%s")) < %.1f | STATS COUNT(*) - """, spatialFunction, castingFunction, wkt, searchDistance()); - try ( - EsqlQueryResponse response1 = EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query1).get(); - EsqlQueryResponse response2 = EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query2).get(); - ) { - Object indexedResult = response1.response().column(0).iterator().next(); - Object notIndexedResult = response2.response().column(0).iterator().next(); - assertEquals(spatialFunction, indexedResult, notIndexedResult); + List queries = getQueries(String.format(Locale.ROOT, """ + FROM index | WHERE %s(location, %s("%s")) < %.1f | STATS COUNT(*) + """, spatialFunction, castingFunction, wkt, searchDistance())); + try (TestQueryResponseCollection responses = new TestQueryResponseCollection(queries)) { + Object indexedResult = responses.getResponse(0, 0); + for (int i = 1; i < ALL_INDEXES.length; i++) { + Object result = responses.getResponse(i, 0); + assertEquals(spatialFunction + " for " + ALL_INDEXES[i], indexedResult, result); + } + long allIndexesResult = (long) responses.getResponse(ALL_INDEXES.length, 0); + assertEquals(spatialFunction + " for all indexes", (long) indexedResult * 4, allIndexesResult); } } diff --git a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownTestCase.java b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownTestCase.java index e7e0c785f50e..90e8bb713552 100644 --- a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownTestCase.java +++ b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialPushDownTestCase.java @@ -18,6 +18,7 @@ import org.elasticsearch.xpack.core.esql.action.EsqlQueryResponse; import org.elasticsearch.xpack.esql.plugin.EsqlPlugin; import org.elasticsearch.xpack.spatial.SpatialPlugin; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; import java.util.List; @@ -35,6 +36,8 @@ import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcke */ public abstract class SpatialPushDownTestCase extends ESIntegTestCase { + protected static final String[] ALL_INDEXES = new String[] { "indexed", "not-indexed", "not-indexed-nor-doc-values", "no-doc-values" }; + protected Collection> nodePlugins() { return List.of(EsqlPlugin.class, SpatialPlugin.class); } @@ -78,12 +81,34 @@ public abstract class SpatialPushDownTestCase extends ESIntegTestCase { """, fieldType()))); assertAcked(prepareCreate("not-indexed").setMapping(String.format(Locale.ROOT, """ + { + "properties" : { + "location": { "type" : "%s", "index" : false, "doc_values" : true } + } + } + """, fieldType()))); + + assertAcked(prepareCreate("not-indexed-nor-doc-values").setMapping(String.format(Locale.ROOT, """ { "properties" : { "location": { "type" : "%s", "index" : false, "doc_values" : false } } } """, fieldType()))); + + assertAcked(prepareCreate("no-doc-values").setMapping(String.format(Locale.ROOT, """ + { + "properties" : { + "location": { "type" : "%s", "index" : true, "doc_values" : false } + } + } + """, fieldType()))); + } + + protected void addToIndexes(int id, String values, String... indexes) { + for (String index : indexes) { + index(index, id + "", "{\"location\" : " + values + " }"); + } } private void assertPushedDownQueries(boolean multiValue) throws RuntimeException { @@ -94,16 +119,14 @@ public abstract class SpatialPushDownTestCase extends ESIntegTestCase { for (int j = 0; j < values.length; j++) { values[j] = "\"" + WellKnownText.toWKT(getIndexGeometry()) + "\""; } - index("indexed", i + "", "{\"location\" : " + Arrays.toString(values) + " }"); - index("not-indexed", i + "", "{\"location\" : " + Arrays.toString(values) + " }"); + addToIndexes(i, Arrays.toString(values), ALL_INDEXES); } else { final String value = WellKnownText.toWKT(getIndexGeometry()); - index("indexed", i + "", "{\"location\" : \"" + value + "\" }"); - index("not-indexed", i + "", "{\"location\" : \"" + value + "\" }"); + addToIndexes(i, "\"" + value + "\"", ALL_INDEXES); } } - refresh("indexed", "not-indexed"); + refresh(ALL_INDEXES); String smallRectangleCW = "POLYGON ((-10 -10, -10 10, 10 10, 10 -10, -10 -10))"; assertFunction("ST_WITHIN", smallRectangleCW); @@ -115,27 +138,57 @@ public abstract class SpatialPushDownTestCase extends ESIntegTestCase { assertFunction("ST_INTERSECTS", wkt); assertFunction("ST_DISJOINT", wkt); assertFunction("ST_CONTAINS", wkt); - // within and lines are not globally supported so we avoid it here + // within and lines are not globally supported, so we avoid it here if (containsLine(geometry) == false) { assertFunction("ST_WITHIN", wkt); } } } + protected List getQueries(String query) { + ArrayList queries = new ArrayList<>(); + Arrays.stream(ALL_INDEXES).forEach(index -> queries.add(query.replaceAll("FROM (\\w+) \\|", "FROM " + index + " |"))); + queries.add(query.replaceAll("FROM (\\w+) \\|", "FROM " + String.join(",", ALL_INDEXES) + " |")); + return queries; + } + protected void assertFunction(String spatialFunction, String wkt) { - final String query1 = String.format(Locale.ROOT, """ - FROM indexed | WHERE %s(location, %s("%s")) | STATS COUNT(*) - """, spatialFunction, castingFunction(), wkt); - final String query2 = String.format(Locale.ROOT, """ - FROM not-indexed | WHERE %s(location, %s("%s")) | STATS COUNT(*) - """, spatialFunction, castingFunction(), wkt); - try ( - EsqlQueryResponse response1 = EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query1).get(); - EsqlQueryResponse response2 = EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query2).get(); - ) { - Object indexedResult = response1.response().column(0).iterator().next(); - Object notIndexedResult = response2.response().column(0).iterator().next(); - assertEquals(spatialFunction, indexedResult, notIndexedResult); + List queries = getQueries(String.format(Locale.ROOT, """ + FROM index | WHERE %s(location, %s("%s")) | STATS COUNT(*) + """, spatialFunction, castingFunction(), wkt)); + try (TestQueryResponseCollection responses = new TestQueryResponseCollection(queries)) { + Object indexedResult = responses.getResponse(0, 0); + for (int i = 1; i < ALL_INDEXES.length; i++) { + Object result = responses.getResponse(i, 0); + assertEquals(spatialFunction + " for " + ALL_INDEXES[i], indexedResult, result); + } + long allIndexesResult = (long) responses.getResponse(ALL_INDEXES.length, 0); + assertEquals(spatialFunction + " for all indexes", (long) indexedResult * 4, allIndexesResult); + } + } + + protected static class TestQueryResponseCollection implements AutoCloseable { + private final List responses; + + public TestQueryResponseCollection(List queries) { + this.responses = queries.stream().map(query -> { + try { + return EsqlQueryRequestBuilder.newRequestBuilder(client()).query(query).get(); + } catch (Exception e) { + throw new RuntimeException(e); + } + }).toList(); + } + + protected Object getResponse(int index, int column) { + return responses.get(index).response().column(column).iterator().next(); + } + + @Override + public void close() { + for (EsqlQueryResponse response : responses) { + response.close(); + } } } diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/EnableSpatialDistancePushdown.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/EnableSpatialDistancePushdown.java index cde305e52a70..dfb1dbc8bc8f 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/EnableSpatialDistancePushdown.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/EnableSpatialDistancePushdown.java @@ -76,15 +76,15 @@ public class EnableSpatialDistancePushdown extends PhysicalOptimizerRules.Parame protected PhysicalPlan rule(FilterExec filterExec, LocalPhysicalOptimizerContext ctx) { PhysicalPlan plan = filterExec; if (filterExec.child() instanceof EsQueryExec esQueryExec) { - plan = rewrite(filterExec, esQueryExec); + plan = rewrite(filterExec, esQueryExec, LucenePushdownPredicates.from(ctx.searchStats())); } else if (filterExec.child() instanceof EvalExec evalExec && evalExec.child() instanceof EsQueryExec esQueryExec) { - plan = rewriteBySplittingFilter(filterExec, evalExec, esQueryExec); + plan = rewriteBySplittingFilter(filterExec, evalExec, esQueryExec, LucenePushdownPredicates.from(ctx.searchStats())); } return plan; } - private FilterExec rewrite(FilterExec filterExec, EsQueryExec esQueryExec) { + private FilterExec rewrite(FilterExec filterExec, EsQueryExec esQueryExec, LucenePushdownPredicates lucenePushdownPredicates) { // Find and rewrite any binary comparisons that involve a distance function and a literal var rewritten = filterExec.condition().transformDown(EsqlBinaryComparison.class, comparison -> { ComparisonType comparisonType = ComparisonType.from(comparison.getFunctionType()); @@ -95,7 +95,7 @@ public class EnableSpatialDistancePushdown extends PhysicalOptimizerRules.Parame } return comparison; }); - if (rewritten.equals(filterExec.condition()) == false && canPushToSource(rewritten, x -> false)) { + if (rewritten.equals(filterExec.condition()) == false && canPushToSource(rewritten, lucenePushdownPredicates)) { return new FilterExec(filterExec.source(), esQueryExec, rewritten); } return filterExec; @@ -119,9 +119,14 @@ public class EnableSpatialDistancePushdown extends PhysicalOptimizerRules.Parame * | WHERE other > 10 * */ - private PhysicalPlan rewriteBySplittingFilter(FilterExec filterExec, EvalExec evalExec, EsQueryExec esQueryExec) { + private PhysicalPlan rewriteBySplittingFilter( + FilterExec filterExec, + EvalExec evalExec, + EsQueryExec esQueryExec, + LucenePushdownPredicates lucenePushdownPredicates + ) { // Find all pushable distance functions in the EVAL - Map distances = getPushableDistances(evalExec.fields()); + Map distances = getPushableDistances(evalExec.fields(), lucenePushdownPredicates); // Don't do anything if there are no distances to push down if (distances.isEmpty()) { @@ -139,7 +144,7 @@ public class EnableSpatialDistancePushdown extends PhysicalOptimizerRules.Parame // Find and rewrite any binary comparisons that involve a distance function and a literal var rewritten = rewriteDistanceFilters(resExp, distances); // If all pushable StDistance functions were found and re-written, we need to re-write the FILTER/EVAL combination - if (rewritten.equals(resExp) == false && canPushToSource(rewritten, x -> false)) { + if (rewritten.equals(resExp) == false && canPushToSource(rewritten, lucenePushdownPredicates)) { pushable.add(rewritten); } else { nonPushable.add(exp); @@ -163,10 +168,10 @@ public class EnableSpatialDistancePushdown extends PhysicalOptimizerRules.Parame } } - private Map getPushableDistances(List aliases) { + private Map getPushableDistances(List aliases, LucenePushdownPredicates lucenePushdownPredicates) { Map distances = new LinkedHashMap<>(); aliases.forEach(alias -> { - if (alias.child() instanceof StDistance distance && canPushSpatialFunctionToSource(distance)) { + if (alias.child() instanceof StDistance distance && canPushSpatialFunctionToSource(distance, lucenePushdownPredicates)) { distances.put(alias.id(), distance); } else if (alias.child() instanceof ReferenceAttribute ref && distances.containsKey(ref.id())) { StDistance distance = distances.get(ref.id()); diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/LucenePushDownUtils.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/LucenePushDownUtils.java deleted file mode 100644 index 1242629c1da3..000000000000 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/LucenePushDownUtils.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -package org.elasticsearch.xpack.esql.optimizer.rules.physical.local; - -import org.elasticsearch.xpack.esql.core.expression.Expression; -import org.elasticsearch.xpack.esql.core.expression.FieldAttribute; -import org.elasticsearch.xpack.esql.core.type.DataType; -import org.elasticsearch.xpack.esql.stats.SearchStats; - -import java.util.function.Predicate; - -class LucenePushDownUtils { - /** - * this method is supposed to be used to define if a field can be used for exact push down (eg. sort or filter). - * "aggregatable" is the most accurate information we can have from field_caps as of now. - * Pushing down operations on fields that are not aggregatable would result in an error. - */ - public static boolean isAggregatable(FieldAttribute f) { - return f.exactAttribute().field().isAggregatable(); - } - - public static boolean hasIdenticalDelegate(FieldAttribute attr, SearchStats stats) { - return stats.hasIdenticalDelegate(attr.name()); - } - - public static boolean isPushableFieldAttribute(Expression exp, Predicate hasIdenticalDelegate) { - if (exp instanceof FieldAttribute fa && fa.getExactInfo().hasExact() && isAggregatable(fa)) { - return fa.dataType() != DataType.TEXT || hasIdenticalDelegate.test(fa); - } - return false; - } -} diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/LucenePushdownPredicates.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/LucenePushdownPredicates.java new file mode 100644 index 000000000000..feb8717f007b --- /dev/null +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/LucenePushdownPredicates.java @@ -0,0 +1,111 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.esql.optimizer.rules.physical.local; + +import org.elasticsearch.xpack.esql.core.expression.Expression; +import org.elasticsearch.xpack.esql.core.expression.FieldAttribute; +import org.elasticsearch.xpack.esql.core.type.DataType; +import org.elasticsearch.xpack.esql.stats.SearchStats; + +/** + * When deciding if a filter or topN can be pushed down to Lucene, we need to check a few things on the field. + * Exactly what is checked depends on the type of field and the query. For example, we have the following possible combinations: + *
    + *
  1. A normal filter on a normal field will be pushed down using SingleValueQuery to remove multi-valued results, + * and this requires knowing if the field is indexed and has doc-values.
    2. + *
  2. A filter using a spatial function will allow multi-valued fields and we only need to know if the field is indexed, + * and do not need doc values.
    3. + *
  3. A TopN will be pushed down if the field is indexed and has doc values.
    4. + *
  4. Filters with TEXT fields can only be pushed down if the TEXT field has a nested KEYWORD field, + * referred to here as ExactSubfield. This that this is related to normal ES|QL predicates, + * not the full-text search provided by the MATCH and QSTR functions, which are pushed down separately.
    5. + *
+ */ +public interface LucenePushdownPredicates { + /** + * For TEXT fields, we need to check if the field has a subfield of type KEYWORD that can be used instead. + */ + boolean hasExactSubfield(FieldAttribute attr); + + /** + * For pushing down TopN and for pushing down filters with SingleValueQuery, + * we need to check if the field is indexed and has doc values. + */ + boolean isIndexedAndHasDocValues(FieldAttribute attr); + + /** + * For pushing down filters when multi-value results are allowed (spatial functions like ST_INTERSECTS), + * we only need to know if the field is indexed. + */ + boolean isIndexed(FieldAttribute attr); + + /** + * We see fields as pushable if either they are aggregatable or they are indexed. + * This covers non-indexed cases like AbstractScriptFieldType which hard-coded isAggregatable to true, + * as well as normal FieldAttribute's which can only be pushed down if they are indexed. + * The reason we don't just rely entirely on isAggregatable is because this is often false for normal fields, and could + * also differ from node to node, and we can physically plan each node separately, allowing Lucene pushdown on the nodes that + * support it, and relying on the compute engine for the nodes that do not. + */ + default boolean isPushableFieldAttribute(Expression exp) { + if (exp instanceof FieldAttribute fa && fa.getExactInfo().hasExact() && isIndexedAndHasDocValues(fa)) { + return (fa.dataType() != DataType.TEXT && fa.dataType() != DataType.SEMANTIC_TEXT) || hasExactSubfield(fa); + } + return false; + } + + /** + * The default implementation of this has no access to SearchStats, so it can only make decisions based on the FieldAttribute itself. + * In particular, it assumes TEXT fields have no exact subfields (underlying keyword field), + * and that isAggregatable means indexed and has hasDocValues. + */ + LucenePushdownPredicates DEFAULT = new LucenePushdownPredicates() { + @Override + public boolean hasExactSubfield(FieldAttribute attr) { + return false; + } + + @Override + public boolean isIndexedAndHasDocValues(FieldAttribute attr) { + // Is the FieldType.isAggregatable() check correct here? In FieldType isAggregatable usually only means hasDocValues + return attr.field().isAggregatable(); + } + + @Override + public boolean isIndexed(FieldAttribute attr) { + // TODO: This is the original behaviour, but is it correct? In FieldType isAggregatable usually only means hasDocValues + return attr.field().isAggregatable(); + } + }; + + /** + * If we have access to SearchStats over a collection of shards, we can make more fine-grained decisions about what can be pushed down. + * This should open up more opportunities for lucene pushdown. + */ + static LucenePushdownPredicates from(SearchStats stats) { + return new LucenePushdownPredicates() { + @Override + public boolean hasExactSubfield(FieldAttribute attr) { + return stats.hasExactSubfield(attr.name()); + } + + @Override + public boolean isIndexedAndHasDocValues(FieldAttribute attr) { + // We still consider the value of isAggregatable here, because some fields like ScriptFieldTypes are always aggregatable + // But this could hide issues with fields that are not indexed but are aggregatable + // This is the original behaviour for ES|QL, but is it correct? + return attr.field().isAggregatable() || stats.isIndexed(attr.name()) && stats.hasDocValues(attr.name()); + } + + @Override + public boolean isIndexed(FieldAttribute attr) { + return stats.isIndexed(attr.name()); + } + }; + } +} diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushFiltersToSource.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushFiltersToSource.java index 626ef5e83bd6..f01e7c4b1f3a 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushFiltersToSource.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushFiltersToSource.java @@ -55,11 +55,9 @@ import org.elasticsearch.xpack.esql.planner.PlannerUtils; import java.util.ArrayList; import java.util.List; -import java.util.function.Predicate; import static java.util.Arrays.asList; import static org.elasticsearch.xpack.esql.core.expression.predicate.Predicates.splitAnd; -import static org.elasticsearch.xpack.esql.optimizer.rules.physical.local.LucenePushDownUtils.isAggregatable; public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOptimizerRule { @@ -78,7 +76,7 @@ public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOpt List pushable = new ArrayList<>(); List nonPushable = new ArrayList<>(); for (Expression exp : splitAnd(filterExec.condition())) { - (canPushToSource(exp, x -> LucenePushDownUtils.hasIdenticalDelegate(x, ctx.searchStats())) ? pushable : nonPushable).add(exp); + (canPushToSource(exp, LucenePushdownPredicates.from(ctx.searchStats())) ? pushable : nonPushable).add(exp); } return rewrite(filterExec, queryExec, pushable, nonPushable, List.of()); } @@ -94,9 +92,7 @@ public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOpt List nonPushable = new ArrayList<>(); for (Expression exp : splitAnd(filterExec.condition())) { Expression resExp = exp.transformUp(ReferenceAttribute.class, r -> aliasReplacedBy.resolve(r, r)); - (canPushToSource(resExp, x -> LucenePushDownUtils.hasIdenticalDelegate(x, ctx.searchStats())) ? pushable : nonPushable).add( - exp - ); + (canPushToSource(resExp, LucenePushdownPredicates.from(ctx.searchStats())) ? pushable : nonPushable).add(exp); } // Replace field references with their actual field attributes pushable.replaceAll(e -> e.transformDown(ReferenceAttribute.class, r -> aliasReplacedBy.resolve(r, r))); @@ -222,17 +218,27 @@ public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOpt return changed ? CollectionUtils.combine(others, bcs, ranges) : pushable; } - public static boolean canPushToSource(Expression exp, Predicate hasIdenticalDelegate) { + /** + * Check if the given expression can be pushed down to the source. + * This version of the check is called when we do not have SearchStats available. It assumes no exact subfields for TEXT fields, + * and makes the indexed/doc-values check using the isAggregatable flag only, which comes from field-caps, represents the field state + * over the entire cluster (is not node specific), and has risks for indexed=false/doc_values=true fields. + */ + public static boolean canPushToSource(Expression exp) { + return canPushToSource(exp, LucenePushdownPredicates.DEFAULT); + } + + static boolean canPushToSource(Expression exp, LucenePushdownPredicates lucenePushdownPredicates) { if (exp instanceof BinaryComparison bc) { - return isAttributePushable(bc.left(), bc, hasIdenticalDelegate) && bc.right().foldable(); + return isAttributePushable(bc.left(), bc, lucenePushdownPredicates) && bc.right().foldable(); } else if (exp instanceof InsensitiveBinaryComparison bc) { - return isAttributePushable(bc.left(), bc, hasIdenticalDelegate) && bc.right().foldable(); + return isAttributePushable(bc.left(), bc, lucenePushdownPredicates) && bc.right().foldable(); } else if (exp instanceof BinaryLogic bl) { - return canPushToSource(bl.left(), hasIdenticalDelegate) && canPushToSource(bl.right(), hasIdenticalDelegate); + return canPushToSource(bl.left(), lucenePushdownPredicates) && canPushToSource(bl.right(), lucenePushdownPredicates); } else if (exp instanceof In in) { - return isAttributePushable(in.value(), null, hasIdenticalDelegate) && Expressions.foldable(in.list()); + return isAttributePushable(in.value(), null, lucenePushdownPredicates) && Expressions.foldable(in.list()); } else if (exp instanceof Not not) { - return canPushToSource(not.field(), hasIdenticalDelegate); + return canPushToSource(not.field(), lucenePushdownPredicates); } else if (exp instanceof UnaryScalarFunction usf) { if (usf instanceof RegexMatch || usf instanceof IsNull || usf instanceof IsNotNull) { if (usf instanceof IsNull || usf instanceof IsNotNull) { @@ -240,12 +246,13 @@ public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOpt return true; } } - return isAttributePushable(usf.field(), usf, hasIdenticalDelegate); + return isAttributePushable(usf.field(), usf, lucenePushdownPredicates); } } else if (exp instanceof CIDRMatch cidrMatch) { - return isAttributePushable(cidrMatch.ipField(), cidrMatch, hasIdenticalDelegate) && Expressions.foldable(cidrMatch.matches()); + return isAttributePushable(cidrMatch.ipField(), cidrMatch, lucenePushdownPredicates) + && Expressions.foldable(cidrMatch.matches()); } else if (exp instanceof SpatialRelatesFunction spatial) { - return canPushSpatialFunctionToSource(spatial); + return canPushSpatialFunctionToSource(spatial, lucenePushdownPredicates); } else if (exp instanceof StringQueryPredicate) { return true; } else if (exp instanceof QueryString) { @@ -259,23 +266,24 @@ public class PushFiltersToSource extends PhysicalOptimizerRules.ParameterizedOpt /** * Push-down to Lucene is only possible if one field is an indexed spatial field, and the other is a constant spatial or string column. */ - public static boolean canPushSpatialFunctionToSource(BinarySpatialFunction s) { + public static boolean canPushSpatialFunctionToSource(BinarySpatialFunction s, LucenePushdownPredicates lucenePushdownPredicates) { // The use of foldable here instead of SpatialEvaluatorFieldKey.isConstant is intentional to match the behavior of the // Lucene pushdown code in EsqlTranslationHandler::SpatialRelatesTranslator // We could enhance both places to support ReferenceAttributes that refer to constants, but that is a larger change - return isPushableSpatialAttribute(s.left()) && s.right().foldable() || isPushableSpatialAttribute(s.right()) && s.left().foldable(); + return isPushableSpatialAttribute(s.left(), lucenePushdownPredicates) && s.right().foldable() + || isPushableSpatialAttribute(s.right(), lucenePushdownPredicates) && s.left().foldable(); } - private static boolean isPushableSpatialAttribute(Expression exp) { - return exp instanceof FieldAttribute fa && fa.getExactInfo().hasExact() && isAggregatable(fa) && DataType.isSpatial(fa.dataType()); + private static boolean isPushableSpatialAttribute(Expression exp, LucenePushdownPredicates p) { + return exp instanceof FieldAttribute fa && DataType.isSpatial(fa.dataType()) && fa.getExactInfo().hasExact() && p.isIndexed(fa); } private static boolean isAttributePushable( Expression expression, Expression operation, - Predicate hasIdenticalDelegate + LucenePushdownPredicates lucenePushdownPredicates ) { - if (LucenePushDownUtils.isPushableFieldAttribute(expression, hasIdenticalDelegate)) { + if (lucenePushdownPredicates.isPushableFieldAttribute(expression)) { return true; } if (expression instanceof MetadataAttribute ma && ma.searchable()) { diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSource.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSource.java index 2ae496b55ac0..925e144b69fc 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSource.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSource.java @@ -30,7 +30,6 @@ import org.elasticsearch.xpack.esql.plan.physical.TopNExec; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.List; -import java.util.function.Predicate; /** * We handle two main scenarios here: @@ -60,7 +59,7 @@ import java.util.function.Predicate; public class PushTopNToSource extends PhysicalOptimizerRules.ParameterizedOptimizerRule { @Override protected PhysicalPlan rule(TopNExec topNExec, LocalPhysicalOptimizerContext ctx) { - Pushable pushable = evaluatePushable(topNExec, x -> LucenePushDownUtils.hasIdenticalDelegate(x, ctx.searchStats())); + Pushable pushable = evaluatePushable(topNExec, LucenePushdownPredicates.from(ctx.searchStats())); return pushable.rewrite(topNExec); } @@ -121,11 +120,11 @@ public class PushTopNToSource extends PhysicalOptimizerRules.ParameterizedOptimi } } - private static Pushable evaluatePushable(TopNExec topNExec, Predicate hasIdenticalDelegate) { + private static Pushable evaluatePushable(TopNExec topNExec, LucenePushdownPredicates lucenePushdownPredicates) { PhysicalPlan child = topNExec.child(); if (child instanceof EsQueryExec queryExec && queryExec.canPushSorts() - && canPushDownOrders(topNExec.order(), hasIdenticalDelegate)) { + && canPushDownOrders(topNExec.order(), lucenePushdownPredicates)) { // With the simplest case of `FROM index | SORT ...` we only allow pushing down if the sort is on a field return new PushableQueryExec(queryExec); } @@ -148,7 +147,7 @@ public class PushTopNToSource extends PhysicalOptimizerRules.ParameterizedOptimi List pushableSorts = new ArrayList<>(); for (Order order : orders) { - if (LucenePushDownUtils.isPushableFieldAttribute(order.child(), hasIdenticalDelegate)) { + if (lucenePushdownPredicates.isPushableFieldAttribute(order.child())) { pushableSorts.add( new EsQueryExec.FieldSort( ((FieldAttribute) order.child()).exactAttribute(), @@ -169,7 +168,7 @@ public class PushTopNToSource extends PhysicalOptimizerRules.ParameterizedOptimi break; } } else if (aliasReplacedBy.resolve(referenceAttribute, referenceAttribute) instanceof FieldAttribute fieldAttribute - && LucenePushDownUtils.isPushableFieldAttribute(fieldAttribute, hasIdenticalDelegate)) { + && lucenePushdownPredicates.isPushableFieldAttribute(fieldAttribute)) { // If the SORT refers to a reference to a pushable field, we can push it down pushableSorts.add( new EsQueryExec.FieldSort(fieldAttribute.exactAttribute(), order.direction(), order.nullsPosition()) @@ -191,9 +190,9 @@ public class PushTopNToSource extends PhysicalOptimizerRules.ParameterizedOptimi return NO_OP; } - private static boolean canPushDownOrders(List orders, Predicate hasIdenticalDelegate) { + private static boolean canPushDownOrders(List orders, LucenePushdownPredicates lucenePushdownPredicates) { // allow only exact FieldAttributes (no expressions) for sorting - return orders.stream().allMatch(o -> LucenePushDownUtils.isPushableFieldAttribute(o.child(), hasIdenticalDelegate)); + return orders.stream().allMatch(o -> lucenePushdownPredicates.isPushableFieldAttribute(o.child())); } private static List buildFieldSorts(List orders) { diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/SpatialDocValuesExtraction.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/SpatialDocValuesExtraction.java index d03cd9ef7cb0..0f1c32e94f86 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/SpatialDocValuesExtraction.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/SpatialDocValuesExtraction.java @@ -15,6 +15,7 @@ import org.elasticsearch.xpack.esql.core.expression.NamedExpression; import org.elasticsearch.xpack.esql.expression.function.aggregate.SpatialAggregateFunction; import org.elasticsearch.xpack.esql.expression.function.scalar.spatial.BinarySpatialFunction; import org.elasticsearch.xpack.esql.expression.function.scalar.spatial.SpatialRelatesFunction; +import org.elasticsearch.xpack.esql.optimizer.LocalPhysicalOptimizerContext; import org.elasticsearch.xpack.esql.optimizer.PhysicalOptimizerRules; import org.elasticsearch.xpack.esql.plan.physical.AggregateExec; import org.elasticsearch.xpack.esql.plan.physical.EvalExec; @@ -22,6 +23,7 @@ import org.elasticsearch.xpack.esql.plan.physical.FieldExtractExec; import org.elasticsearch.xpack.esql.plan.physical.FilterExec; import org.elasticsearch.xpack.esql.plan.physical.PhysicalPlan; import org.elasticsearch.xpack.esql.plan.physical.UnaryExec; +import org.elasticsearch.xpack.esql.stats.SearchStats; import java.util.ArrayList; import java.util.HashSet; @@ -63,9 +65,11 @@ import java.util.Set; * is the only place where this information is available. This also means that the knowledge of the usage of doc-values does not need * to be serialized between nodes, and is only used locally. */ -public class SpatialDocValuesExtraction extends PhysicalOptimizerRules.OptimizerRule { +public class SpatialDocValuesExtraction extends PhysicalOptimizerRules.ParameterizedOptimizerRule< + AggregateExec, + LocalPhysicalOptimizerContext> { @Override - protected PhysicalPlan rule(AggregateExec aggregate) { + protected PhysicalPlan rule(AggregateExec aggregate, LocalPhysicalOptimizerContext ctx) { var foundAttributes = new HashSet(); PhysicalPlan plan = aggregate.transformDown(UnaryExec.class, exec -> { @@ -75,7 +79,7 @@ public class SpatialDocValuesExtraction extends PhysicalOptimizerRules.Optimizer for (NamedExpression aggExpr : agg.aggregates()) { if (aggExpr instanceof Alias as && as.child() instanceof SpatialAggregateFunction af) { if (af.field() instanceof FieldAttribute fieldAttribute - && allowedForDocValues(fieldAttribute, agg, foundAttributes)) { + && allowedForDocValues(fieldAttribute, ctx.searchStats(), agg, foundAttributes)) { // We need to both mark the field to load differently, and change the spatial function to know to use it foundAttributes.add(fieldAttribute); changedAggregates = true; @@ -153,8 +157,13 @@ public class SpatialDocValuesExtraction extends PhysicalOptimizerRules.Optimizer * This function disallows the use of more than one field for doc-values extraction in the same spatial relation function. * This is because comparing two doc-values fields is not supported in the current implementation. */ - private boolean allowedForDocValues(FieldAttribute fieldAttribute, AggregateExec agg, Set foundAttributes) { - if (fieldAttribute.field().isAggregatable() == false) { + private boolean allowedForDocValues( + FieldAttribute fieldAttribute, + SearchStats stats, + AggregateExec agg, + Set foundAttributes + ) { + if (stats.hasDocValues(fieldAttribute.fieldName()) == false) { return false; } var candidateDocValuesAttributes = new HashSet<>(foundAttributes); diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java index 1758edb386e5..c998af221516 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java @@ -20,7 +20,6 @@ import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.xpack.esql.EsqlIllegalArgumentException; import org.elasticsearch.xpack.esql.core.expression.AttributeSet; import org.elasticsearch.xpack.esql.core.expression.Expression; -import org.elasticsearch.xpack.esql.core.expression.FieldAttribute; import org.elasticsearch.xpack.esql.core.expression.predicate.Predicates; import org.elasticsearch.xpack.esql.core.tree.Source; import org.elasticsearch.xpack.esql.core.type.DataType; @@ -52,13 +51,13 @@ import org.elasticsearch.xpack.esql.plan.physical.TopNExec; import org.elasticsearch.xpack.esql.planner.mapper.LocalMapper; import org.elasticsearch.xpack.esql.planner.mapper.Mapper; import org.elasticsearch.xpack.esql.session.Configuration; +import org.elasticsearch.xpack.esql.stats.SearchContextStats; import org.elasticsearch.xpack.esql.stats.SearchStats; import java.util.ArrayList; import java.util.LinkedHashSet; import java.util.List; import java.util.Set; -import java.util.function.Predicate; import static java.util.Arrays.asList; import static org.elasticsearch.index.mapper.MappedFieldType.FieldExtractPreference.DOC_VALUES; @@ -138,7 +137,7 @@ public class PlannerUtils { } public static PhysicalPlan localPlan(List searchContexts, Configuration configuration, PhysicalPlan plan) { - return localPlan(configuration, plan, new SearchStats(searchContexts)); + return localPlan(configuration, plan, SearchContextStats.from(searchContexts)); } public static PhysicalPlan localPlan(Configuration configuration, PhysicalPlan plan, SearchStats searchStats) { @@ -174,17 +173,18 @@ public class PlannerUtils { } /** - * Extracts the ES query provided by the filter parameter - * @param plan - * @param hasIdenticalDelegate a lambda that given a field attribute sayis if it has - * a synthetic source delegate with the exact same value - * @return + * Extracts the ES query for the @timestamp field for the passed plan. */ - public static QueryBuilder requestFilter(PhysicalPlan plan, Predicate hasIdenticalDelegate) { - return detectFilter(plan, "@timestamp", hasIdenticalDelegate); + public static QueryBuilder requestTimestampFilter(PhysicalPlan plan) { + return detectFilter(plan, "@timestamp"); } - static QueryBuilder detectFilter(PhysicalPlan plan, String fieldName, Predicate hasIdenticalDelegate) { + /** + * Note that since this filter does not have access to SearchStats, it cannot detect if the field is a text field with a delegate. + * We currently only use this filter for the @timestamp field, which is always a date field. Any tests that wish to use this should + * take care to not use it with TEXT fields. + */ + static QueryBuilder detectFilter(PhysicalPlan plan, String fieldName) { // first position is the REST filter, the second the query filter var requestFilter = new QueryBuilder[] { null, null }; @@ -205,7 +205,7 @@ public class PlannerUtils { boolean matchesField = refs.removeIf(e -> fieldName.equals(e.name())); // the expression only contains the target reference // and the expression is pushable (functions can be fully translated) - if (matchesField && refs.isEmpty() && canPushToSource(exp, hasIdenticalDelegate)) { + if (matchesField && refs.isEmpty() && canPushToSource(exp)) { matches.add(exp); } } diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java index 108e70d7d3a5..ffad379001ed 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java @@ -309,11 +309,7 @@ public class ComputeService { return reductionNode == null ? f : f.withReducer(reductionNode); }); - // The lambda is to say if a TEXT field has an identical exact subfield - // We cannot use SearchContext because we don't have it yet. - // Since it's used only for @timestamp, it is relatively safe to assume it's not needed - // but it would be better to have a proper impl. - QueryBuilder requestFilter = PlannerUtils.requestFilter(planWithReducer, x -> true); + QueryBuilder requestFilter = PlannerUtils.requestTimestampFilter(planWithReducer); var lookupListener = ActionListener.releaseAfter(computeListener.acquireAvoid(), exchangeSource.addEmptySink()); // SearchShards API can_match is done in lookupDataNodes lookupDataNodes(parentTask, clusterAlias, requestFilter, concreteIndices, originalIndices, ActionListener.wrap(dataNodeResult -> { diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchContextStats.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchContextStats.java new file mode 100644 index 000000000000..1f895c43f5dd --- /dev/null +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchContextStats.java @@ -0,0 +1,357 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.esql.stats; + +import org.apache.lucene.index.DocValuesType; +import org.apache.lucene.index.FieldInfo; +import org.apache.lucene.index.FieldInfos; +import org.apache.lucene.index.IndexOptions; +import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.LeafReader; +import org.apache.lucene.index.LeafReaderContext; +import org.apache.lucene.index.PointValues; +import org.apache.lucene.index.Term; +import org.apache.lucene.index.Terms; +import org.apache.lucene.util.BytesRef; +import org.elasticsearch.index.mapper.ConstantFieldType; +import org.elasticsearch.index.mapper.DocCountFieldMapper.DocCountFieldType; +import org.elasticsearch.index.mapper.IdFieldMapper; +import org.elasticsearch.index.mapper.MappedFieldType; +import org.elasticsearch.index.mapper.NumberFieldMapper.NumberFieldType; +import org.elasticsearch.index.mapper.SeqNoFieldMapper; +import org.elasticsearch.index.mapper.TextFieldMapper; +import org.elasticsearch.index.query.SearchExecutionContext; +import org.elasticsearch.xpack.esql.EsqlIllegalArgumentException; +import org.elasticsearch.xpack.esql.core.type.DataType; + +import java.io.IOException; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import static org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.TimestampFieldType; +import static org.elasticsearch.index.mapper.DateFieldMapper.DateFieldType; +import static org.elasticsearch.index.mapper.KeywordFieldMapper.KeywordFieldType; + +/** + * This class provides SearchStats from a list of SearchExecutionContext's. + * It contains primarily a cache of FieldStats which is dynamically updated as needed. + * Each FieldStats contains FieldConfig information which is populated once at creation time. + * The remaining statistics are lazily computed and cached only on demand. + * This cache is not thread-safe. + */ +public class SearchContextStats implements SearchStats { + + private final List contexts; + + private record FieldConfig(boolean exists, boolean hasExactSubfield, boolean indexed, boolean hasDocValues) {} + + private static class FieldStats { + private Long count; + private Object min, max; + private Boolean singleValue; + private FieldConfig config; + } + + private static final int CACHE_SIZE = 32; + + // simple non-thread-safe cache for avoiding unnecessary IO (which while fast is still I/O) + private final Map cache = new LinkedHashMap<>(CACHE_SIZE, 0.75f, true) { + @Override + protected boolean removeEldestEntry(Map.Entry eldest) { + return size() > CACHE_SIZE; + } + }; + + public static SearchStats from(List contexts) { + if (contexts == null || contexts.isEmpty()) { + return SearchStats.EMPTY; + } + return new SearchContextStats(contexts); + } + + private SearchContextStats(List contexts) { + this.contexts = contexts; + assert contexts != null && contexts.isEmpty() == false; + } + + public boolean exists(String field) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + return stat.config.exists; + } + + private FieldStats makeFieldStats(String field) { + var stat = new FieldStats(); + stat.config = makeFieldConfig(field); + return stat; + } + + private FieldConfig makeFieldConfig(String field) { + boolean exists = false; + boolean hasExactSubfield = true; + boolean indexed = true; + boolean hasDocValues = true; + // even if there are deleted documents, check the existence of a field + // since if it's missing, deleted documents won't change that + for (SearchExecutionContext context : contexts) { + if (context.isFieldMapped(field)) { + exists = exists || true; + MappedFieldType type = context.getFieldType(field); + indexed = indexed && type.isIndexed(); + hasDocValues = hasDocValues && type.hasDocValues(); + if (type instanceof TextFieldMapper.TextFieldType t) { + hasExactSubfield = hasExactSubfield && t.canUseSyntheticSourceDelegateForQuerying(); + } else { + hasExactSubfield = false; + } + } else { + indexed = false; + hasDocValues = false; + hasExactSubfield = false; + } + } + if (exists == false) { + // if it does not exist on any context, no other settings are valid + return new FieldConfig(false, false, false, false); + } else { + return new FieldConfig(exists, hasExactSubfield, indexed, hasDocValues); + } + } + + public boolean isIndexed(String field) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + return stat.config.indexed; + } + + public boolean hasDocValues(String field) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + return stat.config.hasDocValues; + } + + public boolean hasExactSubfield(String field) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + return stat.config.hasExactSubfield; + } + + public long count() { + var count = new long[] { 0 }; + boolean completed = doWithContexts(r -> { + count[0] += r.numDocs(); + return true; + }, false); + return completed ? count[0] : -1; + } + + public long count(String field) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + if (stat.count == null) { + var count = new long[] { 0 }; + boolean completed = doWithContexts(r -> { + count[0] += countEntries(r, field); + return true; + }, false); + stat.count = completed ? count[0] : -1; + } + return stat.count; + } + + public long count(String field, BytesRef value) { + var count = new long[] { 0 }; + Term term = new Term(field, value); + boolean completed = doWithContexts(r -> { + count[0] += r.docFreq(term); + return true; + }, false); + return completed ? count[0] : -1; + } + + public byte[] min(String field, DataType dataType) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + if (stat.min == null) { + var min = new byte[][] { null }; + doWithContexts(r -> { + byte[] localMin = PointValues.getMinPackedValue(r, field); + // TODO: how to compare with the previous min + if (localMin != null) { + if (min[0] == null) { + min[0] = localMin; + } else { + throw new EsqlIllegalArgumentException("Don't know how to compare with previous min"); + } + } + return true; + }, true); + stat.min = min[0]; + } + // return stat.min; + return null; + } + + public byte[] max(String field, DataType dataType) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + if (stat.max == null) { + var max = new byte[][] { null }; + doWithContexts(r -> { + byte[] localMax = PointValues.getMaxPackedValue(r, field); + // TODO: how to compare with the previous max + if (localMax != null) { + if (max[0] == null) { + max[0] = localMax; + } else { + throw new EsqlIllegalArgumentException("Don't know how to compare with previous max"); + } + } + return true; + }, true); + stat.max = max[0]; + } + // return stat.max; + return null; + } + + public boolean isSingleValue(String field) { + var stat = cache.computeIfAbsent(field, this::makeFieldStats); + if (stat.singleValue == null) { + // there's no such field so no need to worry about multi-value fields + if (exists(field) == false) { + stat.singleValue = true; + } else { + // fields are MV per default + var sv = new boolean[] { false }; + for (SearchExecutionContext context : contexts) { + MappedFieldType mappedType = context.isFieldMapped(field) ? context.getFieldType(field) : null; + if (mappedType != null) { + sv[0] = true; + doWithContexts(r -> { + sv[0] &= detectSingleValue(r, mappedType, field); + return sv[0]; + }, true); + break; + } + } + stat.singleValue = sv[0]; + } + } + return stat.singleValue; + } + + private boolean detectSingleValue(IndexReader r, MappedFieldType fieldType, String name) throws IOException { + // types that are always single value (and are accessible through instanceof) + if (fieldType instanceof ConstantFieldType || fieldType instanceof DocCountFieldType || fieldType instanceof TimestampFieldType) { + return true; + } + + var typeName = fieldType.typeName(); + + // non-visible fields, check their names + boolean found = switch (typeName) { + case IdFieldMapper.NAME, SeqNoFieldMapper.NAME -> true; + default -> false; + }; + + if (found) { + return true; + } + + // check against doc size + DocCountTester tester = null; + if (fieldType instanceof DateFieldType || fieldType instanceof NumberFieldType) { + tester = lr -> { + PointValues values = lr.getPointValues(name); + return values == null || values.size() == values.getDocCount(); + }; + } else if (fieldType instanceof KeywordFieldType) { + tester = lr -> { + Terms terms = lr.terms(name); + return terms == null || terms.size() == terms.getDocCount(); + }; + } + + if (tester != null) { + // check each leaf + for (LeafReaderContext context : r.leaves()) { + if (tester.test(context.reader()) == false) { + return false; + } + } + // field is missing or single value + return true; + } + + // unsupported type - default to MV + return false; + } + + private interface DocCountTester { + Boolean test(LeafReader leafReader) throws IOException; + } + + // + // @see org.elasticsearch.search.query.QueryPhaseCollectorManager#shortcutTotalHitCount(IndexReader, Query) + // + private static long countEntries(IndexReader indexReader, String field) { + long count = 0; + try { + for (LeafReaderContext context : indexReader.leaves()) { + LeafReader reader = context.reader(); + FieldInfos fieldInfos = reader.getFieldInfos(); + FieldInfo fieldInfo = fieldInfos.fieldInfo(field); + + if (fieldInfo != null) { + if (fieldInfo.getDocValuesType() == DocValuesType.NONE) { + // no shortcut possible: it's a text field, empty values are counted as no value. + return -1; + } + if (fieldInfo.getPointIndexDimensionCount() > 0) { + PointValues points = reader.getPointValues(field); + if (points != null) { + count += points.size(); + } + } else if (fieldInfo.getIndexOptions() != IndexOptions.NONE) { + Terms terms = reader.terms(field); + if (terms != null) { + count += terms.getSumTotalTermFreq(); + } + } else { + return -1; // no shortcut possible for fields that are not indexed + } + } + } + } catch (IOException ex) { + throw new EsqlIllegalArgumentException("Cannot access data storage", ex); + } + return count; + } + + private interface IndexReaderConsumer { + /** + * Returns true if the consumer should keep on going, false otherwise. + */ + boolean consume(IndexReader reader) throws IOException; + } + + private boolean doWithContexts(IndexReaderConsumer consumer, boolean acceptsDeletions) { + try { + for (SearchExecutionContext context : contexts) { + for (LeafReaderContext leafContext : context.searcher().getLeafContexts()) { + var reader = leafContext.reader(); + if (acceptsDeletions == false && reader.hasDeletions()) { + return false; + } + // check if the looping continues or not + if (consumer.consume(reader) == false) { + return false; + } + } + } + return true; + } catch (IOException ex) { + throw new EsqlIllegalArgumentException("Cannot access data storage", ex); + } + } +} diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchStats.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchStats.java index 73935cea540b..ca24bd54ee67 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchStats.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/stats/SearchStats.java @@ -7,363 +7,90 @@ package org.elasticsearch.xpack.esql.stats; -import org.apache.lucene.index.DocValuesType; -import org.apache.lucene.index.FieldInfo; -import org.apache.lucene.index.FieldInfos; -import org.apache.lucene.index.IndexOptions; -import org.apache.lucene.index.IndexReader; -import org.apache.lucene.index.LeafReader; -import org.apache.lucene.index.LeafReaderContext; -import org.apache.lucene.index.PointValues; -import org.apache.lucene.index.Term; -import org.apache.lucene.index.Terms; import org.apache.lucene.util.BytesRef; -import org.elasticsearch.index.mapper.AbstractScriptFieldType; -import org.elasticsearch.index.mapper.ConstantFieldType; -import org.elasticsearch.index.mapper.DocCountFieldMapper.DocCountFieldType; -import org.elasticsearch.index.mapper.IdFieldMapper; -import org.elasticsearch.index.mapper.MappedFieldType; -import org.elasticsearch.index.mapper.NumberFieldMapper.NumberFieldType; -import org.elasticsearch.index.mapper.SeqNoFieldMapper; -import org.elasticsearch.index.mapper.TextFieldMapper; -import org.elasticsearch.index.query.SearchExecutionContext; -import org.elasticsearch.xpack.esql.EsqlIllegalArgumentException; import org.elasticsearch.xpack.esql.core.type.DataType; -import java.io.IOException; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; +/** + * Interface for determining information about fields in the index. + * This is used by the optimizer to make decisions about how to optimize queries. + */ +public interface SearchStats { + SearchStats EMPTY = new EmptySearchStats(); -import static org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.TimestampFieldType; -import static org.elasticsearch.index.mapper.DateFieldMapper.DateFieldType; -import static org.elasticsearch.index.mapper.KeywordFieldMapper.KeywordFieldType; + boolean exists(String field); -public class SearchStats { + boolean isIndexed(String field); - private final List contexts; + boolean hasDocValues(String field); - private static class FieldStat { - private Long count; - private Object min, max; - // TODO: use a multi-bitset instead - private Boolean exists; - private Boolean singleValue; - private Boolean hasIdenticalDelegate; - private Boolean indexed; - private Boolean runtime; - } + boolean hasExactSubfield(String field); - private static final int CACHE_SIZE = 32; + long count(); + + long count(String field); + + long count(String field, BytesRef value); + + byte[] min(String field, DataType dataType); + + byte[] max(String field, DataType dataType); + + boolean isSingleValue(String field); + + /** + * When there are no search stats available, for example when there are no search contexts, we have static results. + */ + record EmptySearchStats() implements SearchStats { - // simple non-thread-safe cache for avoiding unnecessary IO (which while fast it still I/O) - private final Map cache = new LinkedHashMap<>(CACHE_SIZE, 0.75f, true) { @Override - protected boolean removeEldestEntry(Map.Entry eldest) { - return size() > CACHE_SIZE; + public boolean exists(String field) { + return false; } - }; - public SearchStats(List contexts) { - this.contexts = contexts; - } - - public long count() { - var count = new long[] { 0 }; - boolean completed = doWithContexts(r -> { - count[0] += r.numDocs(); - return true; - }, false); - return completed ? count[0] : -1; - } - - public long count(String field) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.count == null) { - var count = new long[] { 0 }; - boolean completed = doWithContexts(r -> { - count[0] += countEntries(r, field); - return true; - }, false); - stat.count = completed ? count[0] : -1; + @Override + public boolean isIndexed(String field) { + return false; } - return stat.count; - } - public long count(String field, BytesRef value) { - var count = new long[] { 0 }; - Term term = new Term(field, value); - boolean completed = doWithContexts(r -> { - count[0] += r.docFreq(term); - return true; - }, false); - return completed ? count[0] : -1; - } - - public boolean exists(String field) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.exists == null) { - stat.exists = false; - // even if there are deleted documents, check the existence of a field - // since if it's missing, deleted documents won't change that - for (SearchExecutionContext context : contexts) { - if (context.isFieldMapped(field)) { - stat.exists = true; - break; - } - } - - // populate additional properties to save on the lookups - if (stat.exists == false) { - stat.indexed = false; - stat.singleValue = true; - } + @Override + public boolean hasDocValues(String field) { + return false; } - return stat.exists; - } - public boolean hasIdenticalDelegate(String field) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.hasIdenticalDelegate == null) { - stat.hasIdenticalDelegate = true; - for (SearchExecutionContext context : contexts) { - if (context.isFieldMapped(field)) { - MappedFieldType type = context.getFieldType(field); - if (type instanceof TextFieldMapper.TextFieldType t) { - if (t.canUseSyntheticSourceDelegateForQuerying() == false) { - stat.hasIdenticalDelegate = false; - break; - } - } else { - stat.hasIdenticalDelegate = false; - break; - } - } - } + @Override + public boolean hasExactSubfield(String field) { + return false; } - return stat.hasIdenticalDelegate; - } - public byte[] min(String field, DataType dataType) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.min == null) { - var min = new byte[][] { null }; - doWithContexts(r -> { - byte[] localMin = PointValues.getMinPackedValue(r, field); - // TODO: how to compare with the previous min - if (localMin != null) { - if (min[0] == null) { - min[0] = localMin; - } else { - throw new EsqlIllegalArgumentException("Don't know how to compare with previous min"); - } - } - return true; - }, true); - stat.min = min[0]; + @Override + public long count() { + return 0; } - // return stat.min; - return null; - } - public byte[] max(String field, DataType dataType) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.max == null) { - var max = new byte[][] { null }; - doWithContexts(r -> { - byte[] localMax = PointValues.getMaxPackedValue(r, field); - // TODO: how to compare with the previous max - if (localMax != null) { - if (max[0] == null) { - max[0] = localMax; - } else { - throw new EsqlIllegalArgumentException("Don't know how to compare with previous max"); - } - } - return true; - }, true); - stat.max = max[0]; + @Override + public long count(String field) { + return 0; } - // return stat.max; - return null; - } - public boolean isSingleValue(String field) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.singleValue == null) { - // there's no such field so no need to worry about multi-value fields - if (exists(field) == false) { - stat.singleValue = true; - } else { - // fields are MV per default - var sv = new boolean[] { false }; - for (SearchExecutionContext context : contexts) { - MappedFieldType mappedType = context.isFieldMapped(field) ? context.getFieldType(field) : null; - if (mappedType != null) { - sv[0] = true; - doWithContexts(r -> { - sv[0] &= detectSingleValue(r, mappedType, field); - return sv[0]; - }, true); - break; - } - } - stat.singleValue = sv[0]; - } + @Override + public long count(String field, BytesRef value) { + return 0; } - return stat.singleValue; - } - public boolean isRuntimeField(String field) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.runtime == null) { - stat.runtime = false; - if (exists(field)) { - for (SearchExecutionContext context : contexts) { - if (context.isFieldMapped(field)) { - if (context.getFieldType(field) instanceof AbstractScriptFieldType) { - stat.runtime = true; - break; - } - } - } - } + @Override + public byte[] min(String field, DataType dataType) { + return null; } - return stat.runtime; - } - public boolean isIndexed(String field) { - var stat = cache.computeIfAbsent(field, s -> new FieldStat()); - if (stat.indexed == null) { - stat.indexed = false; - if (exists(field)) { - boolean indexed = true; - for (SearchExecutionContext context : contexts) { - if (context.isFieldMapped(field)) { - if (context.getFieldType(field).isIndexed() == false) { - indexed = false; - break; - } - } - } - stat.indexed = indexed; - } + @Override + public byte[] max(String field, DataType dataType) { + return null; } - return stat.indexed; - } - private boolean detectSingleValue(IndexReader r, MappedFieldType fieldType, String name) throws IOException { - // types that are always single value (and are accessible through instanceof) - if (fieldType instanceof ConstantFieldType || fieldType instanceof DocCountFieldType || fieldType instanceof TimestampFieldType) { + @Override + public boolean isSingleValue(String field) { return true; } - var typeName = fieldType.typeName(); - - // non-visible fields, check their names - boolean found = switch (typeName) { - case IdFieldMapper.NAME, SeqNoFieldMapper.NAME -> true; - default -> false; - }; - - if (found) { - return true; - } - - // check against doc size - DocCountTester tester = null; - if (fieldType instanceof DateFieldType || fieldType instanceof NumberFieldType) { - tester = lr -> { - PointValues values = lr.getPointValues(name); - return values == null || values.size() == values.getDocCount(); - }; - } else if (fieldType instanceof KeywordFieldType) { - tester = lr -> { - Terms terms = lr.terms(name); - return terms == null || terms.size() == terms.getDocCount(); - }; - } - - if (tester != null) { - // check each leaf - for (LeafReaderContext context : r.leaves()) { - if (tester.test(context.reader()) == false) { - return false; - } - } - // field is missing or single value - return true; - } - - // unsupported type - default to MV - return false; - } - - private interface DocCountTester { - Boolean test(LeafReader leafReader) throws IOException; - } - - // - // @see org.elasticsearch.search.query.QueryPhaseCollectorManager#shortcutTotalHitCount(IndexReader, Query) - // - private static long countEntries(IndexReader indexReader, String field) { - long count = 0; - try { - for (LeafReaderContext context : indexReader.leaves()) { - LeafReader reader = context.reader(); - FieldInfos fieldInfos = reader.getFieldInfos(); - FieldInfo fieldInfo = fieldInfos.fieldInfo(field); - - if (fieldInfo != null) { - if (fieldInfo.getDocValuesType() == DocValuesType.NONE) { - // no shortcut possible: it's a text field, empty values are counted as no value. - return -1; - } - if (fieldInfo.getPointIndexDimensionCount() > 0) { - PointValues points = reader.getPointValues(field); - if (points != null) { - count += points.size(); - } - } else if (fieldInfo.getIndexOptions() != IndexOptions.NONE) { - Terms terms = reader.terms(field); - if (terms != null) { - count += terms.getSumTotalTermFreq(); - } - } else { - return -1; // no shortcut possible for fields that are not indexed - } - } - } - } catch (IOException ex) { - throw new EsqlIllegalArgumentException("Cannot access data storage", ex); - } - return count; - } - - private interface IndexReaderConsumer { - /** - * Returns true if the consumer should keep on going, false otherwise. - */ - boolean consume(IndexReader reader) throws IOException; - } - - private boolean doWithContexts(IndexReaderConsumer consumer, boolean acceptsDeletions) { - try { - for (SearchExecutionContext context : contexts) { - for (LeafReaderContext leafContext : context.searcher().getLeafContexts()) { - var reader = leafContext.reader(); - if (acceptsDeletions == false && reader.hasDeletions()) { - return false; - } - // check if the looping continues or not - if (consumer.consume(reader) == false) { - return false; - } - } - } - return true; - } catch (IOException ex) { - throw new EsqlIllegalArgumentException("Cannot access data storage", ex); - } } } diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java index 905ca190ebe7..073a51ee6911 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java @@ -60,6 +60,7 @@ import org.elasticsearch.xpack.esql.plugin.QueryPragmas; import org.elasticsearch.xpack.esql.querydsl.query.SingleValueQuery; import org.elasticsearch.xpack.esql.session.Configuration; import org.elasticsearch.xpack.esql.stats.Metrics; +import org.elasticsearch.xpack.esql.stats.SearchContextStats; import org.elasticsearch.xpack.esql.stats.SearchStats; import org.junit.Before; @@ -330,7 +331,7 @@ public class LocalPhysicalPlanOptimizerTests extends MapperServiceTestCase { }, directoryReader -> { IndexSearcher searcher = newSearcher(directoryReader); SearchExecutionContext ctx = createSearchExecutionContext(mapperService, searcher); - plan.set(plannerOptimizer.plan(query, new SearchStats(List.of(ctx)))); + plan.set(plannerOptimizer.plan(query, SearchContextStats.from(List.of(ctx)))); }); return plan.get(); diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java index 9f5d6440e4a0..eb115ed7b294 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java @@ -33,6 +33,8 @@ import org.elasticsearch.search.sort.GeoDistanceSortBuilder; import org.elasticsearch.test.ESTestCase; import org.elasticsearch.xpack.core.enrich.EnrichPolicy; import org.elasticsearch.xpack.esql.EsqlTestUtils; +import org.elasticsearch.xpack.esql.EsqlTestUtils.TestConfigurableSearchStats; +import org.elasticsearch.xpack.esql.EsqlTestUtils.TestConfigurableSearchStats.Config; import org.elasticsearch.xpack.esql.VerificationException; import org.elasticsearch.xpack.esql.analysis.Analyzer; import org.elasticsearch.xpack.esql.analysis.AnalyzerContext; @@ -141,6 +143,7 @@ import static org.elasticsearch.index.query.QueryBuilders.boolQuery; import static org.elasticsearch.index.query.QueryBuilders.existsQuery; import static org.elasticsearch.test.ListMatcher.matchesList; import static org.elasticsearch.test.MapMatcher.assertMap; +import static org.elasticsearch.xpack.esql.EsqlTestUtils.TEST_SEARCH_STATS; import static org.elasticsearch.xpack.esql.EsqlTestUtils.TEST_VERIFIER; import static org.elasticsearch.xpack.esql.EsqlTestUtils.as; import static org.elasticsearch.xpack.esql.EsqlTestUtils.configuration; @@ -189,14 +192,16 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { private TestDataSource testData; private int allFieldRowSize; // TODO: Move this into testDataSource so tests that load other indexes can also assert on this private TestDataSource airports; - private TestDataSource airportsNoDocValues; - private TestDataSource airportsWeb; - private TestDataSource countriesBbox; - private TestDataSource countriesBboxWeb; + private TestDataSource airportsNoDocValues; // Test when spatial field is indexed but has no doc values + private TestDataSource airportsNotIndexed; // Test when spatial field has doc values but is not indexed + private TestDataSource airportsNotIndexedNorDocValues; // Test when spatial field is neither indexed nor has doc-values + private TestDataSource airportsWeb; // Cartesian point field tests + private TestDataSource countriesBbox; // geo_shape field tests + private TestDataSource countriesBboxWeb; // cartesian_shape field tests private final Configuration config; - private record TestDataSource(Map mapping, EsIndex index, Analyzer analyzer) {} + private record TestDataSource(Map mapping, EsIndex index, Analyzer analyzer, SearchStats stats) {} @ParametersFactory(argumentFormatting = PARAM_FORMATTING) public static List readScriptSpec() { @@ -240,9 +245,24 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { this.airports = makeTestDataSource("airports", "mapping-airports.json", functionRegistry, enrichResolution); this.airportsNoDocValues = makeTestDataSource( "airports-no-doc-values", - "mapping-airports-no-doc-values.json", + "mapping-airports_no_doc_values.json", functionRegistry, - enrichResolution + enrichResolution, + new TestConfigurableSearchStats().exclude(Config.DOC_VALUES, "location") + ); + this.airportsNotIndexed = makeTestDataSource( + "airports-not-indexed", + "mapping-airports_not_indexed.json", + functionRegistry, + enrichResolution, + new TestConfigurableSearchStats().exclude(Config.INDEXED, "location") + ); + this.airportsNotIndexedNorDocValues = makeTestDataSource( + "airports-not-indexed-nor-doc-values", + "mapping-airports_not_indexed_nor_doc_values.json", + functionRegistry, + enrichResolution, + new TestConfigurableSearchStats().exclude(Config.INDEXED, "location").exclude(Config.DOC_VALUES, "location") ); this.airportsWeb = makeTestDataSource("airports_web", "mapping-airports_web.json", functionRegistry, enrichResolution); this.countriesBbox = makeTestDataSource("countriesBbox", "mapping-countries_bbox.json", functionRegistry, enrichResolution); @@ -258,13 +278,23 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { String indexName, String mappingFileName, EsqlFunctionRegistry functionRegistry, - EnrichResolution enrichResolution + EnrichResolution enrichResolution, + SearchStats stats ) { Map mapping = loadMapping(mappingFileName); EsIndex index = new EsIndex(indexName, mapping, Map.of("test", IndexMode.STANDARD)); IndexResolution getIndexResult = IndexResolution.valid(index); Analyzer analyzer = new Analyzer(new AnalyzerContext(config, functionRegistry, getIndexResult, enrichResolution), TEST_VERIFIER); - return new TestDataSource(mapping, index, analyzer); + return new TestDataSource(mapping, index, analyzer, stats); + } + + TestDataSource makeTestDataSource( + String indexName, + String mappingFileName, + EsqlFunctionRegistry functionRegistry, + EnrichResolution enrichResolution + ) { + return makeTestDataSource(indexName, mappingFileName, functionRegistry, enrichResolution, TEST_SEARCH_STATS); } private static EnrichResolution setupEnrichResolution() { @@ -2132,7 +2162,7 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { | where long_noidx == 1 """); - var optimized = optimizedPlan(plan); + var optimized = optimizedPlan(plan, statsWithIndexedFields()); var limit = as(optimized, LimitExec.class); var exchange = asRemoteExchange(limit.child()); var project = as(exchange.child(), ProjectExec.class); @@ -2183,7 +2213,7 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { | sort long_noidx """); - var optimized = optimizedPlan(plan); + var optimized = optimizedPlan(plan, statsWithIndexedFields()); var topN = as(optimized, TopNExec.class); var exchange = as(topN.child(), ExchangeExec.class); var project = as(exchange.child(), ProjectExec.class); @@ -2656,7 +2686,8 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { "from airports | stats centroid = st_centroid_agg(to_geopoint(location))", "from airports | eval location = to_geopoint(location) | stats centroid = st_centroid_agg(location)" }) { for (boolean withDocValues : new boolean[] { false, true }) { - var plan = withDocValues ? physicalPlan(query, airports) : physicalPlan(query, airportsNoDocValues); + var testData = withDocValues ? airports : airportsNoDocValues; + var plan = physicalPlan(query, testData); var limit = as(plan, LimitExec.class); var agg = as(limit.child(), AggregateExec.class); @@ -2669,7 +2700,7 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { as(fAgg.child(), EsRelation.class); // Now optimize the plan and assert the aggregation uses doc-values - var optimized = optimizedPlan(plan); + var optimized = optimizedPlan(plan, testData.stats); limit = as(optimized, LimitExec.class); agg = as(limit.child(), AggregateExec.class); // Above the exchange (in coordinator) the aggregation is not using doc-values @@ -2943,11 +2974,12 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { * Note the FieldExtractExec has 'location' set for stats: FieldExtractExec[location{f}#9][location{f}#9] */ public void testSpatialTypesAndStatsUseDocValuesMultiAggregationsGrouped() { - for (boolean useDocValues : new boolean[] { true, false }) { + for (boolean useDocValues : new boolean[] { false }) { + var testData = useDocValues ? airports : airportsNoDocValues; var plan = this.physicalPlan(""" FROM airports | STATS centroid=ST_CENTROID_AGG(location), count=COUNT() BY scalerank - """, useDocValues ? airports : airportsNoDocValues); + """, testData); var limit = as(plan, LimitExec.class); var agg = as(limit.child(), AggregateExec.class); @@ -2964,7 +2996,7 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { as(fAgg.child(), EsRelation.class); // Now optimize the plan and assert the aggregation uses doc-values - var optimized = optimizedPlan(plan); + var optimized = optimizedPlan(plan, testData.stats); limit = as(optimized, LimitExec.class); agg = as(limit.child(), AggregateExec.class); att = as(agg.groupings().get(0), Attribute.class); @@ -3519,44 +3551,63 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { | STATS centroid=ST_CENTROID_AGG(location), count=COUNT() """ }) { - for (boolean useDocValues : new boolean[] { true, false }) { - var plan = this.physicalPlan(query, useDocValues ? airports : airportsNoDocValues); - var limit = as(plan, LimitExec.class); - var agg = as(limit.child(), AggregateExec.class); - assertThat("No groupings in aggregation", agg.groupings().size(), equalTo(0)); - // Before optimization the aggregation does not use doc-values - assertAggregation(agg, "count", Count.class); - assertAggregation(agg, "centroid", SpatialCentroid.class, GEO_POINT, false); + for (boolean isIndexed : new boolean[] { true, false }) { + for (boolean useDocValues : new boolean[] { true, false }) { + var testData = useDocValues + ? (isIndexed ? airports : airportsNotIndexed) + : (isIndexed ? airportsNoDocValues : airportsNotIndexedNorDocValues); + var plan = this.physicalPlan(query, testData); + var limit = as(plan, LimitExec.class); + var agg = as(limit.child(), AggregateExec.class); + assertThat("No groupings in aggregation", agg.groupings().size(), equalTo(0)); + // Before optimization the aggregation does not use doc-values + assertAggregation(agg, "count", Count.class); + assertAggregation(agg, "centroid", SpatialCentroid.class, GEO_POINT, false); - var exchange = as(agg.child(), ExchangeExec.class); - var fragment = as(exchange.child(), FragmentExec.class); - var fAgg = as(fragment.fragment(), Aggregate.class); - var filter = as(fAgg.child(), Filter.class); - assertThat("filter contains ST_INTERSECTS", filter.condition(), instanceOf(SpatialIntersects.class)); + var exchange = as(agg.child(), ExchangeExec.class); + var fragment = as(exchange.child(), FragmentExec.class); + var fAgg = as(fragment.fragment(), Aggregate.class); + var filter = as(fAgg.child(), Filter.class); + assertThat("filter contains ST_INTERSECTS", filter.condition(), instanceOf(SpatialIntersects.class)); - // Now verify that optimization re-writes the ExchangeExec and pushed down the filter into the Lucene query - var optimized = optimizedPlan(plan); - limit = as(optimized, LimitExec.class); - agg = as(limit.child(), AggregateExec.class); - // Above the exchange (in coordinator) the aggregation is not using doc-values - assertAggregation(agg, "count", Count.class); - assertAggregation(agg, "centroid", SpatialCentroid.class, GEO_POINT, false); - exchange = as(agg.child(), ExchangeExec.class); - agg = as(exchange.child(), AggregateExec.class); - assertThat("Aggregation is PARTIAL", agg.getMode(), equalTo(INITIAL)); - // below the exchange (in data node) the aggregation is using doc-values - assertAggregation(agg, "count", Count.class); - assertAggregation(agg, "centroid", SpatialCentroid.class, GEO_POINT, useDocValues); - var source = assertChildIsGeoPointExtract(useDocValues ? agg : as(agg.child(), FilterExec.class), useDocValues); - if (useDocValues) { - // Query is only pushed to lucene if indexing/doc-values are enabled - var condition = as(source.query(), SpatialRelatesQuery.ShapeQueryBuilder.class); - assertThat("Geometry field name", condition.fieldName(), equalTo("location")); - assertThat("Spatial relationship", condition.relation(), equalTo(ShapeRelation.INTERSECTS)); - assertThat("Geometry is Polygon", condition.shape().type(), equalTo(ShapeType.POLYGON)); - var polygon = as(condition.shape(), Polygon.class); - assertThat("Polygon shell length", polygon.getPolygon().length(), equalTo(5)); - assertThat("Polygon holes", polygon.getNumberOfHoles(), equalTo(0)); + // Now verify that optimization re-writes the ExchangeExec and pushed down the filter into the Lucene query + var optimized = optimizedPlan(plan, testData.stats); + limit = as(optimized, LimitExec.class); + agg = as(limit.child(), AggregateExec.class); + // Above the exchange (in coordinator) the aggregation is not using doc-values + assertAggregation(agg, "count", Count.class); + assertAggregation(agg, "centroid", SpatialCentroid.class, GEO_POINT, false); + exchange = as(agg.child(), ExchangeExec.class); + agg = as(exchange.child(), AggregateExec.class); + assertThat("Aggregation is PARTIAL", agg.getMode(), equalTo(INITIAL)); + // below the exchange (in data node) the aggregation is using doc-values + assertAggregation(agg, "count", Count.class); + assertAggregation(agg, "centroid", SpatialCentroid.class, GEO_POINT, useDocValues); + if (isIndexed) { + var source = assertChildIsGeoPointExtract(agg, useDocValues); + // Query is pushed to lucene if field is indexed (and does not require doc-values or isAggregatable) + var condition = as(source.query(), SpatialRelatesQuery.ShapeQueryBuilder.class); + assertThat("Geometry field name", condition.fieldName(), equalTo("location")); + assertThat("Spatial relationship", condition.relation(), equalTo(ShapeRelation.INTERSECTS)); + assertThat("Geometry is Polygon", condition.shape().type(), equalTo(ShapeType.POLYGON)); + var polygon = as(condition.shape(), Polygon.class); + assertThat("Polygon shell length", polygon.getPolygon().length(), equalTo(5)); + assertThat("Polygon holes", polygon.getNumberOfHoles(), equalTo(0)); + } else { + // If the field is not indexed, we cannot push the filter down to source, so assert that we need to have an explicit + // filter as well as extract the field needed for that filter. + var filterExec = as(agg.child(), FilterExec.class); + assertThat("filter contains ST_INTERSECTS", filterExec.condition(), instanceOf(SpatialIntersects.class)); + var fieldExtractLocation = as(filterExec.child(), FieldExtractExec.class); + assertThat("location field is extracted", fieldExtractLocation.attributesToExtract().size(), equalTo(1)); + assertThat( + "location field is extracted", + fieldExtractLocation.attributesToExtract().get(0).name(), + equalTo("location") + ); + var source = source(fieldExtractLocation.child()); + assertThat("source query is null", source.query(), equalTo(null)); + } } } } @@ -6651,14 +6702,7 @@ public class PhysicalPlanOptimizerTests extends ESTestCase { } static SearchStats statsWithIndexedFields(String... names) { - return new EsqlTestUtils.TestSearchStats() { - private final Set indexedFields = Set.of(names); - - @Override - public boolean isIndexed(String field) { - return indexedFields.contains(field); - } - }; + return new TestConfigurableSearchStats().include(Config.INDEXED, names); } static PhysicalPlan localRelationshipAlignment(PhysicalPlan l) { diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSourceTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSourceTests.java index 0fe7eb6b3d43..98f0af8e4b8e 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSourceTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/rules/physical/local/PushTopNToSourceTests.java @@ -34,7 +34,7 @@ import org.elasticsearch.xpack.esql.plan.physical.EsQueryExec; import org.elasticsearch.xpack.esql.plan.physical.EvalExec; import org.elasticsearch.xpack.esql.plan.physical.PhysicalPlan; import org.elasticsearch.xpack.esql.plan.physical.TopNExec; -import org.elasticsearch.xpack.esql.stats.DisabledSearchStats; +import org.elasticsearch.xpack.esql.stats.SearchStats; import java.io.IOException; import java.nio.ByteOrder; @@ -256,8 +256,7 @@ public class PushTopNToSourceTests extends ESTestCase { private static PhysicalPlan pushTopNToSource(TopNExec topNExec) { var configuration = EsqlTestUtils.configuration("from test"); - var searchStats = new DisabledSearchStats(); - var ctx = new LocalPhysicalOptimizerContext(configuration, searchStats); + var ctx = new LocalPhysicalOptimizerContext(configuration, SearchStats.EMPTY); var pushTopNToSource = new PushTopNToSource(); return pushTopNToSource.rule(topNExec, ctx); } diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/FilterTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/FilterTests.java index bb937700ef77..8d819f9dbcd6 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/FilterTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/planner/FilterTests.java @@ -318,7 +318,7 @@ public class FilterTests extends ESTestCase { } private QueryBuilder filterQueryForTransportNodes(PhysicalPlan plan) { - return PlannerUtils.detectFilter(plan, EMP_NO, x -> true); + return PlannerUtils.detectFilter(plan, EMP_NO); } @Override diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/stats/DisabledSearchStats.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/stats/DisabledSearchStats.java index 564d34149da0..fce05b07a6a4 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/stats/DisabledSearchStats.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/stats/DisabledSearchStats.java @@ -10,12 +10,26 @@ package org.elasticsearch.xpack.esql.stats; import org.apache.lucene.util.BytesRef; import org.elasticsearch.xpack.esql.core.type.DataType; -import static java.util.Collections.emptyList; +public class DisabledSearchStats implements SearchStats { -public class DisabledSearchStats extends SearchStats { + @Override + public boolean exists(String field) { + return true; + } - public DisabledSearchStats() { - super(emptyList()); + @Override + public boolean isIndexed(String field) { + return true; + } + + @Override + public boolean hasDocValues(String field) { + return true; + } + + @Override + public boolean hasExactSubfield(String field) { + return true; } @Override @@ -33,11 +47,6 @@ public class DisabledSearchStats extends SearchStats { return -1; } - @Override - public boolean exists(String field) { - return true; - } - @Override public byte[] min(String field, DataType dataType) { return null; diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java index b8a64be5dfd3..82f0ebf31650 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java @@ -21,8 +21,6 @@ import org.elasticsearch.xpack.esql.core.capabilities.UnresolvedException; import org.elasticsearch.xpack.esql.core.expression.Expression; import org.elasticsearch.xpack.esql.core.expression.FieldAttribute; import org.elasticsearch.xpack.esql.core.expression.Literal; -import org.elasticsearch.xpack.esql.core.expression.MetadataAttribute; -import org.elasticsearch.xpack.esql.core.expression.ReferenceAttribute; import org.elasticsearch.xpack.esql.core.expression.UnresolvedAttribute; import org.elasticsearch.xpack.esql.core.expression.UnresolvedAttributeTests; import org.elasticsearch.xpack.esql.core.expression.UnresolvedNamedExpression; @@ -114,9 +112,13 @@ import static org.mockito.Mockito.mock; * */ public class EsqlNodeSubclassTests> extends NodeSubclassTests { + private static final String ESQL_CORE_CLASS_PREFIX = "org.elasticsearch.xpack.esql.core"; + private static final String ESQL_CORE_JAR_LOCATION_SUBSTRING = "x-pack-esql-core"; + private static final String ESQL_CLASS_PREFIX = "org.elasticsearch.xpack.esql"; + private static final Predicate CLASSNAME_FILTER = className -> { - boolean esqlCore = className.startsWith("org.elasticsearch.xpack.esql.core") != false; - boolean esqlProper = className.startsWith("org.elasticsearch.xpack.esql") != false; + boolean esqlCore = className.startsWith(ESQL_CORE_CLASS_PREFIX) != false; + boolean esqlProper = className.startsWith(ESQL_CLASS_PREFIX) != false; return (esqlCore || esqlProper); }; @@ -164,15 +166,6 @@ public class EsqlNodeSubclassTests> extends NodeS */ expectedCount -= 1; - // special exceptions with private constructors - if (MetadataAttribute.class.equals(subclass) || ReferenceAttribute.class.equals(subclass)) { - expectedCount++; - } - - if (FieldAttribute.class.equals(subclass)) { - expectedCount += 2; - } - assertEquals(expectedCount, info(node).properties().size()); } @@ -736,7 +729,7 @@ public class EsqlNodeSubclassTests> extends NodeS // NIO FileSystem API is not used since it trips the SecurityManager // https://bugs.openjdk.java.net/browse/JDK-8160798 // so iterate the jar "by hand" - if (path.endsWith(".jar") && path.contains("x-pack-ql")) { + if (path.endsWith(".jar") && path.contains(ESQL_CORE_JAR_LOCATION_SUBSTRING)) { try (JarInputStream jar = jarStream(root)) { JarEntry je = null; while ((je = jar.getNextJarEntry()) != null) { diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceFeatures.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceFeatures.java index 216b5c984eca..10ffedef14e2 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceFeatures.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceFeatures.java @@ -38,6 +38,9 @@ public class InferenceFeatures implements FeatureSpecification { @Override public Set getTestFeatures() { - return Set.of(SemanticTextFieldMapper.SEMANTIC_TEXT_IN_OBJECT_FIELD_FIX); + return Set.of( + SemanticTextFieldMapper.SEMANTIC_TEXT_IN_OBJECT_FIELD_FIX, + SemanticTextFieldMapper.SEMANTIC_TEXT_SINGLE_FIELD_UPDATE_FIX + ); } } diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/action/filter/ShardBulkInferenceActionFilter.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/action/filter/ShardBulkInferenceActionFilter.java index 1a6e4760fe12..b3bbe3a7df9b 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/action/filter/ShardBulkInferenceActionFilter.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/action/filter/ShardBulkInferenceActionFilter.java @@ -446,7 +446,8 @@ public class ShardBulkInferenceActionFilter implements MappedActionFilter { String field = entry.getName(); String inferenceId = entry.getInferenceId(); var originalFieldValue = XContentMapValues.extractValue(field, docMap); - if (originalFieldValue instanceof Map) { + if (originalFieldValue instanceof Map || (originalFieldValue == null && entry.getSourceFields().length == 1)) { + // Inference has already been computed, or there is no inference required. continue; } int order = 0; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java index 4c0751605128..f0cb612c9082 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java @@ -91,6 +91,8 @@ public class SemanticTextFieldMapper extends FieldMapper implements InferenceFie public static final NodeFeature SEMANTIC_TEXT_DEFAULT_ELSER_2 = new NodeFeature("semantic_text.default_elser_2"); public static final NodeFeature SEMANTIC_TEXT_IN_OBJECT_FIELD_FIX = new NodeFeature("semantic_text.in_object_field_fix"); + public static final NodeFeature SEMANTIC_TEXT_SINGLE_FIELD_UPDATE_FIX = new NodeFeature("semantic_text.single_field_update_fix"); + public static final String CONTENT_TYPE = "semantic_text"; public static final String DEFAULT_ELSER_2_INFERENCE_ID = DEFAULT_ELSER_ID; diff --git a/x-pack/plugin/inference/src/yamlRestTest/resources/rest-api-spec/test/inference/60_semantic_text_inference_update.yml b/x-pack/plugin/inference/src/yamlRestTest/resources/rest-api-spec/test/inference/60_semantic_text_inference_update.yml index 59ce439d954a..294761608ee8 100644 --- a/x-pack/plugin/inference/src/yamlRestTest/resources/rest-api-spec/test/inference/60_semantic_text_inference_update.yml +++ b/x-pack/plugin/inference/src/yamlRestTest/resources/rest-api-spec/test/inference/60_semantic_text_inference_update.yml @@ -610,3 +610,59 @@ setup: - exists: _source.dense_field.inference.chunks.0.embeddings - match: { _source.dense_field.inference.chunks.0.text: "another updated inference test" } - match: { _source.non_inference_field: "updated non inference test" } + +--- +"Bypass inference on bulk update operation": + - requires: + cluster_features: semantic_text.single_field_update_fix + reason: Standalone semantic text fields are now optional in a bulk update operation + + # Update as upsert + - do: + bulk: + body: + - '{"update": {"_index": "test-index", "_id": "doc_1"}}' + - '{"doc": { "sparse_field": "inference test", "dense_field": "another inference test", "non_inference_field": "non inference test" }, "doc_as_upsert": true}' + + - match: { errors: false } + - match: { items.0.update.result: "created" } + + - do: + bulk: + body: + - '{"update": {"_index": "test-index", "_id": "doc_1"}}' + - '{"doc": { "non_inference_field": "another value" }, "doc_as_upsert": true}' + + - match: { errors: false } + - match: { items.0.update.result: "updated" } + + - do: + get: + index: test-index + id: doc_1 + + - match: { _source.sparse_field.text: "inference test" } + - exists: _source.sparse_field.inference.chunks.0.embeddings + - match: { _source.sparse_field.inference.chunks.0.text: "inference test" } + - match: { _source.dense_field.text: "another inference test" } + - exists: _source.dense_field.inference.chunks.0.embeddings + - match: { _source.dense_field.inference.chunks.0.text: "another inference test" } + - match: { _source.non_inference_field: "another value" } + + - do: + bulk: + body: + - '{"update": {"_index": "test-index", "_id": "doc_1"}}' + - '{"doc": { "sparse_field": null, "dense_field": null, "non_inference_field": "updated value" }, "doc_as_upsert": true}' + + - match: { errors: false } + - match: { items.0.update.result: "updated" } + + - do: + get: + index: test-index + id: doc_1 + + - match: { _source.sparse_field: null } + - match: { _source.dense_field: null } + - match: { _source.non_inference_field: "updated value" } diff --git a/x-pack/plugin/kql/build.gradle b/x-pack/plugin/kql/build.gradle index 9d0860346b18..7e4df5654f22 100644 --- a/x-pack/plugin/kql/build.gradle +++ b/x-pack/plugin/kql/build.gradle @@ -1,8 +1,10 @@ import org.elasticsearch.gradle.internal.info.BuildParams + import static org.elasticsearch.gradle.util.PlatformUtils.normalize apply plugin: 'elasticsearch.internal-es-plugin' apply plugin: 'elasticsearch.internal-cluster-test' +apply plugin: 'elasticsearch.internal-yaml-rest-test' apply plugin: 'elasticsearch.publish' esplugin { @@ -17,19 +19,21 @@ base { dependencies { compileOnly project(path: xpackModule('core')) - api "org.antlr:antlr4-runtime:${versions.antlr4}" + implementation "org.antlr:antlr4-runtime:${versions.antlr4}" testImplementation "org.antlr:antlr4-runtime:${versions.antlr4}" testImplementation project(':test:framework') testImplementation(testArtifact(project(xpackModule('core')))) } -/**************************************************************** - * Enable QA/rest integration tests for snapshot builds only * - * TODO: Enable for all builds upon this feature release * - ****************************************************************/ -if (BuildParams.isSnapshotBuild()) { - addQaCheckDependencies(project) +tasks.named('yamlRestTest') { + usesDefaultDistribution() +}.configure { + /**************************************************************** + * Enable QA/rest integration tests for snapshot builds only * + * TODO: Enable for all builds upon this feature release * + ****************************************************************/ + enabled = BuildParams.isSnapshotBuild() } /********************************** diff --git a/x-pack/plugin/kql/src/main/antlr/KqlBase.g4 b/x-pack/plugin/kql/src/main/antlr/KqlBase.g4 index 52a70b9d4c01..da015b699cb1 100644 --- a/x-pack/plugin/kql/src/main/antlr/KqlBase.g4 +++ b/x-pack/plugin/kql/src/main/antlr/KqlBase.g4 @@ -88,7 +88,7 @@ fieldQueryValue ; fieldName - : value=UNQUOTED_LITERAL+ + : value=UNQUOTED_LITERAL | value=QUOTED_STRING | value=WILDCARD ; diff --git a/x-pack/plugin/kql/src/main/java/module-info.java b/x-pack/plugin/kql/src/main/java/module-info.java index c4dd539508f3..41e51033b9c7 100644 --- a/x-pack/plugin/kql/src/main/java/module-info.java +++ b/x-pack/plugin/kql/src/main/java/module-info.java @@ -16,4 +16,5 @@ module org.elasticsearch.kql { exports org.elasticsearch.xpack.kql; exports org.elasticsearch.xpack.kql.parser; + exports org.elasticsearch.xpack.kql.query; } diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/KqlPlugin.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/KqlPlugin.java index 4734924b2361..217513bd2c0d 100644 --- a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/KqlPlugin.java +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/KqlPlugin.java @@ -7,10 +7,21 @@ package org.elasticsearch.xpack.kql; +import org.elasticsearch.Build; import org.elasticsearch.plugins.ExtensiblePlugin; import org.elasticsearch.plugins.Plugin; import org.elasticsearch.plugins.SearchPlugin; +import org.elasticsearch.xpack.kql.query.KqlQueryBuilder; + +import java.util.List; public class KqlPlugin extends Plugin implements SearchPlugin, ExtensiblePlugin { + @Override + public List> getQueries() { + if (Build.current().isSnapshot()) { + return List.of(new SearchPlugin.QuerySpec<>(KqlQueryBuilder.NAME, KqlQueryBuilder::new, KqlQueryBuilder::fromXContent)); + } + return List.of(); + } } diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlAstBuilder.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlAstBuilder.java index a6de28104e31..5fe3a61c0a76 100644 --- a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlAstBuilder.java +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlAstBuilder.java @@ -9,6 +9,7 @@ package org.elasticsearch.xpack.kql.parser; import org.antlr.v4.runtime.ParserRuleContext; import org.antlr.v4.runtime.Token; +import org.elasticsearch.common.regex.Regex; import org.elasticsearch.index.mapper.MappedFieldType; import org.elasticsearch.index.query.BoolQueryBuilder; import org.elasticsearch.index.query.MatchAllQueryBuilder; @@ -16,29 +17,34 @@ import org.elasticsearch.index.query.MatchNoneQueryBuilder; import org.elasticsearch.index.query.MultiMatchQueryBuilder; import org.elasticsearch.index.query.QueryBuilder; import org.elasticsearch.index.query.QueryBuilders; +import org.elasticsearch.index.query.QueryStringQueryBuilder; import org.elasticsearch.index.query.RangeQueryBuilder; +import java.util.Set; import java.util.function.BiConsumer; import java.util.function.BiFunction; import static org.elasticsearch.common.logging.LoggerMessageFormat.format; -import static org.elasticsearch.xpack.kql.parser.KqlParserExecutionContext.isDateField; -import static org.elasticsearch.xpack.kql.parser.KqlParserExecutionContext.isKeywordField; -import static org.elasticsearch.xpack.kql.parser.KqlParserExecutionContext.isRuntimeField; +import static org.elasticsearch.xpack.kql.parser.KqlParsingContext.isDateField; +import static org.elasticsearch.xpack.kql.parser.KqlParsingContext.isKeywordField; +import static org.elasticsearch.xpack.kql.parser.KqlParsingContext.isRuntimeField; +import static org.elasticsearch.xpack.kql.parser.KqlParsingContext.isSearchableField; import static org.elasticsearch.xpack.kql.parser.ParserUtils.escapeLuceneQueryString; +import static org.elasticsearch.xpack.kql.parser.ParserUtils.extractText; import static org.elasticsearch.xpack.kql.parser.ParserUtils.hasWildcard; +import static org.elasticsearch.xpack.kql.parser.ParserUtils.typedParsing; class KqlAstBuilder extends KqlBaseBaseVisitor { - private final KqlParserExecutionContext kqlParserExecutionContext; + private final KqlParsingContext kqlParsingContext; - KqlAstBuilder(KqlParserExecutionContext kqlParserExecutionContext) { - this.kqlParserExecutionContext = kqlParserExecutionContext; + KqlAstBuilder(KqlParsingContext kqlParsingContext) { + this.kqlParsingContext = kqlParsingContext; } public QueryBuilder toQueryBuilder(ParserRuleContext ctx) { if (ctx instanceof KqlBaseParser.TopLevelQueryContext topLeveQueryContext) { if (topLeveQueryContext.query() != null) { - return ParserUtils.typedParsing(this, topLeveQueryContext.query(), QueryBuilder.class); + return typedParsing(this, topLeveQueryContext.query(), QueryBuilder.class); } return new MatchAllQueryBuilder(); @@ -59,9 +65,9 @@ class KqlAstBuilder extends KqlBaseBaseVisitor { // TODO: KQLContext has an option to wrap the clauses into a filter instead of a must clause. Do we need it? for (ParserRuleContext subQueryCtx : ctx.query()) { if (subQueryCtx instanceof KqlBaseParser.BooleanQueryContext booleanSubQueryCtx && isAndQuery(booleanSubQueryCtx)) { - ParserUtils.typedParsing(this, subQueryCtx, BoolQueryBuilder.class).must().forEach(builder::must); + typedParsing(this, subQueryCtx, BoolQueryBuilder.class).must().forEach(builder::must); } else { - builder.must(ParserUtils.typedParsing(this, subQueryCtx, QueryBuilder.class)); + builder.must(typedParsing(this, subQueryCtx, QueryBuilder.class)); } } @@ -73,9 +79,9 @@ class KqlAstBuilder extends KqlBaseBaseVisitor { for (ParserRuleContext subQueryCtx : ctx.query()) { if (subQueryCtx instanceof KqlBaseParser.BooleanQueryContext booleanSubQueryCtx && isOrQuery(booleanSubQueryCtx)) { - ParserUtils.typedParsing(this, subQueryCtx, BoolQueryBuilder.class).should().forEach(builder::should); + typedParsing(this, subQueryCtx, BoolQueryBuilder.class).should().forEach(builder::should); } else { - builder.should(ParserUtils.typedParsing(this, subQueryCtx, QueryBuilder.class)); + builder.should(typedParsing(this, subQueryCtx, QueryBuilder.class)); } } @@ -84,12 +90,12 @@ class KqlAstBuilder extends KqlBaseBaseVisitor { @Override public QueryBuilder visitNotQuery(KqlBaseParser.NotQueryContext ctx) { - return QueryBuilders.boolQuery().mustNot(ParserUtils.typedParsing(this, ctx.simpleQuery(), QueryBuilder.class)); + return QueryBuilders.boolQuery().mustNot(typedParsing(this, ctx.simpleQuery(), QueryBuilder.class)); } @Override public QueryBuilder visitParenthesizedQuery(KqlBaseParser.ParenthesizedQueryContext ctx) { - return ParserUtils.typedParsing(this, ctx.query(), QueryBuilder.class); + return typedParsing(this, ctx.query(), QueryBuilder.class); } @Override @@ -121,12 +127,16 @@ class KqlAstBuilder extends KqlBaseBaseVisitor { public QueryBuilder visitRangeQuery(KqlBaseParser.RangeQueryContext ctx) { BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery().minimumShouldMatch(1); - String queryText = ParserUtils.extractText(ctx.rangeQueryValue()); + String queryText = extractText(ctx.rangeQueryValue()); BiFunction rangeOperation = rangeOperation(ctx.operator); withFields(ctx.fieldName(), (fieldName, mappedFieldType) -> { RangeQueryBuilder rangeQuery = rangeOperation.apply(QueryBuilders.rangeQuery(fieldName), queryText); - // TODO: add timezone for date fields + + if (kqlParsingContext.timeZone() != null) { + rangeQuery.timeZone(kqlParsingContext.timeZone().getId()); + } + boolQueryBuilder.should(rangeQuery); }); @@ -135,42 +145,54 @@ class KqlAstBuilder extends KqlBaseBaseVisitor { @Override public QueryBuilder visitFieldLessQuery(KqlBaseParser.FieldLessQueryContext ctx) { - String queryText = ParserUtils.extractText(ctx.fieldQueryValue()); + String queryText = extractText(ctx.fieldQueryValue()); if (hasWildcard(ctx.fieldQueryValue())) { - // TODO: set default fields. - return QueryBuilders.queryStringQuery(escapeLuceneQueryString(queryText, true)); + QueryStringQueryBuilder queryString = QueryBuilders.queryStringQuery(escapeLuceneQueryString(queryText, true)); + if (kqlParsingContext.defaultField() != null) { + queryString.defaultField(kqlParsingContext.defaultField()); + } + return queryString; } boolean isPhraseMatch = ctx.fieldQueryValue().QUOTED_STRING() != null; - return QueryBuilders.multiMatchQuery(queryText) - // TODO: add default fields? + MultiMatchQueryBuilder multiMatchQuery = QueryBuilders.multiMatchQuery(queryText) .type(isPhraseMatch ? MultiMatchQueryBuilder.Type.PHRASE : MultiMatchQueryBuilder.Type.BEST_FIELDS) .lenient(true); + + if (kqlParsingContext.defaultField() != null) { + kqlParsingContext.resolveDefaultFieldNames() + .stream() + .filter(kqlParsingContext::isSearchableField) + .forEach(multiMatchQuery::field); + } + + return multiMatchQuery; } @Override public QueryBuilder visitFieldQuery(KqlBaseParser.FieldQueryContext ctx) { BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery().minimumShouldMatch(1); - String queryText = ParserUtils.extractText(ctx.fieldQueryValue()); + String queryText = extractText(ctx.fieldQueryValue()); boolean hasWildcard = hasWildcard(ctx.fieldQueryValue()); withFields(ctx.fieldName(), (fieldName, mappedFieldType) -> { QueryBuilder fieldQuery = null; if (hasWildcard && isKeywordField(mappedFieldType)) { - fieldQuery = QueryBuilders.wildcardQuery(fieldName, queryText) - .caseInsensitive(kqlParserExecutionContext.isCaseSensitive() == false); + fieldQuery = QueryBuilders.wildcardQuery(fieldName, queryText).caseInsensitive(kqlParsingContext.caseInsensitive()); } else if (hasWildcard) { fieldQuery = QueryBuilders.queryStringQuery(escapeLuceneQueryString(queryText, true)).field(fieldName); } else if (isDateField(mappedFieldType)) { - // TODO: add timezone - fieldQuery = QueryBuilders.rangeQuery(fieldName).gte(queryText).lte(queryText); + RangeQueryBuilder rangeFieldQuery = QueryBuilders.rangeQuery(fieldName).gte(queryText).lte(queryText); + if (kqlParsingContext.timeZone() != null) { + rangeFieldQuery.timeZone(kqlParsingContext.timeZone().getId()); + } + fieldQuery = rangeFieldQuery; } else if (isKeywordField(mappedFieldType)) { - fieldQuery = QueryBuilders.termQuery(fieldName, queryText) - .caseInsensitive(kqlParserExecutionContext.isCaseSensitive() == false); + fieldQuery = QueryBuilders.termQuery(fieldName, queryText).caseInsensitive(kqlParsingContext.caseInsensitive()); } else if (ctx.fieldQueryValue().QUOTED_STRING() != null) { fieldQuery = QueryBuilders.matchPhraseQuery(fieldName, queryText); } else { @@ -194,7 +216,26 @@ class KqlAstBuilder extends KqlBaseBaseVisitor { } private void withFields(KqlBaseParser.FieldNameContext ctx, BiConsumer fieldConsummer) { - kqlParserExecutionContext.resolveFields(ctx).forEach(fieldDef -> fieldConsummer.accept(fieldDef.v1(), fieldDef.v2())); + assert ctx != null : "Field ctx cannot be null"; + String fieldNamePattern = extractText(ctx); + Set fieldNames = kqlParsingContext.resolveFieldNames(fieldNamePattern); + + if (ctx.value.getType() == KqlBaseParser.QUOTED_STRING && Regex.isSimpleMatchPattern(fieldNamePattern)) { + // When using quoted string, wildcards are not expanded. + // No field can match and we can return early. + return; + } + + if (ctx.value.getType() == KqlBaseParser.QUOTED_STRING) { + assert fieldNames.size() < 2 : "expecting only one matching field"; + } + + fieldNames.forEach(fieldName -> { + MappedFieldType fieldType = kqlParsingContext.fieldType(fieldName); + if (isSearchableField(fieldName, fieldType)) { + fieldConsummer.accept(fieldName, fieldType); + } + }); } private QueryBuilder rewriteDisjunctionQuery(BoolQueryBuilder boolQueryBuilder) { diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBase.interp b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBase.interp index 2b09dd52e95b..7af37d7e3c3b 100644 --- a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBase.interp +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBase.interp @@ -54,4 +54,4 @@ fieldName atn: -[4, 1, 16, 140, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 2, 4, 7, 4, 2, 5, 7, 5, 2, 6, 7, 6, 2, 7, 7, 7, 2, 8, 7, 8, 2, 9, 7, 9, 2, 10, 7, 10, 2, 11, 7, 11, 2, 12, 7, 12, 2, 13, 7, 13, 1, 0, 3, 0, 30, 8, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5, 1, 40, 8, 1, 10, 1, 12, 1, 43, 9, 1, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 53, 8, 2, 1, 3, 1, 3, 1, 3, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 5, 1, 5, 3, 5, 66, 8, 5, 1, 5, 1, 5, 1, 6, 1, 6, 1, 6, 1, 6, 1, 7, 1, 7, 1, 7, 1, 7, 1, 8, 4, 8, 79, 8, 8, 11, 8, 12, 8, 80, 1, 8, 3, 8, 84, 8, 8, 1, 9, 1, 9, 1, 9, 1, 9, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 3, 10, 100, 8, 10, 1, 11, 1, 11, 1, 11, 1, 11, 1, 11, 3, 11, 107, 8, 11, 1, 12, 3, 12, 110, 8, 12, 1, 12, 4, 12, 113, 8, 12, 11, 12, 12, 12, 114, 1, 12, 3, 12, 118, 8, 12, 1, 12, 1, 12, 3, 12, 122, 8, 12, 1, 12, 1, 12, 3, 12, 126, 8, 12, 1, 12, 3, 12, 129, 8, 12, 1, 13, 4, 13, 132, 8, 13, 11, 13, 12, 13, 133, 1, 13, 1, 13, 3, 13, 138, 8, 13, 1, 13, 0, 1, 2, 14, 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 0, 4, 1, 0, 2, 3, 1, 0, 6, 9, 2, 0, 14, 14, 16, 16, 1, 0, 2, 4, 150, 0, 29, 1, 0, 0, 0, 2, 33, 1, 0, 0, 0, 4, 52, 1, 0, 0, 0, 6, 54, 1, 0, 0, 0, 8, 57, 1, 0, 0, 0, 10, 65, 1, 0, 0, 0, 12, 69, 1, 0, 0, 0, 14, 73, 1, 0, 0, 0, 16, 83, 1, 0, 0, 0, 18, 85, 1, 0, 0, 0, 20, 99, 1, 0, 0, 0, 22, 106, 1, 0, 0, 0, 24, 128, 1, 0, 0, 0, 26, 137, 1, 0, 0, 0, 28, 30, 3, 2, 1, 0, 29, 28, 1, 0, 0, 0, 29, 30, 1, 0, 0, 0, 30, 31, 1, 0, 0, 0, 31, 32, 5, 0, 0, 1, 32, 1, 1, 0, 0, 0, 33, 34, 6, 1, -1, 0, 34, 35, 3, 4, 2, 0, 35, 41, 1, 0, 0, 0, 36, 37, 10, 2, 0, 0, 37, 38, 7, 0, 0, 0, 38, 40, 3, 2, 1, 2, 39, 36, 1, 0, 0, 0, 40, 43, 1, 0, 0, 0, 41, 39, 1, 0, 0, 0, 41, 42, 1, 0, 0, 0, 42, 3, 1, 0, 0, 0, 43, 41, 1, 0, 0, 0, 44, 53, 3, 6, 3, 0, 45, 53, 3, 8, 4, 0, 46, 53, 3, 12, 6, 0, 47, 53, 3, 10, 5, 0, 48, 53, 3, 18, 9, 0, 49, 53, 3, 14, 7, 0, 50, 53, 3, 20, 10, 0, 51, 53, 3, 22, 11, 0, 52, 44, 1, 0, 0, 0, 52, 45, 1, 0, 0, 0, 52, 46, 1, 0, 0, 0, 52, 47, 1, 0, 0, 0, 52, 48, 1, 0, 0, 0, 52, 49, 1, 0, 0, 0, 52, 50, 1, 0, 0, 0, 52, 51, 1, 0, 0, 0, 53, 5, 1, 0, 0, 0, 54, 55, 5, 4, 0, 0, 55, 56, 3, 4, 2, 0, 56, 7, 1, 0, 0, 0, 57, 58, 3, 26, 13, 0, 58, 59, 5, 5, 0, 0, 59, 60, 5, 12, 0, 0, 60, 61, 3, 2, 1, 0, 61, 62, 5, 13, 0, 0, 62, 9, 1, 0, 0, 0, 63, 64, 5, 16, 0, 0, 64, 66, 5, 5, 0, 0, 65, 63, 1, 0, 0, 0, 65, 66, 1, 0, 0, 0, 66, 67, 1, 0, 0, 0, 67, 68, 5, 16, 0, 0, 68, 11, 1, 0, 0, 0, 69, 70, 5, 10, 0, 0, 70, 71, 3, 2, 1, 0, 71, 72, 5, 11, 0, 0, 72, 13, 1, 0, 0, 0, 73, 74, 3, 26, 13, 0, 74, 75, 7, 1, 0, 0, 75, 76, 3, 16, 8, 0, 76, 15, 1, 0, 0, 0, 77, 79, 7, 2, 0, 0, 78, 77, 1, 0, 0, 0, 79, 80, 1, 0, 0, 0, 80, 78, 1, 0, 0, 0, 80, 81, 1, 0, 0, 0, 81, 84, 1, 0, 0, 0, 82, 84, 5, 15, 0, 0, 83, 78, 1, 0, 0, 0, 83, 82, 1, 0, 0, 0, 84, 17, 1, 0, 0, 0, 85, 86, 3, 26, 13, 0, 86, 87, 5, 5, 0, 0, 87, 88, 5, 16, 0, 0, 88, 19, 1, 0, 0, 0, 89, 90, 3, 26, 13, 0, 90, 91, 5, 5, 0, 0, 91, 92, 3, 24, 12, 0, 92, 100, 1, 0, 0, 0, 93, 94, 3, 26, 13, 0, 94, 95, 5, 5, 0, 0, 95, 96, 5, 10, 0, 0, 96, 97, 3, 24, 12, 0, 97, 98, 5, 11, 0, 0, 98, 100, 1, 0, 0, 0, 99, 89, 1, 0, 0, 0, 99, 93, 1, 0, 0, 0, 100, 21, 1, 0, 0, 0, 101, 107, 3, 24, 12, 0, 102, 103, 5, 10, 0, 0, 103, 104, 3, 24, 12, 0, 104, 105, 5, 11, 0, 0, 105, 107, 1, 0, 0, 0, 106, 101, 1, 0, 0, 0, 106, 102, 1, 0, 0, 0, 107, 23, 1, 0, 0, 0, 108, 110, 7, 3, 0, 0, 109, 108, 1, 0, 0, 0, 109, 110, 1, 0, 0, 0, 110, 112, 1, 0, 0, 0, 111, 113, 7, 2, 0, 0, 112, 111, 1, 0, 0, 0, 113, 114, 1, 0, 0, 0, 114, 112, 1, 0, 0, 0, 114, 115, 1, 0, 0, 0, 115, 117, 1, 0, 0, 0, 116, 118, 7, 3, 0, 0, 117, 116, 1, 0, 0, 0, 117, 118, 1, 0, 0, 0, 118, 129, 1, 0, 0, 0, 119, 121, 7, 0, 0, 0, 120, 122, 7, 3, 0, 0, 121, 120, 1, 0, 0, 0, 121, 122, 1, 0, 0, 0, 122, 129, 1, 0, 0, 0, 123, 125, 5, 4, 0, 0, 124, 126, 7, 0, 0, 0, 125, 124, 1, 0, 0, 0, 125, 126, 1, 0, 0, 0, 126, 129, 1, 0, 0, 0, 127, 129, 5, 15, 0, 0, 128, 109, 1, 0, 0, 0, 128, 119, 1, 0, 0, 0, 128, 123, 1, 0, 0, 0, 128, 127, 1, 0, 0, 0, 129, 25, 1, 0, 0, 0, 130, 132, 5, 14, 0, 0, 131, 130, 1, 0, 0, 0, 132, 133, 1, 0, 0, 0, 133, 131, 1, 0, 0, 0, 133, 134, 1, 0, 0, 0, 134, 138, 1, 0, 0, 0, 135, 138, 5, 15, 0, 0, 136, 138, 5, 16, 0, 0, 137, 131, 1, 0, 0, 0, 137, 135, 1, 0, 0, 0, 137, 136, 1, 0, 0, 0, 138, 27, 1, 0, 0, 0, 16, 29, 41, 52, 65, 80, 83, 99, 106, 109, 114, 117, 121, 125, 128, 133, 137] \ No newline at end of file +[4, 1, 16, 136, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 2, 4, 7, 4, 2, 5, 7, 5, 2, 6, 7, 6, 2, 7, 7, 7, 2, 8, 7, 8, 2, 9, 7, 9, 2, 10, 7, 10, 2, 11, 7, 11, 2, 12, 7, 12, 2, 13, 7, 13, 1, 0, 3, 0, 30, 8, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5, 1, 40, 8, 1, 10, 1, 12, 1, 43, 9, 1, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 53, 8, 2, 1, 3, 1, 3, 1, 3, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 4, 1, 5, 1, 5, 3, 5, 66, 8, 5, 1, 5, 1, 5, 1, 6, 1, 6, 1, 6, 1, 6, 1, 7, 1, 7, 1, 7, 1, 7, 1, 8, 4, 8, 79, 8, 8, 11, 8, 12, 8, 80, 1, 8, 3, 8, 84, 8, 8, 1, 9, 1, 9, 1, 9, 1, 9, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 3, 10, 100, 8, 10, 1, 11, 1, 11, 1, 11, 1, 11, 1, 11, 3, 11, 107, 8, 11, 1, 12, 3, 12, 110, 8, 12, 1, 12, 4, 12, 113, 8, 12, 11, 12, 12, 12, 114, 1, 12, 3, 12, 118, 8, 12, 1, 12, 1, 12, 3, 12, 122, 8, 12, 1, 12, 1, 12, 3, 12, 126, 8, 12, 1, 12, 3, 12, 129, 8, 12, 1, 13, 1, 13, 1, 13, 3, 13, 134, 8, 13, 1, 13, 0, 1, 2, 14, 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 0, 4, 1, 0, 2, 3, 1, 0, 6, 9, 2, 0, 14, 14, 16, 16, 1, 0, 2, 4, 145, 0, 29, 1, 0, 0, 0, 2, 33, 1, 0, 0, 0, 4, 52, 1, 0, 0, 0, 6, 54, 1, 0, 0, 0, 8, 57, 1, 0, 0, 0, 10, 65, 1, 0, 0, 0, 12, 69, 1, 0, 0, 0, 14, 73, 1, 0, 0, 0, 16, 83, 1, 0, 0, 0, 18, 85, 1, 0, 0, 0, 20, 99, 1, 0, 0, 0, 22, 106, 1, 0, 0, 0, 24, 128, 1, 0, 0, 0, 26, 133, 1, 0, 0, 0, 28, 30, 3, 2, 1, 0, 29, 28, 1, 0, 0, 0, 29, 30, 1, 0, 0, 0, 30, 31, 1, 0, 0, 0, 31, 32, 5, 0, 0, 1, 32, 1, 1, 0, 0, 0, 33, 34, 6, 1, -1, 0, 34, 35, 3, 4, 2, 0, 35, 41, 1, 0, 0, 0, 36, 37, 10, 2, 0, 0, 37, 38, 7, 0, 0, 0, 38, 40, 3, 2, 1, 2, 39, 36, 1, 0, 0, 0, 40, 43, 1, 0, 0, 0, 41, 39, 1, 0, 0, 0, 41, 42, 1, 0, 0, 0, 42, 3, 1, 0, 0, 0, 43, 41, 1, 0, 0, 0, 44, 53, 3, 6, 3, 0, 45, 53, 3, 8, 4, 0, 46, 53, 3, 12, 6, 0, 47, 53, 3, 10, 5, 0, 48, 53, 3, 18, 9, 0, 49, 53, 3, 14, 7, 0, 50, 53, 3, 20, 10, 0, 51, 53, 3, 22, 11, 0, 52, 44, 1, 0, 0, 0, 52, 45, 1, 0, 0, 0, 52, 46, 1, 0, 0, 0, 52, 47, 1, 0, 0, 0, 52, 48, 1, 0, 0, 0, 52, 49, 1, 0, 0, 0, 52, 50, 1, 0, 0, 0, 52, 51, 1, 0, 0, 0, 53, 5, 1, 0, 0, 0, 54, 55, 5, 4, 0, 0, 55, 56, 3, 4, 2, 0, 56, 7, 1, 0, 0, 0, 57, 58, 3, 26, 13, 0, 58, 59, 5, 5, 0, 0, 59, 60, 5, 12, 0, 0, 60, 61, 3, 2, 1, 0, 61, 62, 5, 13, 0, 0, 62, 9, 1, 0, 0, 0, 63, 64, 5, 16, 0, 0, 64, 66, 5, 5, 0, 0, 65, 63, 1, 0, 0, 0, 65, 66, 1, 0, 0, 0, 66, 67, 1, 0, 0, 0, 67, 68, 5, 16, 0, 0, 68, 11, 1, 0, 0, 0, 69, 70, 5, 10, 0, 0, 70, 71, 3, 2, 1, 0, 71, 72, 5, 11, 0, 0, 72, 13, 1, 0, 0, 0, 73, 74, 3, 26, 13, 0, 74, 75, 7, 1, 0, 0, 75, 76, 3, 16, 8, 0, 76, 15, 1, 0, 0, 0, 77, 79, 7, 2, 0, 0, 78, 77, 1, 0, 0, 0, 79, 80, 1, 0, 0, 0, 80, 78, 1, 0, 0, 0, 80, 81, 1, 0, 0, 0, 81, 84, 1, 0, 0, 0, 82, 84, 5, 15, 0, 0, 83, 78, 1, 0, 0, 0, 83, 82, 1, 0, 0, 0, 84, 17, 1, 0, 0, 0, 85, 86, 3, 26, 13, 0, 86, 87, 5, 5, 0, 0, 87, 88, 5, 16, 0, 0, 88, 19, 1, 0, 0, 0, 89, 90, 3, 26, 13, 0, 90, 91, 5, 5, 0, 0, 91, 92, 3, 24, 12, 0, 92, 100, 1, 0, 0, 0, 93, 94, 3, 26, 13, 0, 94, 95, 5, 5, 0, 0, 95, 96, 5, 10, 0, 0, 96, 97, 3, 24, 12, 0, 97, 98, 5, 11, 0, 0, 98, 100, 1, 0, 0, 0, 99, 89, 1, 0, 0, 0, 99, 93, 1, 0, 0, 0, 100, 21, 1, 0, 0, 0, 101, 107, 3, 24, 12, 0, 102, 103, 5, 10, 0, 0, 103, 104, 3, 24, 12, 0, 104, 105, 5, 11, 0, 0, 105, 107, 1, 0, 0, 0, 106, 101, 1, 0, 0, 0, 106, 102, 1, 0, 0, 0, 107, 23, 1, 0, 0, 0, 108, 110, 7, 3, 0, 0, 109, 108, 1, 0, 0, 0, 109, 110, 1, 0, 0, 0, 110, 112, 1, 0, 0, 0, 111, 113, 7, 2, 0, 0, 112, 111, 1, 0, 0, 0, 113, 114, 1, 0, 0, 0, 114, 112, 1, 0, 0, 0, 114, 115, 1, 0, 0, 0, 115, 117, 1, 0, 0, 0, 116, 118, 7, 3, 0, 0, 117, 116, 1, 0, 0, 0, 117, 118, 1, 0, 0, 0, 118, 129, 1, 0, 0, 0, 119, 121, 7, 0, 0, 0, 120, 122, 7, 3, 0, 0, 121, 120, 1, 0, 0, 0, 121, 122, 1, 0, 0, 0, 122, 129, 1, 0, 0, 0, 123, 125, 5, 4, 0, 0, 124, 126, 7, 0, 0, 0, 125, 124, 1, 0, 0, 0, 125, 126, 1, 0, 0, 0, 126, 129, 1, 0, 0, 0, 127, 129, 5, 15, 0, 0, 128, 109, 1, 0, 0, 0, 128, 119, 1, 0, 0, 0, 128, 123, 1, 0, 0, 0, 128, 127, 1, 0, 0, 0, 129, 25, 1, 0, 0, 0, 130, 134, 5, 14, 0, 0, 131, 134, 5, 15, 0, 0, 132, 134, 5, 16, 0, 0, 133, 130, 1, 0, 0, 0, 133, 131, 1, 0, 0, 0, 133, 132, 1, 0, 0, 0, 134, 27, 1, 0, 0, 0, 15, 29, 41, 52, 65, 80, 83, 99, 106, 109, 114, 117, 121, 125, 128, 133] \ No newline at end of file diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBaseParser.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBaseParser.java index b4b0a69a8238..118ac32aadd6 100644 --- a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBaseParser.java +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlBaseParser.java @@ -1194,10 +1194,7 @@ class KqlBaseParser extends Parser { @SuppressWarnings("CheckReturnValue") public static class FieldNameContext extends ParserRuleContext { public Token value; - public List UNQUOTED_LITERAL() { return getTokens(KqlBaseParser.UNQUOTED_LITERAL); } - public TerminalNode UNQUOTED_LITERAL(int i) { - return getToken(KqlBaseParser.UNQUOTED_LITERAL, i); - } + public TerminalNode UNQUOTED_LITERAL() { return getToken(KqlBaseParser.UNQUOTED_LITERAL, 0); } public TerminalNode QUOTED_STRING() { return getToken(KqlBaseParser.QUOTED_STRING, 0); } public TerminalNode WILDCARD() { return getToken(KqlBaseParser.WILDCARD, 0); } public FieldNameContext(ParserRuleContext parent, int invokingState) { @@ -1222,41 +1219,28 @@ class KqlBaseParser extends Parser { public final FieldNameContext fieldName() throws RecognitionException { FieldNameContext _localctx = new FieldNameContext(_ctx, getState()); enterRule(_localctx, 26, RULE_fieldName); - int _la; try { - setState(137); + setState(133); _errHandler.sync(this); switch (_input.LA(1)) { case UNQUOTED_LITERAL: enterOuterAlt(_localctx, 1); { - setState(131); - _errHandler.sync(this); - _la = _input.LA(1); - do { - { - { - setState(130); - ((FieldNameContext)_localctx).value = match(UNQUOTED_LITERAL); - } - } - setState(133); - _errHandler.sync(this); - _la = _input.LA(1); - } while ( _la==UNQUOTED_LITERAL ); + setState(130); + ((FieldNameContext)_localctx).value = match(UNQUOTED_LITERAL); } break; case QUOTED_STRING: enterOuterAlt(_localctx, 2); { - setState(135); + setState(131); ((FieldNameContext)_localctx).value = match(QUOTED_STRING); } break; case WILDCARD: enterOuterAlt(_localctx, 3); { - setState(136); + setState(132); ((FieldNameContext)_localctx).value = match(WILDCARD); } break; @@ -1291,7 +1275,7 @@ class KqlBaseParser extends Parser { } public static final String _serializedATN = - "\u0004\u0001\u0010\u008c\u0002\u0000\u0007\u0000\u0002\u0001\u0007\u0001"+ + "\u0004\u0001\u0010\u0088\u0002\u0000\u0007\u0000\u0002\u0001\u0007\u0001"+ "\u0002\u0002\u0007\u0002\u0002\u0003\u0007\u0003\u0002\u0004\u0007\u0004"+ "\u0002\u0005\u0007\u0005\u0002\u0006\u0007\u0006\u0002\u0007\u0007\u0007"+ "\u0002\b\u0007\b\u0002\t\u0007\t\u0002\n\u0007\n\u0002\u000b\u0007\u000b"+ @@ -1309,70 +1293,67 @@ class KqlBaseParser extends Parser { "\nd\b\n\u0001\u000b\u0001\u000b\u0001\u000b\u0001\u000b\u0001\u000b\u0003"+ "\u000bk\b\u000b\u0001\f\u0003\fn\b\f\u0001\f\u0004\fq\b\f\u000b\f\f\f"+ "r\u0001\f\u0003\fv\b\f\u0001\f\u0001\f\u0003\fz\b\f\u0001\f\u0001\f\u0003"+ - "\f~\b\f\u0001\f\u0003\f\u0081\b\f\u0001\r\u0004\r\u0084\b\r\u000b\r\f"+ - "\r\u0085\u0001\r\u0001\r\u0003\r\u008a\b\r\u0001\r\u0000\u0001\u0002\u000e"+ - "\u0000\u0002\u0004\u0006\b\n\f\u000e\u0010\u0012\u0014\u0016\u0018\u001a"+ - "\u0000\u0004\u0001\u0000\u0002\u0003\u0001\u0000\u0006\t\u0002\u0000\u000e"+ - "\u000e\u0010\u0010\u0001\u0000\u0002\u0004\u0096\u0000\u001d\u0001\u0000"+ - "\u0000\u0000\u0002!\u0001\u0000\u0000\u0000\u00044\u0001\u0000\u0000\u0000"+ - "\u00066\u0001\u0000\u0000\u0000\b9\u0001\u0000\u0000\u0000\nA\u0001\u0000"+ - "\u0000\u0000\fE\u0001\u0000\u0000\u0000\u000eI\u0001\u0000\u0000\u0000"+ - "\u0010S\u0001\u0000\u0000\u0000\u0012U\u0001\u0000\u0000\u0000\u0014c"+ - "\u0001\u0000\u0000\u0000\u0016j\u0001\u0000\u0000\u0000\u0018\u0080\u0001"+ - "\u0000\u0000\u0000\u001a\u0089\u0001\u0000\u0000\u0000\u001c\u001e\u0003"+ - "\u0002\u0001\u0000\u001d\u001c\u0001\u0000\u0000\u0000\u001d\u001e\u0001"+ - "\u0000\u0000\u0000\u001e\u001f\u0001\u0000\u0000\u0000\u001f \u0005\u0000"+ - "\u0000\u0001 \u0001\u0001\u0000\u0000\u0000!\"\u0006\u0001\uffff\uffff"+ - "\u0000\"#\u0003\u0004\u0002\u0000#)\u0001\u0000\u0000\u0000$%\n\u0002"+ - "\u0000\u0000%&\u0007\u0000\u0000\u0000&(\u0003\u0002\u0001\u0002\'$\u0001"+ - "\u0000\u0000\u0000(+\u0001\u0000\u0000\u0000)\'\u0001\u0000\u0000\u0000"+ - ")*\u0001\u0000\u0000\u0000*\u0003\u0001\u0000\u0000\u0000+)\u0001\u0000"+ - "\u0000\u0000,5\u0003\u0006\u0003\u0000-5\u0003\b\u0004\u0000.5\u0003\f"+ - "\u0006\u0000/5\u0003\n\u0005\u000005\u0003\u0012\t\u000015\u0003\u000e"+ - "\u0007\u000025\u0003\u0014\n\u000035\u0003\u0016\u000b\u00004,\u0001\u0000"+ - "\u0000\u00004-\u0001\u0000\u0000\u00004.\u0001\u0000\u0000\u00004/\u0001"+ - "\u0000\u0000\u000040\u0001\u0000\u0000\u000041\u0001\u0000\u0000\u0000"+ - "42\u0001\u0000\u0000\u000043\u0001\u0000\u0000\u00005\u0005\u0001\u0000"+ - "\u0000\u000067\u0005\u0004\u0000\u000078\u0003\u0004\u0002\u00008\u0007"+ - "\u0001\u0000\u0000\u00009:\u0003\u001a\r\u0000:;\u0005\u0005\u0000\u0000"+ - ";<\u0005\f\u0000\u0000<=\u0003\u0002\u0001\u0000=>\u0005\r\u0000\u0000"+ - ">\t\u0001\u0000\u0000\u0000?@\u0005\u0010\u0000\u0000@B\u0005\u0005\u0000"+ - "\u0000A?\u0001\u0000\u0000\u0000AB\u0001\u0000\u0000\u0000BC\u0001\u0000"+ - "\u0000\u0000CD\u0005\u0010\u0000\u0000D\u000b\u0001\u0000\u0000\u0000"+ - "EF\u0005\n\u0000\u0000FG\u0003\u0002\u0001\u0000GH\u0005\u000b\u0000\u0000"+ - "H\r\u0001\u0000\u0000\u0000IJ\u0003\u001a\r\u0000JK\u0007\u0001\u0000"+ - "\u0000KL\u0003\u0010\b\u0000L\u000f\u0001\u0000\u0000\u0000MO\u0007\u0002"+ - "\u0000\u0000NM\u0001\u0000\u0000\u0000OP\u0001\u0000\u0000\u0000PN\u0001"+ - "\u0000\u0000\u0000PQ\u0001\u0000\u0000\u0000QT\u0001\u0000\u0000\u0000"+ - "RT\u0005\u000f\u0000\u0000SN\u0001\u0000\u0000\u0000SR\u0001\u0000\u0000"+ - "\u0000T\u0011\u0001\u0000\u0000\u0000UV\u0003\u001a\r\u0000VW\u0005\u0005"+ - "\u0000\u0000WX\u0005\u0010\u0000\u0000X\u0013\u0001\u0000\u0000\u0000"+ - "YZ\u0003\u001a\r\u0000Z[\u0005\u0005\u0000\u0000[\\\u0003\u0018\f\u0000"+ - "\\d\u0001\u0000\u0000\u0000]^\u0003\u001a\r\u0000^_\u0005\u0005\u0000"+ - "\u0000_`\u0005\n\u0000\u0000`a\u0003\u0018\f\u0000ab\u0005\u000b\u0000"+ - "\u0000bd\u0001\u0000\u0000\u0000cY\u0001\u0000\u0000\u0000c]\u0001\u0000"+ - "\u0000\u0000d\u0015\u0001\u0000\u0000\u0000ek\u0003\u0018\f\u0000fg\u0005"+ - "\n\u0000\u0000gh\u0003\u0018\f\u0000hi\u0005\u000b\u0000\u0000ik\u0001"+ - "\u0000\u0000\u0000je\u0001\u0000\u0000\u0000jf\u0001\u0000\u0000\u0000"+ - "k\u0017\u0001\u0000\u0000\u0000ln\u0007\u0003\u0000\u0000ml\u0001\u0000"+ - "\u0000\u0000mn\u0001\u0000\u0000\u0000np\u0001\u0000\u0000\u0000oq\u0007"+ - "\u0002\u0000\u0000po\u0001\u0000\u0000\u0000qr\u0001\u0000\u0000\u0000"+ - "rp\u0001\u0000\u0000\u0000rs\u0001\u0000\u0000\u0000su\u0001\u0000\u0000"+ - "\u0000tv\u0007\u0003\u0000\u0000ut\u0001\u0000\u0000\u0000uv\u0001\u0000"+ - "\u0000\u0000v\u0081\u0001\u0000\u0000\u0000wy\u0007\u0000\u0000\u0000"+ - "xz\u0007\u0003\u0000\u0000yx\u0001\u0000\u0000\u0000yz\u0001\u0000\u0000"+ - "\u0000z\u0081\u0001\u0000\u0000\u0000{}\u0005\u0004\u0000\u0000|~\u0007"+ - "\u0000\u0000\u0000}|\u0001\u0000\u0000\u0000}~\u0001\u0000\u0000\u0000"+ - "~\u0081\u0001\u0000\u0000\u0000\u007f\u0081\u0005\u000f\u0000\u0000\u0080"+ - "m\u0001\u0000\u0000\u0000\u0080w\u0001\u0000\u0000\u0000\u0080{\u0001"+ - "\u0000\u0000\u0000\u0080\u007f\u0001\u0000\u0000\u0000\u0081\u0019\u0001"+ - "\u0000\u0000\u0000\u0082\u0084\u0005\u000e\u0000\u0000\u0083\u0082\u0001"+ - "\u0000\u0000\u0000\u0084\u0085\u0001\u0000\u0000\u0000\u0085\u0083\u0001"+ - "\u0000\u0000\u0000\u0085\u0086\u0001\u0000\u0000\u0000\u0086\u008a\u0001"+ - "\u0000\u0000\u0000\u0087\u008a\u0005\u000f\u0000\u0000\u0088\u008a\u0005"+ - "\u0010\u0000\u0000\u0089\u0083\u0001\u0000\u0000\u0000\u0089\u0087\u0001"+ - "\u0000\u0000\u0000\u0089\u0088\u0001\u0000\u0000\u0000\u008a\u001b\u0001"+ - "\u0000\u0000\u0000\u0010\u001d)4APScjmruy}\u0080\u0085\u0089"; + "\f~\b\f\u0001\f\u0003\f\u0081\b\f\u0001\r\u0001\r\u0001\r\u0003\r\u0086"+ + "\b\r\u0001\r\u0000\u0001\u0002\u000e\u0000\u0002\u0004\u0006\b\n\f\u000e"+ + "\u0010\u0012\u0014\u0016\u0018\u001a\u0000\u0004\u0001\u0000\u0002\u0003"+ + "\u0001\u0000\u0006\t\u0002\u0000\u000e\u000e\u0010\u0010\u0001\u0000\u0002"+ + "\u0004\u0091\u0000\u001d\u0001\u0000\u0000\u0000\u0002!\u0001\u0000\u0000"+ + "\u0000\u00044\u0001\u0000\u0000\u0000\u00066\u0001\u0000\u0000\u0000\b"+ + "9\u0001\u0000\u0000\u0000\nA\u0001\u0000\u0000\u0000\fE\u0001\u0000\u0000"+ + "\u0000\u000eI\u0001\u0000\u0000\u0000\u0010S\u0001\u0000\u0000\u0000\u0012"+ + "U\u0001\u0000\u0000\u0000\u0014c\u0001\u0000\u0000\u0000\u0016j\u0001"+ + "\u0000\u0000\u0000\u0018\u0080\u0001\u0000\u0000\u0000\u001a\u0085\u0001"+ + "\u0000\u0000\u0000\u001c\u001e\u0003\u0002\u0001\u0000\u001d\u001c\u0001"+ + "\u0000\u0000\u0000\u001d\u001e\u0001\u0000\u0000\u0000\u001e\u001f\u0001"+ + "\u0000\u0000\u0000\u001f \u0005\u0000\u0000\u0001 \u0001\u0001\u0000\u0000"+ + "\u0000!\"\u0006\u0001\uffff\uffff\u0000\"#\u0003\u0004\u0002\u0000#)\u0001"+ + "\u0000\u0000\u0000$%\n\u0002\u0000\u0000%&\u0007\u0000\u0000\u0000&(\u0003"+ + "\u0002\u0001\u0002\'$\u0001\u0000\u0000\u0000(+\u0001\u0000\u0000\u0000"+ + ")\'\u0001\u0000\u0000\u0000)*\u0001\u0000\u0000\u0000*\u0003\u0001\u0000"+ + "\u0000\u0000+)\u0001\u0000\u0000\u0000,5\u0003\u0006\u0003\u0000-5\u0003"+ + "\b\u0004\u0000.5\u0003\f\u0006\u0000/5\u0003\n\u0005\u000005\u0003\u0012"+ + "\t\u000015\u0003\u000e\u0007\u000025\u0003\u0014\n\u000035\u0003\u0016"+ + "\u000b\u00004,\u0001\u0000\u0000\u00004-\u0001\u0000\u0000\u00004.\u0001"+ + "\u0000\u0000\u00004/\u0001\u0000\u0000\u000040\u0001\u0000\u0000\u0000"+ + "41\u0001\u0000\u0000\u000042\u0001\u0000\u0000\u000043\u0001\u0000\u0000"+ + "\u00005\u0005\u0001\u0000\u0000\u000067\u0005\u0004\u0000\u000078\u0003"+ + "\u0004\u0002\u00008\u0007\u0001\u0000\u0000\u00009:\u0003\u001a\r\u0000"+ + ":;\u0005\u0005\u0000\u0000;<\u0005\f\u0000\u0000<=\u0003\u0002\u0001\u0000"+ + "=>\u0005\r\u0000\u0000>\t\u0001\u0000\u0000\u0000?@\u0005\u0010\u0000"+ + "\u0000@B\u0005\u0005\u0000\u0000A?\u0001\u0000\u0000\u0000AB\u0001\u0000"+ + "\u0000\u0000BC\u0001\u0000\u0000\u0000CD\u0005\u0010\u0000\u0000D\u000b"+ + "\u0001\u0000\u0000\u0000EF\u0005\n\u0000\u0000FG\u0003\u0002\u0001\u0000"+ + "GH\u0005\u000b\u0000\u0000H\r\u0001\u0000\u0000\u0000IJ\u0003\u001a\r"+ + "\u0000JK\u0007\u0001\u0000\u0000KL\u0003\u0010\b\u0000L\u000f\u0001\u0000"+ + "\u0000\u0000MO\u0007\u0002\u0000\u0000NM\u0001\u0000\u0000\u0000OP\u0001"+ + "\u0000\u0000\u0000PN\u0001\u0000\u0000\u0000PQ\u0001\u0000\u0000\u0000"+ + "QT\u0001\u0000\u0000\u0000RT\u0005\u000f\u0000\u0000SN\u0001\u0000\u0000"+ + "\u0000SR\u0001\u0000\u0000\u0000T\u0011\u0001\u0000\u0000\u0000UV\u0003"+ + "\u001a\r\u0000VW\u0005\u0005\u0000\u0000WX\u0005\u0010\u0000\u0000X\u0013"+ + "\u0001\u0000\u0000\u0000YZ\u0003\u001a\r\u0000Z[\u0005\u0005\u0000\u0000"+ + "[\\\u0003\u0018\f\u0000\\d\u0001\u0000\u0000\u0000]^\u0003\u001a\r\u0000"+ + "^_\u0005\u0005\u0000\u0000_`\u0005\n\u0000\u0000`a\u0003\u0018\f\u0000"+ + "ab\u0005\u000b\u0000\u0000bd\u0001\u0000\u0000\u0000cY\u0001\u0000\u0000"+ + "\u0000c]\u0001\u0000\u0000\u0000d\u0015\u0001\u0000\u0000\u0000ek\u0003"+ + "\u0018\f\u0000fg\u0005\n\u0000\u0000gh\u0003\u0018\f\u0000hi\u0005\u000b"+ + "\u0000\u0000ik\u0001\u0000\u0000\u0000je\u0001\u0000\u0000\u0000jf\u0001"+ + "\u0000\u0000\u0000k\u0017\u0001\u0000\u0000\u0000ln\u0007\u0003\u0000"+ + "\u0000ml\u0001\u0000\u0000\u0000mn\u0001\u0000\u0000\u0000np\u0001\u0000"+ + "\u0000\u0000oq\u0007\u0002\u0000\u0000po\u0001\u0000\u0000\u0000qr\u0001"+ + "\u0000\u0000\u0000rp\u0001\u0000\u0000\u0000rs\u0001\u0000\u0000\u0000"+ + "su\u0001\u0000\u0000\u0000tv\u0007\u0003\u0000\u0000ut\u0001\u0000\u0000"+ + "\u0000uv\u0001\u0000\u0000\u0000v\u0081\u0001\u0000\u0000\u0000wy\u0007"+ + "\u0000\u0000\u0000xz\u0007\u0003\u0000\u0000yx\u0001\u0000\u0000\u0000"+ + "yz\u0001\u0000\u0000\u0000z\u0081\u0001\u0000\u0000\u0000{}\u0005\u0004"+ + "\u0000\u0000|~\u0007\u0000\u0000\u0000}|\u0001\u0000\u0000\u0000}~\u0001"+ + "\u0000\u0000\u0000~\u0081\u0001\u0000\u0000\u0000\u007f\u0081\u0005\u000f"+ + "\u0000\u0000\u0080m\u0001\u0000\u0000\u0000\u0080w\u0001\u0000\u0000\u0000"+ + "\u0080{\u0001\u0000\u0000\u0000\u0080\u007f\u0001\u0000\u0000\u0000\u0081"+ + "\u0019\u0001\u0000\u0000\u0000\u0082\u0086\u0005\u000e\u0000\u0000\u0083"+ + "\u0086\u0005\u000f\u0000\u0000\u0084\u0086\u0005\u0010\u0000\u0000\u0085"+ + "\u0082\u0001\u0000\u0000\u0000\u0085\u0083\u0001\u0000\u0000\u0000\u0085"+ + "\u0084\u0001\u0000\u0000\u0000\u0086\u001b\u0001\u0000\u0000\u0000\u000f"+ + "\u001d)4APScjmruy}\u0080\u0085"; public static final ATN _ATN = new ATNDeserializer().deserialize(_serializedATN.toCharArray()); static { diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParser.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParser.java index 1064f901cacb..6c2d30860221 100644 --- a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParser.java +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParser.java @@ -14,8 +14,8 @@ import org.antlr.v4.runtime.ParserRuleContext; import org.antlr.v4.runtime.RecognitionException; import org.antlr.v4.runtime.Recognizer; import org.antlr.v4.runtime.atn.PredictionMode; +import org.elasticsearch.core.Strings; import org.elasticsearch.index.query.QueryBuilder; -import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.logging.LogManager; import org.elasticsearch.logging.Logger; @@ -25,22 +25,14 @@ import java.util.function.Function; public class KqlParser { private static final Logger log = LogManager.getLogger(KqlParser.class); - public QueryBuilder parseKqlQuery(String kqlQuery, SearchExecutionContext searchExecutionContext) { - if (log.isDebugEnabled()) { - log.debug("Parsing KQL query: {}", kqlQuery); - } - - return invokeParser( - kqlQuery, - new KqlParserExecutionContext(searchExecutionContext), - KqlBaseParser::topLevelQuery, - KqlAstBuilder::toQueryBuilder - ); + public QueryBuilder parseKqlQuery(String kqlQuery, KqlParsingContext kqlParserContext) { + log.trace("Parsing KQL query: {}", kqlQuery); + return invokeParser(kqlQuery, kqlParserContext, KqlBaseParser::topLevelQuery, KqlAstBuilder::toQueryBuilder); } private T invokeParser( String kqlQuery, - KqlParserExecutionContext kqlParserExecutionContext, + KqlParsingContext kqlParsingContext, Function parseFunction, BiFunction visitor ) { @@ -59,11 +51,9 @@ public class KqlParser { ParserRuleContext tree = parseFunction.apply(parser); - if (log.isTraceEnabled()) { - log.trace("Parse tree: {}", tree.toStringTree()); - } + log.trace(() -> Strings.format("Parse tree: %s", tree.toStringTree())); - return visitor.apply(new KqlAstBuilder(kqlParserExecutionContext), tree); + return visitor.apply(new KqlAstBuilder(kqlParsingContext), tree); } private static final BaseErrorListener ERROR_LISTENER = new BaseErrorListener() { diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParserExecutionContext.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParserExecutionContext.java deleted file mode 100644 index d05c70c6b933..000000000000 --- a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParserExecutionContext.java +++ /dev/null @@ -1,79 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -package org.elasticsearch.xpack.kql.parser; - -import org.elasticsearch.core.Tuple; -import org.elasticsearch.index.mapper.AbstractScriptFieldType; -import org.elasticsearch.index.mapper.DateFieldMapper; -import org.elasticsearch.index.mapper.KeywordFieldMapper; -import org.elasticsearch.index.mapper.MappedFieldType; -import org.elasticsearch.index.query.SearchExecutionContext; - -import java.time.ZoneId; -import java.util.List; -import java.util.function.Predicate; -import java.util.stream.Collectors; - -import static org.elasticsearch.core.Tuple.tuple; - -class KqlParserExecutionContext extends SearchExecutionContext { - - private static final List IGNORED_METADATA_FIELDS = List.of( - "_seq_no", - "_index_mode", - "_routing", - "_ignored", - "_nested_path", - "_field_names" - ); - - private static Predicate> searchableFieldFilter = (fieldDef) -> fieldDef.v2().isSearchable(); - - private static Predicate> ignoredFieldFilter = (fieldDef) -> IGNORED_METADATA_FIELDS.contains( - fieldDef.v1() - ); - - KqlParserExecutionContext(SearchExecutionContext source) { - super(source); - } - - public Iterable> resolveFields(KqlBaseParser.FieldNameContext fieldNameContext) { - // TODO: use index settings default field. - String fieldNamePattern = fieldNameContext != null ? ParserUtils.extractText(fieldNameContext) : "*"; - - if (fieldNameContext != null && fieldNameContext.value != null && fieldNameContext.value.getType() == KqlBaseParser.QUOTED_STRING) { - return isFieldMapped(fieldNamePattern) ? List.of(tuple(fieldNamePattern, getFieldType(fieldNamePattern))) : List.of(); - } - - return getMatchingFieldNames(fieldNamePattern).stream() - .map(fieldName -> tuple(fieldName, getFieldType(fieldName))) - .filter(searchableFieldFilter.and(Predicate.not(ignoredFieldFilter))) - .collect(Collectors.toList()); - } - - public boolean isCaseSensitive() { - // TODO: implementation - return false; - } - - public ZoneId timeZone() { - return null; - } - - public static boolean isRuntimeField(MappedFieldType fieldType) { - return fieldType instanceof AbstractScriptFieldType; - } - - public static boolean isDateField(MappedFieldType fieldType) { - return fieldType.typeName().equals(DateFieldMapper.CONTENT_TYPE); - } - - public static boolean isKeywordField(MappedFieldType fieldType) { - return fieldType.typeName().equals(KeywordFieldMapper.CONTENT_TYPE); - } -} diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParsingContext.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParsingContext.java new file mode 100644 index 000000000000..5f88080fb3ed --- /dev/null +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/parser/KqlParsingContext.java @@ -0,0 +1,121 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.kql.parser; + +import org.elasticsearch.index.mapper.AbstractScriptFieldType; +import org.elasticsearch.index.mapper.DateFieldMapper; +import org.elasticsearch.index.mapper.KeywordFieldMapper; +import org.elasticsearch.index.mapper.MappedFieldType; +import org.elasticsearch.index.query.QueryRewriteContext; + +import java.time.ZoneId; +import java.util.List; +import java.util.Set; + +public class KqlParsingContext { + + private static final List IGNORED_METADATA_FIELDS = List.of( + "_seq_no", + "_index_mode", + "_routing", + "_ignored", + "_nested_path", + "_field_names" + ); + + public static Builder builder(QueryRewriteContext queryRewriteContext) { + return new Builder(queryRewriteContext); + } + + private QueryRewriteContext queryRewriteContext; + private final boolean caseInsensitive; + private final ZoneId timeZone; + private final String defaultField; + + public KqlParsingContext(QueryRewriteContext queryRewriteContext, boolean caseInsensitive, ZoneId timeZone, String defaultField) { + this.queryRewriteContext = queryRewriteContext; + this.caseInsensitive = caseInsensitive; + this.timeZone = timeZone; + this.defaultField = defaultField; + } + + public boolean caseInsensitive() { + return caseInsensitive; + } + + public ZoneId timeZone() { + return timeZone; + } + + public String defaultField() { + return defaultField; + } + + public Set resolveFieldNames(String fieldNamePattern) { + assert fieldNamePattern != null && fieldNamePattern.isEmpty() == false : "fieldNamePattern cannot be null or empty"; + return queryRewriteContext.getMatchingFieldNames(fieldNamePattern); + } + + public Set resolveDefaultFieldNames() { + return resolveFieldNames(defaultField); + } + + public MappedFieldType fieldType(String fieldName) { + return queryRewriteContext.getFieldType(fieldName); + } + + public static boolean isRuntimeField(MappedFieldType fieldType) { + return fieldType instanceof AbstractScriptFieldType; + } + + public static boolean isDateField(MappedFieldType fieldType) { + return fieldType.typeName().equals(DateFieldMapper.CONTENT_TYPE); + } + + public static boolean isKeywordField(MappedFieldType fieldType) { + return fieldType.typeName().equals(KeywordFieldMapper.CONTENT_TYPE); + } + + public static boolean isSearchableField(String fieldName, MappedFieldType fieldType) { + return IGNORED_METADATA_FIELDS.contains(fieldName) == false && fieldType.isSearchable(); + } + + public boolean isSearchableField(String fieldName) { + return isSearchableField(fieldName, fieldType(fieldName)); + } + + public static class Builder { + private final QueryRewriteContext queryRewriteContext; + private boolean caseInsensitive = true; + private ZoneId timeZone = null; + private String defaultField = null; + + private Builder(QueryRewriteContext queryRewriteContext) { + this.queryRewriteContext = queryRewriteContext; + } + + public KqlParsingContext build() { + return new KqlParsingContext(queryRewriteContext, caseInsensitive, timeZone, defaultField); + } + + public Builder caseInsensitive(boolean caseInsensitive) { + this.caseInsensitive = caseInsensitive; + return this; + } + + public Builder timeZone(ZoneId timeZone) { + this.timeZone = timeZone; + return this; + } + + public Builder defaultField(String defaultField) { + this.defaultField = defaultField; + return this; + } + } +} diff --git a/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/query/KqlQueryBuilder.java b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/query/KqlQueryBuilder.java new file mode 100644 index 000000000000..5dff9126b6be --- /dev/null +++ b/x-pack/plugin/kql/src/main/java/org/elasticsearch/xpack/kql/query/KqlQueryBuilder.java @@ -0,0 +1,199 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.kql.query; + +import org.apache.lucene.search.Query; +import org.elasticsearch.TransportVersion; +import org.elasticsearch.TransportVersions; +import org.elasticsearch.common.ParsingException; +import org.elasticsearch.common.Strings; +import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.index.query.AbstractQueryBuilder; +import org.elasticsearch.index.query.QueryBuilder; +import org.elasticsearch.index.query.QueryRewriteContext; +import org.elasticsearch.index.query.SearchExecutionContext; +import org.elasticsearch.logging.LogManager; +import org.elasticsearch.logging.Logger; +import org.elasticsearch.xcontent.ConstructingObjectParser; +import org.elasticsearch.xcontent.ParseField; +import org.elasticsearch.xcontent.XContentBuilder; +import org.elasticsearch.xcontent.XContentParser; +import org.elasticsearch.xpack.kql.parser.KqlParser; +import org.elasticsearch.xpack.kql.parser.KqlParsingContext; + +import java.io.IOException; +import java.time.ZoneId; +import java.util.Objects; + +import static org.elasticsearch.xcontent.ConstructingObjectParser.constructorArg; +import static org.elasticsearch.xcontent.ConstructingObjectParser.optionalConstructorArg; + +public class KqlQueryBuilder extends AbstractQueryBuilder { + public static final String NAME = "kql"; + public static final ParseField QUERY_FIELD = new ParseField("query"); + private static final ParseField CASE_INSENSITIVE_FIELD = new ParseField("case_insensitive"); + private static final ParseField TIME_ZONE_FIELD = new ParseField("time_zone"); + private static final ParseField DEFAULT_FIELD_FIELD = new ParseField("default_field"); + + private static final Logger log = LogManager.getLogger(KqlQueryBuilder.class); + private static final ConstructingObjectParser PARSER = new ConstructingObjectParser<>(NAME, a -> { + KqlQueryBuilder kqlQuery = new KqlQueryBuilder((String) a[0]); + + if (a[1] != null) { + kqlQuery.caseInsensitive((Boolean) a[1]); + } + + if (a[2] != null) { + kqlQuery.timeZone((String) a[2]); + } + + if (a[3] != null) { + kqlQuery.defaultField((String) a[3]); + } + + return kqlQuery; + }); + + static { + PARSER.declareString(constructorArg(), QUERY_FIELD); + PARSER.declareBoolean(optionalConstructorArg(), CASE_INSENSITIVE_FIELD); + PARSER.declareString(optionalConstructorArg(), TIME_ZONE_FIELD); + PARSER.declareString(optionalConstructorArg(), DEFAULT_FIELD_FIELD); + declareStandardFields(PARSER); + } + + private final String query; + private boolean caseInsensitive = true; + private ZoneId timeZone; + private String defaultField; + + public KqlQueryBuilder(String query) { + this.query = Objects.requireNonNull(query, "query can not be null"); + } + + public KqlQueryBuilder(StreamInput in) throws IOException { + super(in); + query = in.readString(); + caseInsensitive = in.readBoolean(); + timeZone = in.readOptionalZoneId(); + defaultField = in.readOptionalString(); + } + + public static KqlQueryBuilder fromXContent(XContentParser parser) { + try { + return PARSER.apply(parser, null); + } catch (IllegalArgumentException e) { + throw new ParsingException(parser.getTokenLocation(), e.getMessage(), e); + } + } + + @Override + public TransportVersion getMinimalSupportedVersion() { + return TransportVersions.KQL_QUERY_ADDED; + } + + public String queryString() { + return query; + } + + public boolean caseInsensitive() { + return caseInsensitive; + } + + public KqlQueryBuilder caseInsensitive(boolean caseInsensitive) { + this.caseInsensitive = caseInsensitive; + return this; + } + + public ZoneId timeZone() { + return timeZone; + } + + public KqlQueryBuilder timeZone(String timeZone) { + this.timeZone = timeZone != null ? ZoneId.of(timeZone) : null; + return this; + } + + public String defaultField() { + return defaultField; + } + + public KqlQueryBuilder defaultField(String defaultField) { + this.defaultField = defaultField; + return this; + } + + @Override + protected void doXContent(XContentBuilder builder, Params params) throws IOException { + builder.startObject(NAME); + { + builder.field(QUERY_FIELD.getPreferredName(), query); + builder.field(CASE_INSENSITIVE_FIELD.getPreferredName(), caseInsensitive); + + if (defaultField != null) { + builder.field(DEFAULT_FIELD_FIELD.getPreferredName(), defaultField); + } + + if (timeZone != null) { + builder.field(TIME_ZONE_FIELD.getPreferredName(), timeZone.getId()); + } + + boostAndQueryNameToXContent(builder); + } + builder.endObject(); + } + + @Override + protected QueryBuilder doIndexMetadataRewrite(QueryRewriteContext context) throws IOException { + KqlParser parser = new KqlParser(); + QueryBuilder rewrittenQuery = parser.parseKqlQuery(query, createKqlParserContext(context)); + + log.trace(() -> Strings.format("KQL query %s translated to Query DSL: %s", query, Strings.toString(rewrittenQuery))); + + return rewrittenQuery; + } + + @Override + protected Query doToQuery(SearchExecutionContext context) throws IOException { + throw new IllegalStateException("The query should have been rewritten"); + } + + protected void doWriteTo(StreamOutput out) throws IOException { + out.writeString(query); + out.writeBoolean(caseInsensitive); + out.writeOptionalZoneId(timeZone); + out.writeOptionalString(defaultField); + } + + @Override + public String getWriteableName() { + return NAME; + } + + @Override + protected int doHashCode() { + return Objects.hash(query, caseInsensitive, timeZone, defaultField); + } + + @Override + protected boolean doEquals(KqlQueryBuilder other) { + return Objects.equals(query, other.query) + && Objects.equals(timeZone, other.timeZone) + && Objects.equals(defaultField, other.defaultField) + && caseInsensitive == other.caseInsensitive; + } + + private KqlParsingContext createKqlParserContext(QueryRewriteContext queryRewriteContext) { + return KqlParsingContext.builder(queryRewriteContext) + .caseInsensitive(caseInsensitive) + .timeZone(timeZone) + .defaultField(defaultField) + .build(); + } +} diff --git a/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/parser/AbstractKqlParserTestCase.java b/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/parser/AbstractKqlParserTestCase.java index 88c63e9a2585..588e60bd4dd7 100644 --- a/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/parser/AbstractKqlParserTestCase.java +++ b/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/parser/AbstractKqlParserTestCase.java @@ -7,19 +7,22 @@ package org.elasticsearch.xpack.kql.parser; +import org.elasticsearch.common.Strings; +import org.elasticsearch.common.compress.CompressedXContent; import org.elasticsearch.common.regex.Regex; import org.elasticsearch.core.Predicates; import org.elasticsearch.core.SuppressForbidden; +import org.elasticsearch.index.mapper.MapperService; import org.elasticsearch.index.query.MatchPhraseQueryBuilder; import org.elasticsearch.index.query.MatchQueryBuilder; import org.elasticsearch.index.query.MultiMatchQueryBuilder; import org.elasticsearch.index.query.QueryBuilder; import org.elasticsearch.index.query.QueryStringQueryBuilder; import org.elasticsearch.index.query.RangeQueryBuilder; -import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.index.query.TermQueryBuilder; import org.elasticsearch.index.query.WildcardQueryBuilder; import org.elasticsearch.test.AbstractBuilderTestCase; +import org.elasticsearch.xcontent.XContentBuilder; import java.io.BufferedReader; import java.io.IOException; @@ -37,6 +40,7 @@ import java.util.function.Predicate; import java.util.stream.Collectors; import java.util.stream.Stream; +import static org.elasticsearch.xcontent.XContentFactory.jsonBuilder; import static org.hamcrest.Matchers.anEmptyMap; import static org.hamcrest.Matchers.contains; import static org.hamcrest.Matchers.equalTo; @@ -47,6 +51,43 @@ public abstract class AbstractKqlParserTestCase extends AbstractBuilderTestCase protected static final String UNSUPPORTED_QUERY_FILE_PATH = "/unsupported-queries"; protected static final Predicate BOOLEAN_QUERY_FILTER = (q) -> q.matches("(?i)[^{]*[^\\\\]*(NOT|AND|OR)[^}]*"); + protected static final String NESTED_FIELD_NAME = "mapped_nested"; + + @Override + protected void initializeAdditionalMappings(MapperService mapperService) throws IOException { + XContentBuilder mapping = jsonBuilder().startObject().startObject("_doc").startObject("properties"); + + mapping.startObject(TEXT_FIELD_NAME).field("type", "text").endObject(); + mapping.startObject(NESTED_FIELD_NAME); + { + mapping.field("type", "nested"); + mapping.startObject("properties"); + { + mapping.startObject(TEXT_FIELD_NAME).field("type", "text").endObject(); + mapping.startObject(KEYWORD_FIELD_NAME).field("type", "keyword").endObject(); + mapping.startObject(INT_FIELD_NAME).field("type", "integer").endObject(); + mapping.startObject(NESTED_FIELD_NAME); + { + mapping.field("type", "nested"); + mapping.startObject("properties"); + { + mapping.startObject(TEXT_FIELD_NAME).field("type", "text").endObject(); + mapping.startObject(KEYWORD_FIELD_NAME).field("type", "keyword").endObject(); + mapping.startObject(INT_FIELD_NAME).field("type", "integer").endObject(); + } + mapping.endObject(); + } + mapping.endObject(); + } + mapping.endObject(); + } + mapping.endObject(); + + mapping.endObject().endObject().endObject(); + + mapperService.merge("_doc", new CompressedXContent(Strings.toString(mapping)), MapperService.MergeReason.MAPPING_UPDATE); + } + protected static String wrapWithRandomWhitespaces(String input) { return String.join("", randomWhitespaces(), input, randomWhitespaces()); } @@ -94,7 +135,18 @@ public abstract class AbstractKqlParserTestCase extends AbstractBuilderTestCase protected List mappedLeafFields() { return Stream.concat( Arrays.stream(MAPPED_LEAF_FIELD_NAMES), - List.of(DATE_FIELD_NAME, INT_FIELD_NAME).stream().map(subfieldName -> OBJECT_FIELD_NAME + "." + subfieldName) + Stream.of( + // Adding mapped_object subfields + Strings.format("%s.%s", OBJECT_FIELD_NAME, INT_FIELD_NAME), + Strings.format("%s.%s", OBJECT_FIELD_NAME, DATE_FIELD_NAME), + // Adding mapped_nested subfields + Strings.format("%s.%s", NESTED_FIELD_NAME, TEXT_FIELD_NAME), + Strings.format("%s.%s", NESTED_FIELD_NAME, KEYWORD_FIELD_NAME), + Strings.format("%s.%s", NESTED_FIELD_NAME, INT_FIELD_NAME), + Strings.format("%s.%s.%s", NESTED_FIELD_NAME, NESTED_FIELD_NAME, TEXT_FIELD_NAME), + Strings.format("%s.%s.%s", NESTED_FIELD_NAME, NESTED_FIELD_NAME, KEYWORD_FIELD_NAME), + Strings.format("%s.%s.%s", NESTED_FIELD_NAME, NESTED_FIELD_NAME, INT_FIELD_NAME) + ) ).toList(); } @@ -111,9 +163,8 @@ public abstract class AbstractKqlParserTestCase extends AbstractBuilderTestCase protected QueryBuilder parseKqlQuery(String kqlQuery) { KqlParser parser = new KqlParser(); - SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); - - return parser.parseKqlQuery(kqlQuery, searchExecutionContext); + KqlParsingContext kqlParserContext = KqlParsingContext.builder(createQueryRewriteContext()).build(); + return parser.parseKqlQuery(kqlQuery, kqlParserContext); } protected static void assertMultiMatchQuery(QueryBuilder query, String expectedValue, MultiMatchQueryBuilder.Type expectedType) { diff --git a/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/query/KqlQueryBuilderTests.java b/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/query/KqlQueryBuilderTests.java new file mode 100644 index 000000000000..2bc23c7d457d --- /dev/null +++ b/x-pack/plugin/kql/src/test/java/org/elasticsearch/xpack/kql/query/KqlQueryBuilderTests.java @@ -0,0 +1,286 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.kql.query; + +import org.apache.lucene.search.Query; +import org.elasticsearch.core.Strings; +import org.elasticsearch.index.query.MultiMatchQueryBuilder; +import org.elasticsearch.index.query.QueryBuilder; +import org.elasticsearch.index.query.QueryRewriteContext; +import org.elasticsearch.index.query.QueryStringQueryBuilder; +import org.elasticsearch.index.query.RangeQueryBuilder; +import org.elasticsearch.index.query.SearchExecutionContext; +import org.elasticsearch.index.query.TermQueryBuilder; +import org.elasticsearch.index.query.WildcardQueryBuilder; +import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.test.AbstractQueryTestCase; +import org.elasticsearch.xpack.kql.KqlPlugin; +import org.hamcrest.Matchers; + +import java.io.IOException; +import java.util.Collection; +import java.util.List; +import java.util.Locale; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.hamcrest.Matchers.contains; +import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.nullValue; + +public class KqlQueryBuilderTests extends AbstractQueryTestCase { + + @Override + protected Collection> getPlugins() { + return List.of(KqlPlugin.class); + } + + @Override + protected KqlQueryBuilder doCreateTestQueryBuilder() { + KqlQueryBuilder kqlQueryBuilder = new KqlQueryBuilder(generateRandomKqlQuery()); + + if (randomBoolean()) { + kqlQueryBuilder.caseInsensitive(randomBoolean()); + } + + if (randomBoolean()) { + kqlQueryBuilder.timeZone(randomTimeZone().getID()); + } + + if (randomBoolean()) { + kqlQueryBuilder.defaultField(randomFrom("*", "mapped_*", KEYWORD_FIELD_NAME, TEXT_FIELD_NAME)); + } + + return kqlQueryBuilder; + } + + @Override + public KqlQueryBuilder mutateInstance(KqlQueryBuilder instance) throws IOException { + if (randomBoolean()) { + // Change name or boost. + return super.mutateInstance(instance); + } + + KqlQueryBuilder kqlQueryBuilder = new KqlQueryBuilder(randomValueOtherThan(instance.queryString(), this::generateRandomKqlQuery)) + .caseInsensitive(instance.caseInsensitive()) + .timeZone(instance.timeZone() != null ? instance.timeZone().getId() : null) + .defaultField(instance.defaultField()); + + if (kqlQueryBuilder.queryString().equals(instance.queryString()) == false) { + return kqlQueryBuilder; + } + + switch (randomInt() % 3) { + case 0 -> { + kqlQueryBuilder.caseInsensitive(instance.caseInsensitive() == false); + } + case 1 -> { + if (randomBoolean() && instance.defaultField() != null) { + kqlQueryBuilder.defaultField(null); + } else { + kqlQueryBuilder.defaultField( + randomValueOtherThan( + instance.defaultField(), + () -> randomFrom("*", "mapped_*", KEYWORD_FIELD_NAME, TEXT_FIELD_NAME) + ) + ); + } + } + default -> { + if (randomBoolean() && instance.timeZone() != null) { + kqlQueryBuilder.timeZone(null); + } else if (instance.timeZone() != null) { + kqlQueryBuilder.timeZone(randomValueOtherThan(instance.timeZone().getId(), () -> randomTimeZone().getID())); + } else { + kqlQueryBuilder.timeZone(randomTimeZone().getID()); + } + } + } + ; + + return kqlQueryBuilder; + } + + @Override + protected void doAssertLuceneQuery(KqlQueryBuilder queryBuilder, Query query, SearchExecutionContext context) throws IOException { + // We're not validating the query content here because it would be too complex. + // Instead, we use ad-hoc parser tests with a predictable output. + } + + private String generateRandomKqlQuery() { + return Stream.generate(() -> { + Stream terms = Stream.generate( + () -> randomValueOtherThanMany(s -> s.toLowerCase(Locale.ROOT).contains("now"), () -> randomAlphaOfLengthBetween(4, 10)) + ).limit(randomIntBetween(1, 5)); + + String subQuery = terms.collect(Collectors.joining(" ")); + + if (randomBoolean() && subQuery.isEmpty() == false) { + String operator = randomFrom(":", "<", "<=", ">", ">="); + String fieldName = randomFrom(KEYWORD_FIELD_NAME, TEXT_FIELD_NAME); + if (operator.equals(":")) { + subQuery = switch (randomFrom(0, 2)) { + case 0 -> subQuery; + case 1 -> '(' + subQuery + ')'; + default -> '"' + subQuery + '"'; + }; + } else { + fieldName = randomFrom(KEYWORD_FIELD_NAME, TEXT_FIELD_NAME, DOUBLE_FIELD_NAME, INT_FIELD_NAME); + if (List.of(DOUBLE_FIELD_NAME, INT_FIELD_NAME).contains(fieldName)) { + subQuery = String.valueOf(randomDouble()); + } + subQuery = randomBoolean() ? '"' + subQuery + '"' : subQuery; + } + + subQuery = fieldName + operator + subQuery; + } + + if (randomBoolean() && subQuery.isEmpty() == false) { + subQuery = '(' + subQuery + ')'; + } + + if (randomBoolean()) { + subQuery = "NOT " + subQuery; + } + + if (randomBoolean() && subQuery.isEmpty() == false) { + subQuery = '(' + subQuery + ')'; + } + + return subQuery; + }).limit(randomIntBetween(0, 5)).collect(Collectors.joining(randomFrom(" OR ", " AND "))); + } + + @Override + public void testMustRewrite() throws IOException { + SearchExecutionContext context = createSearchExecutionContext(); + context.setAllowUnmappedFields(true); + KqlQueryBuilder queryBuilder = createTestQueryBuilder(); + IllegalStateException e = assertThrows(IllegalStateException.class, () -> queryBuilder.toQuery(context)); + assertThat(e.getMessage(), Matchers.containsString("The query should have been rewritten")); + } + + public void testCaseInsensitiveWildcardQuery() throws IOException { + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + + for (boolean caseInsensitive : List.of(true, false)) { + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(KEYWORD_FIELD_NAME + ": foo*"); + // Check case case_insensitive is true by default + assertThat(kqlQuery.caseInsensitive(), equalTo(true)); + + kqlQuery.caseInsensitive(caseInsensitive); + + ; + assertThat( + asInstanceOf(WildcardQueryBuilder.class, rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext)) + .caseInsensitive(), + equalTo(caseInsensitive) + ); + } + } + + public void testCaseInsensitiveTermQuery() throws IOException { + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + + for (boolean caseInsensitive : List.of(true, false)) { + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(KEYWORD_FIELD_NAME + ": foo"); + // Check case case_insensitive is true by default + assertThat(kqlQuery.caseInsensitive(), equalTo(true)); + + kqlQuery.caseInsensitive(caseInsensitive); + + assertThat( + asInstanceOf(TermQueryBuilder.class, rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext)).caseInsensitive(), + equalTo(caseInsensitive) + ); + } + } + + public void testTimeZone() throws IOException { + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + String timeZone = randomTimeZone().getID(); + + for (String operator : List.of(":", "<", "<=", ">", ">=")) { + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(Strings.format("%s %s %s", DATE_FIELD_NAME, operator, "2018-03-28")); + assertThat(kqlQuery.timeZone(), nullValue()); // timeZone is not set by default. + kqlQuery.timeZone(timeZone); + + assertThat( + asInstanceOf(RangeQueryBuilder.class, rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext)).timeZone(), + equalTo(timeZone) + ); + } + } + + public void testDefaultFieldWildcardQuery() throws IOException { + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(Strings.format("foo*")); + assertThat(kqlQuery.defaultField(), nullValue()); // default_field is not set by default. + + kqlQuery.defaultField(TEXT_FIELD_NAME); + + assertThat( + asInstanceOf(QueryStringQueryBuilder.class, rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext)).defaultField(), + equalTo(TEXT_FIELD_NAME) + ); + } + + public void testDefaultFieldMatchQuery() throws IOException { + + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + + { + // Using a specific field name + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(Strings.format("foo")); + assertThat(kqlQuery.defaultField(), nullValue()); // default_field is not set by default. + + kqlQuery.defaultField(TEXT_FIELD_NAME); + MultiMatchQueryBuilder rewritenQuery = asInstanceOf( + MultiMatchQueryBuilder.class, + rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext) + ); + assertThat(rewritenQuery.fields().keySet(), contains(TEXT_FIELD_NAME)); + } + + { + // Using a pattern for as the field name + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(Strings.format("foo")); + assertThat(kqlQuery.defaultField(), nullValue()); // default_field is not set by default. + + kqlQuery.defaultField("mapped_object.*"); + MultiMatchQueryBuilder rewritenQuery = asInstanceOf( + MultiMatchQueryBuilder.class, + rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext) + ); + assertThat(rewritenQuery.fields().keySet(), contains("mapped_object.mapped_date", "mapped_object.mapped_int")); + } + } + + public void testQueryNameIsPreserved() throws IOException { + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(generateRandomKqlQuery()).queryName(randomIdentifier()); + QueryBuilder rewrittenQuery = rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext); + assertThat(rewrittenQuery.queryName(), equalTo(kqlQuery.queryName())); + } + + public void testQueryBoostIsPreserved() throws IOException { + QueryRewriteContext queryRewriteContext = createQueryRewriteContext(); + SearchExecutionContext searchExecutionContext = createSearchExecutionContext(); + + KqlQueryBuilder kqlQuery = new KqlQueryBuilder(generateRandomKqlQuery()).boost(randomFloatBetween(0, Float.MAX_VALUE, true)); + QueryBuilder rewrittenQuery = rewriteQuery(kqlQuery, queryRewriteContext, searchExecutionContext); + assertThat(rewrittenQuery.boost(), equalTo(kqlQuery.boost())); + } +} diff --git a/x-pack/plugin/kql/src/test/resources/supported-queries b/x-pack/plugin/kql/src/test/resources/supported-queries index 4911c9e3ebec..b659b1ae5b1d 100644 --- a/x-pack/plugin/kql/src/test/resources/supported-queries +++ b/x-pack/plugin/kql/src/test/resources/supported-queries @@ -23,54 +23,54 @@ f*oo *:"foo bar" // Querying a field -foo_field:200 -foo_field:foo -foo_field:foo bar -foo_field:(foo bar) -foo_field:foo* -foo_field: f*oo -foo_field: *foo -foo_field:"foo bar" -foo_field.subfield:foo -foo_*_field:foo -foo_field:* -foo_*:* +mapped_int:200 +mapped_string_2:foo +mapped_string:foo bar +mapped_string:(foo bar) +mapped_string:foo* +mapped_string_2: f*oo +mapped_string: *foo +mapped_string:"foo bar" +mapped_object.subfield:foo +mapped_str*:foo +mapped_string:* +mapped_str_*:* // Range queries -foo_field<200 -foo_field=200 -foo_field>=foo -foo_field>"foo bar" -foo_field<=foo -foo_field>=foo +mapped_int<200 +mapped_string_2=200 +mapped_string_alias>=foo +mapped_string>"foo bar" +mapped_string<=foo +mapped_string_2>=foo // Boolean queries NOT foo NOT foo bar -NOT foo_field:foo -NOT foo_fieldbar -(foo_field:foo) AND (foo_field:foo bar) -foo_field:foo OR foo_field:foo bar -NOT(foo_field:foo OR foo_field:foo bar) -NOT(foo_field:foo AND foo_field:foo bar) -NOT foo_field:foo AND NOT foo_field:foo bar -(NOT foo_field:foo) AND (NOT foo_field:foo bar) -NOT(foo_field:foo) AND NOT(foo_field:foo bar) -foo_field:foo AND foo_field:foo bar AND foo bar -foo_field:foo AND foo_field:foo bar OR foo bar -foo_field:foo OR foo_field:foo bar OR foo bar -foo_field:foo OR foo_field:foo bar AND foo bar -foo_field:foo AND (foo_field:foo bar OR foo bar) -foo_field:foo AND (foo_field:foo bar OR foo bar) -foo_field:foo OR (foo_field:foo bar OR foo bar) +NOT mapped_string:foo +NOT mapped_string_2bar +(mapped_string:foo) AND (mapped_string:foo bar) +mapped_string:foo OR mapped_string_2:foo bar +NOT(mapped_string:foo OR mapped_string:foo bar) +NOT(mapped_string:foo AND mapped_string:foo bar) +NOT mapped_string:foo AND NOT mapped_string_2:foo bar +(NOT mapped_string_alias:foo) AND (NOT mapped_string:foo bar) +NOT(mapped_string:foo) AND NOT(mapped_string:foo bar) +mapped_string:foo AND mapped_string_2:foo bar AND foo bar +mapped_string:foo AND mapped_string_2:foo bar OR foo bar +mapped_string:foo OR mapped_string_2:foo bar OR foo bar +mapped_string:foo OR mapped_string:foo bar AND foo bar +mapped_string:foo AND (mapped_string_2:foo bar OR foo bar) +mapped_string:foo AND (mapped_string_2:foo bar OR foo bar) +mapped_string:foo OR (mapped_string_2:foo bar OR foo bar) -foo:AND -foo:OR -foo:NOT +mapped_string:AND +mapped_string:OR +mapped_string:NOT foo AND foo OR foo NOT @@ -79,43 +79,51 @@ OR foo NOT // Nested queries -nested_field: { NOT foo } -nested_field: { NOT foo bar } -nested_field: { NOT foo_field:foo } -nested_field: { foo_field:foo AND foo_field:foo bar } -nested_field: { foo_fieldbar } -nested_field: { (foo_field:foo) AND (foo_field:foo bar) } -nested_field: { foo_field:foo OR foo_field:foo bar } -nested_field: { NOT(foo_field:foo OR foo_field:foo bar) } -nested_field: { NOT(foo_field:foo AND foo_field:foo bar) } -nested_field: { NOT foo_field:foo AND NOT foo_field:foo bar } -nested_field: { (NOT foo_field:foo) AND (NOT foo_field:foo bar) } -nested_field: { NOT(foo_field:foo) AND NOT(foo_field:foo bar) } -nested_field: { foo_field:foo AND foo_field:foo bar AND foo bar } -nested_field: { foo_field:foo AND foo_field:foo bar OR foo bar } -nested_field: { foo_field:foo OR foo_field:foo bar OR foo bar } -nested_field: { foo_field:foo OR foo_field:foo bar AND foo bar } -nested_field: { foo_field:foo AND (foo_field:foo bar OR foo bar) } -nested_field: { foo_field:foo AND (foo_field:foo bar OR foo bar) } -nested_field: { foo_field:foo OR (foo_field:foo bar OR foo bar) } -nested_field: { sub_nested_field : { foo_field:foo } AND foo_field:foo bar } +mapped_nested: { NOT foo } +mapped_nested: { NOT foo bar } +mapped_nested: { NOT mapped_string:foo } +mapped_nested: { mapped_string:foo AND mapped_string_2:foo bar } +mapped_nested: { mapped_string2 } +mapped_nested: { (mapped_string:foo) AND (mapped_string_2:foo bar) } +mapped_nested: { mapped_string:foo OR mapped_string_2:foo bar } +mapped_nested: { NOT(mapped_string:foo OR mapped_string_2:foo bar) } +mapped_nested: { NOT(mapped_string:foo AND mapped_string_2:foo bar) } +mapped_nested: { NOT mapped_string:foo AND NOT mapped_string_2:foo bar } +mapped_nested: { (NOT mapped_string:foo) AND (NOT mapped_string_2:foo bar) } +mapped_nested: { NOT(mapped_string:foo) AND NOT(mapped_string_2:foo bar) } +mapped_nested: { mapped_string:foo AND mapped_string_2:foo bar AND foo bar } +mapped_nested: { mapped_string:foo AND mapped_string_2:foo bar OR foo bar } +mapped_nested: { mapped_string:foo OR mapped_string_2:foo bar OR foo bar } +mapped_nested: { mapped_string:foo OR mapped_string_2:foo bar AND foo bar } +mapped_nested: { mapped_string:foo AND (mapped_string_2:foo bar OR foo bar) } +mapped_nested: { mapped_string:foo AND (mapped_string_2:foo bar OR foo bar) } +mapped_nested: { mapped_string:foo OR (mapped_string_2:foo bar OR foo bar) } +mapped_nested: { mapped_str*:foo } +mapped_nested: { mapped_nested : { mapped_string:foo AND mapped_int < 3 } AND mapped_string_2:foo bar } +mapped_nested: { mapped_nested.mapped_string:foo AND mapped_string_2:foo bar } + +// Inline nested queries +mapped_nested.mapped_string:foo AND mapped_nested.mapped_int < 2 +mapped_nested.mapped_nested.mapped_string:foo AND mapped_nested.mapped_int < 2 +mapped_nested.mapped_str*: foo + // Queries with escape sequences -foo_field : (foo\(bar\)) -foo_field : foo\:bar -foo_field : (foo \\and bar) -foo_field : (foo \\or bar) -foo_field : foo \\not bar -foo_field : foo \{bar\} -foo_field : foo \(bar\) -foo_field : foo \\ bar -foo_field : foo \"bar\" +mapped_string:(foo\(bar\)) +mapped_string:foo\:bar +mapped_string:(foo \\and bar) +mapped_string:(foo \\or bar) +mapped_string:foo \\not bar +mapped_string:foo \{bar\} +mapped_string:foo \(bar\) +mapped_string:foo \\ bar +mapped_string:foo \"bar\" -foo_field : "foo and bar" -foo_field : "foo not bar" -foo_field : "foo or bar" -foo_field : "foo : bar" -foo_field : "foo { bar }" -foo_field : "foo (bar)" -foo_field : "foo \\ bar" -foo_field : "foo \"bar\"" +mapped_string:"foo and bar" +mapped_string:"foo not bar" +mapped_string:"foo or bar" +mapped_string:"foo : bar" +mapped_string:"foo { bar }" +mapped_string:"foo (bar)" +mapped_string:"foo \\ bar" +mapped_string:"foo \"bar\"" diff --git a/x-pack/plugin/kql/src/test/resources/unsupported-queries b/x-pack/plugin/kql/src/test/resources/unsupported-queries index 64901891c678..149bcf5bd2b5 100644 --- a/x-pack/plugin/kql/src/test/resources/unsupported-queries +++ b/x-pack/plugin/kql/src/test/resources/unsupported-queries @@ -1,36 +1,36 @@ // Incomplete expressions -foo_field : -foo_field < -foo_field > -foo_field >= -foo_field <= +mapped_string : +mapped_string < +mapped_string > +mapped_string >= +mapped_string <= >= foo : "foo" : foo // Parentheses mismatch -foo_field: (foo bar -foo_field: foo bar) -NOT foo_field:foo OR foo_field:foo bar) -NOT (foo_field:foo AND) foo_field:foo bar +mapped_string: (foo bar +mapped_string: foo bar) +NOT mapped_string:foo OR mapped_string_2:foo bar) +NOT (mapped_string:foo AND) mapped_string_2:foo bar // Quotes mismatch -foo_field: "foo bar -foo_field: foo bar" +mapped_string: "foo bar +mapped_string: foo bar" // Can't nest grouping terms parentheses -foo_field:(foo (bar)) +mapped_string:(foo (bar)) // Bad syntax for nested fields: -nested_field { foo: bar } +mapped_nested { mapped_string: bar } // Missing escape sequences: -foo_field: foo:bar -foo_field: (foo and bar) -foo_field: (foo or bar) -foo_field: foo not bar -foo_field: foo { bar } -foo_field: foo (bar) -foo_field: foo "bar" -foo_field: "foo "bar"" +mapped_string: foo:bar +mapped_string: (foo and bar) +mapped_string: (foo or bar) +mapped_string: foo not bar +mapped_string: foo { bar } +mapped_string: foo (bar) +mapped_string: foo "bar" +mapped_string: "foo "bar"" diff --git a/x-pack/plugin/kql/src/yamlRestTest/java/org/elasticsearch/xpack/kql/KqlRestIT.java b/x-pack/plugin/kql/src/yamlRestTest/java/org/elasticsearch/xpack/kql/KqlRestIT.java new file mode 100644 index 000000000000..35df46b0fdcb --- /dev/null +++ b/x-pack/plugin/kql/src/yamlRestTest/java/org/elasticsearch/xpack/kql/KqlRestIT.java @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.kql; + +import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; + +import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.elasticsearch.test.cluster.local.distribution.DistributionType; +import org.elasticsearch.test.rest.yaml.ClientYamlTestCandidate; +import org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase; +import org.junit.ClassRule; + +public class KqlRestIT extends ESClientYamlSuiteTestCase { + + @ClassRule + public static ElasticsearchCluster cluster = ElasticsearchCluster.local() + .setting("xpack.security.enabled", "false") + .setting("xpack.security.http.ssl.enabled", "false") + .distribution(DistributionType.DEFAULT) + .build(); + + public KqlRestIT(final ClientYamlTestCandidate testCandidate) { + super(testCandidate); + } + + @Override + protected String getTestRestCluster() { + return cluster.getHttpAddresses(); + } + + @ParametersFactory + public static Iterable parameters() throws Exception { + return ESClientYamlSuiteTestCase.createParameters(); + } +} diff --git a/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/10_kql_basic_query.yml b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/10_kql_basic_query.yml new file mode 100644 index 000000000000..bb59c6a48b61 --- /dev/null +++ b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/10_kql_basic_query.yml @@ -0,0 +1,212 @@ +setup: + - requires: + capabilities: + - method: POST + path: /_search + capabilities: [ kql_query ] + test_runner_features: capabilities + reason: KQL query is not available + + - do: + indices.create: + index: test-index + body: + mappings: + properties: + date_field: + type: date + text_field: + type: text + keyword_field: + type: keyword + integer_field: + type: integer + double_field: + type: double + + - do: + bulk: + index: test-index + refresh: true + body: | + { "index" : { "_id": "doc-1" } } + { "text_field": "foo bar", "integer_field": 1, "double_field": 3.5, "date_field": "2010-03-06T14:15:00", "keyword_field": "foo bar" } + { "index" : { "_id": "doc-42" } } + { "text_field": "foo baz", "integer_field": 2, "double_field": 18.9, "date_field": "2018-03-28T20:30:00", "keyword_field": "foo baz" } + +--- +"KQL match all queries": + # KQL empty query are supposed to match all. + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "" } } + } + - match: { hits.total: 2 } + + # Using the *:* syntax + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "*" } } + } + - match: { hits.total: 2 } + + # Using the *:* syntax + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "*:*" } } + } + - match: { hits.total: 2 } + +--- +"KQL match term queries (no field specified)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "bar" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "foo bar" } } + } + - match: { hits.total: 2 } + - match: { hits.hits.0._id: "doc-1" } + + # KQL does not match on the _id field when no field is specified. + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "doc-42" } } + } + - match: { hits.total: 0 } + +--- +"KQL match multiple terms queries (no field specified)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "foo bar" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "(foo bar)" } } + } + - match: { hits.total: 2 } + +--- +"KQL match phrase queries (no field specified)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "\"foo bar\"" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + +--- +"KQL match number queries (no field specified)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "2" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "3.5" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + +--- +"KQL match multiple terms queries (no matches)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "no match" } } + } + - match: { hits.total: 0 } + + +--- +"KQL boolean queries": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field: foo AND integer_field > 1" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field: baz OR keyword_field: foo bar" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "NOT text_field: baz" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + diff --git a/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/20_kql_match_query.yml b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/20_kql_match_query.yml new file mode 100644 index 000000000000..2e40c73ecf82 --- /dev/null +++ b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/20_kql_match_query.yml @@ -0,0 +1,266 @@ +setup: + - requires: + capabilities: + - method: POST + path: /_search + capabilities: [ kql_query ] + test_runner_features: capabilities + reason: KQL query is not available + + - requires: + "test_runner_features": "contains" + + - do: + indices.create: + index: test-index + body: + mappings: + properties: + date_field: + type: date + text_field: + type: text + keyword_field: + type: keyword + integer_field: + type: integer + double_field: + type: double + + - do: + bulk: + index: test-index + refresh: true + body: | + { "index" : { "_id": "doc-1" } } + { "text_field": "foo bar", "integer_field": 1, "double_field": 3.5, "date_field": "2010-03-06T14:15:00", "keyword_field": "foo bar" } + { "index" : { "_id": "doc-42" } } + { "text_field": "foo baz", "integer_field": 2, "double_field": 18.9, "date_field": "2018-03-28T20:30:00", "keyword_field": "foo baz" } + + +--- +"KQL match term queries (text field)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field:bar" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field: foo bar" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field: (foo bar)" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field: \"foo bar\"" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field: bar*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + +--- +"KQL match term queries (integer field)": + - do: + catch: bad_request + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field: foo" } } + } + - match: { error.type: "search_phase_execution_exception" } + - match: { error.root_cause.0.type: "query_shard_exception" } + - match: { error.root_cause.0.reason: "failed to create query: For input string: \"foo\"" } + - contains: { error.root_cause.0.stack_trace: "Caused by: java.lang.NumberFormatException: For input string: \"foo\"" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field: 2" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field: \"2\"" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + +--- +"KQL match term queries (double field)": + - do: + catch: bad_request + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field: foo" } } + } + - match: { error.type: "search_phase_execution_exception" } + - match: { error.root_cause.0.type: "query_shard_exception" } + - match: { error.root_cause.0.reason: "failed to create query: For input string: \"foo\"" } + - contains: { error.root_cause.0.stack_trace: "Caused by: java.lang.NumberFormatException: For input string: \"foo\"" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field: 18.9" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field: \"18.9\"" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + +--- +"KQL match term queries (keyword field)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field:foo bar" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field: \"foo bar\"" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field: foo ba*" } } + } + - match: { hits.total: 2 } + + +--- +"KQL match term queries (date field)": + - do: + catch: bad_request + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field: foo" } } + } + - match: { error.type: "search_phase_execution_exception" } + - match: { error.root_cause.0.type: "parse_exception" } + - contains: { error.root_cause.0.reason: "failed to parse date field [foo]" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field: 2010-03-06" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field: now" } } + } + - match: { hits.total: 0 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field: now/1d" } } + } + - match: { hits.total: 0 } + +--- +"KQL match term queries (search by id)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "_id:doc-1" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } diff --git a/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/30_kql_range_query.yml b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/30_kql_range_query.yml new file mode 100644 index 000000000000..e03fd41306ba --- /dev/null +++ b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/30_kql_range_query.yml @@ -0,0 +1,343 @@ +setup: + - requires: + capabilities: + - method: POST + path: /_search + capabilities: [ kql_query ] + test_runner_features: capabilities + reason: KQL query is not available + + - requires: + "test_runner_features": "contains" + + - do: + indices.create: + index: test-index + body: + mappings: + properties: + date_field: + type: date + text_field: + type: text + keyword_field: + type: keyword + integer_field: + type: integer + double_field: + type: double + + - do: + bulk: + index: test-index + refresh: true + body: | + { "index" : { "_id": "doc-1" } } + { "text_field": "bar", "integer_field": 1, "double_field": 3.5, "date_field": "2010-03-06T14:15:00", "keyword_field": "foo bar" } + { "index" : { "_id": "doc-42" } } + { "text_field": "baz", "integer_field": 2, "double_field": 18.9, "date_field": "2018-03-28T20:30:00", "keyword_field": "foo baz" } + + +--- +"KQL match term queries (text field)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field < baz" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field <= baz" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field > bar" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field >= bar" } } + } + - match: { hits.total: 2 } + + +--- +"KQL match term queries (integer field)": + - do: + catch: bad_request + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field < foo" } } + } + - match: { error.type: "search_phase_execution_exception" } + - match: { error.root_cause.0.type: "query_shard_exception" } + - match: { error.root_cause.0.reason: "failed to create query: For input string: \"foo\"" } + - contains: { error.root_cause.0.stack_trace: "Caused by: java.lang.NumberFormatException: For input string: \"foo\"" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field >= 1" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field > 1" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field <= 2" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field < 2" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + +--- +"KQL match term queries (double field)": + - do: + catch: bad_request + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field < foo" } } + } + - match: { error.type: "search_phase_execution_exception" } + - match: { error.root_cause.0.type: "query_shard_exception" } + - match: { error.root_cause.0.reason: "failed to create query: For input string: \"foo\"" } + - contains: { error.root_cause.0.stack_trace: "Caused by: java.lang.NumberFormatException: For input string: \"foo\"" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field >= 3.5" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field > 3.5" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field <= 18.9" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field < 18.9" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + +--- +"KQL match term queries (keyword field)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field < foo baz" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field <= foo baz" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field > foo bar" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field >= foo bar" } } + } + - match: { hits.total: 2 } + + +--- +"KQL match term queries (date field)": + - do: + catch: bad_request + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field: foo" } } + } + - match: { error.type: "search_phase_execution_exception" } + - match: { error.root_cause.0.type: "parse_exception" } + - contains: { error.root_cause.0.reason: "failed to parse date field [foo]" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field < 2018-03-28" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field <= 2018-03-28" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field > 2010-03-06" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-42" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field >= 2010-03-06" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field < now" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field <= now" } } + } + - match: { hits.total: 2 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field > now" } } + } + - match: { hits.total: 0 } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field >= now" } } + } + - match: { hits.total: 0 } diff --git a/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/40_kql_exist_query.yml b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/40_kql_exist_query.yml new file mode 100644 index 000000000000..ca9197d382f6 --- /dev/null +++ b/x-pack/plugin/kql/src/yamlRestTest/resources/rest-api-spec/test/kql/40_kql_exist_query.yml @@ -0,0 +1,182 @@ +setup: + - requires: + capabilities: + - method: POST + path: /_search + capabilities: [ kql_query ] + test_runner_features: capabilities + reason: KQL query is not available + + - do: + indices.create: + index: test-index + body: + mappings: + properties: + date_field: + type: date + text_field: + type: text + keyword_field: + type: keyword + integer_field: + type: integer + double_field: + type: double + + - do: + bulk: + index: test-index + refresh: true + body: | + { "index" : { "_id": "doc-1" } } + { "text_field": "foo bar", "integer_field": 1, "double_field": 3.5, "date_field": "2010-03-06T14:15:00", "keyword_field": "foo bar" } + { "index" : { "_id": "doc-42" } } + { "another_field": "foo"} + +--- +"KQL exists queries - Existing field": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_*:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "*_field:*" } } + } + - match: { hits.total: 2 } + +--- +"KQL exists queries (existing field)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "text_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "integer_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "double_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "date_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "keyword_field:*" } } + } + - match: { hits.total: 1 } + - match: { hits.hits.0._id: "doc-1" } + +--- +"KQL exists queries (non-existing field)": + - do: + search: + index: test-index + rest_total_hits_as_int: true + body: > + { + "query": { "kql": { "query": "non_existing_field:*" } } + } + - match: { hits.total: 0 } diff --git a/x-pack/plugin/otel-data/src/main/resources/component-templates/semconv-resource-to-ecs@mappings.yaml b/x-pack/plugin/otel-data/src/main/resources/component-templates/semconv-resource-to-ecs@mappings.yaml index eb5cd6d37af8..ee5145ea4287 100644 --- a/x-pack/plugin/otel-data/src/main/resources/component-templates/semconv-resource-to-ecs@mappings.yaml +++ b/x-pack/plugin/otel-data/src/main/resources/component-templates/semconv-resource-to-ecs@mappings.yaml @@ -95,6 +95,9 @@ template: k8s.statefulset.name: type: keyword ignore_above: 1024 + k8s.cluster.name: + type: keyword + ignore_above: 1024 service.node.name: type: alias path: resource.attributes.service.instance.id @@ -131,6 +134,10 @@ template: host.os.version: type: alias path: resource.attributes.os.version + orchestrator.cluster.name: + type: alias + path: resource.attributes.k8s.cluster.name +# Below are non-ECS fields that may be used by Kibana. kubernetes.deployment.name: type: alias path: resource.attributes.k8s.deployment.name @@ -170,7 +177,6 @@ template: kubernetes.node.hostname: type: alias path: resource.attributes.k8s.node.hostname -# Below are non-ECS fields that may be used by Kibana. service.language.name: type: alias path: resource.attributes.telemetry.sdk.language diff --git a/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml b/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml index 63966e601a3c..95a42b137df5 100644 --- a/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml +++ b/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs_tests.yml @@ -209,12 +209,13 @@ host.name pass-through: k8s.replicaset.name: myReplicasetName k8s.node.uid: myNodeUid k8s.node.hostname: myNodeHostname + k8s.cluster.name: myClusterName - is_false: errors - do: search: index: logs-generic.otel-default body: - fields: ["kubernetes.container.name", "kubernetes.cronjob.name", "kubernetes.job.name", "kubernetes.statefulset.name", "kubernetes.daemonset.name", "kubernetes.replicaset.name", "kubernetes.node.uid", "kubernetes.node.hostname" ] + fields: ["kubernetes.container.name", "kubernetes.cronjob.name", "kubernetes.job.name", "kubernetes.statefulset.name", "kubernetes.daemonset.name", "kubernetes.replicaset.name", "kubernetes.node.uid", "kubernetes.node.hostname", "orchestrator.cluster.name" ] - length: { hits.hits: 1 } - match: { hits.hits.0.fields.kubernetes\.container\.name : ["myContainerName"] } - match: { hits.hits.0.fields.kubernetes\.cronjob\.name : ["myCronJobName"] } @@ -224,3 +225,4 @@ host.name pass-through: - match: { hits.hits.0.fields.kubernetes\.replicaset\.name : ["myReplicasetName"] } - match: { hits.hits.0.fields.kubernetes\.node\.uid : ["myNodeUid"] } - match: { hits.hits.0.fields.kubernetes\.node\.hostname : ["myNodeHostname"] } + - match: { hits.hits.0.fields.orchestrator\.cluster\.name : ["myClusterName"] } diff --git a/x-pack/plugin/profiling/src/main/java/org/elasticsearch/xpack/profiling/persistence/AbstractProfilingPersistenceManager.java b/x-pack/plugin/profiling/src/main/java/org/elasticsearch/xpack/profiling/persistence/AbstractProfilingPersistenceManager.java index 528d6f28a711..0c005e8472cd 100644 --- a/x-pack/plugin/profiling/src/main/java/org/elasticsearch/xpack/profiling/persistence/AbstractProfilingPersistenceManager.java +++ b/x-pack/plugin/profiling/src/main/java/org/elasticsearch/xpack/profiling/persistence/AbstractProfilingPersistenceManager.java @@ -9,7 +9,6 @@ package org.elasticsearch.xpack.profiling.persistence; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; -import org.elasticsearch.Version; import org.elasticsearch.action.ActionListener; import org.elasticsearch.action.ActionRequest; import org.elasticsearch.action.ActionResponse; @@ -22,7 +21,6 @@ import org.elasticsearch.client.internal.Client; import org.elasticsearch.cluster.ClusterChangedEvent; import org.elasticsearch.cluster.ClusterState; import org.elasticsearch.cluster.ClusterStateListener; -import org.elasticsearch.cluster.node.DiscoveryNodes; import org.elasticsearch.cluster.service.ClusterService; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.core.TimeValue; @@ -73,18 +71,6 @@ abstract class AbstractProfilingPersistenceManager indicesToSearch = List.of(regularIndex, partiallyMountedIndex); SearchRequest request = new SearchRequest().indices(indicesToSearch.toArray(new String[0])) + // we randomise the partial search results because if shards that do NOT match the query are unavailable + // the search is not partial + .allowPartialSearchResults(randomBoolean()) .source(new SearchSourceBuilder().query(termQueryBuilder)); assertResponse(client().search(request), searchResponse -> { @@ -1045,6 +1048,7 @@ public class SearchableSnapshotsCanMatchOnCoordinatorIntegTests extends BaseFroz TermsQueryBuilder termsQueryBuilder = QueryBuilders.termsQuery("_tier", "data_hot", "data_content"); List indicesToSearch = List.of(regularIndex, partiallyMountedIndex); SearchRequest request = new SearchRequest().indices(indicesToSearch.toArray(new String[0])) + .allowPartialSearchResults(randomBoolean()) .source(new SearchSourceBuilder().query(termsQueryBuilder)); assertResponse(client().search(request), searchResponse -> { @@ -1061,6 +1065,7 @@ public class SearchableSnapshotsCanMatchOnCoordinatorIntegTests extends BaseFroz BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery("_tier", "data_frozen")); List indicesToSearch = List.of(regularIndex, partiallyMountedIndex); SearchRequest request = new SearchRequest().indices(indicesToSearch.toArray(new String[0])) + .allowPartialSearchResults(randomBoolean()) .source(new SearchSourceBuilder().query(boolQueryBuilder)); assertResponse(client().search(request), searchResponse -> { @@ -1078,6 +1083,7 @@ public class SearchableSnapshotsCanMatchOnCoordinatorIntegTests extends BaseFroz .mustNot(randomFrom(QueryBuilders.wildcardQuery("_tier", "dat*ozen"), QueryBuilders.prefixQuery("_tier", "data_fro"))); List indicesToSearch = List.of(regularIndex, partiallyMountedIndex); SearchRequest request = new SearchRequest().indices(indicesToSearch.toArray(new String[0])) + .allowPartialSearchResults(randomBoolean()) .source(new SearchSourceBuilder().query(boolQueryBuilder)); assertResponse(client().search(request), searchResponse -> { diff --git a/x-pack/plugin/security/qa/multi-cluster/build.gradle b/x-pack/plugin/security/qa/multi-cluster/build.gradle index c7b8f81bb787..b8eccb14819a 100644 --- a/x-pack/plugin/security/qa/multi-cluster/build.gradle +++ b/x-pack/plugin/security/qa/multi-cluster/build.gradle @@ -31,13 +31,15 @@ dependencies { tasks.named("javaRestTest") { enabled = true // This is tested explicitly in bwc test tasks. - exclude '**/RemoteClusterSecurityBwcRestIT.class' + exclude '**/RemoteClusterSecurityBWCToRCS1ClusterRestIT.class' + exclude '**/RemoteClusterSecurityBWCToRCS2ClusterRestIT.class' } -BuildParams.bwcVersions.withWireCompatible(v -> v.before(BuildParams.isSnapshotBuild() ? '8.8.0' : '8.9.1')) { bwcVersion, baseName -> +BuildParams.bwcVersions.withWireCompatible() { bwcVersion, baseName -> tasks.register(bwcTaskName(bwcVersion), StandaloneRestIntegTestTask) { usesBwcDistribution(bwcVersion) systemProperty("tests.old_cluster_version", bwcVersion) - include '**/RemoteClusterSecurityBwcRestIT.class' + include '**/RemoteClusterSecurityBWCToRCS1ClusterRestIT.class' + include '**/RemoteClusterSecurityBWCToRCS2ClusterRestIT.class' } } diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBwcRestIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/AbstractRemoteClusterSecurityBWCRestIT.java similarity index 65% rename from x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBwcRestIT.java rename to x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/AbstractRemoteClusterSecurityBWCRestIT.java index 17acd258ed34..20cdbb9f8b0d 100644 --- a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBwcRestIT.java +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/AbstractRemoteClusterSecurityBWCRestIT.java @@ -7,6 +7,7 @@ package org.elasticsearch.xpack.remotecluster; +import org.apache.http.util.EntityUtils; import org.elasticsearch.action.search.SearchResponse; import org.elasticsearch.client.Request; import org.elasticsearch.client.RequestOptions; @@ -15,14 +16,9 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.core.Strings; import org.elasticsearch.search.SearchHit; import org.elasticsearch.search.SearchResponseUtils; -import org.elasticsearch.test.cluster.ElasticsearchCluster; -import org.elasticsearch.test.cluster.local.distribution.DistributionType; -import org.elasticsearch.test.cluster.util.Version; -import org.elasticsearch.test.cluster.util.resource.Resource; import org.elasticsearch.test.rest.ObjectPath; -import org.junit.ClassRule; -import org.junit.rules.RuleChain; -import org.junit.rules.TestRule; +import org.elasticsearch.xcontent.XContentBuilder; +import org.elasticsearch.xcontent.json.JsonXContent; import java.io.IOException; import java.util.Arrays; @@ -32,48 +28,21 @@ import java.util.Map; import java.util.stream.Collectors; import static org.hamcrest.Matchers.containsInAnyOrder; +import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.hasKey; import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.notNullValue; +import static org.hamcrest.Matchers.nullValue; /** - * BWC test which ensures that users and API keys with defined {@code remote_indices} privileges can be used to query legacy remote clusters + * A set of BWC tests that can be executed with either RCS 1 or RCS 2 against an older fulfilling cluster. */ -public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurityTestCase { +public abstract class AbstractRemoteClusterSecurityBWCRestIT extends AbstractRemoteClusterSecurityTestCase { - private static final Version OLD_CLUSTER_VERSION = Version.fromString(System.getProperty("tests.old_cluster_version")); + protected abstract boolean isRCS2(); - static { - fulfillingCluster = ElasticsearchCluster.local() - .version(OLD_CLUSTER_VERSION) - .distribution(DistributionType.DEFAULT) - .name("fulfilling-cluster") - .apply(commonClusterConfig) - .setting("xpack.ml.enabled", "false") - .build(); - - queryCluster = ElasticsearchCluster.local() - .version(Version.CURRENT) - .distribution(DistributionType.INTEG_TEST) - .name("query-cluster") - .apply(commonClusterConfig) - .setting("xpack.security.remote_cluster_client.ssl.enabled", "true") - .setting("xpack.security.remote_cluster_client.ssl.certificate_authorities", "remote-cluster-ca.crt") - .rolesFile(Resource.fromClasspath("roles.yml")) - .build(); - } - - @ClassRule - // Use a RuleChain to ensure that fulfilling cluster is started before query cluster - public static TestRule clusterRule = RuleChain.outerRule(fulfillingCluster).around(queryCluster); - - public void testBwcWithLegacyCrossClusterSearch() throws Exception { - final boolean useProxyMode = randomBoolean(); - // Update remote cluster settings on QC. - setupQueryClusterRemoteClusters(useProxyMode); - // Ensure remote cluster is connected - ensureRemoteFulfillingClusterIsConnected(useProxyMode); + public void testBwcCCSViaRCS1orRCS2() throws Exception { // Fulfilling cluster { @@ -122,19 +91,22 @@ public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurit ] }"""); assertOK(adminClient().performRequest(putRoleRequest)); - // We need to define the same role on QC and FC in order for CCS to work. - final var putRoleRequestFulfilling = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); - putRoleRequestFulfilling.setJsonEntity(""" - { - "cluster": ["manage_own_api_key"], - "indices": [ + if (isRCS2() == false) { + // We need to define the same role on QC and FC in order for CCS to work. + final var putRoleRequestFulfilling = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); + putRoleRequestFulfilling.setJsonEntity(""" { - "names": ["remote_index1"], - "privileges": ["read", "read_cross_cluster"] - } - ] - }"""); - assertOK(performRequestAgainstFulfillingCluster(putRoleRequestFulfilling)); + "cluster": ["manage_own_api_key"], + "indices": [ + { + "names": ["remote_index1"], + "privileges": ["read", "read_cross_cluster"] + } + ] + }"""); + assertOK(performRequestAgainstFulfillingCluster(putRoleRequestFulfilling)); + } + final var putUserRequest = new Request("PUT", "/_security/user/" + REMOTE_SEARCH_USER); putUserRequest.setJsonEntity(""" { @@ -166,7 +138,7 @@ public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurit ], "remote_cluster": [ { - "privileges": ["monitor_enrich"], + "privileges": ["monitor_enrich", "monitor_stats"], "clusters": ["*"] } ] @@ -187,38 +159,35 @@ public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurit // Check that we can search the fulfilling cluster from the querying cluster final boolean alsoSearchLocally = randomBoolean(); + final String remoteClusterName = randomFrom("my_remote_cluster", "*", "my_remote_*"); + final String remoteIndexName = randomFrom("remote_index1", "*"); final var searchRequest = new Request( "GET", String.format( Locale.ROOT, "/%s%s:%s/_search?ccs_minimize_roundtrips=%s", alsoSearchLocally ? "local_index," : "", - randomFrom("my_remote_cluster", "*", "my_remote_*"), - randomFrom("remote_index1", "*"), + remoteClusterName, + remoteIndexName, randomBoolean() ) ); - final String sendRequestWith = randomFrom("user", "apikey"); - final Response response = sendRequestWith.equals("user") - ? performRequestWithRemoteAccessUser(searchRequest) - : performRequestWithApiKey(searchRequest, apiKeyEncoded); + String esqlCommand = String.format(Locale.ROOT, "FROM %s,%s:%s | LIMIT 10", "local_index", remoteClusterName, remoteIndexName); + // send request with user + Response response = performRequestWithRemoteAccessUser(searchRequest); assertOK(response); - final SearchResponse searchResponse; try (var parser = responseAsParser(response)) { - searchResponse = SearchResponseUtils.parseSearchResponse(parser); + assertSearchResponse(SearchResponseUtils.parseSearchResponse(parser), alsoSearchLocally); } - try { - final List actualIndices = Arrays.stream(searchResponse.getHits().getHits()) - .map(SearchHit::getIndex) - .collect(Collectors.toList()); - if (alsoSearchLocally) { - assertThat(actualIndices, containsInAnyOrder("remote_index1", "local_index")); - } else { - assertThat(actualIndices, containsInAnyOrder("remote_index1")); - } - } finally { - searchResponse.decRef(); + assertEsqlResponse(performRequestWithRemoteAccessUser(esqlRequest(esqlCommand))); + + // send request with apikey + response = performRequestWithApiKey(searchRequest, apiKeyEncoded); + assertOK(response); + try (var parser = responseAsParser(response)) { + assertSearchResponse(SearchResponseUtils.parseSearchResponse(parser), alsoSearchLocally); } + assertEsqlResponse(performRequestWithApiKey(esqlRequest(esqlCommand), apiKeyEncoded)); } } @@ -231,6 +200,14 @@ public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurit final Map remoteInfoMap = responseAsMap(remoteInfoResponse); assertThat(remoteInfoMap, hasKey("my_remote_cluster")); assertThat(org.elasticsearch.xcontent.ObjectPath.eval("my_remote_cluster.connected", remoteInfoMap), is(true)); + if (isRCS2()) { + assertThat( + org.elasticsearch.xcontent.ObjectPath.eval("my_remote_cluster.cluster_credentials", remoteInfoMap), + is("::es_redacted::") // RCS 2.0 + ); + } else { + assertThat(org.elasticsearch.xcontent.ObjectPath.eval("my_remote_cluster.cluster_credentials", remoteInfoMap), nullValue()); + } if (false == useProxyMode) { assertThat( org.elasticsearch.xcontent.ObjectPath.eval("my_remote_cluster.num_nodes_connected", remoteInfoMap), @@ -240,18 +217,6 @@ public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurit }); } - private void setupQueryClusterRemoteClusters(boolean useProxyMode) throws IOException { - final Settings.Builder builder = Settings.builder(); - if (useProxyMode) { - builder.put("cluster.remote.my_remote_cluster.mode", "proxy") - .put("cluster.remote.my_remote_cluster.proxy_address", fulfillingCluster.getTransportEndpoint(0)); - } else { - builder.put("cluster.remote.my_remote_cluster.mode", "sniff") - .putList("cluster.remote.my_remote_cluster.seeds", fulfillingCluster.getTransportEndpoint(0)); - } - updateClusterSettings(builder.build()); - } - private Response performRequestWithRemoteAccessUser(final Request request) throws IOException { request.setOptions(RequestOptions.DEFAULT.toBuilder().addHeader("Authorization", basicAuthHeaderValue(REMOTE_SEARCH_USER, PASS))); return client().performRequest(request); @@ -262,4 +227,49 @@ public class RemoteClusterSecurityBwcRestIT extends AbstractRemoteClusterSecurit return client().performRequest(request); } + private void setupQueryClusterRCS1(boolean useProxyMode) throws IOException { + final Settings.Builder builder = Settings.builder(); + if (useProxyMode) { + builder.put("cluster.remote.my_remote_cluster.mode", "proxy") + .put("cluster.remote.my_remote_cluster.proxy_address", fulfillingCluster.getTransportEndpoint(0)); + } else { + builder.put("cluster.remote.my_remote_cluster.mode", "sniff") + .putList("cluster.remote.my_remote_cluster.seeds", fulfillingCluster.getTransportEndpoint(0)); + } + updateClusterSettings(builder.build()); + } + + private Request esqlRequest(String command) throws IOException { + XContentBuilder body = JsonXContent.contentBuilder(); + body.startObject(); + body.field("query", command); + body.field("include_ccs_metadata", true); + body.endObject(); + Request request = new Request("POST", "_query"); + request.setJsonEntity(org.elasticsearch.common.Strings.toString(body)); + return request; + } + + private void assertSearchResponse(SearchResponse searchResponse, boolean alsoSearchLocally) { + try { + final List actualIndices = Arrays.stream(searchResponse.getHits().getHits()) + .map(SearchHit::getIndex) + .collect(Collectors.toList()); + if (alsoSearchLocally) { + assertThat(actualIndices, containsInAnyOrder("remote_index1", "local_index")); + } else { + assertThat(actualIndices, containsInAnyOrder("remote_index1")); + } + } finally { + searchResponse.decRef(); + } + } + + private void assertEsqlResponse(Response response) throws IOException { + assertOK(response); + String responseAsString = EntityUtils.toString(response.getEntity()); + assertThat(responseAsString, containsString("\"my_remote_cluster\":{\"status\":\"successful\"")); + assertThat(responseAsString, containsString("local_bar")); + assertThat(responseAsString, containsString("bar")); + } } diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS1ClusterRestIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS1ClusterRestIT.java new file mode 100644 index 000000000000..73e0f096039f --- /dev/null +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS1ClusterRestIT.java @@ -0,0 +1,69 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.remotecluster; + +import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.elasticsearch.test.cluster.local.distribution.DistributionType; +import org.elasticsearch.test.cluster.util.Version; +import org.elasticsearch.test.cluster.util.resource.Resource; +import org.junit.Before; +import org.junit.ClassRule; +import org.junit.rules.RuleChain; +import org.junit.rules.TestRule; + +/** + * BWC test which ensures that users and API keys with defined {@code remote_indices}/{@code remote_cluster} privileges can be used + * to query legacy remote clusters when using RCS 1.0. We send the request the to an older fulfilling cluster using RCS 1.0 with a user/role + * and API key where the {@code remote_indices}/{@code remote_cluster} are defined in the newer query cluster. + * All RCS 2.0 config should be effectively ignored when using RCS 1 for CCS. We send to an elder fulfil cluster to help ensure that + * newly introduced RCS 2.0 artifacts are forward compatible from the perspective of the old cluster. For example, a new privilege + * sent to an old cluster should be ignored. + */ +public class RemoteClusterSecurityBWCToRCS1ClusterRestIT extends AbstractRemoteClusterSecurityBWCRestIT { + + private static final Version OLD_CLUSTER_VERSION = Version.fromString(System.getProperty("tests.old_cluster_version")); + + static { + fulfillingCluster = ElasticsearchCluster.local() + .version(OLD_CLUSTER_VERSION) + .distribution(DistributionType.DEFAULT) + .name("fulfilling-cluster") + .apply(commonClusterConfig) + .setting("xpack.ml.enabled", "false") + // .setting("logger.org.elasticsearch.xpack.core", "trace") //useful for human debugging + // .setting("logger.org.elasticsearch.xpack.security", "trace") //useful for human debugging + .build(); + + queryCluster = ElasticsearchCluster.local() + .version(Version.CURRENT) + .distribution(DistributionType.DEFAULT) + .setting("xpack.ml.enabled", "false") + .name("query-cluster") + .apply(commonClusterConfig) + .setting("xpack.security.remote_cluster_client.ssl.enabled", "true") + .setting("xpack.security.remote_cluster_client.ssl.certificate_authorities", "remote-cluster-ca.crt") + .rolesFile(Resource.fromClasspath("roles.yml")) + .build(); + } + + @ClassRule + // Use a RuleChain to ensure that fulfilling cluster is started before query cluster + public static TestRule clusterRule = RuleChain.outerRule(fulfillingCluster).around(queryCluster); + + @Override + protected boolean isRCS2() { + return false; + } + + @Before + @Override + public void setUp() throws Exception { + configureRemoteCluster(REMOTE_CLUSTER_ALIAS, fulfillingCluster, true, randomBoolean(), false); + super.setUp(); + } +} diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS2ClusterRestIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS2ClusterRestIT.java new file mode 100644 index 000000000000..5e173b72c66d --- /dev/null +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityBWCToRCS2ClusterRestIT.java @@ -0,0 +1,90 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.remotecluster; + +import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.elasticsearch.test.cluster.local.distribution.DistributionType; +import org.elasticsearch.test.cluster.util.Version; +import org.junit.Before; +import org.junit.ClassRule; +import org.junit.rules.RuleChain; +import org.junit.rules.TestRule; + +import java.util.Map; +import java.util.concurrent.atomic.AtomicReference; + +/** + * BWC test which ensures that users and API keys with defined {@code remote_indices}/{@code remote_cluster} privileges can be used + * to query older remote clusters when using RCS 2.0. We send the request the to an older fulfilling cluster using RCS 2.0 with a user/role + * and API key where the {@code remote_indices}/{@code remote_cluster} are defined in the newer query cluster. + * All new RCS 2.0 config should be effectively ignored when sending to older RCS 2.0. For example, a new privilege + * sent to an old cluster should be ignored. + */ +public class RemoteClusterSecurityBWCToRCS2ClusterRestIT extends AbstractRemoteClusterSecurityBWCRestIT { + + private static final Version OLD_CLUSTER_VERSION = Version.fromString(System.getProperty("tests.old_cluster_version")); + private static final AtomicReference> API_KEY_MAP_REF = new AtomicReference<>(); + + static { + + fulfillingCluster = ElasticsearchCluster.local() + .name("fulfilling-cluster") + .version(OLD_CLUSTER_VERSION) + .distribution(DistributionType.DEFAULT) + .apply(commonClusterConfig) + .setting("xpack.ml.enabled", "false") + .setting("remote_cluster_server.enabled", "true") + .setting("remote_cluster.port", "0") + .setting("xpack.security.remote_cluster_server.ssl.enabled", "true") + .setting("xpack.security.remote_cluster_server.ssl.key", "remote-cluster.key") + .setting("xpack.security.remote_cluster_server.ssl.certificate", "remote-cluster.crt") + .keystore("xpack.security.remote_cluster_server.ssl.secure_key_passphrase", "remote-cluster-password") + // .setting("logger.org.elasticsearch.xpack.core", "trace") //useful for human debugging + // .setting("logger.org.elasticsearch.xpack.security", "trace") //useful for human debugging + .build(); + + queryCluster = ElasticsearchCluster.local() + .name("query-cluster") + .distribution(DistributionType.DEFAULT) + .setting("xpack.ml.enabled", "false") + .apply(commonClusterConfig) + .setting("xpack.security.remote_cluster_client.ssl.enabled", "true") + .setting("xpack.security.remote_cluster_client.ssl.certificate_authorities", "remote-cluster-ca.crt") + .keystore("cluster.remote.my_remote_cluster.credentials", () -> { + if (API_KEY_MAP_REF.get() == null) { + final Map apiKeyMap = createCrossClusterAccessApiKey(""" + { + "search": [ + { + "names": ["*"] + } + ] + }"""); + API_KEY_MAP_REF.set(apiKeyMap); + } + return (String) API_KEY_MAP_REF.get().get("encoded"); + }) + .build(); + } + + @ClassRule + // Use a RuleChain to ensure that fulfilling cluster is started before query cluster + public static TestRule clusterRule = RuleChain.outerRule(fulfillingCluster).around(queryCluster); + + @Override + protected boolean isRCS2() { + return true; + } + + @Before + @Override + public void setUp() throws Exception { + configureRemoteCluster(REMOTE_CLUSTER_ALIAS, fulfillingCluster, false, randomBoolean(), false); + super.setUp(); + } +} diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRestStatsIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRestStatsIT.java new file mode 100644 index 000000000000..e98fcf6f7288 --- /dev/null +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRestStatsIT.java @@ -0,0 +1,266 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.remotecluster; + +import org.apache.http.util.EntityUtils; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.elasticsearch.action.search.SearchResponse; +import org.elasticsearch.client.Request; +import org.elasticsearch.client.RequestOptions; +import org.elasticsearch.client.Response; +import org.elasticsearch.common.bytes.BytesArray; +import org.elasticsearch.common.xcontent.XContentHelper; +import org.elasticsearch.core.Strings; +import org.elasticsearch.search.SearchHit; +import org.elasticsearch.search.SearchResponseUtils; +import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.elasticsearch.test.cluster.local.distribution.DistributionType; +import org.elasticsearch.test.cluster.util.resource.Resource; +import org.elasticsearch.test.junit.RunnableTestRuleAdapter; +import org.elasticsearch.test.rest.ObjectPath; +import org.elasticsearch.xcontent.XContentType; +import org.junit.ClassRule; +import org.junit.rules.RuleChain; +import org.junit.rules.TestRule; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.Arrays; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import java.util.stream.Collectors; + +import static org.hamcrest.Matchers.containsInAnyOrder; +import static org.hamcrest.Matchers.equalTo; + +public class RemoteClusterSecurityRestStatsIT extends AbstractRemoteClusterSecurityTestCase { + + private static final AtomicReference> API_KEY_MAP_REF = new AtomicReference<>(); + private static final AtomicReference> REST_API_KEY_MAP_REF = new AtomicReference<>(); + private static final AtomicBoolean SSL_ENABLED_REF = new AtomicBoolean(); + private static final AtomicBoolean NODE1_RCS_SERVER_ENABLED = new AtomicBoolean(); + private static final AtomicBoolean NODE2_RCS_SERVER_ENABLED = new AtomicBoolean(); + private static final int FULFILL_NODE_COUNT = 3; + private static final Logger logger = LogManager.getLogger(RemoteClusterSecurityRestStatsIT.class); + + static { + fulfillingCluster = ElasticsearchCluster.local() + .distribution(DistributionType.DEFAULT) + .name("fulfilling-cluster") + .nodes(FULFILL_NODE_COUNT) + .apply(commonClusterConfig) + .setting("remote_cluster.port", "0") + .setting("xpack.security.remote_cluster_server.ssl.enabled", () -> String.valueOf(SSL_ENABLED_REF.get())) + .setting("xpack.security.remote_cluster_server.ssl.key", "remote-cluster.key") + .setting("xpack.security.remote_cluster_server.ssl.certificate", "remote-cluster.crt") + .setting("xpack.security.authc.token.enabled", "true") + .keystore("xpack.security.remote_cluster_server.ssl.secure_key_passphrase", "remote-cluster-password") + .node(0, spec -> spec.setting("remote_cluster_server.enabled", "true")) + .node(1, spec -> spec.setting("remote_cluster_server.enabled", () -> String.valueOf(NODE1_RCS_SERVER_ENABLED.get()))) + .node(2, spec -> spec.setting("remote_cluster_server.enabled", () -> String.valueOf(NODE2_RCS_SERVER_ENABLED.get()))) + .build(); + + queryCluster = ElasticsearchCluster.local() + .distribution(DistributionType.DEFAULT) + .name("query-cluster") + .apply(commonClusterConfig) + .setting("xpack.security.remote_cluster_client.ssl.enabled", () -> String.valueOf(SSL_ENABLED_REF.get())) + .setting("xpack.security.remote_cluster_client.ssl.certificate_authorities", "remote-cluster-ca.crt") + .setting("xpack.security.authc.token.enabled", "true") + .keystore("cluster.remote.my_remote_cluster.credentials", () -> { + if (API_KEY_MAP_REF.get() == null) { + final Map apiKeyMap = createCrossClusterAccessApiKey(""" + { + "search": [ + { + "names": ["*"] + } + ] + }"""); + API_KEY_MAP_REF.set(apiKeyMap); + } + return (String) API_KEY_MAP_REF.get().get("encoded"); + }) + // Define a bogus API key for another remote cluster + .keystore("cluster.remote.invalid_remote.credentials", randomEncodedApiKey()) + // Define remote with a REST API key to observe expected failure + .keystore("cluster.remote.wrong_api_key_type.credentials", () -> { + if (REST_API_KEY_MAP_REF.get() == null) { + initFulfillingClusterClient(); + final var createApiKeyRequest = new Request("POST", "/_security/api_key"); + createApiKeyRequest.setJsonEntity(""" + { + "name": "rest_api_key" + }"""); + try { + final Response createApiKeyResponse = performRequestWithAdminUser(fulfillingClusterClient, createApiKeyRequest); + assertOK(createApiKeyResponse); + REST_API_KEY_MAP_REF.set(responseAsMap(createApiKeyResponse)); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + return (String) REST_API_KEY_MAP_REF.get().get("encoded"); + }) + .rolesFile(Resource.fromClasspath("roles.yml")) + .user(REMOTE_METRIC_USER, PASS.toString(), "read_remote_shared_metrics", false) + .build(); + } + + @ClassRule + // Use a RuleChain to ensure that fulfilling cluster is started before query cluster + // `SSL_ENABLED_REF` is used to control the SSL-enabled setting on the test clusters + // We set it here, since randomization methods are not available in the static initialize context above + public static TestRule clusterRule = RuleChain.outerRule(new RunnableTestRuleAdapter(() -> { + SSL_ENABLED_REF.set(usually()); + NODE1_RCS_SERVER_ENABLED.set(randomBoolean()); + NODE2_RCS_SERVER_ENABLED.set(randomBoolean()); + })).around(fulfillingCluster).around(queryCluster); + + public void testCrossClusterStats() throws Exception { + configureRemoteCluster(); + setupRoleAndUserQueryCluster(); + addDocToIndexFulfillingCluster("index1"); + + // search #1 + searchFulfillingClusterFromQueryCluster("index1"); + Map statsResponseAsMap = getFulfillingClusterStatsFromQueryCluster(); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs.clusters.my_remote_cluster.nodes_count"), equalTo(FULFILL_NODE_COUNT)); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.clusters.my_remote_cluster.total"), equalTo(1)); + int initialIndexCount = ObjectPath.evaluate(statsResponseAsMap, "ccs.clusters.my_remote_cluster.indices_count"); + + // search #2 + searchFulfillingClusterFromQueryCluster("index1"); + statsResponseAsMap = getFulfillingClusterStatsFromQueryCluster(); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.total"), equalTo(2)); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.clusters.my_remote_cluster.total"), equalTo(2)); + + // search #3 + expectThrows(Exception.class, () -> searchFulfillingClusterFromQueryCluster("junk")); + statsResponseAsMap = getFulfillingClusterStatsFromQueryCluster(); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.total"), equalTo(3)); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.clusters.my_remote_cluster.total"), equalTo(2)); + + // search #4 + addDocToIndexFulfillingCluster("index2"); + searchFulfillingClusterFromQueryCluster("index2"); + statsResponseAsMap = getFulfillingClusterStatsFromQueryCluster(); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.total"), equalTo(4)); + assertThat(ObjectPath.evaluate(statsResponseAsMap, "ccs._search.clusters.my_remote_cluster.total"), equalTo(3)); + int updatedIndexCount = ObjectPath.evaluate(statsResponseAsMap, "ccs.clusters.my_remote_cluster.indices_count"); + assertThat(updatedIndexCount, equalTo(initialIndexCount + 1)); + } + + private Map getFulfillingClusterStatsFromQueryCluster() throws IOException { + return getFulfillingClusterStatsFromQueryCluster(false); + } + + private Map getFulfillingClusterStatsFromQueryCluster(boolean humanDebug) throws IOException { + Request stats = new Request("GET", "_cluster/stats?include_remotes=true&filter_path=ccs"); + Response statsResponse = performRequestWithRemoteSearchUser(stats); + if (humanDebug) { + debugResponse(statsResponse); + } + return entityAsMap(statsResponse.getEntity()); + } + + private void searchFulfillingClusterFromQueryCluster(String index, boolean humanDebug) throws IOException { + final var searchRequest = new Request( + "GET", + String.format( + Locale.ROOT, + "/%s:%s/_search?ccs_minimize_roundtrips=%s", + randomFrom("my_remote_cluster", "*", "my_remote_*"), + index, + randomBoolean() + ) + ); + Response response = performRequestWithRemoteSearchUser(searchRequest); + if (humanDebug) { + debugResponse(response); + } + assertOK(response); + final SearchResponse searchResponse = SearchResponseUtils.parseSearchResponse(responseAsParser(response)); + try { + final List actualIndices = Arrays.stream(searchResponse.getHits().getHits()) + .map(SearchHit::getIndex) + .collect(Collectors.toList()); + assertThat(actualIndices, containsInAnyOrder(index)); + + } finally { + searchResponse.decRef(); + } + } + + private void searchFulfillingClusterFromQueryCluster(String index) throws IOException { + searchFulfillingClusterFromQueryCluster(index, false); + } + + private void addDocToIndexFulfillingCluster(String index) throws IOException { + // Index some documents, so we can attempt to search them from the querying cluster + final Request bulkRequest = new Request("POST", "/_bulk?refresh=true"); + bulkRequest.setJsonEntity(Strings.format(""" + { "index": { "_index": "%s" } } + { "foo": "bar" } + """, index)); + assertOK(performRequestAgainstFulfillingCluster(bulkRequest)); + } + + private void setupRoleAndUserQueryCluster() throws IOException { + final var putRoleRequest = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); + putRoleRequest.setJsonEntity(""" + { + "description": "Role with privileges for remote indices and stats.", + "cluster": ["monitor_stats"], + "remote_indices": [ + { + "names": ["*"], + "privileges": ["read", "read_cross_cluster"], + "clusters": ["*"] + } + ], + "remote_cluster": [ + { + "privileges": ["monitor_stats"], + "clusters": ["*"] + } + ] + }"""); + assertOK(adminClient().performRequest(putRoleRequest)); + final var putUserRequest = new Request("PUT", "/_security/user/" + REMOTE_SEARCH_USER); + putUserRequest.setJsonEntity(""" + { + "password": "x-pack-test-password", + "roles" : ["remote_search"] + }"""); + assertOK(adminClient().performRequest(putUserRequest)); + } + + private Response performRequestWithRemoteSearchUser(final Request request) throws IOException { + request.setOptions( + RequestOptions.DEFAULT.toBuilder().addHeader("Authorization", headerFromRandomAuthMethod(REMOTE_SEARCH_USER, PASS)) + ); + return client().performRequest(request); + } + + // helper method for humans see the responses for debug purposes, when used will always fail the test + private void debugResponse(Response response) throws IOException { + String jsonString = XContentHelper.convertToJson( + new BytesArray(EntityUtils.toString(response.getEntity())), + true, + true, + XContentType.JSON + ); + logger.error(jsonString); + assertFalse(true); // boom + } +} diff --git a/x-pack/plugin/security/qa/security-trial/src/javaRestTest/java/org/elasticsearch/xpack/security/apikey/ApiKeyRestIT.java b/x-pack/plugin/security/qa/security-trial/src/javaRestTest/java/org/elasticsearch/xpack/security/apikey/ApiKeyRestIT.java index 667140b84995..8ce7fc77fe4f 100644 --- a/x-pack/plugin/security/qa/security-trial/src/javaRestTest/java/org/elasticsearch/xpack/security/apikey/ApiKeyRestIT.java +++ b/x-pack/plugin/security/qa/security-trial/src/javaRestTest/java/org/elasticsearch/xpack/security/apikey/ApiKeyRestIT.java @@ -828,7 +828,7 @@ public class ApiKeyRestIT extends SecurityOnTrialLicenseRestTestCase { assertOK(response); assertAPIKeyWithRemoteClusterPermissions(apiKeyId, includeRemoteCluster, false, null, new String[] { "foo", "bar" }); - // create API key as the remote user which does remote_cluster limited_by permissions + // create API key as the remote user which has all remote_cluster permissions via limited_by response = sendRequestAsRemoteUser(createApiKeyRequest); apiKeyId = ObjectPath.createFromResponse(response).evaluate("id"); assertThat(apiKeyId, notNullValue()); @@ -922,7 +922,7 @@ public class ApiKeyRestIT extends SecurityOnTrialLicenseRestTestCase { assertNotNull(limitedByRole); List>> remoteCluster = (List>>) limitedByRole.get("remote_cluster"); - assertThat(remoteCluster.get(0).get("privileges"), containsInAnyOrder("monitor_enrich")); + assertThat(remoteCluster.get(0).get("privileges"), containsInAnyOrder("monitor_stats", "monitor_enrich")); assertThat(remoteCluster.get(0).get("clusters"), containsInAnyOrder("remote")); } else { // no limited by permissions diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStore.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStore.java index d79a3e31c1bc..2e1a643bf4f4 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStore.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStore.java @@ -572,7 +572,7 @@ public class CompositeRolesStore { ); }); - if (remoteClusterPermissions.hasPrivileges()) { + if (remoteClusterPermissions.hasAnyPrivileges()) { builder.addRemoteClusterPermissions(remoteClusterPermissions); } else { builder.addRemoteClusterPermissions(RemoteClusterPermissions.NONE); diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/RoleDescriptorStore.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/RoleDescriptorStore.java index ac8d84d95fd1..a64cef366926 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/RoleDescriptorStore.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/RoleDescriptorStore.java @@ -150,7 +150,7 @@ public class RoleDescriptorStore implements RoleReferenceResolver { + "but other privileges found for subject [" + crossClusterAccessRoleReference.getUserPrincipal() + "]"; - logger.debug("{}. Invalid role descriptor: [{}]", message, roleDescriptor); + logger.warn("{}. Invalid role descriptor: [{}]", message, roleDescriptor); listener.onFailure(new IllegalArgumentException(message)); return; } diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/RBACEngineTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/RBACEngineTests.java index d71c2b0d1907..a41c54ada781 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/RBACEngineTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/RBACEngineTests.java @@ -92,6 +92,7 @@ import org.elasticsearch.xpack.core.security.authz.permission.SimpleRole; import org.elasticsearch.xpack.core.security.authz.privilege.ApplicationPrivilege; import org.elasticsearch.xpack.core.security.authz.privilege.ApplicationPrivilegeDescriptor; import org.elasticsearch.xpack.core.security.authz.privilege.ApplicationPrivilegeTests; +import org.elasticsearch.xpack.core.security.authz.privilege.ClusterPrivilegeResolver; import org.elasticsearch.xpack.core.security.authz.privilege.ConfigurableClusterPrivileges.ManageApplicationPrivileges; import org.elasticsearch.xpack.core.security.authz.privilege.IndexPrivilege; import org.elasticsearch.xpack.core.security.authz.privilege.Privilege; @@ -1312,10 +1313,7 @@ public class RBACEngineTests extends ESTestCase { ) .addRemoteClusterPermissions( new RemoteClusterPermissions().addGroup( - new RemoteClusterPermissionGroup( - RemoteClusterPermissions.getSupportedRemoteClusterPermissions().toArray(new String[0]), - new String[] { "remote-1" } - ) + new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "remote-1" }) ) .addGroup( new RemoteClusterPermissionGroup( @@ -1383,26 +1381,33 @@ public class RBACEngineTests extends ESTestCase { RemoteClusterPermissions remoteClusterPermissions = response.getRemoteClusterPermissions(); String[] allRemoteClusterPermissions = RemoteClusterPermissions.getSupportedRemoteClusterPermissions().toArray(new String[0]); - assert allRemoteClusterPermissions.length == 1 - : "if more remote cluster permissions are added this test needs to be updated to ensure the correct remotes receive the " - + "correct permissions. "; - // 2 groups with 3 aliases + assertThat(response.getRemoteClusterPermissions().groups(), iterableWithSize(2)); - assertEquals( - 3, - response.getRemoteClusterPermissions() - .groups() - .stream() - .map(RemoteClusterPermissionGroup::remoteClusterAliases) - .flatMap(Arrays::stream) - .distinct() - .count() + // remote-1 has monitor_enrich permission + // remote-2 and remote-3 have all permissions + assertThat( + response.getRemoteClusterPermissions().groups(), + containsInAnyOrder( + new RemoteClusterPermissionGroup(new String[] { "monitor_enrich" }, new String[] { "remote-1" }), + new RemoteClusterPermissionGroup(allRemoteClusterPermissions, new String[] { "remote-2", "remote-3" }) + ) + ); + + // ensure that all permissions are valid for the current transport version + assertThat( + Arrays.asList(remoteClusterPermissions.collapseAndRemoveUnsupportedPrivileges("remote-1", TransportVersion.current())), + hasItem("monitor_enrich") ); for (String permission : RemoteClusterPermissions.getSupportedRemoteClusterPermissions()) { - assertThat(Arrays.asList(remoteClusterPermissions.privilegeNames("remote-1", TransportVersion.current())), hasItem(permission)); - assertThat(Arrays.asList(remoteClusterPermissions.privilegeNames("remote-2", TransportVersion.current())), hasItem(permission)); - assertThat(Arrays.asList(remoteClusterPermissions.privilegeNames("remote-3", TransportVersion.current())), hasItem(permission)); + assertThat( + Arrays.asList(remoteClusterPermissions.collapseAndRemoveUnsupportedPrivileges("remote-2", TransportVersion.current())), + hasItem(permission) + ); + assertThat( + Arrays.asList(remoteClusterPermissions.collapseAndRemoveUnsupportedPrivileges("remote-3", TransportVersion.current())), + hasItem(permission) + ); } } @@ -1782,7 +1787,10 @@ public class RBACEngineTests extends ESTestCase { new RoleDescriptorsIntersection( new RoleDescriptor( Role.REMOTE_USER_ROLE_NAME, - null, + RemoteClusterPermissions.getSupportedRemoteClusterPermissions() + .stream() + .filter(s -> s.equals(ClusterPrivilegeResolver.MONITOR_STATS.name())) + .toArray(String[]::new), new IndicesPrivileges[] { IndicesPrivileges.builder().indices(".monitoring-*").privileges("read", "read_cross_cluster").build(), IndicesPrivileges.builder().indices("apm-*").privileges("read", "read_cross_cluster").build(), diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStoreTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStoreTests.java index 35da312f2629..f2fd9b301aec 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStoreTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/CompositeRolesStoreTests.java @@ -1158,7 +1158,7 @@ public class CompositeRolesStoreTests extends ESTestCase { assertHasRemoteIndexGroupsForClusters(forRemote, Set.of("*"), indexGroup("remote-idx-2-*")); assertValidRemoteClusterPermissions(role.remoteCluster(), new String[] { "remote-*" }); assertThat( - role.remoteCluster().privilegeNames("remote-foobar", TransportVersion.current()), + role.remoteCluster().collapseAndRemoveUnsupportedPrivileges("remote-foobar", TransportVersion.current()), equalTo(RemoteClusterPermissions.getSupportedRemoteClusterPermissions().toArray(new String[0])) ); } @@ -3334,12 +3334,12 @@ public class CompositeRolesStoreTests extends ESTestCase { } private void assertValidRemoteClusterPermissionsParent(RemoteClusterPermissions permissions, String[] aliases) { - assertTrue(permissions.hasPrivileges()); + assertTrue(permissions.hasAnyPrivileges()); for (String alias : aliases) { - assertTrue(permissions.hasPrivileges(alias)); - assertFalse(permissions.hasPrivileges(randomValueOtherThan(alias, () -> randomAlphaOfLength(5)))); + assertTrue(permissions.hasAnyPrivileges(alias)); + assertFalse(permissions.hasAnyPrivileges(randomValueOtherThan(alias, () -> randomAlphaOfLength(5)))); assertThat( - permissions.privilegeNames(alias, TransportVersion.current()), + permissions.collapseAndRemoveUnsupportedPrivileges(alias, TransportVersion.current()), arrayContaining(RemoteClusterPermissions.getSupportedRemoteClusterPermissions().toArray(new String[0])) ); } diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java index a4d9dacd1a63..af5f44b5989f 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java @@ -388,7 +388,8 @@ public class FileRolesStoreTests extends ESTestCase { events.get(4), startsWith( "failed to parse remote_cluster for role [invalid_role_bad_priv_remote_clusters]. " - + "[monitor_enrich] is the only value allowed for [privileges] within [remote_cluster]. skipping role..." + + "[monitor_enrich, monitor_stats] are the only values allowed for [privileges] within [remote_cluster]. " + + "Found [junk]. skipping role..." ) ); } diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/user/RestGetUserPrivilegesActionTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/user/RestGetUserPrivilegesActionTests.java index e17d651a1974..5b91b774cc43 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/user/RestGetUserPrivilegesActionTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/user/RestGetUserPrivilegesActionTests.java @@ -213,7 +213,7 @@ public class RestGetUserPrivilegesActionTests extends ESTestCase { ,"remote_cluster":[ { "privileges":[ - "monitor_enrich" + "monitor_enrich", "monitor_stats" ], "clusters":[ "remote-1" @@ -221,7 +221,7 @@ public class RestGetUserPrivilegesActionTests extends ESTestCase { }, { "privileges":[ - "monitor_enrich" + "monitor_enrich", "monitor_stats" ], "clusters":[ "remote-2", diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/transport/netty4/SimpleSecurityNetty4ServerTransportTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/transport/netty4/SimpleSecurityNetty4ServerTransportTests.java index 888e858f2b03..c5c5e1493440 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/transport/netty4/SimpleSecurityNetty4ServerTransportTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/transport/netty4/SimpleSecurityNetty4ServerTransportTests.java @@ -454,7 +454,7 @@ public class SimpleSecurityNetty4ServerTransportTests extends AbstractSimpleTran DiscoveryNode node = DiscoveryNodeUtils.create( service.getLocalNode().getId(), clientAddress, - service.getLocalNode().getVersion() + service.getLocalNode().getVersionInformation() ); try (Transport.Connection connection2 = openConnection(serviceA, node, TestProfiles.LIGHT_PROFILE)) { sslEngine = getEngineFromAcceptedChannel(originalTransport, connection2); @@ -486,7 +486,7 @@ public class SimpleSecurityNetty4ServerTransportTests extends AbstractSimpleTran DiscoveryNode node = DiscoveryNodeUtils.create( service.getLocalNode().getId(), clientAddress, - service.getLocalNode().getVersion() + service.getLocalNode().getVersionInformation() ); try (Transport.Connection connection2 = openConnection(serviceA, node, TestProfiles.LIGHT_PROFILE)) { sslEngine = getEngineFromAcceptedChannel(originalTransport, connection2); @@ -518,7 +518,7 @@ public class SimpleSecurityNetty4ServerTransportTests extends AbstractSimpleTran DiscoveryNode node = DiscoveryNodeUtils.create( service.getLocalNode().getId(), clientAddress, - service.getLocalNode().getVersion() + service.getLocalNode().getVersionInformation() ); try (Transport.Connection connection2 = openConnection(serviceA, node, TestProfiles.LIGHT_PROFILE)) { sslEngine = getEngineFromAcceptedChannel(originalTransport, connection2); @@ -562,7 +562,7 @@ public class SimpleSecurityNetty4ServerTransportTests extends AbstractSimpleTran final DiscoveryNode node = DiscoveryNodeUtils.create( fcService.getLocalNode().getId(), remoteAccessAddress, - fcService.getLocalNode().getVersion() + fcService.getLocalNode().getVersionInformation() ); // 1. Connection will fail because FC server certificate is not trusted by default @@ -679,7 +679,7 @@ public class SimpleSecurityNetty4ServerTransportTests extends AbstractSimpleTran final DiscoveryNode node = DiscoveryNodeUtils.create( fcService.getLocalNode().getId(), remoteAccessAddress, - fcService.getLocalNode().getVersion() + fcService.getLocalNode().getVersionInformation() ); final Settings qcSettings = Settings.builder().put("xpack.security.remote_cluster_client.ssl.enabled", "false").build(); try ( diff --git a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/enrich/10_basic.yml b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/enrich/10_basic.yml index c9b05c4e13a8..17e5e0cfb075 100644 --- a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/enrich/10_basic.yml +++ b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/enrich/10_basic.yml @@ -69,7 +69,7 @@ setup: - do: warnings: - - "the [elasticsearch_version] field of an enrich policy has no effect and will be removed in Elasticsearch 9.0" + - "the [elasticsearch_version] field of an enrich policy has no effect and will be removed in a future version of Elasticsearch" enrich.put_policy: name: policy-crud-warning body: diff --git a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml index bb3345f4118b..6e7098da3380 100644 --- a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml +++ b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml @@ -162,4 +162,4 @@ setup: - match: {esql.functions.cos: $functions_cos} - gt: {esql.functions.to_long: $functions_to_long} - match: {esql.functions.coalesce: $functions_coalesce} - - length: {esql.functions: 118} # check the "sister" test above for a likely update to the same esql.functions length check + - length: {esql.functions: 116} # check the "sister" test above for a likely update to the same esql.functions length check diff --git a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/privileges/11_builtin.yml b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/privileges/11_builtin.yml index ef8fab9ca7b6..d03e6925cab1 100644 --- a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/privileges/11_builtin.yml +++ b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/privileges/11_builtin.yml @@ -15,5 +15,5 @@ setup: # This is fragile - it needs to be updated every time we add a new cluster/index privilege # I would much prefer we could just check that specific entries are in the array, but we don't have # an assertion for that - - length: { "cluster" : 61 } + - length: { "cluster" : 62 } - length: { "index" : 22 }