Merge remote-tracking branch 'origin/main' into lucene_snapshot

docs/changelog/106820.yaml

@@ -0,0 +1,5 @@
+pr: 106820
+summary: Add a capabilities API to check node and cluster capabilities
+area: Infra/REST API
+type: feature
+issues: []

docs/changelog/107891.yaml

@@ -0,0 +1,6 @@
+pr: 107891
+summary: Fix `startOffset` must be non-negative error in XLMRoBERTa tokenizer
+area: Machine Learning
+type: bug
+issues:
+ - 104626

docs/changelog/108238.yaml

@@ -0,0 +1,6 @@
+pr: 108238
+summary: "Nativeaccess: try to load all located libsystemds"
+area: Infra/Core
+type: bug
+issues:
+ - 107878

docs/changelog/108300.yaml

@@ -0,0 +1,5 @@
+pr: 108300
+summary: "ESQL: Add more time span units"
+area: ES|QL
+type: enhancement
+issues: []

docs/changelog/108431.yaml

@@ -0,0 +1,5 @@
+pr: 108431
+summary: "ESQL: Disable quoting in FROM command"
+area: ES|QL
+type: bug
+issues: []

docs/changelog/108444.yaml

@@ -0,0 +1,5 @@
+pr: 108444
+summary: "Apm-data: ignore malformed fields, and too many dynamic fields"
+area: Data streams
+type: enhancement
+issues: []

docs/internal/DistributedArchitectureGuide.md

@@ -10,70 +10,7 @@
 
 ### ActionListener
 
-Callbacks are used extensively throughout Elasticsearch because they enable us to write asynchronous and nonblocking code, i.e. code which
-doesn't necessarily compute a result straight away but also doesn't block the calling thread waiting for the result to become available.
-They support several useful control flows:
-
-- They can be completed immediately on the calling thread.
-- They can be completed concurrently on a different thread.
-- They can be stored in a data structure and completed later on when the system reaches a particular state.
-- Most commonly, they can be passed on to other methods that themselves require a callback.
-- They can be wrapped in another callback which modifies the behaviour of the original callback, perhaps adding some extra code to run
-  before or after completion, before passing them on.
-
-`ActionListener` is a general-purpose callback interface that is used extensively across the Elasticsearch codebase. `ActionListener` is
-used pretty much everywhere that needs to perform some asynchronous and nonblocking computation. The uniformity makes it easier to compose
-parts of the system together without needing to build adapters to convert back and forth between different kinds of callback. It also makes
-it easier to develop the skills needed to read and understand all the asynchronous code, although this definitely takes practice and is
-certainly not easy in an absolute sense. Finally, it has allowed us to build a rich library for working with `ActionListener` instances
-themselves, creating new instances out of existing ones and completing them in interesting ways. See for instance:
-
-- all the static methods on [ActionListener](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/ActionListener.java) itself
-- [`ThreadedActionListener`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/support/ThreadedActionListener.java) for forking work elsewhere
-- [`RefCountingListener`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/support/RefCountingListener.java) for running work in parallel
-- [`SubscribableListener`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/support/SubscribableListener.java) for constructing flexible workflows
-
-Callback-based asynchronous code can easily call regular synchronous code, but synchronous code cannot run callback-based asynchronous code
-without blocking the calling thread until the callback is called back. This blocking is at best undesirable (threads are too expensive to
-waste with unnecessary blocking) and at worst outright broken (the blocking can lead to deadlock). Unfortunately this means that most of our
-code ends up having to be written with callbacks, simply because it's ultimately calling into some other code that takes a callback. The
-entry points for all Elasticsearch APIs are callback-based (e.g. REST APIs all start at
-[`org.elasticsearch.rest.BaseRestHandler#prepareRequest`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/rest/BaseRestHandler.java#L158-L171),
-and transport APIs all start at
-[`org.elasticsearch.action.support.TransportAction#doExecute`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/support/TransportAction.java#L65))
-and the whole system fundamentally works in terms of an event loop (a `io.netty.channel.EventLoop`) which processes network events via
-callbacks.
-
-`ActionListener` is not an _ad-hoc_ invention. Formally speaking, it is our implementation of the general concept of a continuation in the
-sense of [_continuation-passing style_](https://en.wikipedia.org/wiki/Continuation-passing_style) (CPS): an extra argument to a function
-which defines how to continue the computation when the result is available. This is in contrast to _direct style_ which is the more usual
-style of calling methods that return values directly back to the caller so they can continue executing as normal. There's essentially two
-ways that computation can continue in Java (it can return a value or it can throw an exception) which is why `ActionListener` has both an
-`onResponse()` and an `onFailure()` method.
-
-CPS is strictly more expressive than direct style: direct code can be mechanically translated into continuation-passing style, but CPS also
-enables all sorts of other useful control structures such as forking work onto separate threads, possibly to be executed in parallel,
-perhaps even across multiple nodes, or possibly collecting a list of continuations all waiting for the same condition to be satisfied before
-proceeding (e.g.
-[`SubscribableListener`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/support/SubscribableListener.java)
-amongst many others). Some languages have first-class support for continuations (e.g. the `async` and `await` primitives in C#) allowing the
-programmer to write code in direct style away from those exotic control structures, but Java does not. That's why we have to manipulate all
-the callbacks ourselves.
-
-Strictly speaking, CPS requires that a computation _only_ continues by calling the continuation. In Elasticsearch, this means that
-asynchronous methods must have `void` return type and may not throw any exceptions. This is mostly the case in our code as written today,
-and is a good guiding principle, but we don't enforce void exceptionless methods and there are some deviations from this rule. In
-particular, it's not uncommon to permit some methods to throw an exception, using things like
-[`ActionListener#run`](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/ActionListener.java#L381-L390)
-(or an equivalent `try ... catch ...` block) further up the stack to handle it. Some methods also take (and may complete) an
-`ActionListener` parameter, but still return a value separately for other local synchronous work.
-
-This pattern is often used in the transport action layer with the use of the
-[ChannelActionListener](https://github.com/elastic/elasticsearch/blob/v8.12.2/server/src/main/java/org/elasticsearch/action/support/ChannelActionListener.java)
-class, which wraps a `TransportChannel` produced by the transport layer. `TransportChannel` implementations can hold a reference to a Netty
-channel with which to pass the response back to the network caller. Netty has a many-to-one association of network callers to channels, so a
-call taking a long time generally won't hog resources: it's cheap. A transport action can take hours to respond and that's alright, barring
-caller timeouts.
+See the [Javadocs for `ActionListener`](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/ActionListener.java)
 
 (TODO: add useful starter references and explanations for a range of Listener classes. Reference the Netty section.)
 
@@ -133,6 +70,14 @@ are only used for internode operations/communications.
 
 ### Work Queues
 
+### RestClient
+
+The `RestClient` is primarily used in testing, to send requests against cluster nodes in the same format as would users. There
+are some uses of `RestClient`, via `RestClientBuilder`, in the production code. For example, remote reindex leverages the
+`RestClient` internally as the REST client to the remote elasticsearch cluster, and to take advantage of the compatibility of
+`RestClient` requests with much older elasticsearch versions. The `RestClient` is also used externally by the `Java API Client`
+to communicate with Elasticsearch.
+
 # Cluster Coordination
 
 (Sketch of important classes? Might inform more sections to add for details.)

docs/reference/alias.asciidoc

@@ -358,6 +358,8 @@ POST _aliases
 ----
 // TEST[s/^/PUT my-index-2099.05.06-000001\n/]
 
+NOTE: Filters are only applied when using the <<query-dsl,Query DSL>>, and are not applied when <<docs-get,retrieving a document by ID>>.
+
 [discrete]
 [[alias-routing]]
 === Routing

docs/reference/esql/esql-syntax.asciidoc

@@ -160,14 +160,15 @@ Datetime intervals and timespans can be expressed using timespan literals.
 Timespan literals are a combination of a number and a qualifier. These
 qualifiers are supported:
 
-* `millisecond`/`milliseconds`
-* `second`/`seconds`
-* `minute`/`minutes`
-* `hour`/`hours`
-* `day`/`days`
-* `week`/`weeks`
-* `month`/`months`
-* `year`/`years`
+* `millisecond`/`milliseconds`/`ms`
+* `second`/`seconds`/`sec`/`s`
+* `minute`/`minutes`/`min`
+* `hour`/`hours`/`h`
+* `day`/`days`/`d`
+* `week`/`weeks`/`w`
+* `month`/`months`/`mo`
+* `quarter`/`quarters`/`q`
+* `year`/`years`/`yr`/`y`
 
 Timespan literals are not whitespace sensitive. These expressions are all valid:
 

docs/reference/high-availability/cluster-design.asciidoc

@@ -7,14 +7,14 @@ nodes to take over their responsibilities, an {es} cluster can continue
 operating normally if some of its nodes are unavailable or disconnected.
 
 There is a limit to how small a resilient cluster can be. All {es} clusters
-require:
+require the following components to function:
 
-- One <<modules-discovery-quorums,elected master node>> node
-- At least one node for each <<modules-node,role>>.
-- At least one copy of every <<scalability,shard>>.
+- One <<modules-discovery-quorums,elected master node>>
+- At least one node for each <<modules-node,role>>
+- At least one copy of every <<scalability,shard>>
 
 A resilient cluster requires redundancy for every required cluster component.
-This means a resilient cluster must have:
+This means a resilient cluster must have the following components:
 
 - At least three master-eligible nodes
 - At least two nodes of each role
@@ -375,11 +375,11 @@ The cluster will be resilient to the loss of any zone as long as:
 - There are at least two zones containing data nodes.
 - Every index that is not a <<searchable-snapshots,searchable snapshot index>>
   has at least one replica of each shard, in addition to the primary.
-- Shard allocation awareness is configured to avoid concentrating all copies of
-  a shard within a single zone.
+- <<shard-allocation-awareness,Shard allocation awareness>> is configured to 
+  avoid concentrating all copies of a shard within a single zone.
 - The cluster has at least three master-eligible nodes. At least two of these 
-  nodes are not voting-only master-eligible nodes, and they are spread evenly
-  across at least three zones.
+  nodes are not <<voting-only-node,voting-only master-eligible nodes>>, 
+  and they are spread evenly across at least three zones.
 - Clients are configured to send their requests to nodes in more than one zone
   or are configured to use a load balancer that balances the requests across an
   appropriate set of nodes. The {ess-trial}[Elastic Cloud] service provides such

docs/reference/modules/cluster/allocation_awareness.asciidoc

@@ -5,7 +5,7 @@ You can use custom node attributes as _awareness attributes_ to enable {es}
 to take your physical hardware configuration into account when allocating shards.
 If {es} knows which nodes are on the same physical server, in the same rack, or
 in the same zone, it can distribute the primary shard and its replica shards to
-minimise the risk of losing all shard copies in the event of a failure.
+minimize the risk of losing all shard copies in the event of a failure.
 
 When shard allocation awareness is enabled with the
 <<dynamic-cluster-setting,dynamic>>
@@ -19,6 +19,8 @@ allocated in each location. If the number of nodes in each location is
 unbalanced and there are a lot of replicas, replica shards might be left
 unassigned.
 
+TIP: Learn more about <<high-availability-cluster-design-large-clusters,designing resilient clusters>>.
+
 [[enabling-awareness]]
 ===== Enabling shard allocation awareness
 
@@ -26,15 +28,18 @@ To enable shard allocation awareness:
 
 . Specify the location of each node with a custom node attribute. For example, 
 if you want Elasticsearch to distribute shards across different racks, you might 
-set an awareness attribute called `rack_id` in each node's `elasticsearch.yml`
-config file.
+use an awareness attribute called `rack_id`. 
++
+You can set custom attributes in two ways:
+
+- By editing the `elasticsearch.yml` config file:
 +
 [source,yaml]
 --------------------------------------------------------
 node.attr.rack_id: rack_one
 --------------------------------------------------------
 +
-You can also set custom attributes when you start a node:
+- Using the `-E` command line argument when you start a node:
 +
 [source,sh]
 --------------------------------------------------------
@@ -56,17 +61,33 @@ cluster.routing.allocation.awareness.attributes: rack_id <1>
 +
 You can also use the
 <<cluster-update-settings,cluster-update-settings>> API to set or update
-a cluster's awareness attributes.
+a cluster's awareness attributes:
++
+[source,console]
+--------------------------------------------------
+PUT /_cluster/settings
+{
+  "persistent" : {
+    "cluster.routing.allocation.awareness.attributes" : "rack_id"
+  }
+}
+--------------------------------------------------
 
 With this example configuration, if you start two nodes with
 `node.attr.rack_id` set to `rack_one` and create an index with 5 primary
 shards and 1 replica of each primary, all primaries and replicas are
-allocated across the two nodes.
+allocated across the two node.
+
+.All primaries and replicas allocated across two nodes in the same rack
+image::images/shard-allocation/shard-allocation-awareness-one-rack.png[All primaries and replicas are allocated across two nodes in the same rack]
 
 If you add two nodes with `node.attr.rack_id` set to `rack_two`,
 {es} moves shards to the new nodes, ensuring (if possible)
 that no two copies of the same shard are in the same rack.
 
+.Primaries and replicas allocated across four nodes in two racks, with no two copies of the same shard in the same rack
+image::images/shard-allocation/shard-allocation-awareness-two-racks.png[Primaries and replicas are allocated across four nodes in two racks with no two copies of the same shard in the same rack]
+
 If `rack_two` fails and takes down both its nodes, by default {es}
 allocates the lost shard copies to nodes in `rack_one`. To prevent multiple
 copies of a particular shard from being allocated in the same location, you can

docs/reference/rest-api/common-parms.asciidoc

@@ -1062,8 +1062,8 @@ end::stats[]
 
 tag::stored_fields[]
 `stored_fields`::
-(Optional, Boolean) If `true`, retrieves the document fields stored in the
-index rather than the document `_source`. Defaults to `false`.
+(Optional, string)
+A comma-separated list of <<mapping-store,`stored fields`>> to include in the response.
 end::stored_fields[]
 
 tag::sync[]

docs/reference/search/search-your-data/cohere-es.asciidoc

@@ -0,0 +1,371 @@
+[[cohere-es]]
+=== Tutorial: Using Cohere with {es}
+++++
+<titleabbrev>Using Cohere with {es}</titleabbrev>
+++++
+
+The instructions in this tutorial shows you how to compute embeddings with
+Cohere using the {infer} API and store them for efficient vector or hybrid
+search in {es}. This tutorial will use the Python {es} client to perform the
+operations.
+
+You'll learn how to:
+
+* create an {infer} endpoint for text embedding using the Cohere service,
+* create the necessary index mapping for the {es} index,
+* build an {infer} pipeline to ingest documents into the index together with the
+embeddings,
+* perform hybrid search on the data,
+* rerank search results by using Cohere's rerank model,
+* design a RAG system with Cohere's Chat API.
+
+The tutorial uses the https://huggingface.co/datasets/mteb/scifact[SciFact] data
+set.
+
+Refer to https://docs.cohere.com/docs/elasticsearch-and-cohere[Cohere's tutorial]
+for an example using a different data set.
+
+
+[discrete]
+[[cohere-es-req]]
+==== Requirements
+
+* A https://cohere.com/[Cohere account],
+* an https://www.elastic.co/guide/en/cloud/current/ec-getting-started.html[Elastic Cloud]
+account,
+* Python 3.7 or higher.
+
+
+[discrete]
+[[cohere-es-packages]]
+==== Istall required packages
+
+Install {es} and Cohere:
+
+[source,py]
+------------------------------------------------------------
+!pip install elasticsearch
+!pip install cohere
+------------------------------------------------------------
+
+Import the required packages:
+
+[source,py]
+------------------------------------------------------------
+from elasticsearch import Elasticsearch, helpers
+import cohere
+import json
+import requests
+------------------------------------------------------------
+
+[discrete]
+[[cohere-es-client]]
+==== Create the {es} client
+
+To create your {es} client, you need:
+* https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#finding-your-cloud-id[your Cloud ID],
+* https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#creating-an-api-key[an encoded API key].
+
+[source,py]
+------------------------------------------------------------
+ELASTICSEARCH_ENDPOINT = "elastic_endpoint"
+ELASTIC_API_KEY = "elastic_api_key"
+
+client = Elasticsearch(
+  cloud_id=ELASTICSEARCH_ENDPOINT,
+  api_key=ELASTIC_API_KEY
+)
+
+# Confirm the client has connected
+print(client.info())
+------------------------------------------------------------
+
+
+[discrete]
+[[cohere-es-infer-endpoint]]
+==== Create the {infer} endpoint
+
+<<put-inference-api,Create the {infer} endpoint>> first. In this example, the
+{infer} endpoint uses Cohere's `embed-english-v3.0` model and the
+`embedding_type` is set to `byte`.
+
+[source,py]
+------------------------------------------------------------
+COHERE_API_KEY = "cohere_api_key"
+
+client.inference.put_model(
+    task_type="text_embedding",
+    inference_id="cohere_embeddings",
+    body={
+        "service": "cohere",
+        "service_settings": {
+            "api_key": COHERE_API_KEY,
+            "model_id": "embed-english-v3.0",
+            "embedding_type": "byte"
+        }
+    },
+)
+------------------------------------------------------------
+
+You can find your API keys in your Cohere dashboard under the
+https://dashboard.cohere.com/api-keys[API keys section].
+
+
+[discrete]
+[[cohere-es-index-mapping]]
+==== Create the index mapping
+
+Create the index mapping for the index that will contain the embeddings.
+
+[source,py]
+------------------------------------------------------------
+client.indices.create(
+    index="cohere-embeddings",
+    settings={"index": {"default_pipeline": "cohere_embeddings"}},
+    mappings={
+        "properties": {
+            "text_embedding": {
+                "type": "dense_vector",
+                "dims": 1024,
+                "element_type": "byte",
+            },
+            "text": {"type": "text"},
+            "id": {"type": "integer"},
+            "title": {"type": "text"}
+        }
+    },
+)
+------------------------------------------------------------
+
+
+[discrete]
+[[cohere-es-infer-pipeline]]
+==== Create the {infer} pipeline
+
+Now you have an {infer} endpoint and an index ready to store embeddings. The
+next step is to create an <<ingest,ingest pipeline>> with an
+<<inference-processor,{infer} processor>> that will create the embeddings using
+the {infer} endpoint and stores them in the index.
+
+[source,py]
+--------------------------------------------------
+client.ingest.put_pipeline(
+    id="cohere_embeddings",
+    description="Ingest pipeline for Cohere inference.",
+    processors=[
+        {
+            "inference": {
+                "model_id": "cohere_embeddings",
+                "input_output": {
+                    "input_field": "text",
+                    "output_field": "text_embedding",
+                },
+            }
+        }
+    ],
+)
+--------------------------------------------------
+
+
+[discrete]
+[[cohere-es-insert-documents]]
+==== Prepare data and insert documents
+
+This example uses the https://huggingface.co/datasets/mteb/scifact[SciFact] data
+set that you can find on HuggingFace.
+
+[source,py]
+--------------------------------------------------
+url = 'https://huggingface.co/datasets/mteb/scifact/raw/main/corpus.jsonl'
+
+# Fetch the JSONL data from the URL
+response = requests.get(url)
+response.raise_for_status()  # Ensure noticing bad responses
+
+# Split the content by new lines and parse each line as JSON
+data = [json.loads(line) for line in response.text.strip().split('\n') if line]
+# Now data is a list of dictionaries
+
+# Change `_id` key to `id` as `_id` is a reserved key in Elasticsearch.
+for item in data:
+    if '_id' in item:
+        item['id'] = item.pop('_id')
+
+# Prepare the documents to be indexed
+documents = []
+for line in data:
+    data_dict = line
+    documents.append({
+        "_index": "cohere-embeddings",
+        "_source": data_dict,
+        }
+      )
+
+# Use the bulk endpoint to index
+helpers.bulk(client, documents)
+
+print("Data ingestion completed, text embeddings generated!")
+--------------------------------------------------
+
+Your index is populated with the SciFact data and text embeddings for the text
+field.
+
+
+[discrete]
+[[cohere-es-hybrid-search]]
+==== Hybrid search
+
+Let's start querying the index!
+
+The code below performs a hybrid search. The `kNN` query computes the relevance
+of search results based on vector similarity using the `text_embedding` field,
+the lexical search query uses BM25 retrieval to compute keyword similarity on
+the `title` and `text` fields.
+
+[source,py]
+--------------------------------------------------
+query = "What is biosimilarity?"
+
+response = client.search(
+    index="cohere-embeddings",
+    size=100,
+    knn={
+        "field": "text_embedding",
+        "query_vector_builder": {
+            "text_embedding": {
+                "model_id": "cohere_embeddings",
+                "model_text": query,
+            }
+        },
+        "k": 10,
+        "num_candidates": 50,
+    },
+    query={
+        "multi_match": {
+            "query": query,
+            "fields": ["text", "title"]
+        }
+    }
+)
+
+raw_documents = response["hits"]["hits"]
+
+# Display the first 10 results
+for document in raw_documents[0:10]:
+  print(f'Title: {document["_source"]["title"]}\nText: {document["_source"]["text"]}\n')
+
+# Format the documents for ranking
+documents = []
+for hit in response["hits"]["hits"]:
+    documents.append(hit["_source"]["text"])
+--------------------------------------------------
+
+
+[discrete]
+[[cohere-es-rerank-results]]
+===== Rerank search results
+
+To combine the results more effectively, use 
+https://docs.cohere.com/docs/rerank-2[Cohere's Rerank v3] model through the
+{infer} API to provide a more precise semantic reranking of the results.
+
+Create an {infer} endpoint with your Cohere API key and the used model name as
+the `model_id` (`rerank-english-v3.0` in this example).
+
+[source,py]
+--------------------------------------------------
+client.inference.put_model(
+    task_type="rerank",
+    inference_id="cohere_rerank",
+    body={
+        "service": "cohere",
+        "service_settings":{
+            "api_key": COHERE_API_KEY,
+            "model_id": "rerank-english-v3.0"
+           },
+        "task_settings": {
+            "top_n": 10,
+        },
+    }
+)
+--------------------------------------------------
+
+Rerank the results using the new {infer} endpoint.
+
+[source,py]
+--------------------------------------------------
+# Pass the query and the search results to the service
+response = client.inference.inference(
+    inference_id="cohere_rerank",
+    body={
+        "query": query,
+        "input": documents,
+        "task_settings": {
+            "return_documents": False
+            }
+        }
+)
+
+# Reconstruct the input documents based on the index provided in the rereank response
+ranked_documents = []
+for document in response.body["rerank"]:
+  ranked_documents.append({
+      "title": raw_documents[int(document["index"])]["_source"]["title"],
+      "text": raw_documents[int(document["index"])]["_source"]["text"]
+  })
+
+# Print the top 10 results
+for document in ranked_documents[0:10]:
+  print(f"Title: {document['title']}\nText: {document['text']}\n")
+--------------------------------------------------
+
+The response is a list of documents in descending order of relevance. Each
+document has a corresponding index that reflects the order of the documents when 
+they were sent to the {infer} endpoint.
+
+
+[discrete]
+[[cohere-es-rag]]
+==== Retrieval Augmented Generation (RAG) with Cohere and {es}
+
+RAG is a method for generating text using additional information fetched from an
+external data source. With the ranked results, you can build a RAG system on the
+top of what you previously created by using 
+https://docs.cohere.com/docs/chat-api[Cohere's Chat API].
+
+Pass in the retrieved documents and the query to receive a grounded response
+using Cohere's newest generative model 
+https://docs.cohere.com/docs/command-r-plus[Command R+].
+
+Then pass in the query and the documents to the Chat API, and print out the
+response. 
+
+[source,py]
+--------------------------------------------------
+response = co.chat(message=query, documents=ranked_documents, model='command-r-plus')
+
+source_documents = []
+for citation in response.citations:
+    for document_id in citation.document_ids:
+        if document_id not in source_documents:
+            source_documents.append(document_id)
+
+print(f"Query: {query}")
+print(f"Response: {response.text}")
+print("Sources:")
+for document in response.documents:
+    if document['id'] in source_documents:
+        print(f"{document['title']}: {document['text']}")
+
+--------------------------------------------------
+
+The response will look similar to this:
+
+[source,consol-result]
+--------------------------------------------------
+Query: What is biosimilarity?
+Response: Biosimilarity is based on the comparability concept, which has been used successfully for several decades to ensure close similarity of a biological product before and after a manufacturing change. Over the last 10 years, experience with biosimilars has shown that even complex biotechnology-derived proteins can be copied successfully.
+Sources:
+Interchangeability of Biosimilars: A European Perspective: (...)
+--------------------------------------------------
+// NOTCONSOLE

docs/reference/search/search-your-data/semantic-search.asciidoc

@@ -136,3 +136,4 @@ include::{es-ref-dir}/tab-widgets/semantic-search/hybrid-search-widget.asciidoc[
 
 include::semantic-search-elser.asciidoc[]
 include::semantic-search-inference.asciidoc[]
+include::cohere-es.asciidoc[]

libs/core/src/main/java/org/elasticsearch/core/RestApiVersion.java

@@ -61,4 +61,15 @@ public enum RestApiVersion {
         };
     }
 
+    public static RestApiVersion forMajor(int major) {
+        switch (major) {
+            case 7 -> {
+                return V_7;
+            }
+            case 8 -> {
+                return V_8;
+            }
+            default -> throw new IllegalArgumentException("Unknown REST API version " + major);
+        }
+    }
 }

libs/native/src/main21/java/org/elasticsearch/nativeaccess/jdk/JdkSystemdLibrary.java

@@ -17,7 +17,10 @@ import java.lang.foreign.FunctionDescriptor;
 import java.lang.foreign.MemorySegment;
 import java.lang.invoke.MethodHandle;
 import java.nio.file.Files;
+import java.nio.file.Path;
 import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.List;
 
 import static java.lang.foreign.ValueLayout.ADDRESS;
 import static java.lang.foreign.ValueLayout.JAVA_INT;
@@ -26,31 +29,49 @@ import static org.elasticsearch.nativeaccess.jdk.LinkerHelper.downcallHandle;
 class JdkSystemdLibrary implements SystemdLibrary {
 
     static {
-        System.load(findLibSystemd());
+        // Find and load libsystemd. We attempt all instances of
+        // libsystemd in case of multiarch systems, and stop when
+        // one is successfully loaded. If none can be loaded,
+        // UnsatisfiedLinkError will be thrown.
+        List<String> paths = findLibSystemd();
+        if (paths.isEmpty()) {
+            String libpath = System.getProperty("java.library.path");
+            throw new UnsatisfiedLinkError("Could not find libsystemd in java.library.path: " + libpath);
+        }
+        UnsatisfiedLinkError last = null;
+        for (String path : paths) {
+            try {
+                System.load(path);
+                last = null;
+                break;
+            } catch (UnsatisfiedLinkError e) {
+                last = e;
+            }
+        }
+        if (last != null) {
+            throw last;
+        }
     }
 
-    // On some systems libsystemd does not have a non-versioned symlink. System.loadLibrary only knows how to find
-    // non-versioned library files. So we must manually check the library path to find what we need.
-    static String findLibSystemd() {
-        final String libsystemd = "libsystemd.so.0";
-        String libpath = System.getProperty("java.library.path");
-        for (String basepathStr : libpath.split(":")) {
-            var basepath = Paths.get(basepathStr);
-            if (Files.exists(basepath) == false) {
-                continue;
-            }
-            try (var stream = Files.walk(basepath)) {
-
-                var foundpath = stream.filter(Files::isDirectory).map(p -> p.resolve(libsystemd)).filter(Files::exists).findAny();
-                if (foundpath.isPresent()) {
-                    return foundpath.get().toAbsolutePath().toString();
-                }
+    // findLibSystemd returns a list of paths to instances of libsystemd
+    // found within java.library.path.
+    static List<String> findLibSystemd() {
+        // Note: on some systems libsystemd does not have a non-versioned symlink.
+        // System.loadLibrary only knows how to find non-versioned library files,
+        // so we must manually check the library path to find what we need.
+        final Path libsystemd = Paths.get("libsystemd.so.0");
+        final String libpath = System.getProperty("java.library.path");
+        return Arrays.stream(libpath.split(":")).map(Paths::get).filter(Files::exists).flatMap(p -> {
+            try {
+                return Files.find(
+                    p,
+                    Integer.MAX_VALUE,
+                    (fp, attrs) -> (attrs.isDirectory() == false && fp.getFileName().equals(libsystemd))
+                );
             } catch (IOException e) {
                 throw new UncheckedIOException(e);
             }
-
-        }
-        throw new UnsatisfiedLinkError("Could not find " + libsystemd + " in java.library.path: " + libpath);
+        }).map(p -> p.toAbsolutePath().toString()).toList();
     }
 
     private static final MethodHandle sd_notify$mh = downcallHandle("sd_notify", FunctionDescriptor.of(JAVA_INT, JAVA_INT, ADDRESS));

qa/no-bootstrap-tests/src/test/java/org/elasticsearch/bootstrap/SpawnerNoBootstrapTests.java

@@ -39,8 +39,10 @@ import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.function.Function;
 
+import static org.hamcrest.Matchers.empty;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.is;
 
 /**
  * Create a simple "daemon controller", put it in the right place and check that it runs.
@@ -64,18 +66,19 @@ public class SpawnerNoBootstrapTests extends LuceneTestCase {
     static {
         // normally done by ESTestCase, but need here because spawner depends on logging
         LogConfigurator.loadLog4jPlugins();
+        MockLogAppender.init();
     }
 
     static class ExpectedStreamMessage implements MockLogAppender.LoggingExpectation {
         final String expectedLogger;
         final String expectedMessage;
-        final CountDownLatch matchCalledLatch;
-        boolean saw;
+        final CountDownLatch matched;
+        volatile boolean saw;
 
-        ExpectedStreamMessage(String logger, String message, CountDownLatch matchCalledLatch) {
+        ExpectedStreamMessage(String logger, String message, CountDownLatch matched) {
             this.expectedLogger = logger;
             this.expectedMessage = message;
-            this.matchCalledLatch = matchCalledLatch;
+            this.matched = matched;
         }
 
         @Override
@@ -84,8 +87,8 @@ public class SpawnerNoBootstrapTests extends LuceneTestCase {
                 && event.getLevel().equals(Level.WARN)
                 && event.getMessage().getFormattedMessage().equals(expectedMessage)) {
                 saw = true;
+                matched.countDown();
             }
-            matchCalledLatch.countDown();
         }
 
         @Override
@@ -129,7 +132,7 @@ public class SpawnerNoBootstrapTests extends LuceneTestCase {
 
         try (Spawner spawner = new Spawner()) {
             spawner.spawnNativeControllers(environment);
-            assertThat(spawner.getProcesses(), hasSize(0));
+            assertThat(spawner.getProcesses(), is(empty()));
         }
     }
 
@@ -228,7 +231,7 @@ public class SpawnerNoBootstrapTests extends LuceneTestCase {
                 // fail if the process does not die within one second; usually it will be even quicker but it depends on OS scheduling
                 assertTrue(process.waitFor(1, TimeUnit.SECONDS));
             } else {
-                assertThat(processes, hasSize(0));
+                assertThat(processes, is(empty()));
             }
             appender.assertAllExpectationsMatched();
         }

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/AbstractRollingUpgradeTestCase.java

@@ -0,0 +1,54 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.upgrades;
+
+import com.carrotsearch.randomizedtesting.annotations.Name;
+
+import org.elasticsearch.core.SuppressForbidden;
+import org.elasticsearch.test.cluster.ElasticsearchCluster;
+import org.elasticsearch.test.cluster.FeatureFlag;
+import org.elasticsearch.test.cluster.local.distribution.DistributionType;
+import org.junit.ClassRule;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestRule;
+
+import java.util.function.Supplier;
+
+public abstract class AbstractRollingUpgradeTestCase extends ParameterizedRollingUpgradeTestCase {
+
+    private static final TemporaryFolder repoDirectory = new TemporaryFolder();
+
+    private static final ElasticsearchCluster cluster = ElasticsearchCluster.local()
+        .distribution(DistributionType.DEFAULT)
+        .version(getOldClusterTestVersion())
+        .nodes(NODE_NUM)
+        .setting("path.repo", new Supplier<>() {
+            @Override
+            @SuppressForbidden(reason = "TemporaryFolder only has io.File methods, not nio.File")
+            public String get() {
+                return repoDirectory.getRoot().getPath();
+            }
+        })
+        .setting("xpack.security.enabled", "false")
+        .feature(FeatureFlag.TIME_SERIES_MODE)
+        .build();
+
+    @ClassRule
+    public static TestRule ruleChain = RuleChain.outerRule(repoDirectory).around(cluster);
+
+    protected AbstractRollingUpgradeTestCase(@Name("upgradedNodes") int upgradedNodes) {
+        super(upgradedNodes);
+    }
+
+    @Override
+    protected ElasticsearchCluster getUpgradeCluster() {
+        return cluster;
+    }
+}

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/ClusterFeatureMigrationIT.java

@@ -24,7 +24,7 @@ import java.util.stream.Collectors;
 import static org.hamcrest.Matchers.empty;
 import static org.hamcrest.Matchers.hasSize;
 
-public class ClusterFeatureMigrationIT extends ParameterizedRollingUpgradeTestCase {
+public class ClusterFeatureMigrationIT extends AbstractRollingUpgradeTestCase {
 
     @Before
     public void checkMigrationVersion() {

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/DesiredNodesUpgradeIT.java

@@ -33,7 +33,7 @@ import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.is;
 
-public class DesiredNodesUpgradeIT extends ParameterizedRollingUpgradeTestCase {
+public class DesiredNodesUpgradeIT extends AbstractRollingUpgradeTestCase {
 
     private final int desiredNodesVersion;
 

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/DownsampleIT.java

@@ -25,7 +25,7 @@ import java.util.concurrent.TimeUnit;
 
 import static org.hamcrest.Matchers.equalTo;
 
-public class DownsampleIT extends ParameterizedRollingUpgradeTestCase {
+public class DownsampleIT extends AbstractRollingUpgradeTestCase {
 
     private static final String FIXED_INTERVAL = "1h";
     private String index;

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/FeatureUpgradeIT.java

@@ -23,7 +23,7 @@ import static org.hamcrest.Matchers.aMapWithSize;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
 
-public class FeatureUpgradeIT extends ParameterizedRollingUpgradeTestCase {
+public class FeatureUpgradeIT extends AbstractRollingUpgradeTestCase {
 
     public FeatureUpgradeIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/FieldCapsIT.java

@@ -40,7 +40,7 @@ import static org.hamcrest.Matchers.equalTo;
  * the co-ordinating node if older nodes were included in the system
  */
 @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/103473")
-public class FieldCapsIT extends ParameterizedRollingUpgradeTestCase {
+public class FieldCapsIT extends AbstractRollingUpgradeTestCase {
 
     public FieldCapsIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/HealthNodeUpgradeIT.java

@@ -20,7 +20,7 @@ import java.util.Map;
 
 import static org.hamcrest.CoreMatchers.equalTo;
 
-public class HealthNodeUpgradeIT extends ParameterizedRollingUpgradeTestCase {
+public class HealthNodeUpgradeIT extends AbstractRollingUpgradeTestCase {
 
     public HealthNodeUpgradeIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/IgnoredMetaFieldRollingUpgradeIT.java

@@ -26,7 +26,7 @@ import java.util.List;
 import java.util.Locale;
 import java.util.Map;
 
-public class IgnoredMetaFieldRollingUpgradeIT extends ParameterizedRollingUpgradeTestCase {
+public class IgnoredMetaFieldRollingUpgradeIT extends AbstractRollingUpgradeTestCase {
 
     private static final String TERMS_AGG_QUERY = Strings.format("""
         {

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/IndexingIT.java

@@ -51,7 +51,7 @@ import static org.hamcrest.Matchers.equalTo;
  * xpack rolling restart tests. We should work on a way to remove this
  * duplication but for now we have no real way to share code.
  */
-public class IndexingIT extends ParameterizedRollingUpgradeTestCase {
+public class IndexingIT extends AbstractRollingUpgradeTestCase {
 
     public IndexingIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/ParameterizedRollingUpgradeTestCase.java

@@ -14,73 +14,44 @@ import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
 import org.elasticsearch.client.Request;
 import org.elasticsearch.client.Response;
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.core.SuppressForbidden;
 import org.elasticsearch.features.NodeFeature;
 import org.elasticsearch.index.IndexVersion;
 import org.elasticsearch.index.IndexVersions;
 import org.elasticsearch.test.cluster.ElasticsearchCluster;
-import org.elasticsearch.test.cluster.FeatureFlag;
-import org.elasticsearch.test.cluster.local.distribution.DistributionType;
 import org.elasticsearch.test.cluster.util.Version;
 import org.elasticsearch.test.rest.ESRestTestCase;
 import org.elasticsearch.test.rest.ObjectPath;
 import org.elasticsearch.test.rest.TestFeatureService;
 import org.junit.AfterClass;
 import org.junit.Before;
-import org.junit.ClassRule;
-import org.junit.rules.RuleChain;
-import org.junit.rules.TemporaryFolder;
-import org.junit.rules.TestRule;
 
 import java.util.HashSet;
 import java.util.Map;
 import java.util.Set;
-import java.util.function.Supplier;
 import java.util.stream.IntStream;
 
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.notNullValue;
 
 public abstract class ParameterizedRollingUpgradeTestCase extends ESRestTestCase {
+    protected static final int NODE_NUM = 3;
     private static final String OLD_CLUSTER_VERSION = System.getProperty("tests.old_cluster_version");
+    private static final Set<Integer> upgradedNodes = new HashSet<>();
+    private static TestFeatureService oldClusterTestFeatureService = null;
+    private static boolean upgradeFailed = false;
+    private static IndexVersion oldIndexVersion;
+    private final int requestedUpgradedNodes;
 
-    private static final TemporaryFolder repoDirectory = new TemporaryFolder();
-
-    private static final int NODE_NUM = 3;
-
-    private static final ElasticsearchCluster cluster = ElasticsearchCluster.local()
-        .distribution(DistributionType.DEFAULT)
-        .version(getOldClusterTestVersion())
-        .nodes(NODE_NUM)
-        .setting("path.repo", new Supplier<>() {
-            @Override
-            @SuppressForbidden(reason = "TemporaryFolder only has io.File methods, not nio.File")
-            public String get() {
-                return repoDirectory.getRoot().getPath();
-            }
-        })
-        .setting("xpack.security.enabled", "false")
-        .feature(FeatureFlag.TIME_SERIES_MODE)
-        .build();
-
-    @ClassRule
-    public static TestRule ruleChain = RuleChain.outerRule(repoDirectory).around(cluster);
+    protected ParameterizedRollingUpgradeTestCase(@Name("upgradedNodes") int upgradedNodes) {
+        this.requestedUpgradedNodes = upgradedNodes;
+    }
 
     @ParametersFactory(shuffle = false)
     public static Iterable<Object[]> parameters() {
         return IntStream.rangeClosed(0, NODE_NUM).boxed().map(n -> new Object[] { n }).toList();
     }
 
-    private static final Set<Integer> upgradedNodes = new HashSet<>();
-    private static TestFeatureService oldClusterTestFeatureService = null;
-    private static boolean upgradeFailed = false;
-    private static IndexVersion oldIndexVersion;
-
-    private final int requestedUpgradedNodes;
-
-    protected ParameterizedRollingUpgradeTestCase(@Name("upgradedNodes") int upgradedNodes) {
-        this.requestedUpgradedNodes = upgradedNodes;
-    }
+    protected abstract ElasticsearchCluster getUpgradeCluster();
 
     @Before
     public void extractOldClusterFeatures() {
@@ -135,7 +106,7 @@ public abstract class ParameterizedRollingUpgradeTestCase extends ESRestTestCase
                 if (upgradedNodes.add(n)) {
                     try {
                         logger.info("Upgrading node {} to version {}", n, Version.CURRENT);
-                        cluster.upgradeNodeToVersion(n, Version.CURRENT);
+                        getUpgradeCluster().upgradeNodeToVersion(n, Version.CURRENT);
                     } catch (Exception e) {
                         upgradeFailed = true;
                         throw e;
@@ -199,7 +170,7 @@ public abstract class ParameterizedRollingUpgradeTestCase extends ESRestTestCase
 
     @Override
     protected String getTestRestCluster() {
-        return cluster.getHttpAddresses();
+        return getUpgradeCluster().getHttpAddresses();
     }
 
     @Override

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/SnapshotBasedRecoveryIT.java

@@ -42,7 +42,7 @@ import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.lessThan;
 import static org.hamcrest.Matchers.notNullValue;
 
-public class SnapshotBasedRecoveryIT extends ParameterizedRollingUpgradeTestCase {
+public class SnapshotBasedRecoveryIT extends AbstractRollingUpgradeTestCase {
 
     public SnapshotBasedRecoveryIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/SystemIndicesUpgradeIT.java

@@ -23,7 +23,7 @@ import static org.hamcrest.Matchers.hasKey;
 import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.notNullValue;
 
-public class SystemIndicesUpgradeIT extends ParameterizedRollingUpgradeTestCase {
+public class SystemIndicesUpgradeIT extends AbstractRollingUpgradeTestCase {
 
     public SystemIndicesUpgradeIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/TsdbIT.java

@@ -26,7 +26,7 @@ import static org.hamcrest.Matchers.hasSize;
 import static org.hamcrest.Matchers.notNullValue;
 import static org.hamcrest.Matchers.nullValue;
 
-public class TsdbIT extends ParameterizedRollingUpgradeTestCase {
+public class TsdbIT extends AbstractRollingUpgradeTestCase {
 
     public TsdbIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/UpgradeWithOldIndexSettingsIT.java

@@ -24,7 +24,7 @@ import java.util.Map;
 import static org.elasticsearch.rest.action.search.RestSearchAction.TOTAL_HITS_AS_INT_PARAM;
 import static org.hamcrest.Matchers.is;
 
-public class UpgradeWithOldIndexSettingsIT extends ParameterizedRollingUpgradeTestCase {
+public class UpgradeWithOldIndexSettingsIT extends AbstractRollingUpgradeTestCase {
 
     public UpgradeWithOldIndexSettingsIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/VectorSearchIT.java

@@ -22,7 +22,7 @@ import java.util.Map;
 import static org.hamcrest.Matchers.closeTo;
 import static org.hamcrest.Matchers.equalTo;
 
-public class VectorSearchIT extends ParameterizedRollingUpgradeTestCase {
+public class VectorSearchIT extends AbstractRollingUpgradeTestCase {
     public VectorSearchIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);
     }

qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/XPackIT.java

@@ -22,7 +22,7 @@ import static org.junit.Assume.assumeThat;
  * Basic tests for simple xpack functionality that are only run if the
  * cluster is the on the default distribution.
  */
-public class XPackIT extends ParameterizedRollingUpgradeTestCase {
+public class XPackIT extends AbstractRollingUpgradeTestCase {
 
     public XPackIT(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

server/src/internalClusterTest/java/org/elasticsearch/nodescapabilities/SimpleNodesCapabilitiesIT.java

@@ -0,0 +1,55 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.nodescapabilities;
+
+import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
+import org.elasticsearch.action.admin.cluster.node.capabilities.NodesCapabilitiesRequest;
+import org.elasticsearch.action.admin.cluster.node.capabilities.NodesCapabilitiesResponse;
+import org.elasticsearch.test.ESIntegTestCase;
+
+import java.io.IOException;
+
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.is;
+
+@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 0)
+public class SimpleNodesCapabilitiesIT extends ESIntegTestCase {
+
+    public void testNodesCapabilities() throws IOException {
+        internalCluster().startNodes(2);
+
+        ClusterHealthResponse clusterHealth = clusterAdmin().prepareHealth().setWaitForGreenStatus().setWaitForNodes("2").get();
+        logger.info("--> done cluster_health, status {}", clusterHealth.getStatus());
+
+        // check we support the capabilities API itself. Which we do.
+        NodesCapabilitiesResponse response = clusterAdmin().nodesCapabilities(new NodesCapabilitiesRequest().path("_capabilities"))
+            .actionGet();
+        assertThat(response.getNodes(), hasSize(2));
+        assertThat(response.isSupported(), is(true));
+
+        // check we support some parameters of the capabilities API
+        response = clusterAdmin().nodesCapabilities(new NodesCapabilitiesRequest().path("_capabilities").parameters("method", "path"))
+            .actionGet();
+        assertThat(response.getNodes(), hasSize(2));
+        assertThat(response.isSupported(), is(true));
+
+        // check we don't support some other parameters of the capabilities API
+        response = clusterAdmin().nodesCapabilities(new NodesCapabilitiesRequest().path("_capabilities").parameters("method", "invalid"))
+            .actionGet();
+        assertThat(response.getNodes(), hasSize(2));
+        assertThat(response.isSupported(), is(false));
+
+        // check we don't support a random invalid api
+        // TODO this is not working yet - see https://github.com/elastic/elasticsearch/issues/107425
+        /*response = clusterAdmin().nodesCapabilities(new NodesCapabilitiesRequest().path("_invalid"))
+            .actionGet();
+        assertThat(response.getNodes(), hasSize(2));
+        assertThat(response.isSupported(), is(false));*/
+    }
+}

server/src/main/java/module-info.java

@@ -65,6 +65,7 @@ module org.elasticsearch.server {
     exports org.elasticsearch.action.admin.cluster.desirednodes;
     exports org.elasticsearch.action.admin.cluster.health;
     exports org.elasticsearch.action.admin.cluster.migration;
+    exports org.elasticsearch.action.admin.cluster.node.capabilities;
     exports org.elasticsearch.action.admin.cluster.node.hotthreads;
     exports org.elasticsearch.action.admin.cluster.node.info;
     exports org.elasticsearch.action.admin.cluster.node.reload;

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -195,6 +195,7 @@ public class TransportVersions {
     public static final TransportVersion INDEXING_PRESSURE_REQUEST_REJECTIONS_COUNT = def(8_652_00_0);
     public static final TransportVersion ROLLUP_USAGE = def(8_653_00_0);
     public static final TransportVersion SECURITY_ROLE_DESCRIPTION = def(8_654_00_0);
+    public static final TransportVersion ML_INFERENCE_AZURE_OPENAI_COMPLETIONS = def(8_655_00_0);
 
     /*
      * STOP! READ THIS FIRST! No, really,

server/src/main/java/org/elasticsearch/action/ActionListener.java

@@ -31,17 +31,94 @@ import static org.elasticsearch.action.ActionListenerImplementations.safeAcceptE
 import static org.elasticsearch.action.ActionListenerImplementations.safeOnFailure;
 
 /**
- * A listener for action responses or failures.
+ * <p>
+ * Callbacks are used extensively throughout Elasticsearch because they enable us to write asynchronous and nonblocking code, i.e. code
+ * which doesn't necessarily compute a result straight away but also doesn't block the calling thread waiting for the result to become
+ * available. They support several useful control flows:
+ * </p>
+ * <ul>
+ * <li>They can be completed immediately on the calling thread.</li>
+ * <li>They can be completed concurrently on a different thread.</li>
+ * <li>They can be stored in a data structure and completed later on when the system reaches a particular state.</li>
+ * <li>Most commonly, they can be passed on to other methods that themselves require a callback.</li>
+ * <li>They can be wrapped in another callback which modifies the behaviour of the original callback, perhaps adding some extra code to run
+ * before or after completion, before passing them on.</li>
+ * </ul>
+ * <p>
+ * {@link ActionListener} is a general-purpose callback interface that is used extensively across the Elasticsearch codebase. {@link
+ * ActionListener} is used pretty much everywhere that needs to perform some asynchronous and nonblocking computation. The uniformity makes
+ * it easier to compose parts of the system together without needing to build adapters to convert back and forth between different kinds of
+ * callback. It also makes it easier to develop the skills needed to read and understand all the asynchronous code, although this definitely
+ * takes practice and is certainly not easy in an absolute sense. Finally, it has allowed us to build a rich library for working with {@link
+ * ActionListener} instances themselves, creating new instances out of existing ones and completing them in interesting ways. See for
+ * instance:
+ * </p>
+ * <ul>
+ * <li>All the static methods on {@link ActionListener} itself.</li>
+ * <li>{@link org.elasticsearch.action.support.ThreadedActionListener} for forking work elsewhere.</li>
+ * <li>{@link org.elasticsearch.action.support.RefCountingListener} for running work in parallel.</li>
+ * <li>{@link org.elasticsearch.action.support.SubscribableListener} for constructing flexible workflows.</li>
+ * </ul>
+ * <p>
+ * Callback-based asynchronous code can easily call regular synchronous code, but synchronous code cannot run callback-based asynchronous
+ * code without blocking the calling thread until the callback is called back. This blocking is at best undesirable (threads are too
+ * expensive to waste with unnecessary blocking) and at worst outright broken (the blocking can lead to deadlock). Unfortunately this means
+ * that most of our code ends up having to be written with callbacks, simply because it's ultimately calling into some other code that takes
+ * a callback. The entry points for all Elasticsearch APIs are callback-based (e.g. REST APIs all start at {@link
+ * org.elasticsearch.rest.BaseRestHandler}{@code #prepareRequest} and transport APIs all start at {@link
+ * org.elasticsearch.action.support.TransportAction}{@code #doExecute} and the whole system fundamentally works in terms of an event loop
+ * (an {@code io.netty.channel.EventLoop}) which processes network events via callbacks.
+ * </p>
+ * <p>
+ * {@link ActionListener} is not an <i>ad-hoc</i> invention. Formally speaking, it is our implementation of the general concept of a
+ * continuation in the sense of <a href="https://en.wikipedia.org/wiki/Continuation-passing_style"><i>continuation-passing style</i></a>
+ * (CPS): an extra argument to a function which defines how to continue the computation when the result is available. This is in contrast to
+ * <i>direct style</i> which is the more usual style of calling methods that return values directly back to the caller so they can continue
+ * executing as normal. There's essentially two ways that computation can continue in Java (it can return a value or it can throw an
+ * exception) which is why {@link ActionListener} has both an {@link #onResponse} and an {@link #onFailure} method.
+ * </p>
+ * <p>
+ * CPS is strictly more expressive than direct style: direct code can be mechanically translated into continuation-passing style, but CPS
+ * also enables all sorts of other useful control structures such as forking work onto separate threads, possibly to be executed in
+ * parallel, perhaps even across multiple nodes, or possibly collecting a list of continuations all waiting for the same condition to be
+ * satisfied before proceeding (e.g. {@link org.elasticsearch.action.support.SubscribableListener} amongst many others). Some languages have
+ * first-class support for continuations (e.g. the {@code async} and {@code await} primitives in C#) allowing the programmer to write code
+ * in direct style away from those exotic control structures, but Java does not. That's why we have to manipulate all the callbacks
+ * ourselves.
+ * </p>
+ * <p>
+ * Strictly speaking, CPS requires that a computation <i>only</i> continues by calling the continuation. In Elasticsearch, this means that
+ * asynchronous methods must have {@code void} return type and may not throw any exceptions. This is mostly the case in our code as written
+ * today, and is a good guiding principle, but we don't enforce void exceptionless methods and there are some deviations from this rule. In
+ * particular, it's not uncommon to permit some methods to throw an exception, using things like {@link ActionListener#run} (or an
+ * equivalent {@code try ... catch ...} block) further up the stack to handle it. Some methods also take (and may complete) an {@link
+ * ActionListener} parameter, but still return a value separately for other local synchronous work.
+ * </p>
+ * <p>
+ * This pattern is often used in the transport action layer with the use of the {@link
+ * org.elasticsearch.action.support.ChannelActionListener} class, which wraps a {@link org.elasticsearch.transport.TransportChannel}
+ * produced by the transport layer.{@link org.elasticsearch.transport.TransportChannel} implementations can hold a reference to a Netty
+ * channel with which to pass the response back to the network caller. Netty has a many-to-one association of network callers to channels,
+ * so a call taking a long time generally won't hog resources: it's cheap. A transport action can take hours to respond and that's alright,
+ * barring caller timeouts.
+ * </p>
+ * <p>
+ * Note that we explicitly avoid {@link java.util.concurrent.CompletableFuture} and other similar mechanisms as much as possible. They
+ * can achieve the same goals as {@link ActionListener}, but can also easily be misused in various ways that lead to severe bugs. In
+ * particular, futures support blocking while waiting for a result, but this is almost never appropriate in Elasticsearch's production code
+ * where threads are such a precious resource. Moreover if something throws an {@link Error} then the JVM should exit pretty much straight
+ * away, but {@link java.util.concurrent.CompletableFuture} can catch an {@link Error} which delays the JVM exit until its result is
+ * observed. This may be much later, or possibly even never. It's not possible to introduce such bugs when using {@link ActionListener}.
+ * </p>
  */
 public interface ActionListener<Response> {
     /**
-     * Handle action response. This response may constitute a failure or a
-     * success but it is up to the listener to make that decision.
+     * Complete this listener with a successful (or at least, non-exceptional) response.
      */
     void onResponse(Response response);
 
     /**
-     * A failure caused by an exception at some phase of the task.
+     * Complete this listener with an exceptional response.
      */
     void onFailure(Exception e);
 

server/src/main/java/org/elasticsearch/action/ActionModule.java

@@ -29,6 +29,7 @@ import org.elasticsearch.action.admin.cluster.migration.GetFeatureUpgradeStatusA
 import org.elasticsearch.action.admin.cluster.migration.PostFeatureUpgradeAction;
 import org.elasticsearch.action.admin.cluster.migration.TransportGetFeatureUpgradeStatusAction;
 import org.elasticsearch.action.admin.cluster.migration.TransportPostFeatureUpgradeAction;
+import org.elasticsearch.action.admin.cluster.node.capabilities.TransportNodesCapabilitiesAction;
 import org.elasticsearch.action.admin.cluster.node.hotthreads.TransportNodesHotThreadsAction;
 import org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction;
 import org.elasticsearch.action.admin.cluster.node.reload.TransportNodesReloadSecureSettingsAction;
@@ -284,6 +285,7 @@ import org.elasticsearch.rest.action.admin.cluster.RestGetSnapshotsAction;
 import org.elasticsearch.rest.action.admin.cluster.RestGetStoredScriptAction;
 import org.elasticsearch.rest.action.admin.cluster.RestGetTaskAction;
 import org.elasticsearch.rest.action.admin.cluster.RestListTasksAction;
+import org.elasticsearch.rest.action.admin.cluster.RestNodesCapabilitiesAction;
 import org.elasticsearch.rest.action.admin.cluster.RestNodesHotThreadsAction;
 import org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction;
 import org.elasticsearch.rest.action.admin.cluster.RestNodesStatsAction;
@@ -616,6 +618,7 @@ public class ActionModule extends AbstractModule {
 
         actions.register(TransportNodesInfoAction.TYPE, TransportNodesInfoAction.class);
         actions.register(TransportRemoteInfoAction.TYPE, TransportRemoteInfoAction.class);
+        actions.register(TransportNodesCapabilitiesAction.TYPE, TransportNodesCapabilitiesAction.class);
         actions.register(RemoteClusterNodesAction.TYPE, RemoteClusterNodesAction.TransportAction.class);
         actions.register(TransportNodesStatsAction.TYPE, TransportNodesStatsAction.class);
         actions.register(TransportNodesUsageAction.TYPE, TransportNodesUsageAction.class);
@@ -833,6 +836,7 @@ public class ActionModule extends AbstractModule {
         registerHandler.accept(new RestClearVotingConfigExclusionsAction());
         registerHandler.accept(new RestNodesInfoAction(settingsFilter));
         registerHandler.accept(new RestRemoteClusterInfoAction());
+        registerHandler.accept(new RestNodesCapabilitiesAction());
         registerHandler.accept(new RestNodesStatsAction());
         registerHandler.accept(new RestNodesUsageAction());
         registerHandler.accept(new RestNodesHotThreadsAction());
@@ -1029,6 +1033,7 @@ public class ActionModule extends AbstractModule {
 
     @Override
     protected void configure() {
+        bind(RestController.class).toInstance(restController);
         bind(ActionFilters.class).toInstance(actionFilters);
         bind(DestructiveOperations.class).toInstance(destructiveOperations);
         bind(new TypeLiteral<RequestValidators<PutMappingRequest>>() {

server/src/main/java/org/elasticsearch/action/admin/cluster/node/capabilities/NodeCapability.java

@@ -0,0 +1,43 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.action.admin.cluster.node.capabilities;
+
+import org.elasticsearch.action.support.nodes.BaseNodeResponse;
+import org.elasticsearch.cluster.node.DiscoveryNode;
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+
+import java.io.IOException;
+
+public class NodeCapability extends BaseNodeResponse {
+
+    private final boolean supported;
+
+    public NodeCapability(StreamInput in) throws IOException {
+        super(in);
+
+        supported = in.readBoolean();
+    }
+
+    public NodeCapability(boolean supported, DiscoveryNode node) {
+        super(node);
+        this.supported = supported;
+    }
+
+    public boolean isSupported() {
+        return supported;
+    }
+
+    @Override
+    public void writeTo(StreamOutput out) throws IOException {
+        super.writeTo(out);
+
+        out.writeBoolean(supported);
+    }
+}

server/src/main/java/org/elasticsearch/action/admin/cluster/node/capabilities/NodesCapabilitiesRequest.java

@@ -0,0 +1,75 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.action.admin.cluster.node.capabilities;
+
+import org.elasticsearch.action.support.nodes.BaseNodesRequest;
+import org.elasticsearch.common.Strings;
+import org.elasticsearch.core.RestApiVersion;
+import org.elasticsearch.rest.RestRequest;
+
+import java.util.Set;
+
+public class NodesCapabilitiesRequest extends BaseNodesRequest<NodesCapabilitiesRequest> {
+
+    private RestRequest.Method method = RestRequest.Method.GET;
+    private String path = "/";
+    private Set<String> parameters = Set.of();
+    private Set<String> capabilities = Set.of();
+    private RestApiVersion restApiVersion = RestApiVersion.current();
+
+    public NodesCapabilitiesRequest() {
+        // always send to all nodes
+        super(Strings.EMPTY_ARRAY);
+    }
+
+    public NodesCapabilitiesRequest path(String path) {
+        this.path = path;
+        return this;
+    }
+
+    public String path() {
+        return path;
+    }
+
+    public NodesCapabilitiesRequest method(RestRequest.Method method) {
+        this.method = method;
+        return this;
+    }
+
+    public RestRequest.Method method() {
+        return method;
+    }
+
+    public NodesCapabilitiesRequest parameters(String... parameters) {
+        this.parameters = Set.of(parameters);
+        return this;
+    }
+
+    public Set<String> parameters() {
+        return parameters;
+    }
+
+    public NodesCapabilitiesRequest capabilities(String... capabilities) {
+        this.capabilities = Set.of(capabilities);
+        return this;
+    }
+
+    public Set<String> capabilities() {
+        return capabilities;
+    }
+
+    public NodesCapabilitiesRequest restApiVersion(RestApiVersion restApiVersion) {
+        this.restApiVersion = restApiVersion;
+        return this;
+    }
+
+    public RestApiVersion restApiVersion() {
+        return restApiVersion;
+    }
+}

server/src/main/java/org/elasticsearch/action/admin/cluster/node/capabilities/NodesCapabilitiesResponse.java

@@ -0,0 +1,46 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.action.admin.cluster.node.capabilities;
+
+import org.elasticsearch.action.FailedNodeException;
+import org.elasticsearch.action.support.TransportAction;
+import org.elasticsearch.action.support.nodes.BaseNodesResponse;
+import org.elasticsearch.cluster.ClusterName;
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.xcontent.ToXContentFragment;
+import org.elasticsearch.xcontent.XContentBuilder;
+
+import java.io.IOException;
+import java.util.List;
+
+public class NodesCapabilitiesResponse extends BaseNodesResponse<NodeCapability> implements ToXContentFragment {
+    protected NodesCapabilitiesResponse(ClusterName clusterName, List<NodeCapability> nodes, List<FailedNodeException> failures) {
+        super(clusterName, nodes, failures);
+    }
+
+    @Override
+    protected List<NodeCapability> readNodesFrom(StreamInput in) throws IOException {
+        return TransportAction.localOnly();
+    }
+
+    @Override
+    protected void writeNodesTo(StreamOutput out, List<NodeCapability> nodes) throws IOException {
+        TransportAction.localOnly();
+    }
+
+    public boolean isSupported() {
+        return getNodes().isEmpty() == false && getNodes().stream().allMatch(NodeCapability::isSupported);
+    }
+
+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        return builder.field("supported", isSupported());
+    }
+}

server/src/main/java/org/elasticsearch/action/admin/cluster/node/capabilities/TransportNodesCapabilitiesAction.java

@@ -0,0 +1,140 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.action.admin.cluster.node.capabilities;
+
+import org.elasticsearch.action.ActionType;
+import org.elasticsearch.action.FailedNodeException;
+import org.elasticsearch.action.support.ActionFilters;
+import org.elasticsearch.action.support.nodes.TransportNodesAction;
+import org.elasticsearch.cluster.node.DiscoveryNode;
+import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.core.RestApiVersion;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.tasks.Task;
+import org.elasticsearch.threadpool.ThreadPool;
+import org.elasticsearch.transport.TransportRequest;
+import org.elasticsearch.transport.TransportService;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+public class TransportNodesCapabilitiesAction extends TransportNodesAction<
+    NodesCapabilitiesRequest,
+    NodesCapabilitiesResponse,
+    TransportNodesCapabilitiesAction.NodeCapabilitiesRequest,
+    NodeCapability> {
+
+    public static final ActionType<NodesCapabilitiesResponse> TYPE = new ActionType<>("cluster:monitor/nodes/capabilities");
+
+    private final RestController restController;
+
+    @Inject
+    public TransportNodesCapabilitiesAction(
+        ThreadPool threadPool,
+        ClusterService clusterService,
+        TransportService transportService,
+        ActionFilters actionFilters,
+        RestController restController
+    ) {
+        super(
+            TYPE.name(),
+            clusterService,
+            transportService,
+            actionFilters,
+            NodeCapabilitiesRequest::new,
+            threadPool.executor(ThreadPool.Names.MANAGEMENT)
+        );
+        this.restController = restController;
+    }
+
+    @Override
+    protected NodesCapabilitiesResponse newResponse(
+        NodesCapabilitiesRequest request,
+        List<NodeCapability> responses,
+        List<FailedNodeException> failures
+    ) {
+        return new NodesCapabilitiesResponse(clusterService.getClusterName(), responses, failures);
+    }
+
+    @Override
+    protected NodeCapabilitiesRequest newNodeRequest(NodesCapabilitiesRequest request) {
+        return new NodeCapabilitiesRequest(
+            request.method(),
+            request.path(),
+            request.parameters(),
+            request.capabilities(),
+            request.restApiVersion()
+        );
+    }
+
+    @Override
+    protected NodeCapability newNodeResponse(StreamInput in, DiscoveryNode node) throws IOException {
+        return new NodeCapability(in);
+    }
+
+    @Override
+    protected NodeCapability nodeOperation(NodeCapabilitiesRequest request, Task task) {
+        boolean supported = restController.checkSupported(
+            request.method,
+            request.path,
+            request.parameters,
+            request.capabilities,
+            request.restApiVersion
+        );
+        return new NodeCapability(supported, transportService.getLocalNode());
+    }
+
+    public static class NodeCapabilitiesRequest extends TransportRequest {
+        private final RestRequest.Method method;
+        private final String path;
+        private final Set<String> parameters;
+        private final Set<String> capabilities;
+        private final RestApiVersion restApiVersion;
+
+        public NodeCapabilitiesRequest(StreamInput in) throws IOException {
+            super(in);
+
+            method = in.readEnum(RestRequest.Method.class);
+            path = in.readString();
+            parameters = in.readCollectionAsImmutableSet(StreamInput::readString);
+            capabilities = in.readCollectionAsImmutableSet(StreamInput::readString);
+            restApiVersion = RestApiVersion.forMajor(in.readVInt());
+        }
+
+        public NodeCapabilitiesRequest(
+            RestRequest.Method method,
+            String path,
+            Set<String> parameters,
+            Set<String> capabilities,
+            RestApiVersion restApiVersion
+        ) {
+            this.method = method;
+            this.path = path;
+            this.parameters = Set.copyOf(parameters);
+            this.capabilities = Set.copyOf(capabilities);
+            this.restApiVersion = restApiVersion;
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            super.writeTo(out);
+
+            out.writeEnum(method);
+            out.writeString(path);
+            out.writeCollection(parameters, StreamOutput::writeString);
+            out.writeCollection(capabilities, StreamOutput::writeString);
+            out.writeVInt(restApiVersion.major);
+        }
+    }
+}

server/src/main/java/org/elasticsearch/client/internal/ClusterAdminClient.java

@@ -21,6 +21,9 @@ import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder;
 import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
 import org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction;
+import org.elasticsearch.action.admin.cluster.node.capabilities.NodesCapabilitiesRequest;
+import org.elasticsearch.action.admin.cluster.node.capabilities.NodesCapabilitiesResponse;
+import org.elasticsearch.action.admin.cluster.node.capabilities.TransportNodesCapabilitiesAction;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequest;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoRequestBuilder;
 import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
@@ -248,6 +251,14 @@ public class ClusterAdminClient implements ElasticsearchClient {
         return new NodesStatsRequestBuilder(this).setNodesIds(nodesIds);
     }
 
+    public ActionFuture<NodesCapabilitiesResponse> nodesCapabilities(final NodesCapabilitiesRequest request) {
+        return execute(TransportNodesCapabilitiesAction.TYPE, request);
+    }
+
+    public void nodesCapabilities(final NodesCapabilitiesRequest request, final ActionListener<NodesCapabilitiesResponse> listener) {
+        execute(TransportNodesCapabilitiesAction.TYPE, request, listener);
+    }
+
     public void nodesUsage(final NodesUsageRequest request, final ActionListener<NodesUsageResponse> listener) {
         execute(TransportNodesUsageAction.TYPE, request, listener);
     }

server/src/main/java/org/elasticsearch/rest/BaseRestHandler.java

@@ -12,6 +12,7 @@ import org.apache.lucene.search.spell.LevenshteinDistance;
 import org.elasticsearch.client.internal.node.NodeClient;
 import org.elasticsearch.common.settings.Setting;
 import org.elasticsearch.common.settings.Setting.Property;
+import org.elasticsearch.common.util.set.Sets;
 import org.elasticsearch.core.CheckedConsumer;
 import org.elasticsearch.core.RefCounted;
 import org.elasticsearch.core.Releasable;
@@ -77,6 +78,13 @@ public abstract class BaseRestHandler implements RestHandler {
 
     @Override
     public final void handleRequest(RestRequest request, RestChannel channel, NodeClient client) throws Exception {
+        // check if the query has any parameters that are not in the supported set (if declared)
+        Set<String> supported = supportedQueryParameters();
+        if (supported != null && supported.containsAll(request.params().keySet()) == false) {
+            Set<String> unsupported = Sets.difference(request.params().keySet(), supported);
+            throw new IllegalArgumentException(unrecognized(request, unsupported, supported, "parameter"));
+        }
+
         // prepare the request for execution; has the side effect of touching the request parameters
         try (var action = prepareRequest(request, client)) {
 

server/src/main/java/org/elasticsearch/rest/RestController.java

@@ -365,6 +365,32 @@ public class RestController implements HttpServerTransport.Dispatcher {
         }
     }
 
+    public boolean checkSupported(
+        RestRequest.Method method,
+        String path,
+        Set<String> parameters,
+        Set<String> capabilities,
+        RestApiVersion restApiVersion
+    ) {
+        Iterator<MethodHandlers> allHandlers = getAllHandlers(null, path);
+        while (allHandlers.hasNext()) {
+            RestHandler handler;
+            MethodHandlers handlers = allHandlers.next();
+            if (handlers == null) {
+                handler = null;
+            } else {
+                handler = handlers.getHandler(method, restApiVersion);
+            }
+
+            if (handler != null) {
+                var supportedParams = handler.supportedQueryParameters();
+                return (supportedParams == null || supportedParams.containsAll(parameters))
+                    && handler.supportedCapabilities().containsAll(capabilities);
+            }
+        }
+        return false;
+    }
+
     @Override
     public Map<String, HttpRouteStats> getStats() {
         final Iterator<MethodHandlers> methodHandlersIterator = handlers.allNodeValues();

server/src/main/java/org/elasticsearch/rest/RestHandler.java

@@ -18,6 +18,7 @@ import org.elasticsearch.xcontent.XContent;
 import java.util.Collections;
 import java.util.List;
 import java.util.Objects;
+import java.util.Set;
 
 /**
  * Handler for REST requests
@@ -85,6 +86,22 @@ public interface RestHandler {
         return Collections.emptyList();
     }
 
+    /**
+     * The set of query parameters accepted by this rest handler,
+     * {@code null} if query parameters should not be checked nor validated.
+     * TODO - make this not nullable when all handlers have been updated
+     */
+    default @Nullable Set<String> supportedQueryParameters() {
+        return null;
+    }
+
+    /**
+     * The set of capabilities this rest handler supports.
+     */
+    default Set<String> supportedCapabilities() {
+        return Set.of();
+    }
+
     /**
      * Controls whether requests handled by this class are allowed to to access system indices by default.
      * @return {@code true} if requests handled by this class should be allowed to access system indices.

server/src/main/java/org/elasticsearch/rest/action/admin/cluster/RestNodesCapabilitiesAction.java

@@ -0,0 +1,60 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.rest.action.admin.cluster;
+
+import org.elasticsearch.action.admin.cluster.node.capabilities.NodesCapabilitiesRequest;
+import org.elasticsearch.client.internal.node.NodeClient;
+import org.elasticsearch.common.Strings;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.Scope;
+import org.elasticsearch.rest.ServerlessScope;
+import org.elasticsearch.rest.action.RestActions.NodesResponseRestListener;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+
+@ServerlessScope(Scope.INTERNAL)
+public class RestNodesCapabilitiesAction extends BaseRestHandler {
+
+    @Override
+    public List<Route> routes() {
+        return List.of(new Route(RestRequest.Method.GET, "/_capabilities"));
+    }
+
+    @Override
+    public Set<String> supportedQueryParameters() {
+        return Set.of("timeout", "method", "path", "parameters", "capabilities");
+    }
+
+    @Override
+    public String getName() {
+        return "nodes_capabilities_action";
+    }
+
+    @Override
+    protected RestChannelConsumer prepareRequest(RestRequest request, NodeClient client) throws IOException {
+        NodesCapabilitiesRequest r = new NodesCapabilitiesRequest().timeout(request.paramAsTime("timeout", null))
+            .method(RestRequest.Method.valueOf(request.param("method", "GET")))
+            .path(URLDecoder.decode(request.param("path"), StandardCharsets.UTF_8))
+            .parameters(request.paramAsStringArray("parameters", Strings.EMPTY_ARRAY))
+            .capabilities(request.paramAsStringArray("capabilities", Strings.EMPTY_ARRAY))
+            .restApiVersion(request.getRestApiVersion());
+
+        return channel -> client.admin().cluster().nodesCapabilities(r, new NodesResponseRestListener<>(channel));
+    }
+
+    @Override
+    public boolean canTripCircuitBreaker() {
+        return false;
+    }
+}

server/src/test/java/org/elasticsearch/common/settings/SettingsFilterTests.java

@@ -119,7 +119,6 @@ public class SettingsFilterTests extends ESTestCase {
         Logger testLogger = LogManager.getLogger("org.elasticsearch.test");
         MockLogAppender appender = new MockLogAppender();
         try (var ignored = appender.capturing("org.elasticsearch.test")) {
-            appender.start();
             Arrays.stream(expectations).forEach(appender::addExpectation);
             consumer.accept(testLogger);
             appender.assertAllExpectationsMatched();

server/src/test/java/org/elasticsearch/reservedstate/service/FileSettingsServiceTests.java

@@ -8,8 +8,8 @@
 
 package org.elasticsearch.reservedstate.service;
 
-import org.apache.lucene.tests.util.LuceneTestCase.AwaitsFix;
 import org.elasticsearch.Version;
+import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.cluster.ClusterChangedEvent;
 import org.elasticsearch.cluster.ClusterName;
 import org.elasticsearch.cluster.ClusterState;
@@ -55,7 +55,6 @@ import static org.mockito.Mockito.spy;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
 
-@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/106968")
 public class FileSettingsServiceTests extends ESTestCase {
     private Environment env;
     private ClusterService clusterService;
@@ -234,6 +233,12 @@ public class FileSettingsServiceTests extends ESTestCase {
             return new ReservedStateChunk(Collections.emptyMap(), new ReservedStateVersion(1L, Version.CURRENT));
         }).when(spiedController).parse(any(String.class), any());
 
+        doAnswer((Answer<Void>) invocation -> {
+            var completionListener = invocation.getArgument(1, ActionListener.class);
+            completionListener.onResponse(null);
+            return null;
+        }).when(spiedController).initEmpty(any(String.class), any());
+
         service.start();
         service.clusterChanged(new ClusterChangedEvent("test", clusterService.state(), ClusterState.EMPTY_STATE));
         assertTrue(service.watching());
@@ -255,55 +260,6 @@ public class FileSettingsServiceTests extends ESTestCase {
         deadThreadLatch.countDown();
     }
 
-    public void testStopWorksIfProcessingDidntReturnYet() throws Exception {
-        var spiedController = spy(controller);
-        var service = new FileSettingsService(clusterService, spiedController, env);
-
-        CountDownLatch processFileLatch = new CountDownLatch(1);
-        CountDownLatch deadThreadLatch = new CountDownLatch(1);
-
-        doAnswer((Answer<ReservedStateChunk>) invocation -> {
-            // allow the other thread to continue, but hold on a bit to avoid
-            // completing the task immediately in the main watcher loop
-            try {
-                Thread.sleep(1_000);
-            } catch (InterruptedException e) {
-                // pass it on
-                Thread.currentThread().interrupt();
-            }
-            processFileLatch.countDown();
-            new Thread(() -> {
-                // Simulate a thread that never allows the completion to complete
-                try {
-                    deadThreadLatch.await();
-                } catch (InterruptedException e) {
-                    throw new RuntimeException(e);
-                }
-            }).start();
-            return new ReservedStateChunk(Collections.emptyMap(), new ReservedStateVersion(1L, Version.CURRENT));
-        }).when(spiedController).parse(any(String.class), any());
-
-        service.start();
-        service.clusterChanged(new ClusterChangedEvent("test", clusterService.state(), ClusterState.EMPTY_STATE));
-        assertTrue(service.watching());
-
-        Files.createDirectories(service.watchedFileDir());
-
-        // Make some fake settings file to cause the file settings service to process it
-        writeTestFile(service.watchedFile(), "{}");
-
-        // we need to wait a bit, on MacOS it may take up to 10 seconds for the Java watcher service to notice the file,
-        // on Linux is instantaneous. Windows is instantaneous too.
-        assertTrue(processFileLatch.await(30, TimeUnit.SECONDS));
-
-        // Stopping the service should interrupt the watcher thread, allowing the whole thing to exit
-        service.stop();
-        assertFalse(service.watching());
-        service.close();
-        // let the deadlocked thread end, so we can cleanly exit the test
-        deadThreadLatch.countDown();
-    }
-
     // helpers
     private void writeTestFile(Path path, String contents) throws IOException {
         Path tempFilePath = createTempFile();

test/external-modules/esql-heap-attack/src/javaRestTest/java/org/elasticsearch/xpack/esql/heap_attack/HeapAttackIT.java

@@ -269,7 +269,6 @@ public class HeapAttackIT extends ESRestTestCase {
         assertMap(map, matchesMap().entry("columns", columns).entry("values", hasSize(10_000)));
     }
 
-    @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/108104")
     public void testTooManyEval() throws IOException {
         initManyLongs();
         assertCircuitBreaks(() -> manyEval(490));

test/framework/src/main/java/org/elasticsearch/test/ESTestCase.java

@@ -64,6 +64,7 @@ import org.elasticsearch.common.logging.HeaderWarningAppender;
 import org.elasticsearch.common.logging.LogConfigurator;
 import org.elasticsearch.common.logging.Loggers;
 import org.elasticsearch.common.lucene.Lucene;
+import org.elasticsearch.common.settings.SecureString;
 import org.elasticsearch.common.settings.Setting;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.time.DateUtils;
@@ -259,6 +260,7 @@ public abstract class ESTestCase extends LuceneTestCase {
         // TODO: consolidate logging initialization for tests so it all occurs in logconfigurator
         LogConfigurator.loadLog4jPlugins();
         LogConfigurator.configureESLogging();
+        MockLogAppender.init();
 
         final List<Appender> testAppenders = new ArrayList<>(3);
         for (String leakLoggerName : Arrays.asList("io.netty.util.ResourceLeakDetector", LeakTracker.class.getName())) {
@@ -1058,6 +1060,11 @@ public abstract class ESTestCase extends LuceneTestCase {
         return RandomizedTest.randomAsciiOfLength(codeUnits);
     }
 
+    public static SecureString randomSecureStringOfLength(int codeUnits) {
+        var randomAlpha = randomAlphaOfLength(codeUnits);
+        return new SecureString(randomAlpha.toCharArray());
+    }
+
     public static String randomNullOrAlphaOfLength(int codeUnits) {
         return randomBoolean() ? null : randomAlphaOfLength(codeUnits);
     }

test/framework/src/main/java/org/elasticsearch/test/MockLogAppender.java

@@ -9,7 +9,6 @@ package org.elasticsearch.test;
 
 import org.apache.logging.log4j.Level;
 import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
 import org.apache.logging.log4j.core.LogEvent;
 import org.apache.logging.log4j.core.appender.AbstractAppender;
 import org.apache.logging.log4j.core.config.Property;
@@ -19,9 +18,10 @@ import org.elasticsearch.core.Releasable;
 
 import java.util.Arrays;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CopyOnWriteArrayList;
-import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.regex.Pattern;
 
 import static org.hamcrest.CoreMatchers.equalTo;
@@ -31,12 +31,38 @@ import static org.hamcrest.Matchers.is;
 /**
  * Test appender that can be used to verify that certain events were logged correctly
  */
-public class MockLogAppender extends AbstractAppender {
+public class MockLogAppender {
 
+    private static final Map<String, List<MockLogAppender>> mockAppenders = new ConcurrentHashMap<>();
+    private static final RealMockAppender parent = new RealMockAppender();
     private final List<WrappedLoggingExpectation> expectations;
+    private volatile boolean isAlive = true;
+
+    private static class RealMockAppender extends AbstractAppender {
+
+        RealMockAppender() {
+            super("mock", null, null, false, Property.EMPTY_ARRAY);
+        }
+
+        @Override
+        public void append(LogEvent event) {
+            List<MockLogAppender> appenders = mockAppenders.get(event.getLoggerName());
+            if (appenders == null) {
+                // check if there is a root appender
+                appenders = mockAppenders.getOrDefault("", List.of());
+            }
+            for (MockLogAppender appender : appenders) {
+                if (appender.isAlive == false) {
+                    continue;
+                }
+                for (LoggingExpectation expectation : appender.expectations) {
+                    expectation.match(event);
+                }
+            }
+        }
+    }
 
     public MockLogAppender() {
-        super("mock", null, null, false, Property.EMPTY_ARRAY);
         /*
          * We use a copy-on-write array list since log messages could be appended while we are setting up expectations. When that occurs,
          * we would run into a concurrent modification exception from the iteration over the expectations in #append, concurrent with a
@@ -45,15 +71,16 @@ public class MockLogAppender extends AbstractAppender {
         expectations = new CopyOnWriteArrayList<>();
     }
 
-    public void addExpectation(LoggingExpectation expectation) {
-        expectations.add(new WrappedLoggingExpectation(expectation));
+    /**
+     * Initialize the mock log appender with the log4j system.
+     */
+    public static void init() {
+        parent.start();
+        Loggers.addAppender(LogManager.getLogger(""), parent);
     }
 
-    @Override
-    public void append(LogEvent event) {
-        for (LoggingExpectation expectation : expectations) {
-            expectation.match(event);
-        }
+    public void addExpectation(LoggingExpectation expectation) {
+        expectations.add(new WrappedLoggingExpectation(expectation));
     }
 
     public void assertAllExpectationsMatched() {
@@ -213,7 +240,7 @@ public class MockLogAppender extends AbstractAppender {
      */
     private static class WrappedLoggingExpectation implements LoggingExpectation {
 
-        private final AtomicBoolean assertMatchedCalled = new AtomicBoolean(false);
+        private volatile boolean assertMatchedCalled = false;
         private final LoggingExpectation delegate;
 
         private WrappedLoggingExpectation(LoggingExpectation delegate) {
@@ -230,7 +257,7 @@ public class MockLogAppender extends AbstractAppender {
             try {
                 delegate.assertMatched();
             } finally {
-                assertMatchedCalled.set(true);
+                assertMatchedCalled = true;
             }
         }
 
@@ -243,34 +270,43 @@ public class MockLogAppender extends AbstractAppender {
     /**
      * Adds the list of class loggers to this {@link MockLogAppender}.
      *
-     * Stops ({@link #stop()}) and runs some checks on the {@link MockLogAppender} once the returned object is released.
+     * Stops and runs some checks on the {@link MockLogAppender} once the returned object is released.
      */
     public Releasable capturing(Class<?>... classes) {
-        return appendToLoggers(Arrays.stream(classes).map(LogManager::getLogger).toList());
+        return appendToLoggers(Arrays.stream(classes).map(Class::getCanonicalName).toList());
     }
 
     /**
      * Same as above except takes string class names of each logger.
      */
     public Releasable capturing(String... names) {
-        return appendToLoggers(Arrays.stream(names).map(LogManager::getLogger).toList());
+        return appendToLoggers(Arrays.asList(names));
     }
 
-    private Releasable appendToLoggers(List<Logger> loggers) {
-        start();
-        for (final var logger : loggers) {
-            Loggers.addAppender(logger, this);
+    private Releasable appendToLoggers(List<String> loggers) {
+        for (String logger : loggers) {
+            mockAppenders.compute(logger, (k, v) -> {
+                if (v == null) {
+                    v = new CopyOnWriteArrayList<>();
+                }
+                v.add(this);
+                return v;
+            });
         }
         return () -> {
-            for (final var logger : loggers) {
-                Loggers.removeAppender(logger, this);
+            isAlive = false;
+            for (String logger : loggers) {
+                mockAppenders.compute(logger, (k, v) -> {
+                    assert v != null;
+                    v.remove(this);
+                    return v.isEmpty() ? null : v;
+                });
             }
-            stop();
             // check that all expectations have been evaluated before this is released
             for (WrappedLoggingExpectation expectation : expectations) {
                 assertThat(
                     "Method assertMatched() not called on LoggingExpectation instance before release: " + expectation,
-                    expectation.assertMatchedCalled.get(),
+                    expectation.assertMatchedCalled,
                     is(true)
                 );
             }

test/framework/src/test/java/org/elasticsearch/test/MockLogAppenderTests.java

@@ -0,0 +1,38 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+package org.elasticsearch.test;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class MockLogAppenderTests extends ESTestCase {
+
+    public void testConcurrentLogAndLifecycle() throws Exception {
+        Logger logger = LogManager.getLogger(MockLogAppenderTests.class);
+        final var keepGoing = new AtomicBoolean(true);
+        final var logThread = new Thread(() -> {
+            while (keepGoing.get()) {
+                logger.info("test");
+            }
+        });
+        logThread.start();
+
+        final var appender = new MockLogAppender();
+        for (int i = 0; i < 1000; i++) {
+            try (var ignored = appender.capturing(MockLogAppenderTests.class)) {
+                Thread.yield();
+            }
+        }
+
+        keepGoing.set(false);
+        logThread.join();
+    }
+}

x-pack/plugin/apm-data/src/main/resources/component-templates/apm@settings.yaml

@@ -8,3 +8,7 @@ template:
       sort:
         field: "@timestamp"
         order: desc
+    mapping:
+      ignore_malformed: true
+      total_fields:
+        ignore_dynamic_beyond_limit: true

x-pack/plugin/apm-data/src/main/resources/component-templates/metrics-apm@settings.yaml

@@ -6,3 +6,9 @@ _meta:
 template:
   settings:
     codec: best_compression
+    mapping:
+      # apm@settings sets `ignore_malformed: true`, but we need
+      # to disable this for metrics since they use synthetic source,
+      # and this combination is incompatible with the
+      # aggregate_metric_double field type.
+      ignore_malformed: false

x-pack/plugin/apm-data/src/main/resources/resources.yaml

@@ -1,7 +1,7 @@
 # "version" holds the version of the templates and ingest pipelines installed
 # by xpack-plugin apm-data. This must be increased whenever an existing template or
 # pipeline is changed, in order for it to be updated on Elasticsearch upgrade.
-version: 3
+version: 4
 
 component-templates:
   # Data lifecycle.

x-pack/plugin/apm-data/src/yamlRestTest/resources/rest-api-spec/test/30_lenient_mappings.yml

@@ -0,0 +1,100 @@
+---
+setup:
+  - do:
+      cluster.health:
+        wait_for_events: languid
+
+  - do:
+      cluster.put_component_template:
+        name: "logs-apm.app@custom"
+        body:
+          template:
+            settings:
+              mapping:
+                total_fields:
+                  limit: 20
+
+---
+"Test ignore_malformed":
+  - do:
+      bulk:
+        index: traces-apm-testing
+        refresh: true
+        body:
+          # Passing a (non-coercable) string into a numeric field should not
+          # cause an indexing failure; it should just not be indexed.
+          - create: {}
+          - '{"@timestamp": "2017-06-22", "numeric_labels": {"key": "string"}}'
+          - create: {}
+          - '{"@timestamp": "2017-06-22", "numeric_labels": {"key": 123}}'
+
+  - is_false: errors
+
+  - do:
+      search:
+        index: traces-apm-testing
+        body:
+          fields: ["numeric_labels.*", "_ignored"]
+  - length: { hits.hits: 2 }
+  - match: { hits.hits.0.fields: {"_ignored": ["numeric_labels.key"]} }
+  - match: { hits.hits.1.fields: {"numeric_labels.key": [123.0]} }
+
+---
+"Test ignore_dynamic_beyond_limit":
+  - do:
+      bulk:
+        index: logs-apm.app.svc1-testing
+        refresh: true
+        body:
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k1": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k2": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k3": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k4": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k5": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k6": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k7": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k8": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k9": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k10": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k11": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k12": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k13": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k14": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k15": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k16": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k17": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k18": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k19": ""}
+          - create: {}
+          - {"@timestamp": "2017-06-22", "k20": ""}
+
+  - is_false: errors
+
+  - do:
+      search:
+        index: logs-apm.app.svc1-testing
+        body:
+          query:
+            term:
+              _ignored:
+                value: k20
+  - length: { hits.hits: 1 }

x-pack/plugin/core/template-resources/src/main/resources/kibana-reporting@settings.json

@@ -0,0 +1,14 @@
+{
+  "template": {
+    "settings": {
+      "number_of_shards": 1,
+      "auto_expand_replicas": "0-1"
+    }
+  },
+  "_meta": {
+    "description": "default kibana reporting settings installed by elasticsearch",
+    "managed": true
+  },
+  "version": ${xpack.stack.template.version},
+  "deprecated": ${xpack.stack.template.deprecated}
+}

x-pack/plugin/core/template-resources/src/main/resources/kibana-reporting@template.json

@@ -5,14 +5,10 @@
     "hidden": true
   },
   "allow_auto_create": true,
-  "composed_of": ["kibana-reporting@custom"],
+  "composed_of": ["kibana-reporting@settings", "kibana-reporting@custom"],
   "ignore_missing_component_templates": ["kibana-reporting@custom"],
   "template": {
     "lifecycle": {},
-    "settings": {
-      "number_of_shards": 1,
-      "auto_expand_replicas": "0-1"
-    },
     "mappings": {
       "properties": {
         "meta": {

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/data/BooleanArrayVector.java

@@ -23,7 +23,9 @@ final class BooleanArrayVector extends AbstractVector implements BooleanVector {
 
     static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(BooleanArrayVector.class)
         // TODO: remove these extra bytes once `asBlock` returns a block with a separate reference to the vector.
-        + RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class);
+        + RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class)
+        // TODO: remove this if/when we account for memory used by Pages
+        + Block.PAGE_MEM_OVERHEAD_PER_BLOCK;
 
     private final boolean[] values;
 

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/data/BytesRefArrayVector.java

@@ -25,7 +25,9 @@ final class BytesRefArrayVector extends AbstractVector implements BytesRefVector
 
     static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(BytesRefArrayVector.class)
         // TODO: remove these extra bytes once `asBlock` returns a block with a separate reference to the vector.
-        + RamUsageEstimator.shallowSizeOfInstance(BytesRefVectorBlock.class);
+        + RamUsageEstimator.shallowSizeOfInstance(BytesRefVectorBlock.class)
+        // TODO: remove this if/when we account for memory used by Pages
+        + Block.PAGE_MEM_OVERHEAD_PER_BLOCK;
 
     private final BytesRefArray values;
 

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/data/BytesRefBlockBuilder.java

@@ -21,10 +21,6 @@ final class BytesRefBlockBuilder extends AbstractBlockBuilder implements BytesRe
 
     private BytesRefArray values;
 
-    BytesRefBlockBuilder(int estimatedSize, BlockFactory blockFactory) {
-        this(estimatedSize, BigArrays.NON_RECYCLING_INSTANCE, blockFactory);
-    }
-
     BytesRefBlockBuilder(int estimatedSize, BigArrays bigArrays, BlockFactory blockFactory) {
         super(blockFactory);
         values = new BytesRefArray(Math.max(estimatedSize, 2), bigArrays);

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/data/DoubleArrayVector.java

@@ -23,7 +23,9 @@ final class DoubleArrayVector extends AbstractVector implements DoubleVector {
 
     static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(DoubleArrayVector.class)
         // TODO: remove these extra bytes once `asBlock` returns a block with a separate reference to the vector.
-        + RamUsageEstimator.shallowSizeOfInstance(DoubleVectorBlock.class);
+        + RamUsageEstimator.shallowSizeOfInstance(DoubleVectorBlock.class)
+        // TODO: remove this if/when we account for memory used by Pages
+        + Block.PAGE_MEM_OVERHEAD_PER_BLOCK;
 
     private final double[] values;
 

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/data/IntArrayVector.java

@@ -23,7 +23,9 @@ final class IntArrayVector extends AbstractVector implements IntVector {
 
     static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(IntArrayVector.class)
         // TODO: remove these extra bytes once `asBlock` returns a block with a separate reference to the vector.
-        + RamUsageEstimator.shallowSizeOfInstance(IntVectorBlock.class);
+        + RamUsageEstimator.shallowSizeOfInstance(IntVectorBlock.class)
+        // TODO: remove this if/when we account for memory used by Pages
+        + Block.PAGE_MEM_OVERHEAD_PER_BLOCK;
 
     private final int[] values;
 

x-pack/plugin/esql/compute/src/main/generated-src/org/elasticsearch/compute/data/LongArrayVector.java

@@ -23,7 +23,9 @@ final class LongArrayVector extends AbstractVector implements LongVector {
 
     static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(LongArrayVector.class)
         // TODO: remove these extra bytes once `asBlock` returns a block with a separate reference to the vector.
-        + RamUsageEstimator.shallowSizeOfInstance(LongVectorBlock.class);
+        + RamUsageEstimator.shallowSizeOfInstance(LongVectorBlock.class)
+        // TODO: remove this if/when we account for memory used by Pages
+        + Block.PAGE_MEM_OVERHEAD_PER_BLOCK;
 
     private final long[] values;
 

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/Block.java

@@ -8,6 +8,7 @@
 package org.elasticsearch.compute.data;
 
 import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.RamUsageEstimator;
 import org.elasticsearch.common.io.stream.NamedWriteable;
 import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
 import org.elasticsearch.common.unit.ByteSizeValue;
@@ -44,6 +45,17 @@ public interface Block extends Accountable, BlockLoader.Block, NamedWriteable, R
      */
     long MAX_LOOKUP = 100_000;
 
+    /**
+     * We do not track memory for pages directly (only for single blocks),
+     * but the page memory overhead can still be significant, especially for pages containing thousands of blocks.
+     * For now, we approximate this overhead, per block, using this value.
+     *
+     * The exact overhead per block would be (more correctly) {@link RamUsageEstimator#NUM_BYTES_OBJECT_REF},
+     * but we approximate it with {@link RamUsageEstimator#NUM_BYTES_OBJECT_ALIGNMENT} to avoid further alignments
+     * to object size (at the end of the alignment, it would make no practical difference).
+     */
+    int PAGE_MEM_OVERHEAD_PER_BLOCK = RamUsageEstimator.NUM_BYTES_OBJECT_ALIGNMENT;
+
     /**
      * {@return an efficient dense single-value view of this block}.
      * Null, if the block is not dense single-valued. That is, if

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/X-ArrayVector.java.st

@@ -38,7 +38,9 @@ final class $Type$ArrayVector extends AbstractVector implements $Type$Vector {
 
     static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance($Type$ArrayVector.class)
         // TODO: remove these extra bytes once `asBlock` returns a block with a separate reference to the vector.
-        + RamUsageEstimator.shallowSizeOfInstance($Type$VectorBlock.class);
+        + RamUsageEstimator.shallowSizeOfInstance($Type$VectorBlock.class)
+        // TODO: remove this if/when we account for memory used by Pages
+        + Block.PAGE_MEM_OVERHEAD_PER_BLOCK;
 
 $if(BytesRef)$
     private final BytesRefArray values;

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/X-BlockBuilder.java.st

@@ -31,10 +31,6 @@ final class $Type$BlockBuilder extends AbstractBlockBuilder implements $Type$Blo
 $if(BytesRef)$
     private BytesRefArray values;
 
-    BytesRefBlockBuilder(int estimatedSize, BlockFactory blockFactory) {
-        this(estimatedSize, BigArrays.NON_RECYCLING_INSTANCE, blockFactory);
-    }
-
     BytesRefBlockBuilder(int estimatedSize, BigArrays bigArrays, BlockFactory blockFactory) {
         super(blockFactory);
         values = new BytesRefArray(Math.max(estimatedSize, 2), bigArrays);

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/TimeSeriesAggregationOperatorFactories.java

@@ -0,0 +1,157 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.compute.operator;
+
+import org.elasticsearch.compute.aggregation.AggregatorFunctionSupplier;
+import org.elasticsearch.compute.aggregation.AggregatorMode;
+import org.elasticsearch.compute.aggregation.GroupingAggregator;
+import org.elasticsearch.compute.aggregation.blockhash.BlockHash;
+import org.elasticsearch.compute.aggregation.blockhash.TimeSeriesBlockHash;
+import org.elasticsearch.compute.data.ElementType;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class provides operator factories for time-series aggregations.
+ * A time-series aggregation executes in three stages, deviating from the typical two-stage aggregation.
+ * For example: {@code sum(rate(write_requests)), avg(cpu) BY cluster, time-bucket}
+ *
+ * 1. Initial Stage:
+ * In this stage, a standard hash aggregation is executed, grouped by tsid and time-bucket.
+ * The {@code values} aggregations are added to collect values of the grouping keys excluding the time-bucket,
+ * which are then used for final result grouping.
+ * {@code rate[INITIAL](write_requests), avg[INITIAL](cpu), values[SINGLE](cluster) BY tsid, time-bucket}
+ *
+ * 2. Intermediate Stage:
+ * Equivalent to the final mode of a standard hash aggregation.
+ * This stage merges and reduces the result of the rate aggregations,
+ * but merges (without reducing) the results of non-rate aggregations.
+ * {@code rate[FINAL](write_requests), avg[INTERMEDIATE](cpu), values[SINGLE](cluster) BY tsid, time-bucket}
+ *
+ * 3. Final Stage:
+ * This extra stage performs outer aggregations over the rate results
+ * and combines the intermediate results of non-rate aggregations using the specified user-defined grouping keys.
+ * {@code sum[SINGLE](rate_result), avg[FINAL](cpu) BY cluster, bucket}
+ */
+public final class TimeSeriesAggregationOperatorFactories {
+
+    public record Initial(
+        int tsHashChannel,
+        int timeBucketChannel,
+        List<BlockHash.GroupSpec> groupings,
+        List<AggregatorFunctionSupplier> rates,
+        List<AggregatorFunctionSupplier> nonRates,
+        int maxPageSize
+    ) implements Operator.OperatorFactory {
+        @Override
+        public Operator get(DriverContext driverContext) {
+            List<GroupingAggregator.Factory> aggregators = new ArrayList<>(groupings.size() + rates.size() + nonRates.size());
+            for (AggregatorFunctionSupplier f : rates) {
+                aggregators.add(f.groupingAggregatorFactory(AggregatorMode.INITIAL));
+            }
+            for (AggregatorFunctionSupplier f : nonRates) {
+                aggregators.add(f.groupingAggregatorFactory(AggregatorMode.INITIAL));
+            }
+            aggregators.addAll(valuesAggregatorForGroupings(groupings, timeBucketChannel));
+            return new HashAggregationOperator(
+                aggregators,
+                () -> new TimeSeriesBlockHash(tsHashChannel, timeBucketChannel, driverContext),
+                driverContext
+            );
+        }
+
+        @Override
+        public String describe() {
+            return "TimeSeriesInitialAggregationOperatorFactory";
+        }
+    }
+
+    public record Intermediate(
+        int tsHashChannel,
+        int timeBucketChannel,
+        List<BlockHash.GroupSpec> groupings,
+        List<AggregatorFunctionSupplier> rates,
+        List<AggregatorFunctionSupplier> nonRates,
+        int maxPageSize
+    ) implements Operator.OperatorFactory {
+        @Override
+        public Operator get(DriverContext driverContext) {
+            List<GroupingAggregator.Factory> aggregators = new ArrayList<>(groupings.size() + rates.size() + nonRates.size());
+            for (AggregatorFunctionSupplier f : rates) {
+                aggregators.add(f.groupingAggregatorFactory(AggregatorMode.FINAL));
+            }
+            for (AggregatorFunctionSupplier f : nonRates) {
+                aggregators.add(f.groupingAggregatorFactory(AggregatorMode.INTERMEDIATE));
+            }
+            aggregators.addAll(valuesAggregatorForGroupings(groupings, timeBucketChannel));
+            List<BlockHash.GroupSpec> hashGroups = List.of(
+                new BlockHash.GroupSpec(tsHashChannel, ElementType.BYTES_REF),
+                new BlockHash.GroupSpec(timeBucketChannel, ElementType.LONG)
+            );
+            return new HashAggregationOperator(
+                aggregators,
+                () -> BlockHash.build(hashGroups, driverContext.blockFactory(), maxPageSize, false),
+                driverContext
+            );
+        }
+
+        @Override
+        public String describe() {
+            return "TimeSeriesIntermediateAggregationOperatorFactory";
+        }
+    }
+
+    public record Final(
+        List<BlockHash.GroupSpec> groupings,
+        List<AggregatorFunctionSupplier> outerRates,
+        List<AggregatorFunctionSupplier> nonRates,
+        int maxPageSize
+    ) implements Operator.OperatorFactory {
+        @Override
+        public Operator get(DriverContext driverContext) {
+            List<GroupingAggregator.Factory> aggregators = new ArrayList<>(outerRates.size() + nonRates.size());
+            for (AggregatorFunctionSupplier f : outerRates) {
+                aggregators.add(f.groupingAggregatorFactory(AggregatorMode.SINGLE));
+            }
+            for (AggregatorFunctionSupplier f : nonRates) {
+                aggregators.add(f.groupingAggregatorFactory(AggregatorMode.FINAL));
+            }
+            return new HashAggregationOperator(
+                aggregators,
+                () -> BlockHash.build(groupings, driverContext.blockFactory(), maxPageSize, false),
+                driverContext
+            );
+        }
+
+        @Override
+        public String describe() {
+            return "TimeSeriesFinalAggregationOperatorFactory";
+        }
+    }
+
+    static List<GroupingAggregator.Factory> valuesAggregatorForGroupings(List<BlockHash.GroupSpec> groupings, int timeBucketChannel) {
+        List<GroupingAggregator.Factory> aggregators = new ArrayList<>();
+        for (BlockHash.GroupSpec g : groupings) {
+            if (g.channel() != timeBucketChannel) {
+                final List<Integer> channels = List.of(g.channel());
+                // TODO: perhaps introduce a specialized aggregator for this?
+                var aggregatorSupplier = (switch (g.elementType()) {
+                    case BYTES_REF -> new org.elasticsearch.compute.aggregation.ValuesBytesRefAggregatorFunctionSupplier(channels);
+                    case DOUBLE -> new org.elasticsearch.compute.aggregation.ValuesDoubleAggregatorFunctionSupplier(channels);
+                    case INT -> new org.elasticsearch.compute.aggregation.ValuesIntAggregatorFunctionSupplier(channels);
+                    case LONG -> new org.elasticsearch.compute.aggregation.ValuesLongAggregatorFunctionSupplier(channels);
+                    case BOOLEAN -> new org.elasticsearch.compute.aggregation.ValuesBooleanAggregatorFunctionSupplier(channels);
+                    case NULL, DOC, UNKNOWN -> throw new IllegalArgumentException("unsupported grouping type");
+                });
+                aggregators.add(aggregatorSupplier.groupingAggregatorFactory(AggregatorMode.SINGLE));
+            }
+        }
+        return aggregators;
+    }
+}

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/TimeSeriesAggregationOperatorFactory.java

@@ -1,48 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
- */
-
-package org.elasticsearch.compute.operator;
-
-import org.elasticsearch.compute.aggregation.AggregatorMode;
-import org.elasticsearch.compute.aggregation.GroupingAggregator;
-import org.elasticsearch.compute.aggregation.blockhash.BlockHash;
-import org.elasticsearch.compute.aggregation.blockhash.TimeSeriesBlockHash;
-import org.elasticsearch.core.TimeValue;
-
-import java.util.List;
-
-public record TimeSeriesAggregationOperatorFactory(
-    AggregatorMode mode,
-    int tsHashChannel,
-    int timestampIntervalChannel,
-    TimeValue timeSeriesPeriod,
-    List<GroupingAggregator.Factory> aggregators,
-    int maxPageSize
-) implements Operator.OperatorFactory {
-
-    @Override
-    public String describe() {
-        return "TimeSeriesAggregationOperator[mode="
-            + mode
-            + ", tsHashChannel = "
-            + tsHashChannel
-            + ", timestampIntervalChannel = "
-            + timestampIntervalChannel
-            + ", timeSeriesPeriod = "
-            + timeSeriesPeriod
-            + ", maxPageSize = "
-            + maxPageSize
-            + "]";
-    }
-
-    @Override
-    public Operator get(DriverContext driverContext) {
-        BlockHash blockHash = new TimeSeriesBlockHash(tsHashChannel, timestampIntervalChannel, driverContext);
-        return new HashAggregationOperator(aggregators, () -> blockHash, driverContext);
-    }
-
-}

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/exchange/ExchangeSourceHandler.java

@@ -10,6 +10,7 @@ package org.elasticsearch.compute.operator.exchange;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.ExceptionsHelper;
 import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.action.support.RefCountingListener;
 import org.elasticsearch.action.support.SubscribableListener;
 import org.elasticsearch.common.util.concurrent.AbstractRunnable;
 import org.elasticsearch.compute.data.Page;
@@ -17,6 +18,7 @@ import org.elasticsearch.core.Releasable;
 import org.elasticsearch.tasks.TaskCancelledException;
 import org.elasticsearch.transport.TransportException;
 
+import java.util.List;
 import java.util.concurrent.Executor;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReference;
@@ -89,6 +91,20 @@ public final class ExchangeSourceHandler {
         }
     }
 
+    public void addCompletionListener(ActionListener<Void> listener) {
+        buffer.addCompletionListener(ActionListener.running(() -> {
+            try (RefCountingListener refs = new RefCountingListener(listener)) {
+                for (PendingInstances pending : List.of(outstandingSinks, outstandingSources)) {
+                    // Create an outstanding instance and then finish to complete the completionListener
+                    // if we haven't registered any instances of exchange sinks or exchange sources before.
+                    pending.trackNewInstance();
+                    pending.completion.addListener(refs.acquire());
+                    pending.finishInstance();
+                }
+            }
+        }));
+    }
+
     /**
      * Create a new {@link ExchangeSource} for exchanging data
      *
@@ -253,10 +269,10 @@ public final class ExchangeSourceHandler {
 
     private static class PendingInstances {
         private final AtomicInteger instances = new AtomicInteger();
-        private final Releasable onComplete;
+        private final SubscribableListener<Void> completion = new SubscribableListener<>();
 
-        PendingInstances(Releasable onComplete) {
-            this.onComplete = onComplete;
+        PendingInstances(Runnable onComplete) {
+            completion.addListener(ActionListener.running(onComplete));
         }
 
         void trackNewInstance() {
@@ -268,7 +284,7 @@ public final class ExchangeSourceHandler {
             int refs = instances.decrementAndGet();
             assert refs >= 0;
             if (refs == 0) {
-                onComplete.close();
+                completion.onResponse(null);
             }
         }
     }

x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/data/BlockAccountingTests.java

@@ -42,9 +42,8 @@ public class BlockAccountingTests extends ComputeTestCase {
     public void testBooleanVector() {
         BlockFactory blockFactory = blockFactory();
         Vector empty = blockFactory.newBooleanArrayVector(new boolean[] {}, 0);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            BooleanVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Vector emptyPlusOne = blockFactory.newBooleanArrayVector(new boolean[] { randomBoolean() }, 1);
@@ -62,9 +61,8 @@ public class BlockAccountingTests extends ComputeTestCase {
     public void testIntVector() {
         BlockFactory blockFactory = blockFactory();
         Vector empty = blockFactory.newIntArrayVector(new int[] {}, 0);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            IntVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(IntVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Vector emptyPlusOne = blockFactory.newIntArrayVector(new int[] { randomInt() }, 1);
@@ -82,9 +80,8 @@ public class BlockAccountingTests extends ComputeTestCase {
     public void testLongVector() {
         BlockFactory blockFactory = blockFactory();
         Vector empty = blockFactory.newLongArrayVector(new long[] {}, 0);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            LongVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(LongVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Vector emptyPlusOne = blockFactory.newLongArrayVector(new long[] { randomLong() }, 1);
@@ -103,9 +100,8 @@ public class BlockAccountingTests extends ComputeTestCase {
     public void testDoubleVector() {
         BlockFactory blockFactory = blockFactory();
         Vector empty = blockFactory.newDoubleArrayVector(new double[] {}, 0);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            DoubleVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(DoubleVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Vector emptyPlusOne = blockFactory.newDoubleArrayVector(new double[] { randomDouble() }, 1);
@@ -127,9 +123,8 @@ public class BlockAccountingTests extends ComputeTestCase {
         var emptyArray = new BytesRefArray(0, blockFactory.bigArrays());
         var arrayWithOne = new BytesRefArray(0, blockFactory.bigArrays());
         Vector emptyVector = blockFactory.newBytesRefArrayVector(emptyArray, 0);
-        long expectedEmptyVectorUsed = RamUsageTester.ramUsed(emptyVector, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            BytesRefVectorBlock.class
-        );
+        long expectedEmptyVectorUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(emptyVector, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(BytesRefVectorBlock.class);
         assertThat(emptyVector.ramBytesUsed(), is(expectedEmptyVectorUsed));
 
         var bytesRef = new BytesRef(randomAlphaOfLengthBetween(1, 16));
@@ -146,9 +141,8 @@ public class BlockAccountingTests extends ComputeTestCase {
     public void testBooleanBlock() {
         BlockFactory blockFactory = blockFactory();
         Block empty = new BooleanArrayBlock(new boolean[] {}, 0, new int[] { 0 }, null, Block.MvOrdering.UNORDERED, blockFactory);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            BooleanVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Block emptyPlusOne = new BooleanArrayBlock(
@@ -194,18 +188,16 @@ public class BlockAccountingTests extends ComputeTestCase {
             Block.MvOrdering.UNORDERED,
             blockFactory()
         );
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            BooleanVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class);
         assertThat(empty.ramBytesUsed(), lessThanOrEqualTo(expectedEmptyUsed));
     }
 
     public void testIntBlock() {
         BlockFactory blockFactory = blockFactory();
         Block empty = new IntArrayBlock(new int[] {}, 0, new int[] { 0 }, null, Block.MvOrdering.UNORDERED, blockFactory);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            IntVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(IntVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Block emptyPlusOne = new IntArrayBlock(
@@ -242,18 +234,16 @@ public class BlockAccountingTests extends ComputeTestCase {
     public void testIntBlockWithNullFirstValues() {
         BlockFactory blockFactory = blockFactory();
         Block empty = new IntArrayBlock(new int[] {}, 0, null, BitSet.valueOf(new byte[] { 1 }), Block.MvOrdering.UNORDERED, blockFactory);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            IntVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(IntVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
     }
 
     public void testLongBlock() {
         BlockFactory blockFactory = blockFactory();
         Block empty = new LongArrayBlock(new long[] {}, 0, new int[] { 0 }, null, Block.MvOrdering.UNORDERED, blockFactory);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            LongVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(LongVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Block emptyPlusOne = new LongArrayBlock(
@@ -299,18 +289,16 @@ public class BlockAccountingTests extends ComputeTestCase {
             Block.MvOrdering.UNORDERED,
             blockFactory()
         );
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            LongVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(LongVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
     }
 
     public void testDoubleBlock() {
         BlockFactory blockFactory = blockFactory();
         Block empty = new DoubleArrayBlock(new double[] {}, 0, new int[] { 0 }, null, Block.MvOrdering.UNORDERED, blockFactory);
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            DoubleVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(DoubleVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
 
         Block emptyPlusOne = new DoubleArrayBlock(
@@ -356,9 +344,8 @@ public class BlockAccountingTests extends ComputeTestCase {
             Block.MvOrdering.UNORDERED,
             blockFactory()
         );
-        long expectedEmptyUsed = RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR) + RamUsageEstimator.shallowSizeOfInstance(
-            DoubleVectorBlock.class
-        );
+        long expectedEmptyUsed = Block.PAGE_MEM_OVERHEAD_PER_BLOCK + RamUsageTester.ramUsed(empty, RAM_USAGE_ACCUMULATOR)
+            + RamUsageEstimator.shallowSizeOfInstance(DoubleVectorBlock.class);
         assertThat(empty.ramBytesUsed(), is(expectedEmptyUsed));
     }
 

x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/TimeSeriesAggregationOperatorTests.java

@@ -11,65 +11,49 @@ import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.BytesRef;
 import org.elasticsearch.common.Randomness;
-import org.elasticsearch.compute.aggregation.AggregatorMode;
+import org.elasticsearch.common.util.CollectionUtils;
 import org.elasticsearch.compute.aggregation.RateLongAggregatorFunctionSupplier;
-import org.elasticsearch.compute.data.BytesRefBlock;
-import org.elasticsearch.compute.data.DoubleBlock;
+import org.elasticsearch.compute.aggregation.SumDoubleAggregatorFunctionSupplier;
+import org.elasticsearch.compute.aggregation.blockhash.BlockHash;
+import org.elasticsearch.compute.data.BlockFactory;
+import org.elasticsearch.compute.data.BlockUtils;
 import org.elasticsearch.compute.data.ElementType;
-import org.elasticsearch.compute.data.LongBlock;
 import org.elasticsearch.compute.data.Page;
 import org.elasticsearch.compute.lucene.ValuesSourceReaderOperatorTests;
 import org.elasticsearch.core.IOUtils;
 import org.elasticsearch.core.TimeValue;
+import org.elasticsearch.index.mapper.KeywordFieldMapper;
 import org.elasticsearch.index.mapper.NumberFieldMapper;
-import org.hamcrest.Matcher;
 import org.junit.After;
 
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.HashMap;
 import java.util.List;
-import java.util.Map;
+import java.util.stream.IntStream;
 
 import static org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests.createTimeSeriesSourceOperator;
 import static org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests.writeTS;
-import static org.elasticsearch.index.mapper.DateFieldMapper.DEFAULT_DATE_TIME_FORMATTER;
-import static org.elasticsearch.test.MapMatcher.assertMap;
-import static org.elasticsearch.test.MapMatcher.matchesMap;
 import static org.hamcrest.Matchers.equalTo;
 
-public class TimeSeriesAggregationOperatorTests extends AnyOperatorTestCase {
+public class TimeSeriesAggregationOperatorTests extends ComputeTestCase {
 
-    private IndexReader reader;
-    private final Directory directory = newDirectory();
+    private IndexReader reader = null;
+    private Directory directory = null;
 
     @After
     public void cleanup() throws IOException {
         IOUtils.close(reader, directory);
     }
 
-    @Override
-    protected Operator.OperatorFactory simple() {
-        return new TimeSeriesAggregationOperatorFactory(AggregatorMode.FINAL, 0, 1, TimeValue.ZERO, List.of(), 100);
+    /**
+     * A {@link DriverContext} with a nonBreakingBigArrays.
+     */
+    protected DriverContext driverContext() { // TODO make this final once all operators support memory tracking
+        BlockFactory blockFactory = blockFactory();
+        return new DriverContext(blockFactory.bigArrays(), blockFactory);
     }
 
-    @Override
-    protected Matcher<String> expectedDescriptionOfSimple() {
-        return equalTo(
-            "TimeSeriesAggregationOperator[mode=FINAL, tsHashChannel = 0, timestampIntervalChannel = 1, "
-                + "timeSeriesPeriod = 0s, maxPageSize = 100]"
-        );
-    }
-
-    @Override
-    protected Matcher<String> expectedToStringOfSimple() {
-        return equalTo(
-            "HashAggregationOperator[blockHash=TimeSeriesBlockHash{keys=[BytesRefKey[channel=0], "
-                + "LongKey[channel=1]], entries=-1b}, aggregators=[]]"
-        );
-    }
-
-    public void testBasicRate() {
+    public void testBasicRate() throws Exception {
         long[] v1 = { 1, 1, 3, 0, 2, 9, 21, 3, 7, 7, 9, 12 };
         long[] t1 = { 1, 5, 11, 20, 21, 59, 88, 91, 92, 97, 99, 112 };
 
@@ -78,25 +62,51 @@ public class TimeSeriesAggregationOperatorTests extends AnyOperatorTestCase {
 
         long[] v3 = { 0, 1, 0, 1, 1, 4, 2, 2, 2, 2, 3, 5, 5 };
         long[] t3 = { 2, 3, 5, 7, 8, 9, 10, 12, 14, 15, 18, 20, 22 };
-        List<Pod> pods = List.of(new Pod("p1", t1, v1), new Pod("p2", t2, v2), new Pod("p3", t3, v3));
-        long unit = between(1, 5);
-        Map<Group, Double> actualRates = runRateTest(pods, TimeValue.timeValueMillis(unit), TimeValue.ZERO);
-        assertThat(
-            actualRates,
-            equalTo(
-                Map.of(
-                    new Group("\u0001\u0003pods\u0002p1", 0),
-                    35.0 * unit / 111.0,
-                    new Group("\u0001\u0003pods\u0002p2", 0),
-                    42.0 * unit / 13.0,
-                    new Group("\u0001\u0003pods\u0002p3", 0),
-                    10.0 * unit / 20.0
-                )
-            )
+        List<Pod> pods = List.of(
+            new Pod("p1", "cluster_1", new Interval(2100, t1, v1)),
+            new Pod("p2", "cluster_1", new Interval(600, t2, v2)),
+            new Pod("p3", "cluster_2", new Interval(1100, t3, v3))
         );
+        long unit = between(1, 5);
+        {
+            List<List<Object>> actual = runRateTest(
+                pods,
+                List.of("cluster"),
+                TimeValue.timeValueMillis(unit),
+                TimeValue.timeValueMillis(500)
+            );
+            List<List<Object>> expected = List.of(
+                List.of(new BytesRef("cluster_1"), 35.0 * unit / 111.0 + 42.0 * unit / 13.0),
+                List.of(new BytesRef("cluster_2"), 10.0 * unit / 20.0)
+            );
+            assertThat(actual, equalTo(expected));
+        }
+        {
+            List<List<Object>> actual = runRateTest(pods, List.of("pod"), TimeValue.timeValueMillis(unit), TimeValue.timeValueMillis(500));
+            List<List<Object>> expected = List.of(
+                List.of(new BytesRef("p1"), 35.0 * unit / 111.0),
+                List.of(new BytesRef("p2"), 42.0 * unit / 13.0),
+                List.of(new BytesRef("p3"), 10.0 * unit / 20.0)
+            );
+            assertThat(actual, equalTo(expected));
+        }
+        {
+            List<List<Object>> actual = runRateTest(
+                pods,
+                List.of("cluster", "bucket"),
+                TimeValue.timeValueMillis(unit),
+                TimeValue.timeValueMillis(500)
+            );
+            List<List<Object>> expected = List.of(
+                List.of(new BytesRef("cluster_1"), 2000L, 35.0 * unit / 111.0),
+                List.of(new BytesRef("cluster_1"), 500L, 42.0 * unit / 13.0),
+                List.of(new BytesRef("cluster_2"), 1000L, 10.0 * unit / 20.0)
+            );
+            assertThat(actual, equalTo(expected));
+        }
     }
 
-    public void testRateWithInterval() {
+    public void testRateWithInterval() throws Exception {
         long[] v1 = { 1, 2, 3, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3 };
         long[] t1 = { 0, 10_000, 20_000, 30_000, 40_000, 50_000, 60_000, 70_000, 80_000, 90_000, 100_000, 110_000, 120_000 };
 
@@ -105,59 +115,71 @@ public class TimeSeriesAggregationOperatorTests extends AnyOperatorTestCase {
 
         long[] v3 = { 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192 };
         long[] t3 = { 0, 10_000, 20_000, 30_000, 40_000, 50_000, 60_000, 70_000, 80_000, 90_000, 100_000, 110_000, 120_000 };
-        List<Pod> pods = List.of(new Pod("p1", t1, v1), new Pod("p2", t2, v2), new Pod("p3", t3, v3));
-        Map<Group, Double> actualRates = runRateTest(pods, TimeValue.timeValueMillis(1), TimeValue.timeValueMinutes(1));
-        assertMap(
-            actualRates,
-            matchesMap().entry(new Group("\u0001\u0003pods\u0002p1", 120_000), 0.0D)
-                .entry(new Group("\u0001\u0003pods\u0002p1", 60_000), 8.0E-5D)
-                .entry(new Group("\u0001\u0003pods\u0002p1", 0), 8.0E-5D)
-                .entry(new Group("\u0001\u0003pods\u0002p2", 120_000), 0.0D)
-                .entry(new Group("\u0001\u0003pods\u0002p2", 60_000), 0.0D)
-                .entry(new Group("\u0001\u0003pods\u0002p2", 0), 0.0D)
-                .entry(new Group("\u0001\u0003pods\u0002p3", 120_000), 0.0D)
-                .entry(new Group("\u0001\u0003pods\u0002p3", 60_000), 0.07936D)
-                .entry(new Group("\u0001\u0003pods\u0002p3", 0), 0.00124D)
+        List<Pod> pods = List.of(
+            new Pod("p1", "cluster_1", new Interval(0, t1, v1)),
+            new Pod("p2", "cluster_2", new Interval(0, t2, v2)),
+            new Pod("p3", "cluster_2", new Interval(0, t3, v3))
+        );
+        List<List<Object>> actual = runRateTest(
+            pods,
+            List.of("pod", "bucket"),
+            TimeValue.timeValueMillis(1),
+            TimeValue.timeValueMinutes(1)
+        );
+        List<List<Object>> expected = List.of(
+            List.of(new BytesRef("p1]"), 120_000L, 0.0D),
+            List.of(new BytesRef("p1"), 60_000L, 8.0E-5D),
+            List.of(new BytesRef("p1"), 0, 8.0E-5D),
+            List.of(new BytesRef("p2"), 120_000L, 0.0D),
+            List.of(new BytesRef("p2"), 60_000L, 0.0D),
+            List.of(new BytesRef("p2"), 0L, 0.0D),
+            List.of(new BytesRef("p3"), 120_000L, 0.0D),
+            List.of(new BytesRef("p3"), 60_000L, 0.07936D),
+            List.of(new BytesRef("p3"), 0L, 0.00124D)
         );
     }
 
-    public void testRandomRate() {
+    public void testRandomRate() throws Exception {
         int numPods = between(1, 10);
         List<Pod> pods = new ArrayList<>();
-        Map<Group, Double> expectedRates = new HashMap<>();
         TimeValue unit = TimeValue.timeValueSeconds(1);
+        List<List<Object>> expected = new ArrayList<>();
         for (int p = 0; p < numPods; p++) {
-            int numValues = between(2, 100);
-            long[] values = new long[numValues];
-            long[] times = new long[numValues];
-            long t = DEFAULT_DATE_TIME_FORMATTER.parseMillis("2024-01-01T00:00:00Z");
-            for (int i = 0; i < numValues; i++) {
-                values[i] = randomIntBetween(0, 100);
-                t += TimeValue.timeValueSeconds(between(1, 10)).millis();
-                times[i] = t;
+            int numIntervals = randomIntBetween(1, 3);
+            Interval[] intervals = new Interval[numIntervals];
+            long startTimeInHours = between(10, 100);
+            String podName = "p" + p;
+            for (int interval = 0; interval < numIntervals; interval++) {
+                final long startInterval = TimeValue.timeValueHours(--startTimeInHours).millis();
+                int numValues = between(2, 100);
+                long[] values = new long[numValues];
+                long[] times = new long[numValues];
+                long delta = 0;
+                for (int i = 0; i < numValues; i++) {
+                    values[i] = randomIntBetween(0, 100);
+                    delta += TimeValue.timeValueSeconds(between(1, 10)).millis();
+                    times[i] = delta;
+                }
+                intervals[interval] = new Interval(startInterval, times, values);
+                if (numValues == 1) {
+                    expected.add(List.of(new BytesRef(podName), startInterval, null));
+                } else {
+                    expected.add(List.of(new BytesRef(podName), startInterval, intervals[interval].expectedRate(unit)));
+                }
             }
-            Pod pod = new Pod("p" + p, times, values);
+            Pod pod = new Pod(podName, "cluster", intervals);
             pods.add(pod);
-            if (numValues == 1) {
-                expectedRates.put(new Group("\u0001\u0003pods\u0002" + pod.name, 0), null);
-            } else {
-                expectedRates.put(new Group("\u0001\u0003pods\u0002" + pod.name, 0), pod.expectedRate(unit));
-            }
         }
-        Map<Group, Double> actualRates = runRateTest(pods, unit, TimeValue.ZERO);
-        assertThat(actualRates, equalTo(expectedRates));
+        List<List<Object>> actual = runRateTest(pods, List.of("pod", "bucket"), unit, TimeValue.timeValueHours(1));
+        assertThat(actual, equalTo(expected));
     }
 
-    record Pod(String name, long[] times, long[] values) {
-        Pod {
-            assert times.length == values.length : times.length + "!=" + values.length;
-        }
-
+    record Interval(long offset, long[] times, long[] values) {
         double expectedRate(TimeValue unit) {
             double dv = 0;
-            for (int i = 0; i < values.length - 1; i++) {
-                if (values[i + 1] < values[i]) {
-                    dv += values[i];
+            for (int v = 0; v < values.length - 1; v++) {
+                if (values[v + 1] < values[v]) {
+                    dv += values[v];
                 }
             }
             dv += (values[values.length - 1] - values[0]);
@@ -166,9 +188,13 @@ public class TimeSeriesAggregationOperatorTests extends AnyOperatorTestCase {
         }
     }
 
-    Map<Group, Double> runRateTest(List<Pod> pods, TimeValue unit, TimeValue interval) {
+    record Pod(String name, String cluster, Interval... intervals) {}
+
+    List<List<Object>> runRateTest(List<Pod> pods, List<String> groupings, TimeValue unit, TimeValue bucketInterval) throws IOException {
+        cleanup();
+        directory = newDirectory();
         long unitInMillis = unit.millis();
-        record Doc(String pod, long timestamp, long requests) {
+        record Doc(String pod, String cluster, long timestamp, long requests) {
 
         }
         var sourceOperatorFactory = createTimeSeriesSourceOperator(
@@ -177,70 +203,114 @@ public class TimeSeriesAggregationOperatorTests extends AnyOperatorTestCase {
             Integer.MAX_VALUE,
             between(1, 100),
             randomBoolean(),
-            interval,
+            bucketInterval,
             writer -> {
                 List<Doc> docs = new ArrayList<>();
                 for (Pod pod : pods) {
-                    for (int i = 0; i < pod.times.length; i++) {
-                        docs.add(new Doc(pod.name, pod.times[i], pod.values[i]));
+                    for (Interval interval : pod.intervals) {
+                        for (int i = 0; i < interval.times.length; i++) {
+                            docs.add(new Doc(pod.name, pod.cluster, interval.offset + interval.times[i], interval.values[i]));
+                        }
                     }
                 }
                 Randomness.shuffle(docs);
                 for (Doc doc : docs) {
-                    writeTS(writer, doc.timestamp, new Object[] { "pod", doc.pod }, new Object[] { "requests", doc.requests });
+                    writeTS(
+                        writer,
+                        doc.timestamp,
+                        new Object[] { "pod", doc.pod, "cluster", doc.cluster },
+                        new Object[] { "requests", doc.requests }
+                    );
                 }
                 return docs.size();
             }
         );
         var ctx = driverContext();
 
-        var aggregators = List.of(
-            new RateLongAggregatorFunctionSupplier(List.of(4, 2), unitInMillis).groupingAggregatorFactory(AggregatorMode.INITIAL)
-        );
-        Operator initialHash = new TimeSeriesAggregationOperatorFactory(
-            AggregatorMode.INITIAL,
+        List<Operator> extractOperators = new ArrayList<>();
+        var rateField = new NumberFieldMapper.NumberFieldType("requests", NumberFieldMapper.NumberType.LONG);
+        Operator extractRate = (ValuesSourceReaderOperatorTests.factory(reader, rateField, ElementType.LONG).get(ctx));
+        extractOperators.add(extractRate);
+        List<String> nonBucketGroupings = new ArrayList<>(groupings);
+        nonBucketGroupings.remove("bucket");
+        for (String grouping : nonBucketGroupings) {
+            var groupingField = new KeywordFieldMapper.KeywordFieldType(grouping);
+            extractOperators.add(ValuesSourceReaderOperatorTests.factory(reader, groupingField, ElementType.BYTES_REF).get(ctx));
+        }
+        // _doc, tsid, timestamp, bucket, requests, grouping1, grouping2
+        Operator intialAgg = new TimeSeriesAggregationOperatorFactories.Initial(
             1,
             3,
-            interval,
-            aggregators,
-            randomIntBetween(1, 1000)
+            IntStream.range(0, nonBucketGroupings.size()).mapToObj(n -> new BlockHash.GroupSpec(5 + n, ElementType.BYTES_REF)).toList(),
+            List.of(new RateLongAggregatorFunctionSupplier(List.of(4, 2), unitInMillis)),
+            List.of(),
+            between(1, 100)
         ).get(ctx);
 
-        aggregators = List.of(
-            new RateLongAggregatorFunctionSupplier(List.of(2, 3, 4), unitInMillis).groupingAggregatorFactory(AggregatorMode.FINAL)
-        );
-        Operator finalHash = new TimeSeriesAggregationOperatorFactory(
-            AggregatorMode.FINAL,
+        // tsid, bucket, rate[0][0],rate[0][1],rate[0][2], grouping1, grouping2
+        Operator intermediateAgg = new TimeSeriesAggregationOperatorFactories.Intermediate(
             0,
             1,
-            interval,
-            aggregators,
-            randomIntBetween(1, 1000)
+            IntStream.range(0, nonBucketGroupings.size()).mapToObj(n -> new BlockHash.GroupSpec(5 + n, ElementType.BYTES_REF)).toList(),
+            List.of(new RateLongAggregatorFunctionSupplier(List.of(2, 3, 4), unitInMillis)),
+            List.of(),
+            between(1, 100)
         ).get(ctx);
+        // tsid, bucket, rate, grouping1, grouping2
+        List<BlockHash.GroupSpec> finalGroups = new ArrayList<>();
+        int groupChannel = 3;
+        for (String grouping : groupings) {
+            if (grouping.equals("bucket")) {
+                finalGroups.add(new BlockHash.GroupSpec(1, ElementType.LONG));
+            } else {
+                finalGroups.add(new BlockHash.GroupSpec(groupChannel++, ElementType.BYTES_REF));
+            }
+        }
+        Operator finalAgg = new TimeSeriesAggregationOperatorFactories.Final(
+            finalGroups,
+            List.of(new SumDoubleAggregatorFunctionSupplier(List.of(2))),
+            List.of(),
+            between(1, 100)
+        ).get(ctx);
+
         List<Page> results = new ArrayList<>();
-        var requestsField = new NumberFieldMapper.NumberFieldType("requests", NumberFieldMapper.NumberType.LONG);
         OperatorTestCase.runDriver(
             new Driver(
                 ctx,
                 sourceOperatorFactory.get(ctx),
-                List.of(ValuesSourceReaderOperatorTests.factory(reader, requestsField, ElementType.LONG).get(ctx), initialHash, finalHash),
+                CollectionUtils.concatLists(extractOperators, List.of(intialAgg, intermediateAgg, finalAgg)),
                 new TestResultPageSinkOperator(results::add),
                 () -> {}
             )
         );
-        Map<Group, Double> rates = new HashMap<>();
+        List<List<Object>> values = new ArrayList<>();
         for (Page result : results) {
-            BytesRefBlock keysBlock = result.getBlock(0);
-            LongBlock timestampIntervalsBock = result.getBlock(1);
-            DoubleBlock ratesBlock = result.getBlock(2);
-            for (int i = 0; i < result.getPositionCount(); i++) {
-                var key = new Group(keysBlock.getBytesRef(i, new BytesRef()).utf8ToString(), timestampIntervalsBock.getLong(i));
-                rates.put(key, ratesBlock.getDouble(i));
+            for (int p = 0; p < result.getPositionCount(); p++) {
+                int blockCount = result.getBlockCount();
+                List<Object> row = new ArrayList<>();
+                for (int b = 0; b < blockCount; b++) {
+                    row.add(BlockUtils.toJavaObject(result.getBlock(b), p));
+                }
+                values.add(row);
             }
             result.releaseBlocks();
         }
-        return rates;
+        values.sort((v1, v2) -> {
+            for (int i = 0; i < v1.size(); i++) {
+                if (v1.get(i) instanceof BytesRef b1) {
+                    int cmp = b1.compareTo((BytesRef) v2.get(i));
+                    if (cmp != 0) {
+                        return cmp;
+                    }
+                } else if (v1.get(i) instanceof Long b1) {
+                    int cmp = b1.compareTo((Long) v2.get(i));
+                    if (cmp != 0) {
+                        return -cmp;
+                    }
+                }
+            }
+            return 0;
+        });
+        return values;
     }
-
-    record Group(String tsidHash, long timestampInterval) {}
 }

x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/exchange/ExchangeServiceTests.java

@@ -55,6 +55,7 @@ import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 import java.util.Set;
+import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Function;
 import java.util.function.Supplier;
@@ -94,6 +95,8 @@ public class ExchangeServiceTests extends ESTestCase {
         ExchangeSink sink1 = sinkExchanger.createExchangeSink();
         ExchangeSink sink2 = sinkExchanger.createExchangeSink();
         ExchangeSourceHandler sourceExchanger = new ExchangeSourceHandler(3, threadPool.executor(ESQL_TEST_EXECUTOR));
+        PlainActionFuture<Void> sourceCompletion = new PlainActionFuture<>();
+        sourceExchanger.addCompletionListener(sourceCompletion);
         ExchangeSource source = sourceExchanger.createExchangeSource();
         sourceExchanger.addRemoteSink(sinkExchanger::fetchPageAsync, 1);
         SubscribableListener<Void> waitForReading = source.waitForReading();
@@ -133,7 +136,9 @@ public class ExchangeServiceTests extends ESTestCase {
         sink2.finish();
         assertTrue(sink2.isFinished());
         assertTrue(source.isFinished());
+        assertFalse(sourceCompletion.isDone());
         source.finish();
+        sourceCompletion.actionGet(10, TimeUnit.SECONDS);
         ESTestCase.terminate(threadPool);
         for (Page page : pages) {
             page.releaseBlocks();
@@ -320,7 +325,9 @@ public class ExchangeServiceTests extends ESTestCase {
 
     public void testConcurrentWithHandlers() {
         BlockFactory blockFactory = blockFactory();
+        PlainActionFuture<Void> sourceCompletionFuture = new PlainActionFuture<>();
         var sourceExchanger = new ExchangeSourceHandler(randomExchangeBuffer(), threadPool.executor(ESQL_TEST_EXECUTOR));
+        sourceExchanger.addCompletionListener(sourceCompletionFuture);
         List<ExchangeSinkHandler> sinkHandlers = new ArrayList<>();
         Supplier<ExchangeSink> exchangeSink = () -> {
             final ExchangeSinkHandler sinkHandler;
@@ -336,6 +343,7 @@ public class ExchangeServiceTests extends ESTestCase {
         final int maxInputSeqNo = rarely() ? -1 : randomIntBetween(0, 50_000);
         final int maxOutputSeqNo = rarely() ? -1 : randomIntBetween(0, 50_000);
         runConcurrentTest(maxInputSeqNo, maxOutputSeqNo, sourceExchanger::createExchangeSource, exchangeSink);
+        sourceCompletionFuture.actionGet(10, TimeUnit.SECONDS);
     }
 
     public void testEarlyTerminate() {
@@ -358,7 +366,7 @@ public class ExchangeServiceTests extends ESTestCase {
         assertTrue(sink.isFinished());
     }
 
-    public void testConcurrentWithTransportActions() throws Exception {
+    public void testConcurrentWithTransportActions() {
         MockTransportService node0 = newTransportService();
         ExchangeService exchange0 = new ExchangeService(Settings.EMPTY, threadPool, ESQL_TEST_EXECUTOR, blockFactory());
         exchange0.registerTransportHandler(node0);
@@ -371,12 +379,15 @@ public class ExchangeServiceTests extends ESTestCase {
             String exchangeId = "exchange";
             Task task = new Task(1, "", "", "", null, Collections.emptyMap());
             var sourceHandler = new ExchangeSourceHandler(randomExchangeBuffer(), threadPool.executor(ESQL_TEST_EXECUTOR));
+            PlainActionFuture<Void> sourceCompletionFuture = new PlainActionFuture<>();
+            sourceHandler.addCompletionListener(sourceCompletionFuture);
             ExchangeSinkHandler sinkHandler = exchange1.createSinkHandler(exchangeId, randomExchangeBuffer());
             Transport.Connection connection = node0.getConnection(node1.getLocalNode());
             sourceHandler.addRemoteSink(exchange0.newRemoteSink(task, exchangeId, node0, connection), randomIntBetween(1, 5));
             final int maxInputSeqNo = rarely() ? -1 : randomIntBetween(0, 50_000);
             final int maxOutputSeqNo = rarely() ? -1 : randomIntBetween(0, 50_000);
             runConcurrentTest(maxInputSeqNo, maxOutputSeqNo, sourceHandler::createExchangeSource, sinkHandler::createExchangeSink);
+            sourceCompletionFuture.actionGet(10, TimeUnit.SECONDS);
         }
     }
 
@@ -427,6 +438,8 @@ public class ExchangeServiceTests extends ESTestCase {
             String exchangeId = "exchange";
             Task task = new Task(1, "", "", "", null, Collections.emptyMap());
             var sourceHandler = new ExchangeSourceHandler(randomIntBetween(1, 128), threadPool.executor(ESQL_TEST_EXECUTOR));
+            PlainActionFuture<Void> sourceCompletionFuture = new PlainActionFuture<>();
+            sourceHandler.addCompletionListener(sourceCompletionFuture);
             ExchangeSinkHandler sinkHandler = exchange1.createSinkHandler(exchangeId, randomIntBetween(1, 128));
             Transport.Connection connection = node0.getConnection(node1.getLocalDiscoNode());
             sourceHandler.addRemoteSink(exchange0.newRemoteSink(task, exchangeId, node0, connection), randomIntBetween(1, 5));
@@ -438,6 +451,7 @@ public class ExchangeServiceTests extends ESTestCase {
             assertNotNull(cause);
             assertThat(cause.getMessage(), equalTo("page is too large"));
             sinkHandler.onFailure(new RuntimeException(cause));
+            sourceCompletionFuture.actionGet(10, TimeUnit.SECONDS);
         }
     }
 

x-pack/plugin/esql/qa/testFixtures/src/main/resources/blog.csv-spec

@@ -1,4 +1,7 @@
+# Examples that were published in a blog post
+
 2023-08-08.full-blown-query
+required_feature: esql.enrich_load
 
   FROM employees
 | WHERE still_hired == true

x-pack/plugin/esql/qa/testFixtures/src/main/resources/date.csv-spec

@@ -621,6 +621,40 @@ dt:datetime              |plus_post:datetime |plus_pre:datetime
 2100-01-01T01:01:01.001Z |null               |null
 ;
 
+datePlusQuarter
+# "quarter" introduced in 8.15
+required_feature: esql.timespan_abbreviations
+row dt = to_dt("2100-01-01T01:01:01.000Z")
+| eval plusQuarter = dt + 2 quarters
+;
+
+dt:datetime              | plusQuarter:datetime
+2100-01-01T01:01:01.000Z | 2100-07-01T01:01:01.000Z
+;
+
+datePlusAbbreviatedDurations
+# abbreviations introduced in 8.15
+required_feature: esql.timespan_abbreviations
+row dt = to_dt("2100-01-01T00:00:00.000Z")
+| eval plusDurations = dt + 1 h + 2 min + 2 sec + 1 s + 4 ms
+;
+
+dt:datetime              | plusDurations:datetime
+2100-01-01T00:00:00.000Z | 2100-01-01T01:02:03.004Z
+;
+
+datePlusAbbreviatedPeriods
+# abbreviations introduced in 8.15
+required_feature: esql.timespan_abbreviations
+row dt = to_dt("2100-01-01T00:00:00.000Z")
+| eval plusDurations = dt + 0 yr + 1y + 2 q + 3 mo + 4 w + 3 d
+;
+
+dt:datetime              | plusDurations:datetime
+2100-01-01T00:00:00.000Z | 2101-11-01T00:00:00.000Z
+;
+
+
 dateMinusDuration
 row dt = to_dt("2100-01-01T01:01:01.001Z")
 | eval minus = dt - 1 hour - 1 minute - 1 second - 1 milliseconds;

x-pack/plugin/esql/qa/testFixtures/src/main/resources/enrich-IT_tests_only.csv-spec

@@ -1,350 +0,0 @@
-simple
-row language_code = "1"  
-| enrich languages_policy
-;
-
-language_code:keyword  | language_name:keyword 
-1                      | English 
-;
-
-
-enrichOn
-from employees  | sort emp_no | limit 1 | eval x = to_string(languages) | enrich languages_policy on x  | keep emp_no, language_name;
-
-emp_no:integer | language_name:keyword
-10001          | French
-;
-
-
-enrichOn2
-from employees | eval x = to_string(languages) | enrich languages_policy on x  | keep emp_no, language_name | sort emp_no | limit 1 ;
-
-emp_no:integer | language_name:keyword
-10001          | French
-;
-
-simpleSortLimit
-from employees  | eval x = to_string(languages) | enrich languages_policy on x  | keep emp_no, language_name | sort emp_no | limit 1;
-
-emp_no:integer | language_name:keyword
-10001          | French
-;
-
-
-with
-from employees | eval x = to_string(languages) | keep emp_no, x | sort emp_no | limit 1 
-| enrich languages_policy on x with language_name;
-
-emp_no:integer | x:keyword | language_name:keyword
-10001          | 2         | French
-;
-
-
-withAlias
-from employees  | sort emp_no | limit 3 | eval x = to_string(languages) | keep emp_no, x 
-| enrich languages_policy on x with lang = language_name;
-
-emp_no:integer | x:keyword | lang:keyword
-10001          | 2         | French
-10002          | 5         | null
-10003          | 4         | German
-;
-
-
-withAliasSort
-from employees | eval x = to_string(languages) | keep emp_no, x  | sort emp_no | limit 3 
-| enrich languages_policy on x with lang = language_name;
-
-emp_no:integer | x:keyword | lang:keyword
-10001          | 2         | French
-10002          | 5         | null
-10003          | 4         | German
-;
-
-
-withAliasOverwriteName#[skip:-8.13.0]
-from employees | sort emp_no
-| eval x = to_string(languages) | enrich languages_policy on x with emp_no = language_name
-| keep emp_no | limit 1
-;
-
-emp_no:keyword
-French
-;
-
-
-withAliasAndPlain
-from employees  | sort emp_no desc | limit 3 | eval x = to_string(languages) | keep emp_no, x 
-| enrich languages_policy on x with lang = language_name, language_name;
-
-emp_no:integer | x:keyword | lang:keyword | language_name:keyword
-10100          | 4         | German       | German
-10099          | 2         | French       | French
-10098          | 4         | German       | German
-;
-
-
-withTwoAliasesSameProp
-from employees  | sort emp_no | limit 1 | eval x = to_string(languages) | keep emp_no, x 
-| enrich languages_policy on x with lang = language_name, lang2 = language_name;
-
-emp_no:integer | x:keyword | lang:keyword | lang2:keyword
-10001          | 2         | French       | French
-;
-
-
-redundantWith
-from employees  | sort emp_no | limit 1 | eval x = to_string(languages) | keep emp_no, x 
-| enrich languages_policy on x with language_name, language_name;
-
-emp_no:integer | x:keyword | language_name:keyword 
-10001          | 2         | French
-;
-
-
-nullInput
-from employees  | where emp_no == 10017 | keep emp_no, gender
-| enrich languages_policy on gender with language_name, language_name;
-
-emp_no:integer | gender:keyword | language_name:keyword 
-10017          | null           | null
-;
-
- 
-constantNullInput
-from employees  | where emp_no == 10020 | eval x = to_string(languages) | keep emp_no, x 
-| enrich languages_policy on x with language_name, language_name;
-
-emp_no:integer | x:keyword | language_name:keyword 
-10020          | null      | null
-;
-
-
-multipleEnrich
-row a = "1", b = "2", c = "10"  
-| enrich languages_policy on a with a_lang = language_name  
-| enrich languages_policy on b with b_lang = language_name  
-| enrich languages_policy on c with c_lang = language_name;
-
-a:keyword | b:keyword | c:keyword | a_lang:keyword | b_lang:keyword | c_lang:keyword
-1         | 2         | 10        | English        | French         | null
-;
-
-
-enrichEval
-from employees | eval x = to_string(languages) 
-| enrich languages_policy on x with lang = language_name
-| eval language = concat(x, "-", lang)
-| keep emp_no, x, lang, language 
-| sort emp_no desc | limit 3;
-
-emp_no:integer | x:keyword | lang:keyword | language:keyword
-10100          | 4         | German       | 4-German
-10099          | 2         | French       | 2-French
-10098          | 4         | German       | 4-German
-;
-
-
-multivalue
-required_feature: esql.mv_sort
-row a = ["1", "2"] | enrich languages_policy on a with a_lang = language_name | eval a_lang = mv_sort(a_lang);  
-
-a:keyword   | a_lang:keyword
-["1", "2"]  | ["English", "French"] 
-;
-
-
-enrichCidr#[skip:-8.13.99, reason:enrich for cidr added in 8.14.0]
-FROM sample_data
-| ENRICH client_cidr_policy ON client_ip WITH env
-| EVAL max_env = MV_MAX(env), count_env = MV_COUNT(env)
-| KEEP client_ip, count_env, max_env
-| SORT client_ip
-;
-
-client_ip:ip | count_env:i | max_env:keyword
-172.21.0.5   | 1           | Development
-172.21.2.113 | 2           | QA
-172.21.2.162 | 2           | QA
-172.21.3.15  | 2           | Production
-172.21.3.15  | 2           | Production
-172.21.3.15  | 2           | Production
-172.21.3.15  | 2           | Production
-;
-
-
-enrichCidr2#[skip:-8.99.99, reason:ip_range support not added yet]
-FROM sample_data
-| ENRICH client_cidr_policy ON client_ip WITH env, client_cidr
-| KEEP client_ip, env, client_cidr
-| SORT client_ip
-;
-
-client_ip:ip | env:keyword               | client_cidr:ip_range
-172.21.3.15  | [Development, Production] | 172.21.3.0/24
-172.21.3.15  | [Development, Production] | 172.21.3.0/24
-172.21.3.15  | [Development, Production] | 172.21.3.0/24
-172.21.3.15  | [Development, Production] | 172.21.3.0/24
-172.21.0.5   | Development               | 172.21.0.0/16
-172.21.2.113 | [Development, QA]         | 172.21.2.0/24
-172.21.2.162 | [Development, QA]         | 172.21.2.0/24
-;
-
-
-enrichAgesStatsYear#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM employees
-| WHERE birth_date > "1960-01-01"
-| EVAL birth_year = DATE_EXTRACT("year", birth_date)
-| EVAL age = 2022 - birth_year
-| ENRICH ages_policy ON age WITH age_group = description
-| STATS count=count(age_group) BY age_group, birth_year
-| KEEP birth_year, age_group, count
-| SORT birth_year DESC
-;
-
-birth_year:long | age_group:keyword | count:long
-1965            | Middle-aged       | 1
-1964            | Middle-aged       | 4
-1963            | Middle-aged       | 7
-1962            | Senior            | 6
-1961            | Senior            | 8
-1960            | Senior            | 8
-;
-
-
-enrichAgesStatsAgeGroup#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM employees
-| WHERE birth_date IS NOT NULL
-| EVAL age = 2022 - DATE_EXTRACT("year", birth_date)
-| ENRICH ages_policy ON age WITH age_group = description
-| STATS count=count(age_group) BY age_group
-| SORT count DESC
-;
-
-count:long | age_group:keyword
-78         | Senior
-12         | Middle-aged
-;
-
-
-enrichHeightsStats#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM employees
-| ENRICH heights_policy ON height WITH height_group = description
-| STATS count=count(height_group), min=min(height), max=max(height) BY height_group
-| KEEP height_group, min, max, count
-| SORT min ASC
-;
-
-height_group:k | min:double | max:double | count:long
-Very Short     | 1.41       | 1.48       | 9
-Short          | 1.5        | 1.59       | 20
-Medium Height  | 1.61       | 1.79       | 26
-Tall           | 1.8        | 1.99       | 25
-Very Tall      | 2.0        | 2.1        | 20
-;
-
-
-enrichDecadesStats#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM employees
-| ENRICH decades_policy ON birth_date WITH birth_decade = decade, birth_description = description
-| ENRICH decades_policy ON hire_date WITH hire_decade = decade, hire_description = description
-| STATS count=count(*) BY birth_decade, hire_decade, birth_description, hire_description
-| KEEP birth_decade, hire_decade, birth_description, hire_description, count
-| SORT birth_decade DESC, hire_decade DESC
-;
-
-birth_decade:long | hire_decade:l | birth_description:k | hire_description:k | count:long
-null              | 1990          | null                | Nineties Nostalgia | 6
-null              | 1980          | null                | Radical Eighties   | 4
-1960              | 1990          | Swinging Sixties    | Nineties Nostalgia | 13
-1960              | 1980          | Swinging Sixties    | Radical Eighties   | 21
-1950              | 1990          | Nifty Fifties       | Nineties Nostalgia | 22
-1950              | 1980          | Nifty Fifties       | Radical Eighties   | 34
-;
-
-
-spatialEnrichmentKeywordMatch#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM airports
-| WHERE abbrev == "CPH"
-| ENRICH city_names ON city WITH airport, region, city_boundary
-| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
-| KEEP abbrev, city, city_location, country, location, name, airport, region, boundary_wkt_length
-;
-
-abbrev:keyword  |  city:keyword  |  city_location:geo_point |  country:keyword  |  location:geo_point                       |  name:text   |  airport:text  |  region:text         |  boundary_wkt_length:integer
-CPH             |  Copenhagen    |  POINT(12.5683 55.6761)  |  Denmark          |  POINT(12.6493508684508 55.6285017221528) |  Copenhagen  |  Copenhagen    |  Københavns Kommune  |  265
-;
-
-
-spatialEnrichmentGeoMatch#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM airports
-| WHERE abbrev == "CPH"
-| ENRICH city_boundaries ON city_location WITH airport, region, city_boundary
-| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
-| KEEP abbrev, city, city_location, country, location, name, airport, region, boundary_wkt_length
-;
-
-abbrev:keyword  |  city:keyword  |  city_location:geo_point |  country:keyword  |  location:geo_point                       |  name:text   |  airport:text  |  region:text         |  boundary_wkt_length:integer
-CPH             |  Copenhagen    |  POINT(12.5683 55.6761)  |  Denmark          |  POINT(12.6493508684508 55.6285017221528) |  Copenhagen  |  Copenhagen    |  Københavns Kommune  |  265
-;
-
-
-spatialEnrichmentGeoMatchStats#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-required_feature: esql.mv_warn
-
-FROM airports
-| ENRICH city_boundaries ON city_location WITH airport, region, city_boundary
-| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
-| STATS city_centroid = ST_CENTROID_AGG(city_location), count = COUNT(city_location), min_wkt = MIN(boundary_wkt_length), max_wkt = MAX(boundary_wkt_length)
-;
-warning:Line 3:30: evaluation of [LENGTH(TO_STRING(city_boundary))] failed, treating result as null. Only first 20 failures recorded.
-warning:Line 3:30: java.lang.IllegalArgumentException: single-value function encountered multi-value
-
-city_centroid:geo_point    |  count:long  |  min_wkt:integer  |  max_wkt:integer
-POINT(1.396561 24.127649)  |  872         |  88               |  1044
-;
-
-
-spatialEnrichmentKeywordMatchAndSpatialPredicate#[skip:-8.13.99, reason:st_intersects added in 8.14]
-FROM airports
-| ENRICH city_names ON city WITH airport, region, city_boundary
-| MV_EXPAND city_boundary
-| EVAL airport_in_city = ST_INTERSECTS(location, city_boundary)
-| STATS count=COUNT(*) BY airport_in_city
-| SORT count ASC
-;
-
-count:long  |  airport_in_city:boolean
-114         |  null
-396         |  true
-455         |  false
-;
-
-
-spatialEnrichmentKeywordMatchAndSpatialAggregation#[skip:-8.13.99, reason:st_intersects added in 8.14]
-FROM airports
-| ENRICH city_names ON city WITH airport, region, city_boundary
-| MV_EXPAND city_boundary
-| EVAL airport_in_city = ST_INTERSECTS(location, city_boundary)
-| STATS count=COUNT(*), centroid=ST_CENTROID_AGG(location) BY airport_in_city
-| SORT count ASC
-;
-
-count:long  |  centroid:geo_point            |  airport_in_city:boolean
-114         |  POINT (-24.750062 31.575549)  |  null
-396         |  POINT (-2.534797 20.667712)   |  true
-455         |  POINT (3.090752 27.676442)    |  false
-;
-
-
-spatialEnrichmentTextMatch#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
-FROM airports
-| WHERE abbrev == "IDR"
-| ENRICH city_airports ON name WITH city_name = city, region, city_boundary
-| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
-| KEEP abbrev, city_name, city_location, country, location, name, name, region, boundary_wkt_length
-;
-
-abbrev:k  |  city_name:k  |  city_location:geo_point |  country:k  |  location:geo_point                       |  name:text                    |  region:text  |  boundary_wkt_length:i
-IDR       |  Indore       |  POINT(75.8472 22.7167)  |  India      |  POINT(75.8092915005895 22.727749187571)  |  Devi Ahilyabai Holkar Int'l  |  Indore City  |  231
-;

x-pack/plugin/esql/qa/testFixtures/src/main/resources/enrich.csv-spec

@@ -1,10 +1,10 @@
-simple
+simpleNoLoad
 from employees | eval x = 1, y = to_string(languages) | enrich languages_policy on y | where x > 1 | keep emp_no, language_name | limit 1;
 
 emp_no:integer | language_name:keyword
 ;
 
-docsGettingStartedEnrich
+docsGettingStartedEnrichNoLoad
 // tag::gs-enrich[]
 FROM sample_data
 | KEEP @timestamp, client_ip, event_duration
@@ -30,3 +30,408 @@ FROM sample_data
 
 median_duration:double | env:keyword
 ;
+
+simple
+required_feature: esql.enrich_load
+
+row language_code = "1"  
+| enrich languages_policy
+;
+
+language_code:keyword  | language_name:keyword 
+1                      | English 
+;
+
+
+enrichOn
+required_feature: esql.enrich_load
+
+from employees  | sort emp_no | limit 1 | eval x = to_string(languages) | enrich languages_policy on x  | keep emp_no, language_name;
+
+emp_no:integer | language_name:keyword
+10001          | French
+;
+
+
+enrichOn2
+required_feature: esql.enrich_load
+
+from employees | eval x = to_string(languages) | enrich languages_policy on x  | keep emp_no, language_name | sort emp_no | limit 1 ;
+
+emp_no:integer | language_name:keyword
+10001          | French
+;
+
+
+simpleSortLimit
+required_feature: esql.enrich_load
+
+from employees  | eval x = to_string(languages) | enrich languages_policy on x  | keep emp_no, language_name | sort emp_no | limit 1;
+
+emp_no:integer | language_name:keyword
+10001          | French
+;
+
+with
+required_feature: esql.enrich_load
+
+from employees | eval x = to_string(languages) | keep emp_no, x | sort emp_no | limit 1 
+| enrich languages_policy on x with language_name;
+
+emp_no:integer | x:keyword | language_name:keyword
+10001          | 2         | French
+;
+
+
+withAlias
+required_feature: esql.enrich_load
+
+from employees  | sort emp_no | limit 3 | eval x = to_string(languages) | keep emp_no, x 
+| enrich languages_policy on x with lang = language_name;
+
+emp_no:integer | x:keyword | lang:keyword
+10001          | 2         | French
+10002          | 5         | null
+10003          | 4         | German
+;
+
+
+withAliasSort
+required_feature: esql.enrich_load
+
+from employees | eval x = to_string(languages) | keep emp_no, x  | sort emp_no | limit 3 
+| enrich languages_policy on x with lang = language_name;
+
+emp_no:integer | x:keyword | lang:keyword
+10001          | 2         | French
+10002          | 5         | null
+10003          | 4         | German
+;
+
+
+withAliasOverwriteName#[skip:-8.13.0]
+required_feature: esql.enrich_load
+
+from employees | sort emp_no
+| eval x = to_string(languages) | enrich languages_policy on x with emp_no = language_name
+| keep emp_no | limit 1
+;
+
+emp_no:keyword
+French
+;
+
+withAliasAndPlain
+required_feature: esql.enrich_load
+
+from employees  | sort emp_no desc | limit 3 | eval x = to_string(languages) | keep emp_no, x 
+| enrich languages_policy on x with lang = language_name, language_name;
+
+emp_no:integer | x:keyword | lang:keyword | language_name:keyword
+10100          | 4         | German       | German
+10099          | 2         | French       | French
+10098          | 4         | German       | German
+;
+
+
+withTwoAliasesSameProp
+required_feature: esql.enrich_load
+
+from employees  | sort emp_no | limit 1 | eval x = to_string(languages) | keep emp_no, x 
+| enrich languages_policy on x with lang = language_name, lang2 = language_name;
+
+emp_no:integer | x:keyword | lang:keyword | lang2:keyword
+10001          | 2         | French       | French
+;
+
+
+redundantWith
+required_feature: esql.enrich_load
+
+from employees  | sort emp_no | limit 1 | eval x = to_string(languages) | keep emp_no, x 
+| enrich languages_policy on x with language_name, language_name;
+
+emp_no:integer | x:keyword | language_name:keyword 
+10001          | 2         | French
+;
+
+
+nullInput
+required_feature: esql.enrich_load
+
+from employees  | where emp_no == 10017 | keep emp_no, gender
+| enrich languages_policy on gender with language_name, language_name;
+
+emp_no:integer | gender:keyword | language_name:keyword 
+10017          | null           | null
+;
+
+ 
+constantNullInput
+required_feature: esql.enrich_load
+
+from employees  | where emp_no == 10020 | eval x = to_string(languages) | keep emp_no, x 
+| enrich languages_policy on x with language_name, language_name;
+
+emp_no:integer | x:keyword | language_name:keyword 
+10020          | null      | null
+;
+
+
+multipleEnrich
+required_feature: esql.enrich_load
+
+row a = "1", b = "2", c = "10"  
+| enrich languages_policy on a with a_lang = language_name  
+| enrich languages_policy on b with b_lang = language_name  
+| enrich languages_policy on c with c_lang = language_name;
+
+a:keyword | b:keyword | c:keyword | a_lang:keyword | b_lang:keyword | c_lang:keyword
+1         | 2         | 10        | English        | French         | null
+;
+
+
+enrichEval
+required_feature: esql.enrich_load
+
+from employees | eval x = to_string(languages) 
+| enrich languages_policy on x with lang = language_name
+| eval language = concat(x, "-", lang)
+| keep emp_no, x, lang, language 
+| sort emp_no desc | limit 3;
+
+emp_no:integer | x:keyword | lang:keyword | language:keyword
+10100          | 4         | German       | 4-German
+10099          | 2         | French       | 2-French
+10098          | 4         | German       | 4-German
+;
+
+
+multivalue
+required_feature: esql.enrich_load
+required_feature: esql.mv_sort
+
+row a = ["1", "2"] | enrich languages_policy on a with a_lang = language_name | eval a_lang = mv_sort(a_lang);  
+
+a:keyword   | a_lang:keyword
+["1", "2"]  | ["English", "French"] 
+;
+
+
+enrichCidr#[skip:-8.13.99, reason:enrich for cidr added in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM sample_data
+| ENRICH client_cidr_policy ON client_ip WITH env
+| EVAL max_env = MV_MAX(env), count_env = MV_COUNT(env)
+| KEEP client_ip, count_env, max_env
+| SORT client_ip
+;
+
+client_ip:ip | count_env:i | max_env:keyword
+172.21.0.5   | 1           | Development
+172.21.2.113 | 2           | QA
+172.21.2.162 | 2           | QA
+172.21.3.15  | 2           | Production
+172.21.3.15  | 2           | Production
+172.21.3.15  | 2           | Production
+172.21.3.15  | 2           | Production
+;
+
+
+enrichCidr2#[skip:-8.99.99, reason:ip_range support not added yet]
+required_feature: esql.enrich_load
+
+FROM sample_data
+| ENRICH client_cidr_policy ON client_ip WITH env, client_cidr
+| KEEP client_ip, env, client_cidr
+| SORT client_ip
+;
+
+client_ip:ip | env:keyword               | client_cidr:ip_range
+172.21.3.15  | [Development, Production] | 172.21.3.0/24
+172.21.3.15  | [Development, Production] | 172.21.3.0/24
+172.21.3.15  | [Development, Production] | 172.21.3.0/24
+172.21.3.15  | [Development, Production] | 172.21.3.0/24
+172.21.0.5   | Development               | 172.21.0.0/16
+172.21.2.113 | [Development, QA]         | 172.21.2.0/24
+172.21.2.162 | [Development, QA]         | 172.21.2.0/24
+;
+
+
+enrichAgesStatsYear#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM employees
+| WHERE birth_date > "1960-01-01"
+| EVAL birth_year = DATE_EXTRACT("year", birth_date)
+| EVAL age = 2022 - birth_year
+| ENRICH ages_policy ON age WITH age_group = description
+| STATS count=count(age_group) BY age_group, birth_year
+| KEEP birth_year, age_group, count
+| SORT birth_year DESC
+;
+
+birth_year:long | age_group:keyword | count:long
+1965            | Middle-aged       | 1
+1964            | Middle-aged       | 4
+1963            | Middle-aged       | 7
+1962            | Senior            | 6
+1961            | Senior            | 8
+1960            | Senior            | 8
+;
+
+
+enrichAgesStatsAgeGroup#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM employees
+| WHERE birth_date IS NOT NULL
+| EVAL age = 2022 - DATE_EXTRACT("year", birth_date)
+| ENRICH ages_policy ON age WITH age_group = description
+| STATS count=count(age_group) BY age_group
+| SORT count DESC
+;
+
+count:long | age_group:keyword
+78         | Senior
+12         | Middle-aged
+;
+
+
+enrichHeightsStats#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM employees
+| ENRICH heights_policy ON height WITH height_group = description
+| STATS count=count(height_group), min=min(height), max=max(height) BY height_group
+| KEEP height_group, min, max, count
+| SORT min ASC
+;
+
+height_group:k | min:double | max:double | count:long
+Very Short     | 1.41       | 1.48       | 9
+Short          | 1.5        | 1.59       | 20
+Medium Height  | 1.61       | 1.79       | 26
+Tall           | 1.8        | 1.99       | 25
+Very Tall      | 2.0        | 2.1        | 20
+;
+
+
+enrichDecadesStats#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM employees
+| ENRICH decades_policy ON birth_date WITH birth_decade = decade, birth_description = description
+| ENRICH decades_policy ON hire_date WITH hire_decade = decade, hire_description = description
+| STATS count=count(*) BY birth_decade, hire_decade, birth_description, hire_description
+| KEEP birth_decade, hire_decade, birth_description, hire_description, count
+| SORT birth_decade DESC, hire_decade DESC
+;
+
+birth_decade:long | hire_decade:l | birth_description:k | hire_description:k | count:long
+null              | 1990          | null                | Nineties Nostalgia | 6
+null              | 1980          | null                | Radical Eighties   | 4
+1960              | 1990          | Swinging Sixties    | Nineties Nostalgia | 13
+1960              | 1980          | Swinging Sixties    | Radical Eighties   | 21
+1950              | 1990          | Nifty Fifties       | Nineties Nostalgia | 22
+1950              | 1980          | Nifty Fifties       | Radical Eighties   | 34
+;
+
+
+spatialEnrichmentKeywordMatch#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM airports
+| WHERE abbrev == "CPH"
+| ENRICH city_names ON city WITH airport, region, city_boundary
+| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
+| KEEP abbrev, city, city_location, country, location, name, airport, region, boundary_wkt_length
+;
+
+abbrev:keyword  |  city:keyword  |  city_location:geo_point |  country:keyword  |  location:geo_point                       |  name:text   |  airport:text  |  region:text         |  boundary_wkt_length:integer
+CPH             |  Copenhagen    |  POINT(12.5683 55.6761)  |  Denmark          |  POINT(12.6493508684508 55.6285017221528) |  Copenhagen  |  Copenhagen    |  Københavns Kommune  |  265
+;
+
+
+spatialEnrichmentGeoMatch#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM airports
+| WHERE abbrev == "CPH"
+| ENRICH city_boundaries ON city_location WITH airport, region, city_boundary
+| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
+| KEEP abbrev, city, city_location, country, location, name, airport, region, boundary_wkt_length
+;
+
+abbrev:keyword  |  city:keyword  |  city_location:geo_point |  country:keyword  |  location:geo_point                       |  name:text   |  airport:text  |  region:text         |  boundary_wkt_length:integer
+CPH             |  Copenhagen    |  POINT(12.5683 55.6761)  |  Denmark          |  POINT(12.6493508684508 55.6285017221528) |  Copenhagen  |  Copenhagen    |  Københavns Kommune  |  265
+;
+
+
+spatialEnrichmentGeoMatchStats#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+required_feature: esql.mv_warn
+
+FROM airports
+| ENRICH city_boundaries ON city_location WITH airport, region, city_boundary
+| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
+| STATS city_centroid = ST_CENTROID_AGG(city_location), count = COUNT(city_location), min_wkt = MIN(boundary_wkt_length), max_wkt = MAX(boundary_wkt_length)
+;
+warning:Line 3:30: evaluation of [LENGTH(TO_STRING(city_boundary))] failed, treating result as null. Only first 20 failures recorded.
+warning:Line 3:30: java.lang.IllegalArgumentException: single-value function encountered multi-value
+
+city_centroid:geo_point    |  count:long  |  min_wkt:integer  |  max_wkt:integer
+POINT(1.396561 24.127649)  |  872         |  88               |  1044
+;
+
+
+spatialEnrichmentKeywordMatchAndSpatialPredicate#[skip:-8.13.99, reason:st_intersects added in 8.14]
+required_feature: esql.enrich_load
+
+FROM airports
+| ENRICH city_names ON city WITH airport, region, city_boundary
+| MV_EXPAND city_boundary
+| EVAL airport_in_city = ST_INTERSECTS(location, city_boundary)
+| STATS count=COUNT(*) BY airport_in_city
+| SORT count ASC
+;
+
+count:long  |  airport_in_city:boolean
+114         |  null
+396         |  true
+455         |  false
+;
+
+
+spatialEnrichmentKeywordMatchAndSpatialAggregation#[skip:-8.13.99, reason:st_intersects added in 8.14]
+required_feature: esql.enrich_load
+
+FROM airports
+| ENRICH city_names ON city WITH airport, region, city_boundary
+| MV_EXPAND city_boundary
+| EVAL airport_in_city = ST_INTERSECTS(location, city_boundary)
+| STATS count=COUNT(*), centroid=ST_CENTROID_AGG(location) BY airport_in_city
+| SORT count ASC
+;
+
+count:long  |  centroid:geo_point            |  airport_in_city:boolean
+114         |  POINT (-24.750062 31.575549)  |  null
+396         |  POINT (-2.534797 20.667712)   |  true
+455         |  POINT (3.090752 27.676442)    |  false
+;
+
+
+spatialEnrichmentTextMatch#[skip:-8.13.99, reason:ENRICH extended in 8.14.0]
+required_feature: esql.enrich_load
+
+FROM airports
+| WHERE abbrev == "IDR"
+| ENRICH city_airports ON name WITH city_name = city, region, city_boundary
+| EVAL boundary_wkt_length = LENGTH(TO_STRING(city_boundary))
+| KEEP abbrev, city_name, city_location, country, location, name, name, region, boundary_wkt_length
+;
+
+abbrev:k  |  city_name:k  |  city_location:geo_point |  country:k  |  location:geo_point                       |  name:text                    |  region:text  |  boundary_wkt_length:i
+IDR       |  Indore       |  POINT(75.8472 22.7167)  |  India      |  POINT(75.8092915005895 22.727749187571)  |  Devi Ahilyabai Holkar Int'l  |  Indore City  |  231
+;

x-pack/plugin/esql/src/main/antlr/EsqlBaseLexer.g4

@@ -201,10 +201,6 @@ FROM_UNQUOTED_IDENTIFIER
     : FROM_UNQUOTED_IDENTIFIER_PART+
     ;
 
-FROM_QUOTED_IDENTIFIER
-    : QUOTED_IDENTIFIER -> type(QUOTED_IDENTIFIER)
-    ;
-
 FROM_LINE_COMMENT
     : LINE_COMMENT -> channel(HIDDEN)
     ;

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

@@ -109,7 +109,6 @@ fromCommand
 
 fromIdentifier
     : FROM_UNQUOTED_IDENTIFIER
-    | QUOTED_IDENTIFIER
     ;
 
 fromOptions

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java

@@ -2101,7 +2101,6 @@ public class EsqlBaseParser extends Parser {
   @SuppressWarnings("CheckReturnValue")
   public static class FromIdentifierContext extends ParserRuleContext {
     public TerminalNode FROM_UNQUOTED_IDENTIFIER() { return getToken(EsqlBaseParser.FROM_UNQUOTED_IDENTIFIER, 0); }
-    public TerminalNode QUOTED_IDENTIFIER() { return getToken(EsqlBaseParser.QUOTED_IDENTIFIER, 0); }
     @SuppressWarnings("this-escape")
     public FromIdentifierContext(ParserRuleContext parent, int invokingState) {
       super(parent, invokingState);
@@ -2125,20 +2124,11 @@ public class EsqlBaseParser extends Parser {
   public final FromIdentifierContext fromIdentifier() throws RecognitionException {
     FromIdentifierContext _localctx = new FromIdentifierContext(_ctx, getState());
     enterRule(_localctx, 32, RULE_fromIdentifier);
-    int _la;
     try {
       enterOuterAlt(_localctx, 1);
       {
       setState(296);
-      _la = _input.LA(1);
-      if ( !(_la==QUOTED_IDENTIFIER || _la==FROM_UNQUOTED_IDENTIFIER) ) {
-      _errHandler.recoverInline(this);
-      }
-      else {
-        if ( _input.LA(1)==Token.EOF ) matchedEOF = true;
-        _errHandler.reportMatch(this);
-        consume();
-      }
+      match(FROM_UNQUOTED_IDENTIFIER);
       }
     }
     catch (RecognitionException re) {
@@ -4971,32 +4961,32 @@ public class EsqlBaseParser extends Parser {
     "\u00015\u00015\u00035\u021b\b5\u00015\u00015\u00015\u0000\u0004\u0002"+
     "\n\u0010\u00126\u0000\u0002\u0004\u0006\b\n\f\u000e\u0010\u0012\u0014"+
     "\u0016\u0018\u001a\u001c\u001e \"$&(*,.02468:<>@BDFHJLNPRTVXZ\\^`bdfh"+
-    "j\u0000\b\u0001\u0000<=\u0001\u0000>@\u0002\u0000DDJJ\u0001\u0000CD\u0002"+
-    "\u0000  $$\u0001\u0000\'(\u0002\u0000&&44\u0002\u0000557;\u0238\u0000"+
-    "l\u0001\u0000\u0000\u0000\u0002o\u0001\u0000\u0000\u0000\u0004\u007f\u0001"+
-    "\u0000\u0000\u0000\u0006\u008e\u0001\u0000\u0000\u0000\b\u0090\u0001\u0000"+
-    "\u0000\u0000\n\u00af\u0001\u0000\u0000\u0000\f\u00ca\u0001\u0000\u0000"+
-    "\u0000\u000e\u00d1\u0001\u0000\u0000\u0000\u0010\u00d7\u0001\u0000\u0000"+
-    "\u0000\u0012\u00ec\u0001\u0000\u0000\u0000\u0014\u00f6\u0001\u0000\u0000"+
-    "\u0000\u0016\u0105\u0001\u0000\u0000\u0000\u0018\u0107\u0001\u0000\u0000"+
-    "\u0000\u001a\u010a\u0001\u0000\u0000\u0000\u001c\u0117\u0001\u0000\u0000"+
-    "\u0000\u001e\u0119\u0001\u0000\u0000\u0000 \u0128\u0001\u0000\u0000\u0000"+
-    "\"\u012a\u0001\u0000\u0000\u0000$\u0133\u0001\u0000\u0000\u0000&\u0139"+
-    "\u0001\u0000\u0000\u0000(\u013b\u0001\u0000\u0000\u0000*\u0144\u0001\u0000"+
-    "\u0000\u0000,\u0148\u0001\u0000\u0000\u0000.\u014b\u0001\u0000\u0000\u0000"+
-    "0\u0153\u0001\u0000\u0000\u00002\u0159\u0001\u0000\u0000\u00004\u0161"+
-    "\u0001\u0000\u0000\u00006\u0169\u0001\u0000\u0000\u00008\u016b\u0001\u0000"+
-    "\u0000\u0000:\u0197\u0001\u0000\u0000\u0000<\u0199\u0001\u0000\u0000\u0000"+
-    ">\u019c\u0001\u0000\u0000\u0000@\u01a5\u0001\u0000\u0000\u0000B\u01ad"+
-    "\u0001\u0000\u0000\u0000D\u01b6\u0001\u0000\u0000\u0000F\u01bf\u0001\u0000"+
-    "\u0000\u0000H\u01c8\u0001\u0000\u0000\u0000J\u01cc\u0001\u0000\u0000\u0000"+
-    "L\u01d2\u0001\u0000\u0000\u0000N\u01d6\u0001\u0000\u0000\u0000P\u01d9"+
-    "\u0001\u0000\u0000\u0000R\u01e1\u0001\u0000\u0000\u0000T\u01e5\u0001\u0000"+
-    "\u0000\u0000V\u01e9\u0001\u0000\u0000\u0000X\u01ec\u0001\u0000\u0000\u0000"+
-    "Z\u01f1\u0001\u0000\u0000\u0000\\\u01f5\u0001\u0000\u0000\u0000^\u01f7"+
-    "\u0001\u0000\u0000\u0000`\u01f9\u0001\u0000\u0000\u0000b\u01fc\u0001\u0000"+
-    "\u0000\u0000d\u0200\u0001\u0000\u0000\u0000f\u0203\u0001\u0000\u0000\u0000"+
-    "h\u0206\u0001\u0000\u0000\u0000j\u021a\u0001\u0000\u0000\u0000lm\u0003"+
+    "j\u0000\u0007\u0001\u0000<=\u0001\u0000>@\u0001\u0000CD\u0002\u0000  "+
+    "$$\u0001\u0000\'(\u0002\u0000&&44\u0002\u0000557;\u0238\u0000l\u0001\u0000"+
+    "\u0000\u0000\u0002o\u0001\u0000\u0000\u0000\u0004\u007f\u0001\u0000\u0000"+
+    "\u0000\u0006\u008e\u0001\u0000\u0000\u0000\b\u0090\u0001\u0000\u0000\u0000"+
+    "\n\u00af\u0001\u0000\u0000\u0000\f\u00ca\u0001\u0000\u0000\u0000\u000e"+
+    "\u00d1\u0001\u0000\u0000\u0000\u0010\u00d7\u0001\u0000\u0000\u0000\u0012"+
+    "\u00ec\u0001\u0000\u0000\u0000\u0014\u00f6\u0001\u0000\u0000\u0000\u0016"+
+    "\u0105\u0001\u0000\u0000\u0000\u0018\u0107\u0001\u0000\u0000\u0000\u001a"+
+    "\u010a\u0001\u0000\u0000\u0000\u001c\u0117\u0001\u0000\u0000\u0000\u001e"+
+    "\u0119\u0001\u0000\u0000\u0000 \u0128\u0001\u0000\u0000\u0000\"\u012a"+
+    "\u0001\u0000\u0000\u0000$\u0133\u0001\u0000\u0000\u0000&\u0139\u0001\u0000"+
+    "\u0000\u0000(\u013b\u0001\u0000\u0000\u0000*\u0144\u0001\u0000\u0000\u0000"+
+    ",\u0148\u0001\u0000\u0000\u0000.\u014b\u0001\u0000\u0000\u00000\u0153"+
+    "\u0001\u0000\u0000\u00002\u0159\u0001\u0000\u0000\u00004\u0161\u0001\u0000"+
+    "\u0000\u00006\u0169\u0001\u0000\u0000\u00008\u016b\u0001\u0000\u0000\u0000"+
+    ":\u0197\u0001\u0000\u0000\u0000<\u0199\u0001\u0000\u0000\u0000>\u019c"+
+    "\u0001\u0000\u0000\u0000@\u01a5\u0001\u0000\u0000\u0000B\u01ad\u0001\u0000"+
+    "\u0000\u0000D\u01b6\u0001\u0000\u0000\u0000F\u01bf\u0001\u0000\u0000\u0000"+
+    "H\u01c8\u0001\u0000\u0000\u0000J\u01cc\u0001\u0000\u0000\u0000L\u01d2"+
+    "\u0001\u0000\u0000\u0000N\u01d6\u0001\u0000\u0000\u0000P\u01d9\u0001\u0000"+
+    "\u0000\u0000R\u01e1\u0001\u0000\u0000\u0000T\u01e5\u0001\u0000\u0000\u0000"+
+    "V\u01e9\u0001\u0000\u0000\u0000X\u01ec\u0001\u0000\u0000\u0000Z\u01f1"+
+    "\u0001\u0000\u0000\u0000\\\u01f5\u0001\u0000\u0000\u0000^\u01f7\u0001"+
+    "\u0000\u0000\u0000`\u01f9\u0001\u0000\u0000\u0000b\u01fc\u0001\u0000\u0000"+
+    "\u0000d\u0200\u0001\u0000\u0000\u0000f\u0203\u0001\u0000\u0000\u0000h"+
+    "\u0206\u0001\u0000\u0000\u0000j\u021a\u0001\u0000\u0000\u0000lm\u0003"+
     "\u0002\u0001\u0000mn\u0005\u0000\u0000\u0001n\u0001\u0001\u0000\u0000"+
     "\u0000op\u0006\u0001\uffff\uffff\u0000pq\u0003\u0004\u0002\u0000qw\u0001"+
     "\u0000\u0000\u0000rs\n\u0001\u0000\u0000st\u0005\u001a\u0000\u0000tv\u0003"+
@@ -5105,42 +5095,42 @@ public class EsqlBaseParser extends Parser {
     "\u0000\u0000\u0123\u0124\u0001\u0000\u0000\u0000\u0124\u0126\u0001\u0000"+
     "\u0000\u0000\u0125\u0127\u0003\"\u0011\u0000\u0126\u0125\u0001\u0000\u0000"+
     "\u0000\u0126\u0127\u0001\u0000\u0000\u0000\u0127\u001f\u0001\u0000\u0000"+
-    "\u0000\u0128\u0129\u0007\u0002\u0000\u0000\u0129!\u0001\u0000\u0000\u0000"+
-    "\u012a\u012b\u0005H\u0000\u0000\u012b\u0130\u0003$\u0012\u0000\u012c\u012d"+
-    "\u0005#\u0000\u0000\u012d\u012f\u0003$\u0012\u0000\u012e\u012c\u0001\u0000"+
-    "\u0000\u0000\u012f\u0132\u0001\u0000\u0000\u0000\u0130\u012e\u0001\u0000"+
-    "\u0000\u0000\u0130\u0131\u0001\u0000\u0000\u0000\u0131#\u0001\u0000\u0000"+
-    "\u0000\u0132\u0130\u0001\u0000\u0000\u0000\u0133\u0134\u0003\\.\u0000"+
-    "\u0134\u0135\u0005!\u0000\u0000\u0135\u0136\u0003\\.\u0000\u0136%\u0001"+
-    "\u0000\u0000\u0000\u0137\u013a\u0003(\u0014\u0000\u0138\u013a\u0003*\u0015"+
-    "\u0000\u0139\u0137\u0001\u0000\u0000\u0000\u0139\u0138\u0001\u0000\u0000"+
-    "\u0000\u013a\'\u0001\u0000\u0000\u0000\u013b\u013c\u0005I\u0000\u0000"+
-    "\u013c\u0141\u0003 \u0010\u0000\u013d\u013e\u0005#\u0000\u0000\u013e\u0140"+
-    "\u0003 \u0010\u0000\u013f\u013d\u0001\u0000\u0000\u0000\u0140\u0143\u0001"+
-    "\u0000\u0000\u0000\u0141\u013f\u0001\u0000\u0000\u0000\u0141\u0142\u0001"+
-    "\u0000\u0000\u0000\u0142)\u0001\u0000\u0000\u0000\u0143\u0141\u0001\u0000"+
-    "\u0000\u0000\u0144\u0145\u0005A\u0000\u0000\u0145\u0146\u0003(\u0014\u0000"+
-    "\u0146\u0147\u0005B\u0000\u0000\u0147+\u0001\u0000\u0000\u0000\u0148\u0149"+
-    "\u0005\u0004\u0000\u0000\u0149\u014a\u0003\u001a\r\u0000\u014a-\u0001"+
-    "\u0000\u0000\u0000\u014b\u014d\u0005\u0011\u0000\u0000\u014c\u014e\u0003"+
-    "\u001a\r\u0000\u014d\u014c\u0001\u0000\u0000\u0000\u014d\u014e\u0001\u0000"+
-    "\u0000\u0000\u014e\u0151\u0001\u0000\u0000\u0000\u014f\u0150\u0005\u001e"+
-    "\u0000\u0000\u0150\u0152\u0003\u001a\r\u0000\u0151\u014f\u0001\u0000\u0000"+
-    "\u0000\u0151\u0152\u0001\u0000\u0000\u0000\u0152/\u0001\u0000\u0000\u0000"+
-    "\u0153\u0154\u0005\b\u0000\u0000\u0154\u0157\u0003\u001a\r\u0000\u0155"+
-    "\u0156\u0005\u001e\u0000\u0000\u0156\u0158\u0003\u001a\r\u0000\u0157\u0155"+
-    "\u0001\u0000\u0000\u0000\u0157\u0158\u0001\u0000\u0000\u0000\u01581\u0001"+
-    "\u0000\u0000\u0000\u0159\u015e\u00036\u001b\u0000\u015a\u015b\u0005%\u0000"+
-    "\u0000\u015b\u015d\u00036\u001b\u0000\u015c\u015a\u0001\u0000\u0000\u0000"+
-    "\u015d\u0160\u0001\u0000\u0000\u0000\u015e\u015c\u0001\u0000\u0000\u0000"+
-    "\u015e\u015f\u0001\u0000\u0000\u0000\u015f3\u0001\u0000\u0000\u0000\u0160"+
-    "\u015e\u0001\u0000\u0000\u0000\u0161\u0166\u00038\u001c\u0000\u0162\u0163"+
-    "\u0005%\u0000\u0000\u0163\u0165\u00038\u001c\u0000\u0164\u0162\u0001\u0000"+
-    "\u0000\u0000\u0165\u0168\u0001\u0000\u0000\u0000\u0166\u0164\u0001\u0000"+
-    "\u0000\u0000\u0166\u0167\u0001\u0000\u0000\u0000\u01675\u0001\u0000\u0000"+
-    "\u0000\u0168\u0166\u0001\u0000\u0000\u0000\u0169\u016a\u0007\u0003\u0000"+
-    "\u0000\u016a7\u0001\u0000\u0000\u0000\u016b\u016c\u0005N\u0000\u0000\u016c"+
-    "9\u0001\u0000\u0000\u0000\u016d\u0198\u0005.\u0000\u0000\u016e\u016f\u0003"+
+    "\u0000\u0128\u0129\u0005J\u0000\u0000\u0129!\u0001\u0000\u0000\u0000\u012a"+
+    "\u012b\u0005H\u0000\u0000\u012b\u0130\u0003$\u0012\u0000\u012c\u012d\u0005"+
+    "#\u0000\u0000\u012d\u012f\u0003$\u0012\u0000\u012e\u012c\u0001\u0000\u0000"+
+    "\u0000\u012f\u0132\u0001\u0000\u0000\u0000\u0130\u012e\u0001\u0000\u0000"+
+    "\u0000\u0130\u0131\u0001\u0000\u0000\u0000\u0131#\u0001\u0000\u0000\u0000"+
+    "\u0132\u0130\u0001\u0000\u0000\u0000\u0133\u0134\u0003\\.\u0000\u0134"+
+    "\u0135\u0005!\u0000\u0000\u0135\u0136\u0003\\.\u0000\u0136%\u0001\u0000"+
+    "\u0000\u0000\u0137\u013a\u0003(\u0014\u0000\u0138\u013a\u0003*\u0015\u0000"+
+    "\u0139\u0137\u0001\u0000\u0000\u0000\u0139\u0138\u0001\u0000\u0000\u0000"+
+    "\u013a\'\u0001\u0000\u0000\u0000\u013b\u013c\u0005I\u0000\u0000\u013c"+
+    "\u0141\u0003 \u0010\u0000\u013d\u013e\u0005#\u0000\u0000\u013e\u0140\u0003"+
+    " \u0010\u0000\u013f\u013d\u0001\u0000\u0000\u0000\u0140\u0143\u0001\u0000"+
+    "\u0000\u0000\u0141\u013f\u0001\u0000\u0000\u0000\u0141\u0142\u0001\u0000"+
+    "\u0000\u0000\u0142)\u0001\u0000\u0000\u0000\u0143\u0141\u0001\u0000\u0000"+
+    "\u0000\u0144\u0145\u0005A\u0000\u0000\u0145\u0146\u0003(\u0014\u0000\u0146"+
+    "\u0147\u0005B\u0000\u0000\u0147+\u0001\u0000\u0000\u0000\u0148\u0149\u0005"+
+    "\u0004\u0000\u0000\u0149\u014a\u0003\u001a\r\u0000\u014a-\u0001\u0000"+
+    "\u0000\u0000\u014b\u014d\u0005\u0011\u0000\u0000\u014c\u014e\u0003\u001a"+
+    "\r\u0000\u014d\u014c\u0001\u0000\u0000\u0000\u014d\u014e\u0001\u0000\u0000"+
+    "\u0000\u014e\u0151\u0001\u0000\u0000\u0000\u014f\u0150\u0005\u001e\u0000"+
+    "\u0000\u0150\u0152\u0003\u001a\r\u0000\u0151\u014f\u0001\u0000\u0000\u0000"+
+    "\u0151\u0152\u0001\u0000\u0000\u0000\u0152/\u0001\u0000\u0000\u0000\u0153"+
+    "\u0154\u0005\b\u0000\u0000\u0154\u0157\u0003\u001a\r\u0000\u0155\u0156"+
+    "\u0005\u001e\u0000\u0000\u0156\u0158\u0003\u001a\r\u0000\u0157\u0155\u0001"+
+    "\u0000\u0000\u0000\u0157\u0158\u0001\u0000\u0000\u0000\u01581\u0001\u0000"+
+    "\u0000\u0000\u0159\u015e\u00036\u001b\u0000\u015a\u015b\u0005%\u0000\u0000"+
+    "\u015b\u015d\u00036\u001b\u0000\u015c\u015a\u0001\u0000\u0000\u0000\u015d"+
+    "\u0160\u0001\u0000\u0000\u0000\u015e\u015c\u0001\u0000\u0000\u0000\u015e"+
+    "\u015f\u0001\u0000\u0000\u0000\u015f3\u0001\u0000\u0000\u0000\u0160\u015e"+
+    "\u0001\u0000\u0000\u0000\u0161\u0166\u00038\u001c\u0000\u0162\u0163\u0005"+
+    "%\u0000\u0000\u0163\u0165\u00038\u001c\u0000\u0164\u0162\u0001\u0000\u0000"+
+    "\u0000\u0165\u0168\u0001\u0000\u0000\u0000\u0166\u0164\u0001\u0000\u0000"+
+    "\u0000\u0166\u0167\u0001\u0000\u0000\u0000\u01675\u0001\u0000\u0000\u0000"+
+    "\u0168\u0166\u0001\u0000\u0000\u0000\u0169\u016a\u0007\u0002\u0000\u0000"+
+    "\u016a7\u0001\u0000\u0000\u0000\u016b\u016c\u0005N\u0000\u0000\u016c9"+
+    "\u0001\u0000\u0000\u0000\u016d\u0198\u0005.\u0000\u0000\u016e\u016f\u0003"+
     "Z-\u0000\u016f\u0170\u0005C\u0000\u0000\u0170\u0198\u0001\u0000\u0000"+
     "\u0000\u0171\u0198\u0003X,\u0000\u0172\u0198\u0003Z-\u0000\u0173\u0198"+
     "\u0003T*\u0000\u0174\u0198\u00051\u0000\u0000\u0175\u0198\u0003\\.\u0000"+
@@ -5172,10 +5162,10 @@ public class EsqlBaseParser extends Parser {
     "@ \u0000\u01a0\u019e\u0001\u0000\u0000\u0000\u01a1\u01a4\u0001\u0000\u0000"+
     "\u0000\u01a2\u01a0\u0001\u0000\u0000\u0000\u01a2\u01a3\u0001\u0000\u0000"+
     "\u0000\u01a3?\u0001\u0000\u0000\u0000\u01a4\u01a2\u0001\u0000\u0000\u0000"+
-    "\u01a5\u01a7\u0003\n\u0005\u0000\u01a6\u01a8\u0007\u0004\u0000\u0000\u01a7"+
+    "\u01a5\u01a7\u0003\n\u0005\u0000\u01a6\u01a8\u0007\u0003\u0000\u0000\u01a7"+
     "\u01a6\u0001\u0000\u0000\u0000\u01a7\u01a8\u0001\u0000\u0000\u0000\u01a8"+
     "\u01ab\u0001\u0000\u0000\u0000\u01a9\u01aa\u0005/\u0000\u0000\u01aa\u01ac"+
-    "\u0007\u0005\u0000\u0000\u01ab\u01a9\u0001\u0000\u0000\u0000\u01ab\u01ac"+
+    "\u0007\u0004\u0000\u0000\u01ab\u01a9\u0001\u0000\u0000\u0000\u01ab\u01ac"+
     "\u0001\u0000\u0000\u0000\u01acA\u0001\u0000\u0000\u0000\u01ad\u01ae\u0005"+
     "\t\u0000\u0000\u01ae\u01b3\u00034\u001a\u0000\u01af\u01b0\u0005#\u0000"+
     "\u0000\u01b0\u01b2\u00034\u001a\u0000\u01b1\u01af\u0001\u0000\u0000\u0000"+
@@ -5204,7 +5194,7 @@ public class EsqlBaseParser extends Parser {
     "\u0000\u0000\u0000\u01de\u01df\u0001\u0000\u0000\u0000\u01dfQ\u0001\u0000"+
     "\u0000\u0000\u01e0\u01de\u0001\u0000\u0000\u0000\u01e1\u01e2\u00036\u001b"+
     "\u0000\u01e2\u01e3\u0005!\u0000\u0000\u01e3\u01e4\u0003:\u001d\u0000\u01e4"+
-    "S\u0001\u0000\u0000\u0000\u01e5\u01e6\u0007\u0006\u0000\u0000\u01e6U\u0001"+
+    "S\u0001\u0000\u0000\u0000\u01e5\u01e6\u0007\u0005\u0000\u0000\u01e6U\u0001"+
     "\u0000\u0000\u0000\u01e7\u01ea\u0003X,\u0000\u01e8\u01ea\u0003Z-\u0000"+
     "\u01e9\u01e7\u0001\u0000\u0000\u0000\u01e9\u01e8\u0001\u0000\u0000\u0000"+
     "\u01eaW\u0001\u0000\u0000\u0000\u01eb\u01ed\u0007\u0000\u0000\u0000\u01ec"+
@@ -5214,7 +5204,7 @@ public class EsqlBaseParser extends Parser {
     "\u0001\u0000\u0000\u0000\u01f1\u01f2\u0001\u0000\u0000\u0000\u01f2\u01f3"+
     "\u0001\u0000\u0000\u0000\u01f3\u01f4\u0005\u001c\u0000\u0000\u01f4[\u0001"+
     "\u0000\u0000\u0000\u01f5\u01f6\u0005\u001b\u0000\u0000\u01f6]\u0001\u0000"+
-    "\u0000\u0000\u01f7\u01f8\u0007\u0007\u0000\u0000\u01f8_\u0001\u0000\u0000"+
+    "\u0000\u0000\u01f7\u01f8\u0007\u0006\u0000\u0000\u01f8_\u0001\u0000\u0000"+
     "\u0000\u01f9\u01fa\u0005\u0005\u0000\u0000\u01fa\u01fb\u0003b1\u0000\u01fb"+
     "a\u0001\u0000\u0000\u0000\u01fc\u01fd\u0005A\u0000\u0000\u01fd\u01fe\u0003"+
     "\u0002\u0001\u0000\u01fe\u01ff\u0005B\u0000\u0000\u01ffc\u0001\u0000\u0000"+

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/IdentifierBuilder.java

@@ -25,7 +25,7 @@ abstract class IdentifierBuilder extends AbstractBuilder {
 
     @Override
     public String visitFromIdentifier(FromIdentifierContext ctx) {
-        return ctx == null ? null : unquoteIdentifier(ctx.QUOTED_IDENTIFIER(), ctx.FROM_UNQUOTED_IDENTIFIER());
+        return ctx == null ? null : unquoteIdentifier(null, ctx.FROM_UNQUOTED_IDENTIFIER());
     }
 
     protected static String unquoteIdentifier(TerminalNode quotedNode, TerminalNode unquotedNode) {

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ComputeService.java

@@ -205,6 +205,7 @@ public class ComputeService {
             RefCountingListener refs = new RefCountingListener(listener.map(unused -> new Result(collectedPages, collectedProfiles)))
         ) {
             // run compute on the coordinator
+            exchangeSource.addCompletionListener(refs.acquire());
             runCompute(
                 rootTask,
                 new ComputeContext(sessionId, RemoteClusterAware.LOCAL_CLUSTER_GROUP_KEY, List.of(), configuration, exchangeSource, null),
@@ -722,6 +723,7 @@ public class ComputeService {
             var externalSink = exchangeService.getSinkHandler(externalId);
             task.addListener(() -> exchangeService.finishSinkHandler(externalId, new TaskCancelledException(task.getReasonCancelled())));
             var exchangeSource = new ExchangeSourceHandler(1, esqlExecutor);
+            exchangeSource.addCompletionListener(refs.acquire());
             exchangeSource.addRemoteSink(internalSink::fetchPageAsync, 1);
             ActionListener<Void> reductionListener = cancelOnFailure(task, cancelled, refs.acquire());
             runCompute(
@@ -854,6 +856,7 @@ public class ComputeService {
             RefCountingListener refs = new RefCountingListener(listener.map(unused -> new ComputeResponse(collectedProfiles)))
         ) {
             exchangeSink.addCompletionListener(refs.acquire());
+            exchangeSource.addCompletionListener(refs.acquire());
             PhysicalPlan coordinatorPlan = new ExchangeSinkExec(
                 plan.source(),
                 plan.output(),

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlFeatures.java

@@ -136,6 +136,17 @@ public class EsqlFeatures implements FeatureSpecification {
      */
     public static final NodeFeature METADATA_FIELDS = new NodeFeature("esql.metadata_fields");
 
+    /**
+     * Support for loading values over enrich. This is supported by all versions of ESQL but not
+     * the unit test CsvTests.
+     */
+    public static final NodeFeature ENRICH_LOAD = new NodeFeature("esql.enrich_load");
+
+    /**
+     * Support for timespan units abbreviations
+     */
+    public static final NodeFeature TIMESPAN_ABBREVIATIONS = new NodeFeature("esql.timespan_abbreviations");
+
     @Override
     public Set<NodeFeature> getFeatures() {
         return Set.of(
@@ -157,7 +168,8 @@ public class EsqlFeatures implements FeatureSpecification {
             MV_ORDERING_SORTED_ASCENDING,
             METRICS_COUNTER_FIELDS,
             STRING_LITERAL_AUTO_CASTING_EXTENDED,
-            METADATA_FIELDS
+            METADATA_FIELDS,
+            TIMESPAN_ABBREVIATIONS
         );
     }
 
@@ -168,7 +180,8 @@ public class EsqlFeatures implements FeatureSpecification {
             Map.entry(MV_WARN, Version.V_8_12_0),
             Map.entry(SPATIAL_POINTS, Version.V_8_12_0),
             Map.entry(CONVERT_WARN, Version.V_8_12_0),
-            Map.entry(POW_DOUBLE, Version.V_8_12_0)
+            Map.entry(POW_DOUBLE, Version.V_8_12_0),
+            Map.entry(ENRICH_LOAD, Version.V_8_12_0)
         );
     }
 }

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/type/EsqlDataTypeConverter.java

@@ -234,18 +234,20 @@ public class EsqlDataTypeConverter {
         return DataTypeConverter.commonType(left, right);
     }
 
+    // generally supporting abbreviations from https://en.wikipedia.org/wiki/Unit_of_time
     public static TemporalAmount parseTemporalAmout(Number value, String qualifier, Source source) throws InvalidArgumentException,
         ArithmeticException, ParsingException {
         return switch (qualifier) {
-            case "millisecond", "milliseconds" -> Duration.ofMillis(safeToLong(value));
-            case "second", "seconds" -> Duration.ofSeconds(safeToLong(value));
-            case "minute", "minutes" -> Duration.ofMinutes(safeToLong(value));
-            case "hour", "hours" -> Duration.ofHours(safeToLong(value));
+            case "millisecond", "milliseconds", "ms" -> Duration.ofMillis(safeToLong(value));
+            case "second", "seconds", "sec", "s" -> Duration.ofSeconds(safeToLong(value));
+            case "minute", "minutes", "min" -> Duration.ofMinutes(safeToLong(value));
+            case "hour", "hours", "h" -> Duration.ofHours(safeToLong(value));
 
-            case "day", "days" -> Period.ofDays(safeToInt(safeToLong(value)));
-            case "week", "weeks" -> Period.ofWeeks(safeToInt(safeToLong(value)));
-            case "month", "months" -> Period.ofMonths(safeToInt(safeToLong(value)));
-            case "year", "years" -> Period.ofYears(safeToInt(safeToLong(value)));
+            case "day", "days", "d" -> Period.ofDays(safeToInt(safeToLong(value)));
+            case "week", "weeks", "w" -> Period.ofWeeks(safeToInt(safeToLong(value)));
+            case "month", "months", "mo" -> Period.ofMonths(safeToInt(safeToLong(value)));
+            case "quarter", "quarters", "q" -> Period.ofMonths(safeToInt(Math.multiplyExact(3L, safeToLong(value))));
+            case "year", "years", "yr", "y" -> Period.ofYears(safeToInt(safeToLong(value)));
 
             default -> throw new ParsingException(source, "Unexpected time interval qualifier: '{}'", qualifier);
         };

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/CsvTests.java

@@ -224,6 +224,7 @@ public class CsvTests extends ESTestCase {
              * are tested in integration tests.
              */
             assumeFalse("metadata fields aren't supported", testCase.requiredFeatures.contains(EsqlFeatures.METADATA_FIELDS.id()));
+            assumeFalse("enrich can't load fields in csv tests", testCase.requiredFeatures.contains(EsqlFeatures.ENRICH_LOAD.id()));
             doTest();
         } catch (Throwable th) {
             throw reworkException(th);

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/AbstractFunctionTestCase.java

@@ -307,7 +307,13 @@ public abstract class AbstractFunctionTestCase extends ESTestCase {
      * </p>
      */
     public final void testEvaluateBlockWithoutNulls() {
-        testEvaluateBlock(driverContext().blockFactory(), driverContext(), false);
+        assumeTrue("no warning is expected", testCase.getExpectedWarnings() == null);
+        try {
+            testEvaluateBlock(driverContext().blockFactory(), driverContext(), false);
+        } catch (CircuitBreakingException ex) {
+            assertThat(ex.getMessage(), equalTo(MockBigArrays.ERROR_MESSAGE));
+            assertFalse("Test data is too large to fit in the memory", true);
+        }
     }
 
     /**
@@ -315,7 +321,13 @@ public abstract class AbstractFunctionTestCase extends ESTestCase {
      * some null values inserted between.
      */
     public final void testEvaluateBlockWithNulls() {
-        testEvaluateBlock(driverContext().blockFactory(), driverContext(), true);
+        assumeTrue("no warning is expected", testCase.getExpectedWarnings() == null);
+        try {
+            testEvaluateBlock(driverContext().blockFactory(), driverContext(), true);
+        } catch (CircuitBreakingException ex) {
+            assertThat(ex.getMessage(), equalTo(MockBigArrays.ERROR_MESSAGE));
+            assertFalse("Test data is too large to fit in the memory", true);
+        }
     }
 
     /**
@@ -1543,17 +1555,18 @@ public abstract class AbstractFunctionTestCase extends ESTestCase {
     private final List<CircuitBreaker> breakers = Collections.synchronizedList(new ArrayList<>());
 
     protected final DriverContext driverContext() {
-        MockBigArrays bigArrays = new MockBigArrays(PageCacheRecycler.NON_RECYCLING_INSTANCE, ByteSizeValue.ofGb(1));
+        BigArrays bigArrays = new MockBigArrays(PageCacheRecycler.NON_RECYCLING_INSTANCE, ByteSizeValue.ofMb(256)).withCircuitBreaking();
         CircuitBreaker breaker = bigArrays.breakerService().getBreaker(CircuitBreaker.REQUEST);
         breakers.add(breaker);
-        return new DriverContext(bigArrays.withCircuitBreaking(), new BlockFactory(breaker, bigArrays));
+        return new DriverContext(bigArrays, new BlockFactory(breaker, bigArrays));
     }
 
     protected final DriverContext crankyContext() {
-        BigArrays bigArrays = new MockBigArrays(PageCacheRecycler.NON_RECYCLING_INSTANCE, new CrankyCircuitBreakerService());
+        BigArrays bigArrays = new MockBigArrays(PageCacheRecycler.NON_RECYCLING_INSTANCE, new CrankyCircuitBreakerService())
+            .withCircuitBreaking();
         CircuitBreaker breaker = bigArrays.breakerService().getBreaker(CircuitBreaker.REQUEST);
         breakers.add(breaker);
-        return new DriverContext(bigArrays.withCircuitBreaking(), new BlockFactory(breaker, bigArrays));
+        return new DriverContext(bigArrays, new BlockFactory(breaker, bigArrays));
     }
 
     @After

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/ExpressionTests.java

@@ -380,14 +380,18 @@ public class ExpressionTests extends ESTestCase {
         assertEquals(l(Duration.ZERO, TIME_DURATION), whereExpression("0 second"));
         assertEquals(l(Duration.ofSeconds(value), TIME_DURATION), whereExpression(value + "second"));
         assertEquals(l(Duration.ofSeconds(value), TIME_DURATION), whereExpression(value + " seconds"));
+        assertEquals(l(Duration.ofSeconds(value), TIME_DURATION), whereExpression(value + " sec"));
+        assertEquals(l(Duration.ofSeconds(value), TIME_DURATION), whereExpression(value + " s"));
 
         assertEquals(l(Duration.ZERO, TIME_DURATION), whereExpression("0 minute"));
         assertEquals(l(Duration.ofMinutes(value), TIME_DURATION), whereExpression(value + "minute"));
         assertEquals(l(Duration.ofMinutes(value), TIME_DURATION), whereExpression(value + " minutes"));
+        assertEquals(l(Duration.ofMinutes(value), TIME_DURATION), whereExpression(value + " min"));
 
         assertEquals(l(Duration.ZERO, TIME_DURATION), whereExpression("0 hour"));
         assertEquals(l(Duration.ofHours(value), TIME_DURATION), whereExpression(value + "hour"));
         assertEquals(l(Duration.ofHours(value), TIME_DURATION), whereExpression(value + " hours"));
+        assertEquals(l(Duration.ofHours(value), TIME_DURATION), whereExpression(value + " h"));
 
         assertEquals(l(Duration.ofHours(-value), TIME_DURATION), whereExpression("-" + value + " hours"));
     }
@@ -395,22 +399,33 @@ public class ExpressionTests extends ESTestCase {
     public void testDatePeriodLiterals() {
         int value = randomInt(Integer.MAX_VALUE);
         int weeksValue = randomInt(Integer.MAX_VALUE / 7);
+        int quartersValue = randomInt(Integer.MAX_VALUE / 3);
 
         assertEquals(l(Period.ZERO, DATE_PERIOD), whereExpression("0 day"));
         assertEquals(l(Period.ofDays(value), DATE_PERIOD), whereExpression(value + "day"));
         assertEquals(l(Period.ofDays(value), DATE_PERIOD), whereExpression(value + " days"));
+        assertEquals(l(Period.ofDays(value), DATE_PERIOD), whereExpression(value + " d"));
 
         assertEquals(l(Period.ZERO, DATE_PERIOD), whereExpression("0week"));
         assertEquals(l(Period.ofDays(weeksValue * 7), DATE_PERIOD), whereExpression(weeksValue + "week"));
         assertEquals(l(Period.ofDays(weeksValue * 7), DATE_PERIOD), whereExpression(weeksValue + " weeks"));
+        assertEquals(l(Period.ofDays(weeksValue * 7), DATE_PERIOD), whereExpression(weeksValue + " w"));
 
         assertEquals(l(Period.ZERO, DATE_PERIOD), whereExpression("0 month"));
         assertEquals(l(Period.ofMonths(value), DATE_PERIOD), whereExpression(value + "month"));
         assertEquals(l(Period.ofMonths(value), DATE_PERIOD), whereExpression(value + " months"));
+        assertEquals(l(Period.ofMonths(value), DATE_PERIOD), whereExpression(value + " mo"));
+
+        assertEquals(l(Period.ZERO, DATE_PERIOD), whereExpression("0 quarter"));
+        assertEquals(l(Period.ofMonths(Math.multiplyExact(quartersValue, 3)), DATE_PERIOD), whereExpression(quartersValue + " quarter"));
+        assertEquals(l(Period.ofMonths(Math.multiplyExact(quartersValue, 3)), DATE_PERIOD), whereExpression(quartersValue + " quarters"));
+        assertEquals(l(Period.ofMonths(Math.multiplyExact(quartersValue, 3)), DATE_PERIOD), whereExpression(quartersValue + " q"));
 
         assertEquals(l(Period.ZERO, DATE_PERIOD), whereExpression("0year"));
         assertEquals(l(Period.ofYears(value), DATE_PERIOD), whereExpression(value + "year"));
         assertEquals(l(Period.ofYears(value), DATE_PERIOD), whereExpression(value + " years"));
+        assertEquals(l(Period.ofYears(value), DATE_PERIOD), whereExpression(value + " yr"));
+        assertEquals(l(Period.ofYears(value), DATE_PERIOD), whereExpression(value + " y"));
 
         assertEquals(l(Period.ofYears(-value), DATE_PERIOD), whereExpression("-" + value + " years"));
     }

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

@@ -338,17 +338,17 @@ public class StatementParserTests extends ESTestCase {
     }
 
     public void testIdentifiersAsIndexPattern() {
-        assertIdentifierAsIndexPattern("foo", "from `foo`");
-        assertIdentifierAsIndexPattern("foo,test-*", "from `foo`,`test-*`");
+        // assertIdentifierAsIndexPattern("foo", "from `foo`");
+        // assertIdentifierAsIndexPattern("foo,test-*", "from `foo`,`test-*`");
         assertIdentifierAsIndexPattern("foo,test-*", "from foo,test-*");
         assertIdentifierAsIndexPattern("123-test@foo_bar+baz1", "from 123-test@foo_bar+baz1");
-        assertIdentifierAsIndexPattern("foo,test-*,abc", "from `foo`,`test-*`,abc");
-        assertIdentifierAsIndexPattern("foo,     test-*, abc, xyz", "from `foo,     test-*, abc, xyz`");
-        assertIdentifierAsIndexPattern("foo,     test-*, abc, xyz,test123", "from `foo,     test-*, abc, xyz`,     test123");
+        // assertIdentifierAsIndexPattern("foo,test-*,abc", "from `foo`,`test-*`,abc");
+        // assertIdentifierAsIndexPattern("foo, test-*, abc, xyz", "from `foo, test-*, abc, xyz`");
+        // assertIdentifierAsIndexPattern("foo, test-*, abc, xyz,test123", "from `foo, test-*, abc, xyz`, test123");
         assertIdentifierAsIndexPattern("foo,test,xyz", "from foo,   test,xyz");
         assertIdentifierAsIndexPattern(
-            "<logstash-{now/M{yyyy.MM}}>,<logstash-{now/d{yyyy.MM.dd|+12:00}}>",
-            "from <logstash-{now/M{yyyy.MM}}>, `<logstash-{now/d{yyyy.MM.dd|+12:00}}>`"
+            "<logstash-{now/M{yyyy.MM}}>", // ,<logstash-{now/d{yyyy.MM.dd|+12:00}}>
+            "from <logstash-{now/M{yyyy.MM}}>" // , `<logstash-{now/d{yyyy.MM.dd|+12:00}}>`
         );
     }
 

x-pack/plugin/inference/build.gradle

@@ -4,6 +4,8 @@
  * 2.0; you may not use this file except in compliance with the Elastic License
  * 2.0.
  */
+import org.elasticsearch.gradle.internal.info.BuildParams
+
 apply plugin: 'elasticsearch.internal-es-plugin'
 apply plugin: 'elasticsearch.internal-cluster-test'
 apply plugin: 'elasticsearch.internal-yaml-rest-test'
@@ -36,6 +38,12 @@ dependencies {
   api "com.ibm.icu:icu4j:${versions.icu4j}"
 }
 
+if (BuildParams.isSnapshotBuild() == false) {
+  tasks.named("test").configure {
+    systemProperty 'es.semantic_text_feature_flag_enabled', 'true'
+  }
+}
+
 tasks.named('yamlRestTest') {
   usesDefaultDistribution()
 }

x-pack/plugin/inference/qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/xpack/application/InferenceUpgradeTestCase.java

@@ -13,7 +13,7 @@ import org.elasticsearch.client.Request;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.inference.TaskType;
 import org.elasticsearch.test.http.MockWebServer;
-import org.elasticsearch.upgrades.ParameterizedRollingUpgradeTestCase;
+import org.elasticsearch.upgrades.AbstractRollingUpgradeTestCase;
 
 import java.io.IOException;
 import java.util.List;
@@ -21,7 +21,7 @@ import java.util.Map;
 
 import static org.elasticsearch.core.Strings.format;
 
-public class InferenceUpgradeTestCase extends ParameterizedRollingUpgradeTestCase {
+public class InferenceUpgradeTestCase extends AbstractRollingUpgradeTestCase {
 
     public InferenceUpgradeTestCase(@Name("upgradedNodes") int upgradedNodes) {
         super(upgradedNodes);

x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/external/action/azureopenai/AzureOpenAiActionCreator.java

@@ -10,6 +10,7 @@ package org.elasticsearch.xpack.inference.external.action.azureopenai;
 import org.elasticsearch.xpack.inference.external.action.ExecutableAction;
 import org.elasticsearch.xpack.inference.external.http.sender.Sender;
 import org.elasticsearch.xpack.inference.services.ServiceComponents;
+import org.elasticsearch.xpack.inference.services.azureopenai.completion.AzureOpenAiCompletionModel;
 import org.elasticsearch.xpack.inference.services.azureopenai.embeddings.AzureOpenAiEmbeddingsModel;
 
 import java.util.Map;
@@ -32,4 +33,10 @@ public class AzureOpenAiActionCreator implements AzureOpenAiActionVisitor {
         var overriddenModel = AzureOpenAiEmbeddingsModel.of(model, taskSettings);
         return new AzureOpenAiEmbeddingsAction(sender, overriddenModel, serviceComponents);
     }
+
+    @Override
+    public ExecutableAction create(AzureOpenAiCompletionModel model, Map<String, Object> taskSettings) {
+        var overriddenModel = AzureOpenAiCompletionModel.of(model, taskSettings);
+        return new AzureOpenAiCompletionAction(sender, overriddenModel, serviceComponents);
+    }
 }

x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/external/action/azureopenai/AzureOpenAiActionVisitor.java

@@ -8,10 +8,13 @@
 package org.elasticsearch.xpack.inference.external.action.azureopenai;
 
 import org.elasticsearch.xpack.inference.external.action.ExecutableAction;
+import org.elasticsearch.xpack.inference.services.azureopenai.completion.AzureOpenAiCompletionModel;
 import org.elasticsearch.xpack.inference.services.azureopenai.embeddings.AzureOpenAiEmbeddingsModel;
 
 import java.util.Map;
 
 public interface AzureOpenAiActionVisitor {
     ExecutableAction create(AzureOpenAiEmbeddingsModel model, Map<String, Object> taskSettings);
+
+    ExecutableAction create(AzureOpenAiCompletionModel model, Map<String, Object> taskSettings);
 }

x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/external/action/azureopenai/AzureOpenAiCompletionAction.java

@@ -0,0 +1,67 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.xpack.inference.external.action.azureopenai;
+
+import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.ElasticsearchStatusException;
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.core.TimeValue;
+import org.elasticsearch.inference.InferenceServiceResults;
+import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.xpack.inference.external.action.ExecutableAction;
+import org.elasticsearch.xpack.inference.external.http.sender.AzureOpenAiCompletionRequestManager;
+import org.elasticsearch.xpack.inference.external.http.sender.DocumentsOnlyInput;
+import org.elasticsearch.xpack.inference.external.http.sender.InferenceInputs;
+import org.elasticsearch.xpack.inference.external.http.sender.Sender;
+import org.elasticsearch.xpack.inference.services.ServiceComponents;
+import org.elasticsearch.xpack.inference.services.azureopenai.completion.AzureOpenAiCompletionModel;
+
+import java.util.Objects;
+
+import static org.elasticsearch.xpack.inference.external.action.ActionUtils.constructFailedToSendRequestMessage;
+import static org.elasticsearch.xpack.inference.external.action.ActionUtils.createInternalServerError;
+import static org.elasticsearch.xpack.inference.external.action.ActionUtils.wrapFailuresInElasticsearchException;
+
+public class AzureOpenAiCompletionAction implements ExecutableAction {
+
+    private final String errorMessage;
+    private final AzureOpenAiCompletionRequestManager requestCreator;
+    private final Sender sender;
+
+    public AzureOpenAiCompletionAction(Sender sender, AzureOpenAiCompletionModel model, ServiceComponents serviceComponents) {
+        Objects.requireNonNull(serviceComponents);
+        Objects.requireNonNull(model);
+        this.sender = Objects.requireNonNull(sender);
+        this.requestCreator = new AzureOpenAiCompletionRequestManager(model, serviceComponents.threadPool());
+        this.errorMessage = constructFailedToSendRequestMessage(model.getUri(), "Azure OpenAI completion");
+    }
+
+    @Override
+    public void execute(InferenceInputs inferenceInputs, TimeValue timeout, ActionListener<InferenceServiceResults> listener) {
+        if (inferenceInputs instanceof DocumentsOnlyInput == false) {
+            listener.onFailure(new ElasticsearchStatusException("Invalid inference input type", RestStatus.INTERNAL_SERVER_ERROR));
+            return;
+        }
+
+        var docsOnlyInput = (DocumentsOnlyInput) inferenceInputs;
+        if (docsOnlyInput.getInputs().size() > 1) {
+            listener.onFailure(new ElasticsearchStatusException("Azure OpenAI completion only accepts 1 input", RestStatus.BAD_REQUEST));
+            return;
+        }
+
+        try {
+            ActionListener<InferenceServiceResults> wrappedListener = wrapFailuresInElasticsearchException(errorMessage, listener);
+
+            sender.send(requestCreator, inferenceInputs, timeout, wrappedListener);
+        } catch (ElasticsearchException e) {
+            listener.onFailure(e);
+        } catch (Exception e) {
+            listener.onFailure(createInternalServerError(e, errorMessage));
+        }
+    }
+}

x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/external/azureopenai/AzureOpenAiAccount.java

@@ -1,40 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
- */
-
-package org.elasticsearch.xpack.inference.external.azureopenai;
-
-import org.elasticsearch.common.settings.SecureString;
-import org.elasticsearch.core.Nullable;
-import org.elasticsearch.xpack.inference.services.azureopenai.embeddings.AzureOpenAiEmbeddingsModel;
-
-import java.util.Objects;
-
-public record AzureOpenAiAccount(
-    String resourceName,
-    String deploymentId,
-    String apiVersion,
-    @Nullable SecureString apiKey,
-    @Nullable SecureString entraId
-) {
-
-    public AzureOpenAiAccount {
-        Objects.requireNonNull(resourceName);
-        Objects.requireNonNull(deploymentId);
-        Objects.requireNonNull(apiVersion);
-        Objects.requireNonNullElse(apiKey, entraId);
-    }
-
-    public static AzureOpenAiAccount fromModel(AzureOpenAiEmbeddingsModel model) {
-        return new AzureOpenAiAccount(
-            model.getServiceSettings().resourceName(),
-            model.getServiceSettings().deploymentId(),
-            model.getServiceSettings().apiVersion(),
-            model.getSecretSettings().apiKey(),
-            model.getSecretSettings().entraId()
-        );
-    }
-}