elasticsearch/rest-api-spec
Pete Gillin 1fe3b77a2a
ES-10063 Add multi-project support for more stats APIs (#127650)
* Add multi-project support for more stats APIs

This affects the following APIs:
 - `GET _nodes/stats`:
   - For `indices`, it now prefixes the index name with the project ID (for non-default projects). Previously, it didn't tell you which project an index was in, and it failed if two projects had the same index name.
   - For `ingest`, it now gets the pipeline and processor stats for all projects, and prefixes the pipeline ID with the project ID. Previously, it only got them for the default project.
 - `GET /_cluster/stats`:
   - For `ingest`, it now aggregates the pipeline and processor stats for all projects. Previously, it only got them for the default project.
 - `GET /_info`:
   - For `ingest`, same as for `GET /_nodes/stats`.

This is done by making `IndicesService.stats()` and `IngestService.stats()` include project IDs in the `NodeIndicesStats` and `IngestStats` objects they return, and making those stats objects incorporate the project IDs when converting to XContent.

The transitive callers of these two methods are rather extensive (including all callers to `NodeService.stats()`, all callers of `TransportNodesStatsAction`, and so on). To ensure the change is safe, the callers were all checked out, and they fall into the following cases:
 - The behaviour change is one of the desired enhancements described above.
 - There is no behaviour change because it was getting node stats but neither `indices` nor `ingest` stats were requested.
 - There is no behaviour change because it was getting `indices` and/or `ingest` stats but only using aggregate values.
 - In `MachineLearningUsageTransportAction` and `TransportGetTrainedModelsStatsAction`, the `IngestStats` returned will return stats from all projects instead of just the default with this change, but they have been changed to filter the non-default project stats out, so this change is a noop there. (These actions are not MP-ready yet.)
 - `MonitoringService` will be affected, but this is the legacy monitoring module which is not in use anywhere that MP is going to be enabled. (If anything, the behaviour is probably improved by this change, as it will now include project IDs, rather than producing ambiguous unqualified results and failing in the case of duplicates.)

* Update test/external-modules/multi-project/build.gradle

Change suggested by Niels.

Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>

* Respond to review comments

* fix merge weirdness

* [CI] Auto commit changes from spotless

* Fix test compilation following upstream change to base class

* Update x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/datatiers/DataTierUsageFixtures.java

Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>

* Make projects-by-index map nullable and omit in single-project; always include project prefix in XContent in multip-project, even if default; also incorporate one other review comment

* Add a TODO

* update IT to reflect changed behaviour

* Switch to using XContent.Params to indicate whether it is multi-project or not

* Refactor NodesStatsMultiProjectIT to common up repeated assertions

* Defer use of ProjectIdResolver in REST handlers to keep tests happy

* Include index UUID in "unknown project" case

* Make the index-to-project map empty rather than null in the BWC deserialization case.

This works out fine, for the reasons given in the comment. As it happens, I'd already forgotten to do the null check in the one place it's actively used.

* remove a TODO that is done, and add a comment

* fix typo

* Get REST YAML tests working with project ID prefix TODO finish this

* As a drive-by, fix and un-suppress one of the health REST tests

* [CI] Auto commit changes from spotless

* TODO ugh

* Experiment with different stashing behaviour

* [CI] Auto commit changes from spotless

* Try a more sensible stash behaviour for assertions

* clarify comment

* Make checkstyle happy

* Make the way `Assertion` works more consistent, and simplify implementation

* [CI] Auto commit changes from spotless

* In RestNodesStatsAction, make the XContent params to channel.request(), which is the value it would have had before this change

---------

Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
2025-05-21 19:04:22 +01:00
..
src ES-10063 Add multi-project support for more stats APIs (#127650) 2025-05-21 19:04:22 +01:00
.gitignore
build.gradle Non existing synonyms sets do not fail shard recovery (#125659) 2025-03-27 18:04:20 +02:00
keywords.json Added PERL reserved words in REST keywords (#58535) 2020-06-26 12:11:22 +02:00
README.markdown [Dev Docs] Replacing unsupported **Note** with [!Note] (#127102) 2025-05-05 11:33:04 +02:00

Elasticsearch REST API JSON specification

This repository contains a collection of JSON files which describe the Elasticsearch HTTP API.

Their purpose is to formalize and standardize the API, to facilitate development of libraries and integrations.

Example for the "Create Index" API:

{
  "indices.create": {
    "documentation":{
      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-create-index.html"
    },
    "stability": "stable",
    "url":{
      "paths":[
        {
          "path":"/{index}",
          "method":"PUT",
          "parts":{
            "index":{
              "type":"string",
              "description":"The name of the index"
            }
          }
        }
      ]
    },
    "params": {
      "timeout": {
        "type" : "time",
        "description" : "Explicit operation timeout"
      }
    },
    "body": {
      "description" : "The configuration for the index (`settings` and `mappings`)"
    }
  }
}

The specification contains:

  • The name of the API (indices.create), which usually corresponds to the client calls

  • Link to the documentation at the http://elastic.co website.

    IMPORANT: This should be a live link. Several downstream ES clients use this link to generate their documentation. Using a broken link or linking to yet-to-be-created doc pages can break the Elastic docs build.

  • stability indicating the state of the API, has to be declared explicitly or YAML tests will fail

    • experimental highly likely to break in the near future (minor/patch), no bwc guarantees. Possibly removed in the future.
    • beta less likely to break or be removed but still reserve the right to do so
    • stable No backwards breaking changes in a minor
  • Request URL: HTTP method, path and parts

  • Request parameters

  • Request body specification

Note

If an API is stable but it response should be treated as an arbitrary map of key values please notate this as followed

{
  "api.name": {
    "stability" : "stable",
    "response": {
      "treat_json_as_key_value" : true
    }
  }
}

Type definition

In the documentation, you will find the type field, which documents which type every parameter will accept.

Querystring parameters

Type Description
list An array of strings (represented as a comma separated list in the querystring)
date A string representing a date formatted in ISO8601 or a number representing milliseconds since the epoch (used only in ML)
time A numeric or string value representing duration
string A string value
enum A set of named constants (a single value should be sent in the querystring)
int A signed 32-bit integer with a minimum value of -231 and a maximum value of 231-1.
double A double-precision 64-bit IEEE 754 floating point number, restricted to finite values.
long A signed 64-bit integer with a minimum value of -263 and a maximum value of 263-1. (Note: the max safe integer for JSON is 253-1)
number Alias for double. (deprecated, a more specific type should be used)
boolean Boolean fields accept JSON true and false values

Backwards compatibility

The specification follows the same backward compatibility guarantees as Elasticsearch.

  • Within a Major, additions only.
  • If an item has been documented wrong it should be deprecated instead as removing these might break downstream clients.
  • Major version change, may deprecate pieces or simply remove them given enough deprecation time.

Deprecations

The specification schema allows to codify API deprecations, either for an entire API, or for specific parts of the API, such as paths or parameters.

Entire API:

{
  "api" : {
    "deprecated" : {
      "version" : "7.0.0",
      "description" : "Reason API is being deprecated"
    },
  }
}

Specific paths and their parts:

{
  "api": {
    "url": {
      "paths": [
        {
          "path":"/{index}/{type}/{id}/_create",
          "method":"PUT",
          "parts":{
            "id":{
              "type":"string",
              "description":"Document ID"
            },
            "index":{
              "type":"string",
              "description":"The name of the index"
            },
            "type":{
              "type":"string",
              "description":"The type of the document",
              "deprecated":true
            }
          },
          "deprecated":{
            "version":"7.0.0",
            "description":"Specifying types in urls has been deprecated"
          }
        }
      ]
    }
  }
}

Parameters

{
  "api": {
    "url": {
      "params": {
        "stored_fields": {
          "type": "list",
          "description" : "",
          "deprecated" : {
            "version" : "7.0.0",
            "description" : "Reason parameter is being deprecated"
          }
        }
      }
    }
  }
}

License

This software is dual licensed under the Server Side Public License, v 1 ("SSPL-1.0") and Elastic License 2.0.