[8.x] [Console] Add autocompletion for data streams (#198507) (#199439)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Console] Add autocompletion for data streams
(#198507)](https://github.com/elastic/kibana/pull/198507)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Elena
Stoeva","email":"59341489+ElenaStoeva@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-11-08T10:15:19Z","message":"[Console]
Add autocompletion for data streams (#198507)\n\nCloses
https://github.com/elastic/kibana/issues/190137\r\n\r\n##
Summary\r\n\r\nThis PR adds Console autocompletion for data
streams.\r\n\r\nWe use the patterns in the spec definitions to determine
what\r\nautocomplete entity we should return. For example the `{index}`
pattern\r\nin the endpoint means we should suggest indices and aliases.
This\r\npattern comes directly from the schema in the
[Elasticsearch\r\nspecification\r\nrepo](https://github.com/elastic/elasticsearch-specification/blob/main/output/schema/schema.json),\r\nwhich
is used to generate the Console spec definition.\r\n\r\nPreviously, for
the pattern `{index}`, we would only suggest indices and\r\naliases.
However, for most of the endpoints where this pattern is used,\r\nwe
should suggest indices, aliases, **and data streams** (see the
table\r\nbelow). For example, the [Get
Index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html)\r\naccepts
indices, aliases, and data streams as the target, while the
spec\r\ndefinition for the `GET /<target>` endpoint uses the `{index}`
pattern.\r\n\r\nIn this PR, we make Console also suggest data streams
for the `{index}`\r\npattern.\r\n\r\nList with some endpoints that use
the `{index}` pattern:\r\n\r\nEs API | Endpoint (with link to specs) |
Expected target by Es\r\n(according to docs) | Data stream
compatible?\r\n--- | --- | --- | --- \r\n[Get
index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-get-index.html)\r\n|
[`GET\r\n/<target>`](e0838147bd/src/plugins/console/server/lib/spec_definitions/json/generated/indices.get.json (L35))\r\n|
data streams, indices, and aliases |
✅\r\n[Async\r\nSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html)\r\n|
[`POST\r\n{index}/_async_search`](e0838147bd/src/plugins/console/server/lib/spec_definitions/json/generated/async_search.submit.json (L94))\r\n|
Not specified | ✅ (tested
locally)\r\n[Index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-index_.html)\r\n|
`PUT/POST {index}/_create/{id}` | data stream or index |
✅\r\n[Bulk\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)\r\n|
[`POST/PUT\r\n{index}/_bulk`](3ac902df8a/src/plugins/console/server/lib/spec_definitions/json/generated/bulk.json (L36))\r\n|
data stream, index, or index alias | ✅\r\n[cat
count\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-count.html)\r\n|
[`GET\r\n_cat/count/{index}`](3ac902df8a/src/plugins/console/server/lib/spec_definitions/json/generated/cat.count.json (L27))\r\n|
data streams, indices, and aliases | ✅\r\n[cat
indices\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html)\r\n|
`GET _cat/indices/{index}` | data streams, indices, and aliases |
✅\r\n[cat
recovery\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html)\r\n|
`GET _cat/recovery/{index}` | data streams, indices, and aliases |
✅\r\n\r\n\r\nNote: As discussed with @elastic/devtools-team, it is safe
to assume\r\nthat endpoints with the `{index}` pattern accept data
streams, apart\r\nfrom indices and aliases. There are still a few edge
cases though, such\r\nas alias APIs, which don't accept data streams.
Temporarily, it's okay\r\nto display data stream suggestions for them,
but we could try fixing\r\nthis in a follow-up PR by utilizing the
override specs and adding a\r\npattern for indices and aliases only.
Issue:\r\nhttps://github.com/elastic/kibana/issues/198588","sha":"af2834f5c3cb4cf3e576264ed8193e7d0abfab5f","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Console","Team:Kibana
Management","release_note:skip","v9.0.0","backport:prev-minor","v8.17.0"],"title":"[Console]
Add autocompletion for data
streams","number":198507,"url":"https://github.com/elastic/kibana/pull/198507","mergeCommit":{"message":"[Console]
Add autocompletion for data streams (#198507)\n\nCloses
https://github.com/elastic/kibana/issues/190137\r\n\r\n##
Summary\r\n\r\nThis PR adds Console autocompletion for data
streams.\r\n\r\nWe use the patterns in the spec definitions to determine
what\r\nautocomplete entity we should return. For example the `{index}`
pattern\r\nin the endpoint means we should suggest indices and aliases.
This\r\npattern comes directly from the schema in the
[Elasticsearch\r\nspecification\r\nrepo](https://github.com/elastic/elasticsearch-specification/blob/main/output/schema/schema.json),\r\nwhich
is used to generate the Console spec definition.\r\n\r\nPreviously, for
the pattern `{index}`, we would only suggest indices and\r\naliases.
However, for most of the endpoints where this pattern is used,\r\nwe
should suggest indices, aliases, **and data streams** (see the
table\r\nbelow). For example, the [Get
Index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html)\r\naccepts
indices, aliases, and data streams as the target, while the
spec\r\ndefinition for the `GET /<target>` endpoint uses the `{index}`
pattern.\r\n\r\nIn this PR, we make Console also suggest data streams
for the `{index}`\r\npattern.\r\n\r\nList with some endpoints that use
the `{index}` pattern:\r\n\r\nEs API | Endpoint (with link to specs) |
Expected target by Es\r\n(according to docs) | Data stream
compatible?\r\n--- | --- | --- | --- \r\n[Get
index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-get-index.html)\r\n|
[`GET\r\n/<target>`](e0838147bd/src/plugins/console/server/lib/spec_definitions/json/generated/indices.get.json (L35))\r\n|
data streams, indices, and aliases |
✅\r\n[Async\r\nSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html)\r\n|
[`POST\r\n{index}/_async_search`](e0838147bd/src/plugins/console/server/lib/spec_definitions/json/generated/async_search.submit.json (L94))\r\n|
Not specified | ✅ (tested
locally)\r\n[Index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-index_.html)\r\n|
`PUT/POST {index}/_create/{id}` | data stream or index |
✅\r\n[Bulk\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)\r\n|
[`POST/PUT\r\n{index}/_bulk`](3ac902df8a/src/plugins/console/server/lib/spec_definitions/json/generated/bulk.json (L36))\r\n|
data stream, index, or index alias | ✅\r\n[cat
count\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-count.html)\r\n|
[`GET\r\n_cat/count/{index}`](3ac902df8a/src/plugins/console/server/lib/spec_definitions/json/generated/cat.count.json (L27))\r\n|
data streams, indices, and aliases | ✅\r\n[cat
indices\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html)\r\n|
`GET _cat/indices/{index}` | data streams, indices, and aliases |
✅\r\n[cat
recovery\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html)\r\n|
`GET _cat/recovery/{index}` | data streams, indices, and aliases |
✅\r\n\r\n\r\nNote: As discussed with @elastic/devtools-team, it is safe
to assume\r\nthat endpoints with the `{index}` pattern accept data
streams, apart\r\nfrom indices and aliases. There are still a few edge
cases though, such\r\nas alias APIs, which don't accept data streams.
Temporarily, it's okay\r\nto display data stream suggestions for them,
but we could try fixing\r\nthis in a follow-up PR by utilizing the
override specs and adding a\r\npattern for indices and aliases only.
Issue:\r\nhttps://github.com/elastic/kibana/issues/198588","sha":"af2834f5c3cb4cf3e576264ed8193e7d0abfab5f"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198507","number":198507,"mergeCommit":{"message":"[Console]
Add autocompletion for data streams (#198507)\n\nCloses
https://github.com/elastic/kibana/issues/190137\r\n\r\n##
Summary\r\n\r\nThis PR adds Console autocompletion for data
streams.\r\n\r\nWe use the patterns in the spec definitions to determine
what\r\nautocomplete entity we should return. For example the `{index}`
pattern\r\nin the endpoint means we should suggest indices and aliases.
This\r\npattern comes directly from the schema in the
[Elasticsearch\r\nspecification\r\nrepo](https://github.com/elastic/elasticsearch-specification/blob/main/output/schema/schema.json),\r\nwhich
is used to generate the Console spec definition.\r\n\r\nPreviously, for
the pattern `{index}`, we would only suggest indices and\r\naliases.
However, for most of the endpoints where this pattern is used,\r\nwe
should suggest indices, aliases, **and data streams** (see the
table\r\nbelow). For example, the [Get
Index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html)\r\naccepts
indices, aliases, and data streams as the target, while the
spec\r\ndefinition for the `GET /<target>` endpoint uses the `{index}`
pattern.\r\n\r\nIn this PR, we make Console also suggest data streams
for the `{index}`\r\npattern.\r\n\r\nList with some endpoints that use
the `{index}` pattern:\r\n\r\nEs API | Endpoint (with link to specs) |
Expected target by Es\r\n(according to docs) | Data stream
compatible?\r\n--- | --- | --- | --- \r\n[Get
index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-get-index.html)\r\n|
[`GET\r\n/<target>`](e0838147bd/src/plugins/console/server/lib/spec_definitions/json/generated/indices.get.json (L35))\r\n|
data streams, indices, and aliases |
✅\r\n[Async\r\nSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/async-search.html)\r\n|
[`POST\r\n{index}/_async_search`](e0838147bd/src/plugins/console/server/lib/spec_definitions/json/generated/async_search.submit.json (L94))\r\n|
Not specified | ✅ (tested
locally)\r\n[Index\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-index_.html)\r\n|
`PUT/POST {index}/_create/{id}` | data stream or index |
✅\r\n[Bulk\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)\r\n|
[`POST/PUT\r\n{index}/_bulk`](3ac902df8a/src/plugins/console/server/lib/spec_definitions/json/generated/bulk.json (L36))\r\n|
data stream, index, or index alias | ✅\r\n[cat
count\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-count.html)\r\n|
[`GET\r\n_cat/count/{index}`](3ac902df8a/src/plugins/console/server/lib/spec_definitions/json/generated/cat.count.json (L27))\r\n|
data streams, indices, and aliases | ✅\r\n[cat
indices\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html)\r\n|
`GET _cat/indices/{index}` | data streams, indices, and aliases |
✅\r\n[cat
recovery\r\nAPI](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html)\r\n|
`GET _cat/recovery/{index}` | data streams, indices, and aliases |
✅\r\n\r\n\r\nNote: As discussed with @elastic/devtools-team, it is safe
to assume\r\nthat endpoints with the `{index}` pattern accept data
streams, apart\r\nfrom indices and aliases. There are still a few edge
cases though, such\r\nas alias APIs, which don't accept data streams.
Temporarily, it's okay\r\nto display data stream suggestions for them,
but we could try fixing\r\nthis in a follow-up PR by utilizing the
override specs and adding a\r\npattern for indices and aliases only.
Issue:\r\nhttps://github.com/elastic/kibana/issues/198588","sha":"af2834f5c3cb4cf3e576264ed8193e7d0abfab5f"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Elena Stoeva <59341489+ElenaStoeva@users.noreply.github.com>

src/plugins/console/public/services/autocomplete.ts

@@ -53,7 +53,10 @@ export class AutocompleteInfo {
       case ENTITIES.INDICES:
         const includeAliases = true;
         const collaborator = this.mapping;
-        return () => this.alias.getIndices(includeAliases, collaborator);
+        return () => [
+          ...this.alias.getIndices(includeAliases, collaborator),
+          ...this.dataStream.getDataStreams(),
+        ];
       case ENTITIES.FIELDS:
         return this.mapping.getMappings(
           context.indices,

src/plugins/console/server/routes/api/console/autocomplete_entities/index.ts

@@ -43,7 +43,10 @@ const getAliases = async (settings: SettingsToRetrieve, esClient: IScopedCluster
 
 const getDataStreams = async (settings: SettingsToRetrieve, esClient: IScopedClusterClient) => {
   if (settings.dataStreams) {
-    const dataStreams = await esClient.asCurrentUser.indices.getDataStream();
+    const dataStreams = await esClient.asCurrentUser.indices.getDataStream({
+      name: '*',
+      expand_wildcards: 'all',
+    });
     return dataStreams;
   }
   // If the user doesn't want autocomplete suggestions, then clear any that exist.