fix https://github.com/elastic/kibana/issues/192052
## Summary
Internal APIs will be
[restricted](https://github.com/elastic/kibana/issues/163654) from
public access as of 9.0.0. In non-serverless environments, this breaking
change will result in a 400 error if an external request is made to an
internal Kibana API (route `access` option as `"internal"` or
`"public"`).
This PR allows API owners of non-xpack plugins to run their `ftr` API
integration tests against the restriction and adds examples of how to
handle it.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Note to reviewers: The header needed to allow access to internal apis
shouldn't change your test output, with or without the restriction
enabled.
### How to test the changes work:
#### Non x-pack:
1. Set `server.restrictInternalApis: true` in `test/common/config.js`
2. Ensure your tests pass
#### x-pack:
1. Set `server.restrictInternalApis: true` in
`x-pack/test/api_integration/apis/security/config.ts`
2. Ensure the spaces tests pass
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Moving common services to respective new homes.
Resolves: https://github.com/elastic/kibana/issues/188541
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
### Notes for reviewers
- Currently, autocomplete suggestions for fields don't work with
wildcards and data streams due to the
[bug](https://github.com/elastic/kibana/issues/149496) in the `main`. It
should be addressed separately.
### How to test
In order to spot the loading behaviour, ideally you should create an
index with a heavy mappings definition.
Afterwards, write a query that requires a field from this index, e.g.:
```
GET <my-index>/_search
{
"aggs": {
"my_agg": {
"terms": {
"field": "",
"size": 10
}
}
}
}
```
Place a cursor next to the `field` property, it should trigger mappings
fetch. After that, the mappings definition for this index will be cached
and accessed synchronously.
You can also open the browser's dev tools and enable Network throttling.
It allows noticing loading behaviour for any index.
--------------------
Resolves https://github.com/elastic/kibana/issues/146855
Instead of fetching all mappings upfront, requests mapping definition on
demand per index according to the cursor position.
Considering there is a maximum response size limit of 10MB in the
`/autocomplete_entities` endpoint, field autocompletion wasn't working
at all if the overall mappings definition exceeded this size. Retrieving
mappings per index tackles this and improves the init time.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
Fixes https://github.com/elastic/kibana/issues/144310
### Summary
This PR addresses the issue of the Kibana instance restarting when the
response size is too big for the `autocomplete_entities` API. This
happens when a cluster has a large number of mappings and we try to
retrieve them all on the server side with
`esClient.asInternalUser.indices.getMapping()`. esClient does not handle
large responses well and throws an error that causes the Kibana instance
to restart. As node's max [string
length](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length#description)
is 2^28-1 (~512MB) if the response size is over 512MB, it will throw an
error with the message "Invalid string length".
The fix is to use the raw http request to retrieve the mappings instead
of esClient and check the response size before sending it to the client.
If the response size is too big, we will return an empty object and log
the error in the server logs.
#### Proposed changes
- Remove ES JS client requests and use native Node.js HTTP client
instead
- Limit the response size to 10MB for the `autocomplete_entities` API
#### How to test this PR locally
To test this out, you will need to connect Kibana to a remote cluster
with a large number of mappings. We created a patch file that you can
apply to your local Kibana instance to test this PR. Since the patch
file contains credentials, we can't share it publicly. Please reach out
to me if you would like to test this PR. I will share the patch file and
the instructions to apply it.
Co-authored-by: Muhammad Ibragimov <muhammad.ibragimov@elastic.co>
* Create a specific route for fetching mappings, aliases, templates, etc...
* Encapsulate data streams
* Encapsulate the mappings data into a class
* Setup up autocompleteInfo service and provide its instance through context
* Migrate the logic from mappings.js to Kibana server
* Translate the logic to consume the appropriate ES client method
* Update related test cases
* Lint
* Address comments
* Fix server proxy/mock
* Add API integration tests for /api/console/autocomplete_entities
* Lint
* Add tests
* Add API integration tests for autocomplete_entities API
* Add deleted tests
Co-authored-by: Muhammad Ibragimov <muhammad.ibragimov@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
