## Summary
Version alllll the data view routes.
Best viewed with whitespace hidden -
https://github.com/elastic/kibana/pull/158608/files?diff=unified&w=1
In this PR:
- All REST (public and internal) routes are versioned
- Internal routes are called with version specified
- Internal and public routes are now stored in directories labeled as
such
- All routes have a response schema
- All responses are typed with `response` types, separate from internal
api types. This is to help prevent unacknowledged changes to the api.
- Moves some functional tests from js => ts
For follow up PRs:
- Move to `internal` path for internal routes
- Proper typing and schema for `fields_for_wildcard` filter
Closes https://github.com/elastic/kibana/issues/157099
Closes https://github.com/elastic/kibana/issues/157100
---------
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
## Summary
Fix https://github.com/elastic/kibana/issues/150079
Add support for the `*` wildcard for by-type export, allowing to more
easily export all the exportable SO types
```
POST /api/saved_objects/_export
{
types: '*',
}
```
## Release Note
The savedObjects export API now supports exporting all types using the
`*` wildcard. Please refer to the documentation
for more details and examples.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
- Closes https://github.com/elastic/kibana/issues/147885
- Closes https://github.com/elastic/kibana/issues/157109
## Summary
**Before:**
Unified Field List plugin has internal routes (wrappers for client code)
which exist only to run api functional tests against them:
- `/api/unified_field_list/existing_fields/{dataViewId}`
- `/api/unified_field_list/field_stats`
Client code does not call these routes directly. So there is no reason
in keeping and versioning them.
**After:**
- Internal routes are removed
- A new "Unified Field List Examples" page was created
http://localhost:5601/app/unifiedFieldListExamples
- API functional tests (which used the routes) were converted to
functional tests against this new example page
- Created a new `unifiedFieldList` page object which is used now in
functional tests (methods are extracted from existing `discover` page
object).
**For testing:**
Steps:
1. Run Kibana with examples: `yarn start --run-examples`
2. Install sample data
3. And navigate to Developer Examples > Unified Field List Examples
page.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* Add deferred migrations parameter.
* Update outdated documents query to take into account deferred migrations.
* Update outdated documents query to take into account the core migration version.
* Update read operations in the saved objects repository to perform deferred migrations.
## Summary
The interface for fields allow array of string see below but the
validation with `@kbn/config-schema` only allow string. This will allow
to pass fields as array.
<img width="355" alt="image"
src="04c099c6-6a84-49ef-af45-587efa8e508b">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Fixes https://github.com/elastic/kibana/issues/155611
This PR changes the API prefix to indicate that the endpoints are
intended for internal use only. The plugin has been setup incorrectly
initially with the prefix `/api` that is intended for a public API.
There should not be any changes to the UI or functionality of the
plugin.
### Release Note
Fixed the guided onboarding API prefix to indicate that it's intended
for internal use
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Description
Fix https://github.com/elastic/kibana/issues/104081
This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`
This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)
**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly
## To reviewers
**Overall estimated risk of regressions: low**
But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.
If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.
## .Kibana split
The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)
Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).
#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window
#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event
#### `.kibana_analytics`
- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization
#### `.kibana_ingest`
- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies
## Tasks / PRs
### Sub-PRs
**Implementation**
- 🟣https://github.com/elastic/kibana/pull/154846
- 🟣https://github.com/elastic/kibana/pull/154892
- 🟣https://github.com/elastic/kibana/pull/154882
- 🟣https://github.com/elastic/kibana/pull/154884
- 🟣https://github.com/elastic/kibana/pull/155155
**Individual index split**
- 🟣https://github.com/elastic/kibana/pull/154897
- 🟣https://github.com/elastic/kibana/pull/155129
- 🟣https://github.com/elastic/kibana/pull/155140
- 🟣https://github.com/elastic/kibana/pull/155130
### Improvements / follow-ups
- 👷🏼 Extract logic into
[runV2Migration](https://github.com/elastic/kibana/pull/154151#discussion_r1158470566)
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
https://github.com/elastic/kibana/pull/154151#discussion_r1169289717
- 🚧 Build a more structured
[MigratorSynchronizer](https://github.com/elastic/kibana/pull/154151#discussion_r1158469918)
- 🟣https://github.com/elastic/kibana/pull/155035
- 🟣https://github.com/elastic/kibana/pull/155116
- 🟣https://github.com/elastic/kibana/pull/155366
## Reallocation tweaks
Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8
- `url` should get back to `.kibana` (see
[comment](https://github.com/elastic/kibana/pull/154888#discussion_r1172317133))
## Release Note
For performance purposes, Kibana is now using more system indices to
store its internal data.
The following system indices will be created when upgrading to `8.8.0`:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`
---------
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
## Summary
Fixes https://github.com/elastic/kibana/issues/143322
This PR adds an option to store dynamic parameters when a step is
completed to be used later for dynamically built URLs.
### How to test
1. Add `xpack.cloud.id: 'testID'` to your `/config/kibana.dev.yml` file
2. Start ES with `yarn es snapshot`
3. Start Kibana with `yarn start --run-examples`
4. Navigate to the guided onboarding example plugin
`http://localhost:5601/app/guidedOnboardingExample`
5. Start the test guide and when completing step 1 provide any value for
the paramter `indexName`
6. Continue the guide until step 4 and check that the correct value is
being used for the url of this step.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/integrations/pull/5123. This PR:
1) Syncs hardcoded list of integration categories against the latest
list from [package
spec](24557251ec/categories/categories.yml)
2) Brings the categories used by the "home tutorial" cards and custom
integrations cards in line with the above PR
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
https://github.com/elastic/kibana/pull/151788 established that data
views would no longer error when they failed to retrieve a field list.
This index pattern validation code exists since previously ALL index
pattern segments needed to match in order to avoid an error response
from field caps, rather than just one. Now we can remove the validation
code and simply pass the index pattern to field caps directly.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
### Notes for reviewers
- Currently, autocomplete suggestions for fields don't work with
wildcards and data streams due to the
[bug](https://github.com/elastic/kibana/issues/149496) in the `main`. It
should be addressed separately.
### How to test
In order to spot the loading behaviour, ideally you should create an
index with a heavy mappings definition.
Afterwards, write a query that requires a field from this index, e.g.:
```
GET <my-index>/_search
{
"aggs": {
"my_agg": {
"terms": {
"field": "",
"size": 10
}
}
}
}
```
Place a cursor next to the `field` property, it should trigger mappings
fetch. After that, the mappings definition for this index will be cached
and accessed synchronously.
You can also open the browser's dev tools and enable Network throttling.
It allows noticing loading behaviour for any index.
--------------------
Resolves https://github.com/elastic/kibana/issues/146855
Instead of fetching all mappings upfront, requests mapping definition on
demand per index according to the cursor position.
Considering there is a maximum response size limit of 10MB in the
`/autocomplete_entities` endpoint, field autocompletion wasn't working
at all if the overall mappings definition exceeded this size. Retrieving
mappings per index tackles this and improves the init time.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/149691
Follow up to https://github.com/elastic/kibana/pull/149528
This PR adds 3 separate guide ID values `appSearch`, `websiteSearch` and
`databaseSearch` instead of previously used `search`. With that change,
each card for Search on the landing page has its own independent guide
that can be completed.
Example plugin for guided onboarding is also updated so that all 3
search guides can be managed there. Otherwise there should not be any
UI/UX changes.
Currently, all 3 ids are registered with the same config and that can be
easily updated by the team in the future.
#### Screenshot
<img width="485" alt="Screenshot 2023-01-31 at 18 24 12"
src="https://user-images.githubusercontent.com/6585477/215840343-f9a65be3-0568-40ca-b458-5de92bfe8aaf.png">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR removes the deprecated field list sampling setting which was
marked as deprecated on 8.1. This
https://github.com/elastic/kibana/pull/139828 was the PR that added the
deprecation callout.
This PR moves the test guide used for testing and dev work to the guided
onboarding package. It is now being registered when the guided
onboarding example plugin is being setup. For that a server side is
added to the example plugin.
Registering the test guide from the example plugin demonstrates how the
"register guide" function can be used for production guides. The test
config needs to be in the `kbn-guided-onboarding` package because
example plugins have some problems importing files from other plugins.
That way the test config is available to the example plugin and to the
guided onboarding plugin for unit and functional tests.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds new optional `fields` argument to the data view fields for wildcard
api that allows the consumer to request only specific fields as opposed
to always getting all fields.
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
## Summary
Fixes https://github.com/elastic/kibana/issues/144452
This PR updates the guide IDs used in guided onboarding from generic
values (`observability` and `security`) to more specific values
(`kubernetes` and `siem`) which will allow us to add more guides for
observability and security in the future.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Fixes https://github.com/elastic/kibana/issues/141129
Fixes https://github.com/elastic/kibana/issues/144515
This PR introduces a new state to the guided onboarding plugin. The
state keeps track of the `creationDate` and of the overall `status` of
the plugin. The creation date allows us to detect an "active" period
during which the header button will be displayed more prominently in the
header. Currently, the active period is set to 30 days. During this
time, if the user has not started any guide, has quit a guide before
completion or skipped the guide on the landing page, the header button
will be displayed and when clicked, redirect the user to the landing
page to start/continue a guide.
Also this PR adds a check for Cloud deployments and prevents the code
from sending any API requests when not on Cloud, because guided
onboarding is disabled on prem.
#### Screenshot
<img width="298" alt="Screenshot 2022-11-10 at 18 42 18"
src="https://user-images.githubusercontent.com/6585477/201168414-391a7cd4-0709-492b-9001-1432b5bed3c8.png">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/144310
### Summary
This PR addresses the issue of the Kibana instance restarting when the
response size is too big for the `autocomplete_entities` API. This
happens when a cluster has a large number of mappings and we try to
retrieve them all on the server side with
`esClient.asInternalUser.indices.getMapping()`. esClient does not handle
large responses well and throws an error that causes the Kibana instance
to restart. As node's max [string
length](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length#description)
is 2^28-1 (~512MB) if the response size is over 512MB, it will throw an
error with the message "Invalid string length".
The fix is to use the raw http request to retrieve the mappings instead
of esClient and check the response size before sending it to the client.
If the response size is too big, we will return an empty object and log
the error in the server logs.
#### Proposed changes
- Remove ES JS client requests and use native Node.js HTTP client
instead
- Limit the response size to 10MB for the `autocomplete_entities` API
#### How to test this PR locally
To test this out, you will need to connect Kibana to a remote cluster
with a large number of mappings. We created a patch file that you can
apply to your local Kibana instance to test this PR. Since the patch
file contains credentials, we can't share it publicly. Please reach out
to me if you would like to test this PR. I will share the patch file and
the instructions to apply it.
Co-authored-by: Muhammad Ibragimov <muhammad.ibragimov@elastic.co>
* Use brotli compression
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* Add integration test for brotli support
* Use import instead of require()
* Suppress build error on importing brok
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* add brok as explicit package dep
* add `server.compression.brotli` config settings
* update documentation
* fix test utils
* fix more test configs
* add tests for endpoints too
* remove against endpoint for now
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
