# Backport
This will backport the following commits from `main` to `8.9`:
- [[Flaky #98240] Unskip test
(#165197)](https://github.com/elastic/kibana/pull/165197)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Alejandro Fernández
Haro","email":"alejandro.haro@elastic.co"},"sourceCommit":{"committedDate":"2023-08-30T15:04:22Z","message":"[Flaky
#98240] Unskip test
(#165197)","sha":"0236516351e89ce1591d916c24d45a406bef714d","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","technical
debt","release_note:skip","backport:all-open","v8.11.0"],"number":165197,"url":"https://github.com/elastic/kibana/pull/165197","mergeCommit":{"message":"[Flaky
#98240] Unskip test
(#165197)","sha":"0236516351e89ce1591d916c24d45a406bef714d"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/165197","number":165197,"mergeCommit":{"message":"[Flaky
#98240] Unskip test
(#165197)","sha":"0236516351e89ce1591d916c24d45a406bef714d"}}]}]
BACKPORT-->
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
# Backport
This will backport the following commits from `main` to `8.9`:
- [[Lens] allow removing ad-hoc data view from event annotation group
(#163976)](https://github.com/elastic/kibana/pull/163976)
<!--- Backport version: 8.9.8 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Drew
Tate","email":"drew.tate@elastic.co"},"sourceCommit":{"committedDate":"2023-08-16T15:24:15Z","message":"[Lens]
allow removing ad-hoc data view from event annotation group
(#163976)","sha":"08a48f3cf1a928d2e01dbe1e867326ae54afbb5a","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Visualizations","v8.9.0","v8.10.0"],"number":163976,"url":"https://github.com/elastic/kibana/pull/163976","mergeCommit":{"message":"[Lens]
allow removing ad-hoc data view from event annotation group
(#163976)","sha":"08a48f3cf1a928d2e01dbe1e867326ae54afbb5a"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/163976","number":163976,"mergeCommit":{"message":"[Lens]
allow removing ad-hoc data view from event annotation group
(#163976)","sha":"08a48f3cf1a928d2e01dbe1e867326ae54afbb5a"}}]}]
BACKPORT-->
# Backport
This will backport the following commits from `main` to `8.9`:
- [Saved Object APIs - FTR - Fix flaky test in `_find` API
(#160794)](https://github.com/elastic/kibana/pull/160794)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"gerard.soldevila@elastic.co"},"sourceCommit":{"committedDate":"2023-06-29T08:39:32Z","message":"Saved
Object APIs - FTR - Fix flaky test in `_find` API (#160794)\n\n##
Summary\r\n\r\nAttempt to fix
https://github.com/elastic/kibana/issues/156581\r\n\r\nThe best
explanation I have is that the two retrieved visualisations are\r\nnot
necessarily retrieved in the expected order.\r\nSorting the results
should ensure the expected order.\r\n\r\nWill use the flaky test runner
to check the
fix.","sha":"4e3f6990c2dcb4bf4e3e8edf4e6e2d6fe9e8d99c","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Feature:Saved
Objects","release_note:skip","test-failure-flaky","backport:prev-minor","v8.9.0","v8.10.0"],"number":160794,"url":"https://github.com/elastic/kibana/pull/160794","mergeCommit":{"message":"Saved
Object APIs - FTR - Fix flaky test in `_find` API (#160794)\n\n##
Summary\r\n\r\nAttempt to fix
https://github.com/elastic/kibana/issues/156581\r\n\r\nThe best
explanation I have is that the two retrieved visualisations are\r\nnot
necessarily retrieved in the expected order.\r\nSorting the results
should ensure the expected order.\r\n\r\nWill use the flaky test runner
to check the
fix.","sha":"4e3f6990c2dcb4bf4e3e8edf4e6e2d6fe9e8d99c"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/160794","number":160794,"mergeCommit":{"message":"Saved
Object APIs - FTR - Fix flaky test in `_find` API (#160794)\n\n##
Summary\r\n\r\nAttempt to fix
https://github.com/elastic/kibana/issues/156581\r\n\r\nThe best
explanation I have is that the two retrieved visualisations are\r\nnot
necessarily retrieved in the expected order.\r\nSorting the results
should ensure the expected order.\r\n\r\nWill use the flaky test runner
to check the fix.","sha":"4e3f6990c2dcb4bf4e3e8edf4e6e2d6fe9e8d99c"}}]}]
BACKPORT-->
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
## Summary
Part of [54306](https://github.com/elastic/kibana/issues/154306)
Required #158468 to be merged first.
This PR integrates only the migration part of the new Metric new
formatter task.
This migration implementation defines takes the simplest route for the
migration route of metric visualization formatter:
* if any custom formatter is set in Lens, then apply a `compact` flag on
it to keep it consistent
* if `Default` formatter is set in Lens, then assume nothing and let the
regular formatter hierarchy do its own work
### Example
Original dashboard
After migration
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
Version alllll the data view routes.
Best viewed with whitespace hidden -
https://github.com/elastic/kibana/pull/158608/files?diff=unified&w=1
In this PR:
- All REST (public and internal) routes are versioned
- Internal routes are called with version specified
- Internal and public routes are now stored in directories labeled as
such
- All routes have a response schema
- All responses are typed with `response` types, separate from internal
api types. This is to help prevent unacknowledged changes to the api.
- Moves some functional tests from js => ts
For follow up PRs:
- Move to `internal` path for internal routes
- Proper typing and schema for `fields_for_wildcard` filter
Closes https://github.com/elastic/kibana/issues/157099
Closes https://github.com/elastic/kibana/issues/157100
---------
Co-authored-by: Julia Rechkunova <julia.rechkunova@gmail.com>
## Summary
Fix https://github.com/elastic/kibana/issues/150079
Add support for the `*` wildcard for by-type export, allowing to more
easily export all the exportable SO types
```
POST /api/saved_objects/_export
{
types: '*',
}
```
## Release Note
The savedObjects export API now supports exporting all types using the
`*` wildcard. Please refer to the documentation
for more details and examples.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
- Closes https://github.com/elastic/kibana/issues/147885
- Closes https://github.com/elastic/kibana/issues/157109
## Summary
**Before:**
Unified Field List plugin has internal routes (wrappers for client code)
which exist only to run api functional tests against them:
- `/api/unified_field_list/existing_fields/{dataViewId}`
- `/api/unified_field_list/field_stats`
Client code does not call these routes directly. So there is no reason
in keeping and versioning them.
**After:**
- Internal routes are removed
- A new "Unified Field List Examples" page was created
http://localhost:5601/app/unifiedFieldListExamples
- API functional tests (which used the routes) were converted to
functional tests against this new example page
- Created a new `unifiedFieldList` page object which is used now in
functional tests (methods are extracted from existing `discover` page
object).
**For testing:**
Steps:
1. Run Kibana with examples: `yarn start --run-examples`
2. Install sample data
3. And navigate to Developer Examples > Unified Field List Examples
page.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* Add deferred migrations parameter.
* Update outdated documents query to take into account deferred migrations.
* Update outdated documents query to take into account the core migration version.
* Update read operations in the saved objects repository to perform deferred migrations.
## Summary
The interface for fields allow array of string see below but the
validation with `@kbn/config-schema` only allow string. This will allow
to pass fields as array.
<img width="355" alt="image"
src="04c099c6-6a84-49ef-af45-587efa8e508b">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Fixes https://github.com/elastic/kibana/issues/155611
This PR changes the API prefix to indicate that the endpoints are
intended for internal use only. The plugin has been setup incorrectly
initially with the prefix `/api` that is intended for a public API.
There should not be any changes to the UI or functionality of the
plugin.
### Release Note
Fixed the guided onboarding API prefix to indicate that it's intended
for internal use
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Description
Fix https://github.com/elastic/kibana/issues/104081
This PR move some of the SO types from the `.kibana` index into the
following ones:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`
This split/reallocation will occur during the `8.8.0` Kibana upgrade
(*meaning: from any version older than `8.8.0` to any version greater or
equal to `8.8.0`*)
**This PR main changes are:**
- implement the changes required in the SO migration algorithm to
support this reallocation
- update the FTR tools (looking at you esArchiver) to support these new
indices
- update hardcoded references to `.kibana` and usage of the
`core.savedObjects.getKibanaIndex()` to use new APIs to target the
correct index/indices
- update FTR datasets, tests and utility accordingly
## To reviewers
**Overall estimated risk of regressions: low**
But, still, please take the time to review changes in your code. The
parts of the production code that were the most impacted are the
telemetry collectors, as most of them were performing direct requests
against the `.kibana` index, so we had to adapt them. Most other
contributor-owned changes are in FTR tests and datasets.
If you think a type is misplaced (either we missed some types that
should be moved to a specific index, or some types were moved and
shouldn't have been) please tell us, and we'll fix the reallocation
either in this PR or in a follow-up.
## .Kibana split
The following new indices are introduced by this PR, with the following
SO types being moved to it. (any SO type not listed here will be staying
in its current index)
Note: The complete **_type => index_** breakdown is available in [this
spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788).
#### `.kibana_alerting_cases`
- action
- action_task_params
- alert
- api_key_pending_invalidation
- cases
- cases-comments
- cases-configure
- cases-connector-mappings
- cases-telemetry
- cases-user-actions
- connector_token
- rules-settings
- maintenance-window
#### `.kibana_security_solution`
- csp-rule-template
- endpoint:user-artifact
- endpoint:user-artifact-manifest
- exception-list
- exception-list-agnostic
- osquery-manager-usage-metric
- osquery-pack
- osquery-pack-asset
- osquery-saved-query
- security-rule
- security-solution-signals-migration
- siem-detection-engine-rule-actions
- siem-ui-timeline
- siem-ui-timeline-note
- siem-ui-timeline-pinned-event
#### `.kibana_analytics`
- canvas-element
- canvas-workpad-template
- canvas-workpad
- dashboard
- graph-workspace
- index-pattern
- kql-telemetry
- lens
- lens-ui-telemetry
- map
- search
- search-session
- search-telemetry
- visualization
#### `.kibana_ingest`
- epm-packages
- epm-packages-assets
- fleet-fleet-server-host
- fleet-message-signing-keys
- fleet-preconfiguration-deletion-record
- fleet-proxy
- ingest_manager_settings
- ingest-agent-policies
- ingest-download-sources
- ingest-outputs
- ingest-package-policies
## Tasks / PRs
### Sub-PRs
**Implementation**
- 🟣https://github.com/elastic/kibana/pull/154846
- 🟣https://github.com/elastic/kibana/pull/154892
- 🟣https://github.com/elastic/kibana/pull/154882
- 🟣https://github.com/elastic/kibana/pull/154884
- 🟣https://github.com/elastic/kibana/pull/155155
**Individual index split**
- 🟣https://github.com/elastic/kibana/pull/154897
- 🟣https://github.com/elastic/kibana/pull/155129
- 🟣https://github.com/elastic/kibana/pull/155140
- 🟣https://github.com/elastic/kibana/pull/155130
### Improvements / follow-ups
- 👷🏼 Extract logic into
[runV2Migration](https://github.com/elastic/kibana/pull/154151#discussion_r1158470566)
@gsoldevila
- Make `getCurrentIndexTypesMap` resillient to intermittent failures
https://github.com/elastic/kibana/pull/154151#discussion_r1169289717
- 🚧 Build a more structured
[MigratorSynchronizer](https://github.com/elastic/kibana/pull/154151#discussion_r1158469918)
- 🟣https://github.com/elastic/kibana/pull/155035
- 🟣https://github.com/elastic/kibana/pull/155116
- 🟣https://github.com/elastic/kibana/pull/155366
## Reallocation tweaks
Tweaks to the reallocation can be done after the initial merge, as long
as it's done before the public release of 8.8
- `url` should get back to `.kibana` (see
[comment](https://github.com/elastic/kibana/pull/154888#discussion_r1172317133))
## Release Note
For performance purposes, Kibana is now using more system indices to
store its internal data.
The following system indices will be created when upgrading to `8.8.0`:
- `.kibana_alerting_cases`
- `.kibana_analytics`
- `.kibana_security_solution`
- `.kibana_ingest`
---------
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
## Summary
Fixes https://github.com/elastic/kibana/issues/143322
This PR adds an option to store dynamic parameters when a step is
completed to be used later for dynamically built URLs.
### How to test
1. Add `xpack.cloud.id: 'testID'` to your `/config/kibana.dev.yml` file
2. Start ES with `yarn es snapshot`
3. Start Kibana with `yarn start --run-examples`
4. Navigate to the guided onboarding example plugin
`http://localhost:5601/app/guidedOnboardingExample`
5. Start the test guide and when completing step 1 provide any value for
the paramter `indexName`
6. Continue the guide until step 4 and check that the correct value is
being used for the url of this step.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/integrations/pull/5123. This PR:
1) Syncs hardcoded list of integration categories against the latest
list from [package
spec](24557251ec/categories/categories.yml)
2) Brings the categories used by the "home tutorial" cards and custom
integrations cards in line with the above PR
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
https://github.com/elastic/kibana/pull/151788 established that data
views would no longer error when they failed to retrieve a field list.
This index pattern validation code exists since previously ALL index
pattern segments needed to match in order to avoid an error response
from field caps, rather than just one. Now we can remove the validation
code and simply pass the index pattern to field caps directly.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
### Notes for reviewers
- Currently, autocomplete suggestions for fields don't work with
wildcards and data streams due to the
[bug](https://github.com/elastic/kibana/issues/149496) in the `main`. It
should be addressed separately.
### How to test
In order to spot the loading behaviour, ideally you should create an
index with a heavy mappings definition.
Afterwards, write a query that requires a field from this index, e.g.:
```
GET <my-index>/_search
{
"aggs": {
"my_agg": {
"terms": {
"field": "",
"size": 10
}
}
}
}
```
Place a cursor next to the `field` property, it should trigger mappings
fetch. After that, the mappings definition for this index will be cached
and accessed synchronously.
You can also open the browser's dev tools and enable Network throttling.
It allows noticing loading behaviour for any index.
--------------------
Resolves https://github.com/elastic/kibana/issues/146855
Instead of fetching all mappings upfront, requests mapping definition on
demand per index according to the cursor position.
Considering there is a maximum response size limit of 10MB in the
`/autocomplete_entities` endpoint, field autocompletion wasn't working
at all if the overall mappings definition exceeded this size. Retrieving
mappings per index tackles this and improves the init time.
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
---------
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/149691
Follow up to https://github.com/elastic/kibana/pull/149528
This PR adds 3 separate guide ID values `appSearch`, `websiteSearch` and
`databaseSearch` instead of previously used `search`. With that change,
each card for Search on the landing page has its own independent guide
that can be completed.
Example plugin for guided onboarding is also updated so that all 3
search guides can be managed there. Otherwise there should not be any
UI/UX changes.
Currently, all 3 ids are registered with the same config and that can be
easily updated by the team in the future.
#### Screenshot
<img width="485" alt="Screenshot 2023-01-31 at 18 24 12"
src="https://user-images.githubusercontent.com/6585477/215840343-f9a65be3-0568-40ca-b458-5de92bfe8aaf.png">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR removes the deprecated field list sampling setting which was
marked as deprecated on 8.1. This
https://github.com/elastic/kibana/pull/139828 was the PR that added the
deprecation callout.
This PR moves the test guide used for testing and dev work to the guided
onboarding package. It is now being registered when the guided
onboarding example plugin is being setup. For that a server side is
added to the example plugin.
Registering the test guide from the example plugin demonstrates how the
"register guide" function can be used for production guides. The test
config needs to be in the `kbn-guided-onboarding` package because
example plugins have some problems importing files from other plugins.
That way the test config is available to the example plugin and to the
guided onboarding plugin for unit and functional tests.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds new optional `fields` argument to the data view fields for wildcard
api that allows the consumer to request only specific fields as opposed
to always getting all fields.
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
## Summary
Fixes https://github.com/elastic/kibana/issues/144452
This PR updates the guide IDs used in guided onboarding from generic
values (`observability` and `security`) to more specific values
(`kubernetes` and `siem`) which will allow us to add more guides for
observability and security in the future.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
