Addresses the following feedback:
https://github.com/elastic/kibana/pull/154151#discussion_r1158470566
Similar to what has been done for ZDT, the goal of this PR is to extract
the logic of the `runV2Migration()` from the `KibanaMigrator` into a
separate file.
The PR also fixes some incomplete / incorrect UTs and adds a few missing
ones.
## Summary
Add validation to ensure that mappings added via a `mappings_addition`
type change are also present in the type's full mappings.
(just an extra layer of protection for when the teams will switch to
using the new API)
## Summary
`@elastic/eui@80.0.0` ⏩ `@elastic/eui@81.0.0`
---
## [`81.0.0`](https://github.com/elastic/eui/tree/v81.0.0)
- Added ability to set `options.checked` to "mixed" in `EuiSelectable`
([#6774](https://github.com/elastic/eui/pull/6774))
**Bug fixes**
- Portalled components (e.g. `EuiPopover`, `EuiModal`, `EuiFlyout`) will
correctly inherit text color from its nearest `EuiThemeProvider` parent.
`<EuiText color="default">` is no longer needed.
([#6775](https://github.com/elastic/eui/pull/6775))
**Breaking changes**
- `EuiSelectable` no longer renders a `data-test-selected` attribute on
its list items. Use the `aria-checked` property instead
([#6774](https://github.com/elastic/eui/pull/6774))
- Nested `EuiThemeProvider`s now render a wrapping `<span>` element in
order to correctly set the inherited text `color` of all descendants.
`<EuiText color="default">` is no longer needed.
([#6775](https://github.com/elastic/eui/pull/6775))
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Constance Chen <constance.chen@elastic.co>
Tackles https://github.com/elastic/kibana/issues/155136
When an upgrade fails, a cluster might be on a partially migrated state
(with some indices already updated to the newer version). When rolling
back to the previous version, in ESS, this can cause these indices to be
`write_blocked`.
This PR aims at detecting this situation and failing early, effectively
preventing to `write_block` any indices.
## Summary
Related to https://github.com/elastic/kibana/issues/150296
Use a `range` clause for version filtering for the ZDT outdated doc
query, similar to what is already done for the `v2` algo, to avoid
eventually trying to migrate documents from higher versions (which would
result in a migration failure)
## Summary
Migrations read 1000 documents by default which works well for most
deployments. But if any batch happens to be > ~512MB we hit NodeJS' max
string length limit and cannot process that batch. This forces users to
reduce the batch size to a smaller number which could severely slow down
migrations.
This PR reduces the impact of large batches by catching
elasticsearch-js' `RequestAbortedError` and reducing the batch size in
half. When subsequent batches are successful the batchSize increases by
20%. This means we'll have a sequence like:
1. Read 1000 docs ✅ (small batch)
2. Read 1000 docs 🔴 (too large batch)
3. Read 500 docs ✅
4. Read 600 docs ✅
5. Read 720 docs ✅
6. Read 864 docs ✅
7. Read 1000 docs ✅ (small batch)
This assumes that most clusters just have a few large batches exceeding
the limit. If all batches exceed the limit we'd have 1 failure for every
4 successful reads so we pay a 20% throughput penalty. In such a case it
would be better to configure a lower `migrations.batchSize`.
Tested this manually:
1. Start ES with more heap than the default, otherwise reading large
batches will cause it to run out of memory
`ES_JAVA_OPTS=' -Xms6g -Xmx6g' yarn es snapshot
--data-archive=/Users/rudolf/dev/kibana/src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip`
2. Ingest lots of large documents of ~5mb
```
curl -XPUT
"elastic:changeme@localhost:9200/_security/role/grant_kibana_system_indices"
-H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"indices": [
{
"names": [
".kibana*"
],
"privileges": [
"all"
],
"allow_restricted_indices": true
}
]
}'
curl -XPOST "elastic:changeme@localhost:9200/_security/user/superuser"
-H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"password" : "changeme",
"roles" : [ "superuser", "grant_kibana_system_indices" ]
}'
curl -XPUT
"superuser:changeme@localhost:9200/.kibana_8.4.0_001/_mappings" -H
"kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"dynamic": false,
"properties": {
}
}'
set -B # enable brace expansion
for i in {1..400}; do
curl -k --data-binary
"@/Users/rudolf/dev/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/body.json"
-X PUT
"http://superuser:changeme@localhost:9200/.kibana_8.4.0_001/_doc/cases-comments:"{$i}"?&pretty=true"
-H "Content-Type: application/json"
done
```
3. Start Kibana with a modest batchSize otherwise we could OOM ES `node
scripts/kibana --dev --migrations.batchSize=120`
<details><summary>Example logs. Note the "Processed x documents" only
logs when the next batch is successfull read, so the order seems wrong.
To improve it we'd need to log progress after a batch is successfully
written instead 🤷 </summary>
```
[.kibana] Processed 120 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3667ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1740ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1376ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1402ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1311ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1388ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 900ms.
[.kibana] Read a batch that exceeded the NodeJS maximum string length, retrying by reducing the batch size in half to 60.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_READ. took: 1538ms.
[.kibana] Processed 240 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 2054ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1042ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1310ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1388ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 1130ms.
[.kibana] Processed 300 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 2610ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1262ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1299ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1363ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1341ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 572ms.
[.kibana] Processed 372 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3330ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1488ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1349ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1312ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1380ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1310ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 139ms.
[.kibana] Processed 458 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3278ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1460ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1370ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1303ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1384ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 1298ms.
[.kibana] Processed 542 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 4ms.
```
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risks
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
## Summary
Fix https://github.com/elastic/kibana/issues/158372
Change the logic of the `zdt` algorithm to be able to re-use an existing
"v2 managed" cluster state (a kibana cluster previously running the v2
migration algorithm) if compatible.
Technically the v2 index state is compatible with zdt after 8.8 (when we
introduced `indexTypeMap` and restricted to compatible mapping changes
only).
See https://github.com/elastic/kibana/issues/158372 for more detailed
explanations.
## Summary
Adds build date to `GET kbn:api/status` similar to ES. Example output
running locally:
```json
{
"name": "ES-DMVD5M3",
"uuid": "545ba70c-063e-449b-af21-6c8e7b30f77e",
"version": {
"number": "8.9.0",
"build_hash": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"build_number": 9007199254740991,
"build_snapshot": false,
"build_date": "2023-05-15T23:12:09.000Z"
},
...rest
}
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Release Note
The status endpoint now returns the build date alongside other build
information.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
part of https://github.com/elastic/kibana/issues/154307
PR adds ability to put TSVB into read only mode - preventing TSVB
visualizations from being created and edited.
To test:
* start kibana with `yarn start --serverless=es`
* add `vis_type_timeseries.readOnly: true` to kibana.yml
Visualization public plugin changes:
* Removes `hideTypes` from VisualizationSetup contract. Used by Maps
plugin to set "hidden" to true for tile_map and region_map visualization
types. In 8.0, tile_map and region_map visualization type registration
moved into maps plugin so `hideTypes` no longer needed.
* Renamed vis type definition `hidden` to `disableCreate`.
* Added `disableEdit` to vis type definition.
* Hide edit link in dashboard panel options when `disableEdit` is true
* Does not display links and edit action in listing table when
`disableEdit` is true
Visualization server plugin changes:
* Add `readOnlyVisType` registry to set up contract
* Update visualization savedObject.management.getInAppUrl to return
undefined when vis type has been registered as readOnly.
* Prevents "readOnly "visualization types from being displayed in global
search results
* Prevents "readOnly "visualization types from having links in saved
object management listing table.
Timeseries server plugin changes:
* Add `readOnly` yaml configuration
* Expose `readOnly` yaml configuration to public
* When `readOnly` is true, call
VisualizationsServerSetup.registerReadOnlyVisType to mark vis type as
read only
Timeseries public plugin changes:
* Set disableCreate and disableEdit to true when `readOnly` is true
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
Sets `switchToModelVersionAt` to `8.10.0` for all registered SO types,
forcing them to switch to the model version API once `main` is on
version `8.10.0` (~end of june)
## Summary
Part of https://github.com/elastic/kibana/issues/150312
(next steps depend on https://github.com/elastic/kibana/pull/153117)
**This PR does two things:**
- introduce the concept of version persistence schema
- adapt the document migrator to support downward migrations for
documents of an higher version.
In the follow-up, we will then update the calls from the SOR to the
document migrator to allow downward conversions when we're using the ZDT
migration algorithm (which requires
https://github.com/elastic/kibana/pull/153117 to be merged)
### Model version persistence schema.
*(This is what has also been named 'eviction schema' or 'known fields
schema'.)*
A new `SavedObjectsModelVersion.schemas.backwardConversion` property was
added to the model version definition.
This 'schema' can either be an arbitrary function, or a `schema.object`
from `@kbn/config-schema`
```ts
type SavedObjectModelVersionBackwardConversionSchema<
InAttrs = unknown,
OutAttrs = unknown
> = ObjectType | SavedObjectModelVersionBackwardConversionFn<InAttrs, OutAttrs>;
```
When specified for a version, the document's attributes will go thought
this schema during down conversions by the document migrator.
### Adapt the document migrator to support downward migrations for
documents of an higher version.
Add an `allowDowngrade` option to `DocumentMigrator.migrate` and
`KibanaMigrator.migrateDocument`. When this option is set to `true`, the
document migration will accept to 'downgrade' the document if necessary,
instead of throwing an error as done when the option is `false` or
unspecified (which was the only behavior prior to this PR's changes)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
After changing the UserSettingService to calculate darkmode and return
`boolean | undefined` , the Rendering service `darkMode` logic needed to
be updated to work when a User chooses 'Light' which provides a 'false'
value to the Rendering service.
## Testing
For Space Setting:
1. Set Space Adv. Setting to darkMode: true
2. Set User Profile Setting to 'Light'
3. Observe that Light mode takes precedence
For Config setting:
1. Set User Profile Setting to 'Dark'
2. In `kibana.yml` set `uiSettings.overrides.theme:darkMode: false`
3. Observe that Light mode takes precedence
## Summary
Part of https://github.com/elastic/kibana/issues/158034
This PR moves the TopNavMenu placement to below the breadcrumbs bar for
the serverless project chrome layout.
The app toolbar must appear below the breadcrumbs bar in the layout,
when the app has TopNavMenu items registered with the navigation
service.
To prevent the layout from rendering an empty container, I had to create
a new hook, `useHeaderActionMenuMounter`, that extracts the state from
HeaderActionMenu. The state (`mounter: MountPoint | undefined`) is
hoisted above so that the header can check if it is empty before
rendering the container.
_Future work is still needed to achieve the end goal of the design._
Apps must be able to define custom layouts, which commonly require grid
and spacer components. The TopNavMenu needs augmentation to support the
new API. See the project document from the issue.
## Screenshots
**App showing the toolbar, with the container shown below breadcrumbs**
**App without the toolbar and no empty container**
## Summary
Use migrations.batchSize config for the scroll_size in update_by_query /
updateAndPickupMappings. The default scroll_size=1000 can sometimes
cause Elasticsearch to fail with `RecyclerBytesStreamOutput cannot hold
more than 2GB of data`
On CI our Elasticsearch cluster does not have enough memory to reproduce
`RecyclerBytesStreamOutput` error it OOMs before it's able to load 2GB.
However it's possible to test manually:
1. Start Elasticsearch with 8GB heap `ES_JAVA_OPTS=' -Xms8g -Xmx8g' yarn
es snapshot`
2. Ingest > 2GB of saved objects distributed over batchSize documents (<
1000)
```
curl -XPOST "elastic:changeme@localhost:9200/_security/user/superuser"
-H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"password" : "changeme",
"roles" : [ "superuser", "grant_kibana_system_indices" ]
}'
curl -XPUT
"superuser:changeme@localhost:9200/.kibana_8.4.0_001/_mappings" -H
"kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"dynamic": false,
"properties": {
}
}'
set -B # enable brace expansion
for i in {1..500}; do
curl -k --data-binary "@/Users/rudolf/dev/kibana/body.json" -X PUT
"http://superuser:changeme@localhost:9200/.kibana_8.4.0_001/_doc/cases-comments:"{$i}"?&pretty=true"
-H "Content-Type: application/json"
done
curl -XPOST "superuser:changeme@localhost:9200/_aliases" -H "kbn-xsrf:
reporting" -H "Content-Type: application/json" -d'
{
"actions": [
{
"add": {
"index": ".kibana_8.4.0_001",
"alias": ".kibana_8.4.0"
}
},
{
"add": {
"index": ".kibana_8.4.0_001",
"alias": ".kibana"
}
}
]
}'
```
body.json
```
{
"cases-comments": {
"comment": "...put lots of data here...",
"type": "user",
"owner": "cases",
"created_at": "2023-05-09T08:07:50.121Z",
"created_by": {
"email": null,
"full_name": null,
"username": "elastic"
},
"pushed_at": null,
"pushed_by": null,
"updated_at": null,
"updated_by": null
},
"type": "cases-comments",
"references": [
{
"type": "cases",
"name": "associated-cases",
"id": "9563b290-ee40-11ed-8fcc-e975e7d47f63"
}
],
"namespaces": [
"default"
],
"migrationVersion": {
"cases-comments": "8.6.0"
},
"coreMigrationVersion": "8.7.2",
"updated_at": "2023-05-09T08:07:50.168Z",
"created_at": "2023-05-09T08:07:50.168Z"
}
```
3. Run Kibana with default and smaller migrations.batchSize
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
The spread operator is costly and put pressure on GC. It should be
avoided when possible, especially in loops.
This PR adapts a lot of `reduce` calls in the codebase to remove the
usages of the diabolic spread operator, when possible.
Note: the PR is not fully exhaustive. I focused on the server-side, as
we're more directly impacted than on browser-side code regarding
performances.
## Removing `...` usages in `kittens.reduce()`
For `reduce` loops, the spread operator can usually easily be replaced:
#### - setting a value on the accum object and returning it
#### BAD
```ts
return this.toArray().reduce(
(acc, renderer) => ({
...acc,
[renderer.name]: renderer,
}),
{} as Record<string, ExpressionRenderer>
);
```
#### GOOD
```ts
return this.toArray().reduce((acc, renderer) => {
acc[renderer.name] = renderer;
return acc;
}, {} as Record<string, ExpressionRenderer>);
```
#### - assigning values to the accum object and returning it
#### BAD
```ts
const allAggs: Record<string, any> = fieldAggRequests.reduce(
(aggs: Record<string, any>, fieldAggRequest: unknown | null) => {
return fieldAggRequest ? { ...aggs, ...(fieldAggRequest as Record<string, any>) } : aggs;
},
{}
);
```
#### GOOD
```ts
const allAggs = fieldAggRequests.reduce<Record<string, any>>(
(aggs: Record<string, any>, fieldAggRequest: unknown | null) => {
if (fieldAggRequest) {
Object.assign(aggs, fieldAggRequest);
}
return aggs;
},
{}
);
```
#### - pushing items to the accum list and returning it
#### BAD
```ts
const charsFound = charToArray.reduce(
(acc, char) => (value.includes(char) ? [...acc, char] : acc),
[] as string[]
);
```
#### GOOD
```ts
const charsFound = charToArray.reduce((acc, char) => {
if (value.includes(char)) {
acc.push(char);
}
return acc;
}, [] as string[]);
```
## Questions
#### Are you sure all the changes in this are strictly better for
runtime performances?
Yes, yes I am.
#### How much better?
Likely not much.
#### Are you planning on analyzing the perf gain?
Nope.
#### So why did you do it?
I got tired of seeing badly used spread operators in my team's owned
code, and I had some extra time during on-week, so I spent a few hours
adapting the usages in all our runtime/production codebase.
#### Was it fun?
Take your best guess.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds boilerplate code and a few initial end-to-end tests to
serverless plugins.
Note that the tests defined in this PR are not part of any CI run yet,
this will be done in a follow-up after this PR is merged.
### Details
The serverless test structure corresponds to what we have in
`x-pack/test` with API tests in `api_integration` and UI tests in
`functional`, each with their set of helper methods and sub-directories
for
- `common` functionality shared across serverless projects (core, shared
UX, ...)
- `observability` project specific functionality
- `search` project specific functionality
- `security` project specific functionality
The `shared` directory contains fixtures, services, ... that are shared
across `api_integration` abd `functional` tests.
```
x-pack/test_serverless/
├─ api_integration
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ functional
│ ├─ page_objects
│ ├─ services
│ ├─ test_suites
│ │ ├─ common
│ │ ├─ observability
│ │ ├─ search
│ │ ├─ security
├─ shared
│ ├─ services
│ ├─ types
```
See also `x-pack/test_serverless/README.md`
### Run tests
Similar to how functional tests are run in `x-pack/test`, you can point
the functional tests server and test runner to config files in this
`x-pack/test_serverless` directory, e.g. from the `x-pack` directory
run:
```
node scripts/functional_tests_server.js --config test_serverless/api_integration/test_suites/common/config.ts
```
and
```
node scripts/functional_test_runner.js --config test_serverless/api_integration/test_suites/common/config.ts
```
### Additional changes
- The stateful `common_page` page object used the existence of the
global nav to determine `isChromeVisible` and `isChromeHidden`, which is
not working when the global nav is disabled. To solve this, a
`data-test-subj` that indicates the chrome visible state is added to the
Kibana app wrapper and is used for the checks.
- Add a few `data-test-subj` entries to the Observability overview page.
- Add optional `dataTestSubj` to the `Navigation` component and use that
for the serverless search nav.
- Add optional `titleDataTestSubj` to the `SolutionNav` component and
use it for the serverless security nav.
- Add a data-test-subj entry to the Search overview page.
* Add deferred migrations parameter.
* Update outdated documents query to take into account deferred migrations.
* Update outdated documents query to take into account the core migration version.
* Update read operations in the saved objects repository to perform deferred migrations.
## Summary
Closes https://github.com/elastic/kibana/issues/152833
This PR:
- Removes the visualize:enableLabs setting which doesn't do anything
after the presentation team decided to hide the creation of the legacy
input controls from the UI
- Cleanups wherever the components associated with this feature were
used
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Tackles https://github.com/elastic/kibana/issues/157968
When creating new indices during SO migrations, we used to rely on the
`res.acknowledged && res.shardsAcknowledged` of the
`esClient.indices.create(...)` to determine that the indices are ready
to use.
However, we believe that due to certain race conditions, this can cause
Kibana migrations to fail (refer to the [related
issue](https://github.com/elastic/kibana/issues/157968)).
This PR aims at fixing recent CI failures by adding a systematic
`waitForIndexStatus` after creating an index.
## Summary
The versioned router internally uses the IRouter and currently
incorrectly passes through options. TS was not complaining due to
structural type checking seeing this as valid. Added a test to ensure
all options are being passed through as expected.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Current internal version check uses `parseFloat` which successfully
parses values like `2023-01-23` which is not in keeping with the
intended pattern of whole numbers for internal routes. This PR
simplifies this logic by using a regex `^[0-9]+$` to enforce internal
versions are 1 or more numbers only.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Addresses
https://github.com/elastic/kibana/pull/156600#discussion_r1186071163
> Let's think if there is a way to throw an error when core chrome api
are called from invalid plugins (in this cases only the serverless
plugin would be allowed.
This PR can be a starting point for discussion on the behavior we really
want. This PR has a simple goal to ensure that non-serverless plugins do
not call the `chrome.projects` API. However, it's not complete security,
as the compile-time error would be easy to override.
cc @sebelga @Dosant @clintandrewhall
---
---
### Checklist
Delete any items that are not applicable to this PR.
- [x] Documentation was added for features that require explanation or
tutorials
Internal documentation:
https://docs.google.com/document/d/1ew8KYl6ROs_V7jeIXgeP_C9YgkYK2IPtuceo6KVF_jE/edit#
---------
Co-authored-by: Clint Andrew Hall <clint@clintandrewhall.com>
Steps to view problem
* install sample data set
* Open lens visualization
* Open inspector. Notice console errors
<img width="300" alt="Screen Shot 2023-05-05 at 11 03 25 AM"
src="https://user-images.githubusercontent.com/373691/236521366-d8fb9302-e93b-4047-a0bf-d7c09dcc3ffb.png">
https://github.com/elastic/eui/pull/6566 removed `closeButtonAriaLabel`
prop from [EuiFlyout](https://elastic.github.io/eui/#/layout/flyout) EUI
75.0.0 (Effecting 8.8 and 8.9). FlyoutService spreads options into
`EuiFlyout`, resulting in `closeButtonAriaLabel` getting added to dom
and causing error.
`OverlayFlyoutOpenOptions` type added by
https://github.com/elastic/kibana/issues/37894. I replaced
`OverlayFlyoutOpenOptions` with `EuiFlyoutProps` to make it more clear
what props are accepted and provide stronger typing that stays in sync
with EUI typings
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Follow-up of https://github.com/elastic/kibana/pull/157154
Continue the cleanup started in the previous PR, by moving more things
around.
- Move everything search-related (dsl, aggregations, kql utils) to a
dedicated `search` folder
- Move a few more things to `/apis/internals`
- Remove the 'v5' field compatibility in the field list generation (see
comment)
- Cleanup some files a bit.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Structural cleanup of the `SavedObjectsRepository` code, by extracting
the actual implementation of each API to their individual file (as it
was initiated for some by Joe a while ago, e.g `updateObjectsSpaces`)
### Why doing that, and why now?
I remember discussing about this extraction with Joe for the first time
like, what, almost 3 years ago? The 2.5k line SOR is a beast, and the
only reason we did not refactor that yet is because of (the lack of)
priorization of tech debt (and lack of courage, probably).
So, why now? Well, with the changes we're planning to perform to the SOR
code for serverless, I thought that doing a bit of cleanup beforehand
was probably a wise thing. So I took this on-week time to tackle this (I
know, so much for an on-week project, right?)
### API extraction
All of these APIs in the SOR class now look like:
```ts
/**
* {@inheritDoc ISavedObjectsRepository.create}
*/
public async create<T = unknown>(
type: string,
attributes: T,
options: SavedObjectsCreateOptions = {}
): Promise<SavedObject<T>> {
return await performCreate(
{
type,
attributes,
options,
},
this.apiExecutionContext
);
}
```
This separation allows a better isolation, testability, readability and
therefore maintainability overall.
### Structure
```
@kbn/core-saved-objects-api-server-internal
- /src/lib
- repository.ts
- /apis
- create.ts
- delete.ts
- ....
- /helpers
- /utils
- /internals
```
There was a *massive* amount of helpers, utilities and such, both as
internal functions on the SOR, and as external utilities. Some being
stateless, some requiring access to parts of the SOR to perform calls...
I introduced 3 concepts here, as you can see on the structure:
#### utils
Base utility functions, receiving (mostly) parameters from a given API
call's option (e.g the type or id of a document, but not the type
registry).
#### helpers
'Stateful' helpers. These helpers were mostly here to receive the
utility functions that were extracted from the SOR. They are
instantiated with the SOR's context (e.g type registry, mappings and so
on), to avoid the caller to such helpers to have to pass all the
parameters again.
#### internals
I would call them 'utilities with business logic'. These are the 'big'
chunks of logic called by the APIs. E.g `preflightCheckForCreate`,
`internalBulkResolve` and so on.
Note that given the legacy of the code, the frontier between those
concept is quite thin sometimes, but I wanted to regroups things a bit,
and also I aimed at increasing the developer experience by avoiding to
call methods with 99 parameters (which is why the helpers were created).
### Tests
Test coverage was not altered by this PR. The base repository tests
(`packages/core/saved-objects/core-saved-objects-api-server-internal/src/lib/repository.test.ts`)
and the extension tests
(`packages/core/saved-objects/core-saved-objects-api-server-internal/src/lib/repository.{ext}_extension.test.ts`)
were remain untouched. These tests are performing 'almost unmocked'
tests, somewhat close to integration tests, so it would probably be
worth keeping them.
The new structure allow more low-level, unitary testing of the
individual APIs. I did **NOT** add proper unit test coverage for the
extracted APIs, as the amount of work it represent is way more
significant than the refactor itself (and, once again, the existing
coverage still applies / function here).
The testing utilities and mocks were added in the PR though, and an
example of what the per-API unit test could look like was also added
(`packages/core/saved-objects/core-saved-objects-api-server-internal/src/lib/apis/remove_references_to.test.ts`).
Overall, I think it of course would be beneficial to add the missing
unit test coverage, but given the amount of work it represent, and the
fact that the code is already tested by the repository test and the
(quite exhaustive) FTR test suites, I don't think it's worth the effort
right now given our other priorities.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Changes the way model versions are defined, using a design more
appropriate to the iterative approach we decided to take regarding
delivering/exposing more capabilities to model versions.
A model version's changes is now defined by composition of multiple
low-level 'change' blocks (compared to high-level migration blocks as it
was previously designed).
This PR introduces the following type of 'model changes':
#### - `mappings_addition`
```ts
/**
* A {@link SavedObjectsModelChange | model change} adding new mappings.
*
* @remark when adding mappings, {@link SavedObjectsType.mappings | the type mappings} must also be updated accordingly.
* Overall, the type's mapping represents the latest version of the mappings, where the model changes
* represent the changes of mappings between two versions.
*
* @public
*/
export interface SavedObjectsModelMappingsAdditionChange {
type: 'mappings_addition';
/**
* The new mappings introduced in this version.
*/
addedMappings: SavedObjectsMappingProperties;
}
```
#### - `mappings_deprecation`
```ts
/**
* A {@link SavedObjectsModelChange | model change} flagging mappings as being deprecated.
* Deprecated mappings should no longer be used and will eventually be deleted later.
*/
export interface SavedObjectsModelMappingsDeprecationChange {
type: 'mappings_deprecation';
/**
* A list of paths to mappings to flag as deprecated.
*/
deprecatedMappings: string[];
}
```
#### - `data_backfill`
```ts
/**
* A {@link SavedObjectsModelChange | model change} used to backfill fields introduced in the same model version.
*
* @remark This type of model change should only be used to backfill newly introduced fields.
* Even if no check is performed to ensure that, using such transformations to mutate
* existing data of the document can lead to data corruption or inconsistency.
*/
export interface SavedObjectsModelDataBackfillChange<
PreviousAttributes = any,
NewAttributes = any
> {
type: 'data_backfill';
/**
* The backfill function to run.
*/
transform: SavedObjectModelDataBackfillFn<PreviousAttributes, NewAttributes>;
}
```
#### Example of definition
```ts
const myType = {
name: 'my_type',
mappings: {
properties: {
text: { type: 'text' },
keyword: { type: 'keyword' },
newField: { type: 'text' },
},
},
switchToModelVersionAt: '8.0.0',
modelVersions: {
0: {
changes: [],
},
1: {
changes: [
{
type: 'mappings_addition',
addedMappings: { newField: { type: 'text' } },
},
{
type: 'data_backfill',
transform: (doc) => {
return {
document: {
...doc,
attributes: {
...doc.attributes,
newField: `new ${doc.id}`,
},
},
};
},
},
],
},
},
}
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/156423
We were only using the schema for validation if its version was an
*exact* match with the current stack version, meaning that any previous
schema was ignored (and even introducing a minor was causing the schema
to be ignored).
This PR addresses it, by always using the closest previous schema
available.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of https://github.com/elastic/kibana/issues/150309
A few enhancements to the ZDT migration algorithm.
### 1. Run the 'expand' phase (and only this one) on non-migrator nodes
Given our latests changes to the way we want the algo to function, the
non-migrator nodes will have to run the 'expand' (schema expansion)
phase. However, the document migration phase will have to be run by the
migrator node exclusively.
Note: because it was required for integration tests, a new
`migration.zdt.runOnNonMigratorNodes` option was introduced to change
this behavor and have non-migrator nodes ignore this limitation.
### 2. Don't terminate during `INIT` if higher mapping versions are
found
Any mapping changes are upward compatible, meaning that we can safely
no-op instead of failing of the mapping version check result is
`lesser`. This change is required now that mapping updates will be
performed before all nodes of the previous version are shut down (and is
also required for rollbacks)
### 3. Perform a version check during `DOCUMENTS_UPDATE_INIT`
We were always executing the full doc update cycle when entering this
stage. We're now performing a version check similar to what was done
during `INIT`.
If the check result returns:
- `greater`: we perform the document migration (as it was done before
this change)
- `equal`: we skip the document migration
- `lesser`: we skip the document migration (**NOTE**: this may change
later depending on how we handle rollbacks)
- `conflict`: we terminate with a failure, as done during `INIT`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds the `VersionedRouter` to the public http contract.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
In this PR I've exposed a new api from the Chrome service to set a
custom React component for the side navigation.
Calling this API does not have any effect if Kibana is not started in
"project" ChromeStyle.
## Api
```ts
chrome.project.setSideNavComponent(/* component */);
```
## Example
```ts
// my_nav.tsx
import React, { type FC } from 'react';
import type { SideNavComponent } from '@kbn/core-chrome-browser';
// SideNavComponent will receive in its props Chrome service state (the current active route, the navTree...)
export const MyNav: SideNavComponent = (props) => {
return <div>Custom navigation...</div>;
};
-------------------
// serverless plugin.ts (public)
import { MyNav } from './my_nav';
...
public start(core: CoreStart) {
core.chrome.project.setSideNavComponent(MyNav);
return {};
}
```
## Summary
Adapt the `zdt` migration algorithm to run the v2 migrations in addition
to the model version transformations.
The intent is to be able to use the current migration system in a
zero-downtime upgrade friendly-ish way
### Technicals
https://github.com/elastic/kibana/pull/148656 was a precondition, as the
zdt algo requires that mapping changes are compatible (given we keep the
same index).
Technically, it means we're now storing the `virtualVersion` instead of
the `modelVersion` in the index's meta (`mappingVersions` and
`docVersions`), to allow keeping track of mixed stack and model versions
per type.
Note that switching to model versioning is still a one way,
non-revertible action. Once using model versioning (by specifying
`switchToModelVersionAt` on your type), there is no going back.
