## Summary
Original ticket https://github.com/elastic/kibana/issues/158110
These changes fixes the issue with the exception flyout which can be
stuck in loading state in case `getFieldsForIndexPattern` throws an
exception.
Fixed by putting the `getFieldsForIndexPattern` call in try/catch. We
use this call to fetch extended information about the fields [to show
warning to the user in case there are some index
issues](https://github.com/elastic/kibana/pull/149149). If
`getFieldsForIndexPattern` fails and throws an exception we will
continue using fields without conflicts/unmapped information.
I also, noticed that we do not fetch extended information for the case
where user uses index patterns instead of data views. Fixed this issue
in `useFetchIndex`.
cc @dhurley14 We will need to adjust either of our PRs depending whose
changes will go in first :-)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Addresses: #105371
One can use `--elasticsearch.password ---my-password---`,
`--elasticsearch.password=---mypassword---` or `--elasticsearch.password
\"---mypassword\"` both in these cases. Commander is splitting
single-dash inputs, it can't be prevented, so for those cases, use
quotes or compound options.
The original problem can be reproduced like this:
```shell
yarn start --elasticsearch.password "-some-password"
```
We were using some extra argument parsing prototype extension in
(command.js)[./src/cli/command.js] - this allows for collecting extra
options, but the proprietary logic had a few incorrect assumptions.
(e.g.: extra opts will be already split by `=` by the time we get them,
shorthands will be split to individual flags).
(1) This change makes it explicit in the code, that the single-letter
flags (`-fcz`) are not supported in the extra options.
The core of the problem is now how we treat an ambiguous expression like
this:
```
kibana --password "--kebabs--are--my--favourite--" --somethingElse 283
```
Is the `"--kebabs--are--my--favourite--"` another flag, or is it a value
for the previous option?
Unfortunately, both interpretations will have shortcomings.
- if we treat it as a boolean flag, then values starting with "--" will
be incorrectly treated as flags, and an error will be displayed (in the
case of a very unfortunate random generated string)
- if we treat it as a value always, then incorrectly ordered
options/flags will be consumed in place of values
Maybe the latter one will happen less often. For that, people would have
to incorrectly assume that `--booleanOn` type flags exist, especially
among the non-listed options. Or they'd have to incorrectly parameterize
the command, for that they can review their own usage locally, and see
their error in the ordering.
(2) This change makes it explicit, that extra options will come in
pairs.
Also, added tests for the base use case, and the _dash_ edge case.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Checked and works on 7.17/7.13 (it was originally reported on
7.13).
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR addresses Cypress parallelisation in Security Solution scope. It
is a first step in improving e2e tests reliability and gives us a solid
foundation for further work on flakiness and run times. With this PR
each CI job starts 3 instances of ES, kibana and Cypress runner in
isolation.
Other issues addressed in this PR:
- Use click() instead of trigger('click') in all Cypress tests.
- Use testIsolation in all Cypress tests.
- login before each test to ensure that the correct user is always
logged in. Also, with test isolation enabled, login should be required
before each test.
- use visit() instead of cy.visit() in some cases so that tests wait for
the page to fully load before executing.
- Cypress e2e tests are no longer compatible with Firefox (or any
non-Chromium based browser) due to using cypress-real-events
- Removes uses of cypress-pipe. Tests that used cypress-pipe in order to
retry click events will no longer retry click events.
- Fixes an error related to test files being overwritten by Webpack file
processing
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of: [#156696](https://github.com/elastic/kibana/issues/156696)
## Summary
This PR refactors some of the newly introduced Asset Details embeddable
code to make it easier to consume by other pages, decoupling its props
from what the Host View used to send to the old Host View Flyout.
Besides adjustments to the component responsiveness, this PR also
improves the logic around adding tabs and managing their state.
### How to test this PR
- Run `yarn storybook infra`
- Check all the storybooks related to the Asset Detail Embeddable
- Start Kibana
- Go to the Hosts View page, open the flyout, and play with it.
### For reviewers
The Asset Details Embeddable still depends on a few things from the
`pages/*` folder, and a couple of components are still tightly coupled
with the Host View. We'll address these things to keep this PR small in
the next ones.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Migrations read 1000 documents by default which works well for most
deployments. But if any batch happens to be > ~512MB we hit NodeJS' max
string length limit and cannot process that batch. This forces users to
reduce the batch size to a smaller number which could severely slow down
migrations.
This PR reduces the impact of large batches by catching
elasticsearch-js' `RequestAbortedError` and reducing the batch size in
half. When subsequent batches are successful the batchSize increases by
20%. This means we'll have a sequence like:
1. Read 1000 docs ✅ (small batch)
2. Read 1000 docs 🔴 (too large batch)
3. Read 500 docs ✅
4. Read 600 docs ✅
5. Read 720 docs ✅
6. Read 864 docs ✅
7. Read 1000 docs ✅ (small batch)
This assumes that most clusters just have a few large batches exceeding
the limit. If all batches exceed the limit we'd have 1 failure for every
4 successful reads so we pay a 20% throughput penalty. In such a case it
would be better to configure a lower `migrations.batchSize`.
Tested this manually:
1. Start ES with more heap than the default, otherwise reading large
batches will cause it to run out of memory
`ES_JAVA_OPTS=' -Xms6g -Xmx6g' yarn es snapshot
--data-archive=/Users/rudolf/dev/kibana/src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip`
2. Ingest lots of large documents of ~5mb
```
curl -XPUT
"elastic:changeme@localhost:9200/_security/role/grant_kibana_system_indices"
-H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"indices": [
{
"names": [
".kibana*"
],
"privileges": [
"all"
],
"allow_restricted_indices": true
}
]
}'
curl -XPOST "elastic:changeme@localhost:9200/_security/user/superuser"
-H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"password" : "changeme",
"roles" : [ "superuser", "grant_kibana_system_indices" ]
}'
curl -XPUT
"superuser:changeme@localhost:9200/.kibana_8.4.0_001/_mappings" -H
"kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"dynamic": false,
"properties": {
}
}'
set -B # enable brace expansion
for i in {1..400}; do
curl -k --data-binary
"@/Users/rudolf/dev/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/body.json"
-X PUT
"http://superuser:changeme@localhost:9200/.kibana_8.4.0_001/_doc/cases-comments:"{$i}"?&pretty=true"
-H "Content-Type: application/json"
done
```
3. Start Kibana with a modest batchSize otherwise we could OOM ES `node
scripts/kibana --dev --migrations.batchSize=120`
<details><summary>Example logs. Note the "Processed x documents" only
logs when the next batch is successfull read, so the order seems wrong.
To improve it we'd need to log progress after a batch is successfully
written instead 🤷 </summary>
```
[.kibana] Processed 120 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3667ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1740ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1376ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1402ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1311ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1388ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 900ms.
[.kibana] Read a batch that exceeded the NodeJS maximum string length, retrying by reducing the batch size in half to 60.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_READ. took: 1538ms.
[.kibana] Processed 240 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 2054ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1042ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1310ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1388ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 1130ms.
[.kibana] Processed 300 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 2610ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1262ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1299ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1363ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1341ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 572ms.
[.kibana] Processed 372 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3330ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1488ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1349ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1312ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1380ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1310ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 139ms.
[.kibana] Processed 458 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 3278ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1460ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1370ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1303ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 1384ms.
[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 1298ms.
[.kibana] Processed 542 documents out of 542.
[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 4ms.
```
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risks
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
part of #157985
## 📝 Summary
After implementing [infra
locators](https://github.com/elastic/kibana/pull/155156) to allow
navigation to the logs UI we need to replace all usages of the old
link-to routes so that we have strongly typed navigation to the logs UI.
This PR focuses on replacing `link-to` usages in the Infra Public
plugin.
## ✅ Testing
A)
1. Navigate to Observability -> Infrastructure -> Inventory
2. Choose any of the Host from the waffle
3. Click logs tab
4. Click open in logs link
B)
1. Navigate to Observability -> Infrastructure -> Inventory
2. Switch to table view
3. Click any of the Host from the table
4. Click Host Logs
6646c496-1760-4788-9532-b318487070d1
Resolves https://github.com/elastic/kibana/issues/152279
Separates out the type for saved object `infrastructure-ui-source` so
the HTTP API no longer references the same type.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR refactors a couple things:
1. Moves the decodeOrThrow logic inside of the `try/catch` within the
client. This handles the scenario where another plugin is calling the
client. This way we wrap the error such that it is more consumable by
other plugins. This also makes the code more consistent.
2. Adds `decodeWithExcessOrThrow` to a couple places we didn't have it
in the client
3. Adds `decodeWithExcessOrThrow` to a few places in the HTTP route
handlers where we were access fields from the query params or body but
weren't validating that they existed
4. Ensured we had `decodeOrThrow` in the service layer before returning
to the client
5. Unskipped a few tests now that we have the `strict` io-ts types in
place
Adds several storage explorer improvements with a goal to change the
target to GA.
### Changes
- Adds a download report component that creates a JSON doc with the
service stats breakdown
- Changes `Sample rate` to `Sampling rate` in the services table
- Changes `Disk space used` to `Relative disk space used` and
`Incremental APM size` to `Delta in APM size` in the summary stats panel
- Updates the tooltip of the `Total APM size` summary stat to `Total
storage size of all APM indices including replicas, ignoring the filter
settings`
- Removes the storage explorer link from the header and adds it to the
left navigation component
- Adds a link from the data streams in the indices breakdown table to
data stream details in index management
- Removes the Beta label
- Adds `key` to the card items used in the documentation section
Partially addresses https://github.com/elastic/kibana/issues/157756
## Summary
This PR disables the Search Sessions plugin for serverless.
**How to test:**
1. Start Elasticsearch with `yarn es snapshot` and Kibana with yarn
`serverless-{mode}` where `{mode}` can be `es`, `security`, or `oblt`.
2. Verify that the Search Sessions app is not accessible and its path
(`app/management/kibana/search_sessions`) leads to the Stack Management
landing page.
## Summary
- Adds automated test coverage for Policy Detail page related manual
tests. See linked issue for test cases.
- fixes e2e alerts tests
~Note: 3 more e2e test cases are coming soon.~
Update: ready to review, the 3 e2e test cases will be added in a
separate PR.
part of #157985
## 📝 Summary
After implementing [infra
locators](https://github.com/elastic/kibana/pull/155156) to allow
navigation to the logs UI we need to replace all usages of the old
link-to routes so that we have strongly typed navigation to the logs UI.
This PR focuses on replacing `link-to` usages in the APM plugin.
## ✅ Testing
1. Navigate to APM -> Services
2. Follow video below to navigate to where the logs links are located
3. Links should behave as before and maintain navigation params
c631ea31-c5d9-4d05-9219-7b208daf2c37
## Summary
- Increases timeout as element might take a bit to be displayed
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix https://github.com/elastic/kibana/issues/158372
Change the logic of the `zdt` algorithm to be able to re-use an existing
"v2 managed" cluster state (a kibana cluster previously running the v2
migration algorithm) if compatible.
Technically the v2 index state is compatible with zdt after 8.8 (when we
introduced `indexTypeMap` and restricted to compatible mapping changes
only).
See https://github.com/elastic/kibana/issues/158372 for more detailed
explanations.
Addresses https://github.com/elastic/kibana/issues/158201
## Summary
This PR fixes grid row selection in dark mode.
Before:
<img width="300" alt="Screenshot 2023-05-23 at 09 53 11"
src="e447ebb5-ca75-4778-84a7-ec654aca3709">
After:
<img width="300" alt="Screenshot 2023-05-23 at 09 52 44"
src="86f7ccd0-a306-426b-9fcb-91d9131a7da6">
For testing:
Enable dark mode on User Profile page.
## Summary
Part of quick wins where back to resources button is now right aligned.
**Notes:**
The solution where we are hiding the Vulnerabilities and
Misconfigurations tabs doesn't seem to be a 'Quick win' as such for now
the fix will be this
<img width="1429" alt="Screenshot 2023-05-27 at 1 26 28 AM"
src="a3d261da-96f6-4141-80fa-fd2c98ed7acb">
## Summary
This PR ensures that alert redirection mechanism works the same way it
does for the legacy flyout (eg. that redirects done with
`kibana.alert.url` work),
and it introduces the share button, whenever `kibana.alert.url` is
present:
### Testing
The usual approach with switching the flag applies, it should work with
any alert created with detection rules.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
