## Summary
- Increases the amount of time it waits for the Fleet Server agent to
show up in Fleet
- Add an `AxiosError` formatter utility and makes use of it in CLI
common services
Fixes: https://github.com/elastic/kibana/issues/163447https://github.com/elastic/kibana/issues/163586
## Summary
- Fixes flaky test:
`x-pack/plugins/security_solution/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.cy.ts`
- Test title: `Detection rules, Prebuilt Rules Installation and Update
workflow - Installation of prebuilt rules package via Fleet should
install package from Fleet in the background`
## Details
- Initially ran the flaky test runner with multiple iterations and all
gave succesful results, i.e. no flakiness or failed tests.
- But: after checking the logs for the failed tests in the original
failed build, discovered that the reason the test failed is because:
- when checking Fleet's response for the installation of the
`security_detection_engine`, the API response was not as expected from
the API spec:
```
**Expected:** [{ name: 'security_detection_engine', installSource: 'registry' }]
**Actual:** [{ name: 'security_detection_engine', installSource: undefined }]
```
Since we cannot rely 100% that the Fleet API will return the correct
value for the installSource, this PR deletes this part of the test to
prevent any type of flakiness caused by external factors such as this.
Lots of small fixes, but mainly new (APM) functions for the
Observability AI Assistant.
f489a310-6ba8-4591-8ac9-54a176e0b58d
---------
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Clint Andrew Hall <clint@clintandrewhall.com>
## Summary
- Updates the roles YAML file (used in testing) to match update done to
project-controller for serverless
- adds additional tests to validate `t3_analyst` role in serverless
Closes#163682
## 📝 Summary
Add different data-test-subj for different alert details pages based on
rule type id, this will help us in tracking these pages more easily.
Resolves: #159493
This PR replaces `AlertFactory` in ES Query rule type with
`AlertsClient` so the alerts are persistent in an alert-as-data index.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
fixes https://github.com/elastic/kibana/issues/160126
This PR
- hides owner selection while adding lens to case from dashboard for
observability and securitySolution serverless
- hides add to new case and add to existing case lens action for
Elasticsearch serverless
**How to test**
- Run Observability/Securtiy solution serverless project and add lens
visualization to case from dashboard
- Run ES serverless project and check that lens do not have option to
add to case
- Classic kibana works as before (check dashboard from securitySolution
and generic dashboard as well)
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
https://github.com/elastic/security-team/issues/6531
### Success Criteria [Telemetry]
- [x] Date and time of the check
- [x] Number of indices that were checked
- [x] Time it took for each quality check to complete
- [x] Number of failures, if any
- [x] Number of incompatible fields per index
- [x] Identifying the fields that are incompatible to spot recurring
mapping conflicts in customers' environments - an array of incompatible
fields
- [x] Identifying unallowed values to determine patterns of unallowed
values found in customers' environments - an array of incompatible
fields
- [x] Number of documents per cluster, index pattern, and index -
pattern not tracked, index is tracked by uuid
- [x] Storage information per cluster, index pattern, and index -
pattern not tracked, index is tracked by uuid
### Additional properties tracked:
1. batchId: events triggered by the same action share the same batchId
2. ecsVersion
[sample
data](b0662610-31ff-11ee-adde-d5df298171dd?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-7d%2Fd,to:now))&_a=(columns:!(event_type,properties.pattern,properties.sizeInBytes,properties.numberOfIndices,properties.numberOfDocuments,properties.numberOfIncompatibleFields,properties.timeConsumedMs),filters:!(),grid:(),hideChart:!f,index:c5dc7cd0-2950-4e51-b428-d0451b1b8d9d,interval:auto,query:(language:kuery,query:'event_type%20:%20%22Data%20Quality%20Checked%22%20'),sort:!(!(timestamp,desc))))
### Prerequisit:
```
telemetry.optIn: true
```
### Steps to verify:
Create an index with invalid fields from dev tools, go to data quality
dashboard, click `Check All` button.
You should find `Data Quality Index Checked` in your console verbose
when each index is checked.
When everything completed, you should find `Data Quality Check All
Completed`
```
PUT auditbeat-custom-index-1
PUT auditbeat-custom-index-1/_mapping
{
"properties": {
"@timestamp": {
"type": "date"
},
"event.category": {
"type": "keyword",
"ignore_above": 1024
}
}
}
POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:41:49.668Z",
"host": {
"name": "foo"
},
"event": {
"category": "an_invalid_category"
},
"some.field": "this",
"source": {
"port": 90210,
"ip": "10.1.2.3"
}
}
POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:42:22.123Z",
"host": {
"name": "bar"
},
"event": {
"category": "an_invalid_category"
},
"some.field": "space",
"source": {
"port": 867,
"ip": "10.9.8.7"
}
}
POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:43:35.456Z",
"host": {
"name": "baz"
},
"event": {
"category": "theory"
},
"some.field": "for",
"source": {
"port": 5,
"ip": "10.4.6.6"
}
}
POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:44:36.700Z",
"host": {
"name": "@baz"
},
"event": {
"category": "malware"
},
"some.field": "rent",
"source": {
"port": 309,
"ip": "10.1.1.1"
}
}
```
### event_type:
**Data Quality Index Checked**
```
{
"sizeInBytes": 89517384,
"numberOfIndices": 1,
"numberOfIndicesChecked": 1,
"ilmPhase": "hot",
"ecsVersion": "8.6.1",
"numberOfIncompatibleFields": 0,
"batchId": "43c35a90-e700-45cf-b9e9-822686e16cff",
"isCheckAll": false, // This is true when it's triggered by check all button clicked
"unallowedValueFields": [field1, field2],
"numberOfDocuments": 172327,
"indexId": "m_liWv7CRGWOuY2Op3lHZw",
"errorCount": 0,
"unallowedMappingFields": [field3],
"timeConsumedMs": 120
}
```
**Data Quality Check All Completed:**
```
{
"isCheckAll": true,
"sizeInBytes": 94708547,
"numberOfIndices": 4,
"numberOfIndicesChecked": 4,
"ecsVersion": "8.6.1",
"numberOfDocuments": 178049,
"numberOfIncompatibleFields": 3,
"batchId": "bdbb47ba-ffbf-432c-8212-8c691145f0d3",
"timeConsumedMs": 13786
},
"timestamp": "2023-08-08T14:39:37.380Z"
}
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/163340
This PR disables graph in serverless. Specifically:
- creates a serverless yml setting for disabling graph
- adds the setting in serverless.yml
## Summary
Aaddress https://github.com/elastic/kibana/issues/163337 for discover
saved search
### Context:
In serverless navigation, we changed how breadcrumbs work. Instead of
setting the full path manually, we automatically calculate the main
parts of the path from the side nav + current URL. This was done to keep
side nav and breadcrumbs in sync as much as possible and solve
consistency issues with breadcrumbs across apps.
https://docs.elastic.dev/kibana-dev-docs/serverless-project-navigation#breadcrumbs
Apps can append custom deeper context using the
serverless.setBreadcrumbs API. Regular core.chrome.setBreadcrumbs has no
effect when the serverless nav is rendered.
<img width="1624" alt="Screenshot 2023-08-10 at 15 22 08"
src="269879b0-6fcc-4606-816e-5d76616db60a">
closes [#163491](https://github.com/elastic/kibana/issues/163491)
## Summary
This PR creates a new package that contains a utility API that helps to
generate the JSON with the attributes required to render a Lens chart
with the `EmbeddableComponent`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes#157456
Secret storage requires that fleet servers are 8.10.0 or above.
This PR adds a backend check that all fleet servers are above 8.10.0
before enabling secrets storage. Once all fleet servers are above that
version, secrets are permanently enabled.
the fleet server check checks all agents in policies that contain the
fleet server package.
A flag on the`ingest_manager_settings` saved. object
`secret_storage_requirements_met` is used to make a note that the check
has previously passed, meaning we don't have to keep querying the agents
and policies.
Test scenarios (all covered by integration tests) :
- given a deployment with no fleet servers connected, on creating a
package policy with secret variables, the values should be stored in
plain text not as a secret reference
- given a deployment with at least one fleet server that is below
8.10.0, on creating a package policy with secret variables, the values
should be stored in plain text not as a secret reference
- given a deployment where all fleet servers are 8.10.0 or above,
secrets should be stored as secret references and in the secrets index
- if a package policy was created before secrets were enabled, and since
its creation the fleet server versions pass the check, when updating
that policy, all secrets should move to being secret references.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Julia Bardi <julia.bardi@elastic.co>
Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
## Summary
UI
- [x] When there is no elastic defend integration present, the agent
tamper protection (`is_protected`) switch and instruction link are
disabled and there is an info tooltip explaining why the switch is
disabled
API
- [x] Requires the elastic defend integration to be present, in order to
set `is_protected` to true. Will allow the user to create the agent
policy and not throw an error, but will keep `is_protected` as false and
log a warning in the kibana server. In the next release, the response
will be modified to send back a 201 with the relevant messaging.
- [x] Sets `is_protected` to false when a user deletes the elastic
defend package policy
## Screenshots
### No Elastic Defend integration installed
<img width="970" alt="image"
src="910be766-1a1e-4580-9ace-306089b4626d">
## Summary
Ref: https://github.com/elastic/kibana/issues/162569
The test was trying to load the exceptions tab before the rule details
page loaded. Now we wait for the rule tab to load before continuing.
Something I was unaware of was that `cy.url()` will [automatically
retry](https://docs.cypress.io/api/commands/url#Assertions) until all
chained assertions have passed, which I think can be an easy way to fix
future flake issues where cypress tries to click on an element before
the new page loads.
## Summary
Some simple dev UX improvements to the swap_references data views api -
```
POST /api/data_views/swap_references/_preview
{
"fromId" : "abcd-efg",
"toId" : "xyz-123"
}
returns
{
result: [{ id: "123", type: "visualization" }],
}
```
```
POST /api/data_views/swap_references
{
"fromId" : "abcd-efg",
"toId" : "xyz-123",
"delete" : true // optional, removes data view which is no longer referenced
}
returns
{
result: [{ id: "123", type: "visualization" }],
deleteStatus: {
remainingRefs: 0,
deletePerformed: true
}
```
Additional params -
```
fromType: string - specify the saved object type. Default is `index-pattern` for data view
forId: string | string[] - limit the affected saved objects to one or more by id
forType: string - limit the affected saved objects by type
```
Improves upon https://github.com/elastic/kibana/pull/157665
Docs will be created in follow up PR
## Summary
Adds `All`/`None` RBAC for the Elastic AI Assistant within the Security
app via Kibana feature privileges, and also via serverless PLI App
Features for the Security `complete` product (see
https://github.com/elastic/security-team/issues/7023).
Added as high-level category to enable future support of sub-features
(included sample `Create Conversation` sub-feature plumbed as example).
<p align="center">
<img width="500"
src="b42ab3fe-65e1-49b9-a379-262f2438c0eb"
/>
</p>
Note: Since [`minimumLicense:
'enterprise'`](https://github.com/elastic/kibana/pull/163031/files#diff-56de8b6234daf4e0e69efe680e5a4afc4f88d152243b773d90c3991fa9dabc19R28)
is configured on this privilege, when the license check isn't satisfied,
the privilege will be hidden (not disabled).
## Testing
Feature is available for `enterprise` licenses and when `All` privilege
is set, otherwise Assistant is hidden in Timeline, all `Chat` UI
elements are hidden, and the `cmd/ctrl + ;` shortcut is unavailable.
---
### On Prem Testing:
Create two roles, one for each `all`/`none` Security Elastic AI
Assistant privilege (via Dev Tools):
<details><summary>PUT /_security/role/assistant_all</summary>
<p>
``` ts
PUT /_security/role/assistant_all
{
"cluster": [
"all"
],
"indices": [
{
"names": [
"*"
],
"privileges": [
"all"
],
"field_security": {
"grant": [
"*"
],
"except": []
},
"allow_restricted_indices": false
}
],
"applications": [
{
"application": "kibana-.kibana",
"privileges": [
"feature_securitySolutionAssistant.minimal_all",
"feature_siem.all",
"feature_securitySolutionCases.all",
"feature_actions.all"
],
"resources": [
"*"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
}
}
```
</p>
</details>
<details><summary>PUT /_security/role/assistant_none</summary>
<p>
``` ts
PUT /_security/role/assistant_none
{
"cluster": [
"all"
],
"indices": [
{
"names": [
"*"
],
"privileges": [
"all"
],
"field_security": {
"grant": [
"*"
],
"except": []
},
"allow_restricted_indices": false
}
],
"applications": [
{
"application": "kibana-.kibana",
"privileges": [
"feature_siem.all",
"feature_securitySolutionCases.all",
"feature_actions.all"
],
"resources": [
"*"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
}
}
```
</p>
</details>
Create a new `assistant_user` (assigned both roles above), log in and
test assistant availability, then remove one role at a time testing
each:
<details><summary>POST /_security/user/assistant_user (ALL)</summary>
<p>
``` ts
POST /_security/user/assistant_user
{
"username": "assistant_user",
"password": "changeme",
"roles": [
"assistant_all",
"assistant_none"
],
"full_name": "Assistant User",
"email": "assistant-user@elastic.co",
"metadata": {},
"enabled": true
}
```
</p>
</details>
Test that assistant is available in UI via `Chat` buttons and shortcut
keys.
<details><summary>PUT /_security/user/assistant_user (NONE)</summary>
<p>
``` ts
PUT /_security/user/assistant_user
{
"username": "assistant_user",
"roles": [
"assistant_none"
],
"full_name": "Assistant User",
"email": "assistant-user@elastic.co",
"metadata": {},
"enabled": true
}
```
</p>
</details>
Test that assistant is **NOT** available in UI via `Chat` buttons or
shortcut keys.
---
### Serverless Testing:
To test with the Assistant available, set `productTypes` to `complete`
in `config/serverless.security.yml`
```
xpack.securitySolutionServerless.productTypes:
[
{ product_line: 'security', product_tier: 'complete' },
{ product_line: 'endpoint', product_tier: 'complete' },
]
```
otherwise to test without the Assistant, pick a different product type like `essentials`:
```
xpack.securitySolutionServerless.productTypes:
[
{ product_line: 'security', product_tier: 'essentials' },
{ product_line: 'endpoint', product_tier: 'essentials' },
]
```
Then start Serverless Kibana: `yarn serverless-security`
---
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Closes https://github.com/elastic/kibana/issues/162474
### Changes
* RangeSliderEmbeddable - call setInitializationFinished when
runRangeSliderQuery throws. This fixes the issue
* Investigated if OptionsListEmbeddable is vulnerable to the same issue.
It's not because it uses its own REST API that has a service wrapper
`OptionsListService`. `OptionsListService` handles REST API errors.
* Add unit test verifying OptionsListService.runOptionsListRequest does
not throw when there are REST API errors and always returns a response.
* Add unit tests ensuring setInitializationFinished is called for both
RangeSliderEmbeddable and OptionsListEmbeddable in all cases
* Other clean up
* Fix uses of `dataViewsService.get`. `dataViewsService.get` throws when
data view is not found. It does not return undefined. PR updates
OptionsListEmbeddable, RangeSliderEmbeddable, and mocked data service
* Fix uses of `dataView.getFieldByName`. `dataView.getFieldByName`
returns undefined when field is not found and never throws. PR updates
OptionsListEmbeddable and RangeSliderEmbeddable
* Remove `resp` wrapper around mocked `fetch` results.
### Test instructions
1) In console run
```
PUT test1
PUT test1/_mapping
{
"properties": {
"value": {
"type": "integer"
}
}
}
PUT test1/_doc/1
{
"value" : 1
}
PUT test1/_doc/2
{
"value" : 10
}
```
2) create data view `test*`
3) create dashboard with range slider control on test*.value.
4) select a range in the range slider
5) save dashboard
6) run the following in console
```
PUT test2
PUT test2/_mapping
{
"properties": {
"value": {
"type": "keyword"
}
}
}
PUT test2/_doc/1
{
"value" : "foo"
}
DELETE test1
```
7) Open dashboard saved above. Verify dashboard opens and control
displays an error message about being unable to run aggregation on
keyword field.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Devon Thomson <devon.thomson@elastic.co>
## Summary
- Added Beta tag for GCP option on CSPM
- Fix a bug where setup_access is stuck on google_cloud_shell ( This bug
occur when user clicks on Manual option, click the start of Project ID
name, and then click google cloud shell option again)
- Added unit test for getCspmCloudShellDefaultValue to help with
refactor later
<img width="950" alt="Screenshot 2023-08-10 at 2 02 46 PM"
src="ccd45ed1-8b6c-4631-8a01-22d35d1b62aa">
## Summary
Redo of this PR https://github.com/elastic/kibana/pull/162178 but
without the `native-hdr-histogram` library which caused issues in the
serverless build. In the future we may want to pursue generating a
custom build of this native library but for our current purposes, a
simple bucketed histogram should suffice. The only changes from the
original PR are in this commit:
dde5245ded,
where we create a `SimpleHistogram` class to bucket task claim durations
into `100ms` buckets.
Please reference the original PR for more description about this HTTP
API
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
I was not sure if there are other plans for these stats, but I went
ahead and cleaned those up:
### Issue 1. Sidenav groups are collapsed on a smaller screen
#### Before
#### After
<img width="1456" alt="Screenshot 2023-08-11 at 14 13 23"
src="3e52d05f-12fa-4d38-addb-538239e7d8d1">
### Issue 2. Collapsed sidenav state is empty
We reserved this for icons, but until we have them, I think it makes
sense to just hide the bar:
#### Berfore
#### After
<img width="1456" alt="Screenshot 2023-08-11 at 14 14 35"
src="99adadb4-637b-404b-9909-fe7e78e0224e">
### Issue 3. Navigation is not initialized when Kibana loaded with
hidden navigation
We initialize the navigation as we render the nav tree (the sidenav).
But if the sidenav is hidden, then the navigation is not initialized.
**So, for example, breadcrumbs are not displayed correctly until the nav
is opened.** As a hack, we will always render the tree, but will make it
hidden.
#### Before
<img width="1296" alt="Screenshot 2023-08-11 at 14 35 14"
src="499e4a97-b5c3-405d-968b-bae753f15b99">
#### After
<img width="1296" alt="Screenshot 2023-08-11 at 14 34 37"
src="ae51dea4-8d98-40f6-b3bc-c4d5df8e97fa">
## Summary
This PR adjusts the `data-test-subj` for the global loading indicator in
serverless projects such that at matches the stateful version. This
makes sure that functional tests and corresponding test helper methods
continue to work the same in stateful and serverless environments when
comes to waiting for global loading to finish, which is a key mechanism
to avoid test flakiness.
### Additional information
- The serverless project specific global loading indicator was
introduced with #158523
- The stateful loading indicator `data-test-subj` naming is implemented
here:
https://github.com/elastic/kibana/blob/main/packages/core/chrome/core-chrome-browser-internal/src/ui/loading_indicator.tsx#L61
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
## Summary
This closes https://github.com/elastic/kibana/issues/161960, a basic
integration will now be created whilst onboarding logs (though the
custom logs flow).
This implements the *initial* version of this work, and does not include
things like adding a dataset to an existing integration.
## UI / UX
General:
Naming conflict errors:
Lack of permissions error:
General errors:
Success callout on the next panel:
Delete previous flow (happens in the background):
## Pointers for reviewers / next steps
- This PR also creates a new package for the `useTrackedPromise` hook,
as this is used in several places and I didn't want to just duplicate it
again (I haven't replaced other current uses in this PR, but will as a
followup).
- `useFetcher` was avoided as A) it's very tightly coupled with the
observability onboarding server route repository (and `callApi` is
scoped to this) and I wanted to call an "external" API in Fleet and B) I
wanted explicit control over when the request is dispatched (not on
mount), and whilst this can sort of be achieved by not returning a
promise from the callback it gets quite messy. I also wanted more
granular error handling control.
- Moving forward I think we'll need to enhance the state management of
the plugin. We'll want to add the ability to "add to existing
integration" and this is going to make the state more complex (even with
chunks of this functionality likely moved to it's own package). I did
actually have the Wizard state moved in to a constate container at one
point (as a starter) but I reverted this commit to make the changeset
less intrusive. It's for this same reason that, for now, I haven't
focussed too closely on extracting things like generating the friendly
error messages etc as we'll likely want to extract some of the "create
integration" hooks / UI in to a standalone package so they can be used
elsewhere (not just onboarding). There are also quite a few `
eslint-disable-next-line react-hooks/exhaustive-deps` rules in the
plugin at the moment due to the references not being stable, we could
improve that at the same time as any state changes.
- You can technically navigate directly to
`/fox/app/observabilityOnboarding/customLogs/installElasticAgent`, but
no state is stored in the URL, so nothing is rehydrated resulting in a
very empty configuration. I'm not entirely sure this is a behaviour we
want, but for now I've just made the callout conditional on state
existing (so coming from the previous panel).
- The Fleet custom integrations API now throws a 409 (conflict) when
using a name that already exists.
## Testing
- Head to `/app/observabilityOnboarding` to trigger the onboarding flow
- Select "Stream log files"
- When hitting "continue" an integration should be created in the
background (check the network requests for
`api/fleet/epm/custom_integrations`)
- When continuing (to install shipper), then going back **and** making
changes to your integration options, when clicking continue again there
should be a network request that deletes the previously created
integration (to clean things up). This should be seamless to the user.
- You should not be able to use a name that already exists (for an
existing custom integration)
- General errors (like permission issues, asset installation issues)
should display at the bottom
- When you hit the next panel (install shipper) there should be a
success callout that also contains the name of the integration that was
created
## In progress
~Two changes still in progress, but they don't need to hold up the
review (8.10 coming soon 👀):~
- ~To have a friendlier error for permissions issues (not just
"forbidden")~
- ~Fleet API integration test for the naming collision~
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
