## Summary
Let's automate E2E against Serverless
Changelog:
- updated certs to include additional dns names we are using for testing
locally, `host.docker.internal`, `es01`
- updated certs generation README to include changes related to
`openssl@3`
- added new certs for Fleet server
- added fleet-server service token
- added support for `ca_trusted_fingerprint` in fleet preconfig
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Co-authored-by: Tomasz Ciecierski <tomasz.ciecierski@elastic.co>
Co-authored-by: Kevin Logan <kevin.logan@elastic.co>
## Summary
Implement risk score engine telemetry
Here we use 2 types of telemetry:
Event base telemetry:
- Risk execution success. With parameters `scoresWritten`,
`taskCompletionTimeSeconds`, `isRunMoreThanInteval`
- Risk execution error
Usage telemetry:
- `unique_user_risk_score_total` and `unique_host_risk_score_total` -
Total amount from latest transform index for host and users
- `unique_user_risk_score_day` and `unique_host_risk_score_day` - Last
day amount from the latest transform index for host and users
- `all_host_risk_scores_total` and `all_user_risk_scores_total` - Total
amount from datastream for all risk executions for host and users
- `all_host_risk_scores_total_day` and `all_user_risk_scores_total_day`
- Last day amount from datastream for all risk executions for host and
users
- `all_risk_scores_index_size` and `unique_risk_scores_index_size` -
sizes of datastream of all risk scores and latest transform index
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Updates the exceptions flyout UI `match_any` operator to accept numerous
duplicate values that differ in case. Prior to this change, a user could
not add a field value of `foo` and `FOO` - the UI would display that the
value is a duplicate. We now will allow this as exceptions are case
sensitive and this is a necessary use case for the current exceptions
behavior.
Cypress tests and FTR tests are added.
This PR is next step completing discover timeline integration. All
previous/nest steps have been defined here:
https://github.com/elastic/security-team/issues/6677
## Summary
This PR implements the integration between timeline State v/s Discover
State. The purpose of this PR is to add functionality related to the
persistence of saved search which will always be linked to the timeline
user is working in.
Below diagram shows briefly how saved search is working with timeline.
```mermaid
graph TD;
DS(Discover State) -. user updates .-> SS(Saved Search);
SS(Saved Search) -. updates savedSearchId .-> TS(Timeline State) ;
TS(Timeline State) -. restores Saved Search to App state .->DS(Discover State);
```
Primarily, this PR implements below technical components:
1. `DiscoverInTimleineContext` : provides the ability across security
solution to manipulate discover state.
2. `useDiscoverInTimelineActions`: acts as a helper to provide
short-hand actions to manipulate discover state. For eg.
`resetDiscoverAppState` or `restoreAppStateFromSavedSearch`.
Here is the small demo video:
006465ba-19ce-4209-ac46-21dbb746508d
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
Closes#161754Closes#166807
To make the testing and review easier I merged the old components
[cleanup PR](https://github.com/jennypavlova/kibana/pull/5) into this
one
## Summary
This PR replaces the old node details view with the asset details flyout
### Old
### New
### Testing
1. Go to inventory
2. Click on a host in the waffle map
3. Click on any **host**
- These changes are related only if a `Host` is selected- in the case of
a pod the view shouldn't be changed:
4. Check the new flyout functionality
3557821c-7964-466e-8514-84c2f81bc2fd
Note: the selected host should have a border like in the previous
version (this I fixed in the [last
commit](ff4753aa06))
so it should be added if there is a selected node:
<img width="1193" alt="image"
src="6646fe47-6333-435a-a5ec-248339402224">
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Adds API versioning to all routes involved in new Risk Engine, public
and private
* Adds missing PLI auth headers for some routes
* Updates API invocations to specify an appropriate version header
* Does NOT add header to legacy transform-based EA routes
### Checklist
- [x] Verify no API calls from the UI were missed
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Updated EMS Styles are waiting to be put into production. They are
already available in Elastic staging environment
([preview](maps-staging.elastic.co/?manifest=testing)). This PR is a
safe measure to ensure that this change do not break our CI tests.
The process has been as follows:
1. Momentarily replaces the EMS Tile Service `tileApiUrl` by our staging
server to force the use of the new styles and check which tests break
with the slightly different basemaps at
[12481c6](12481c6ada)
2. Look for related [broken
tests](https://buildkite.com/elastic/kibana-pull-request/builds/161870)
```
Error: expected 0.030813687704837327 to be below 0.03
```
4. Adjust the threshold for the dashboard report, since the new value
was slightly over the limit
[e655b84](e655b84569)
5. Wait for a green CI (this took a few days because of unrelated issues
with Kibana CI)
6. Revert the `tileApiUrl` change to its original value
[c0030bc](c0030bcff1)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Close https://github.com/elastic/kibana/issues/167152
Log a warning instead of throwing an error in
`saved_object_content_storage` when response validation failed.
We decided to do this as a precaution and as a follow up to an issue
found in saved search https://github.com/elastic/kibana/pull/166886
where storage started failing because of too strict validation.
As of this PR the saved_object_content_storage covers and this change
cover:
- `search`
- `index_pattern`
- `dashboard`
- `lens`
- `maps`
For other types we agreed with @dej611 that instead of applying the same
change for other types (visualization, graph, annotation) the team would
look into migrating their types to also use
`saved_object_content_storage`
https://github.com/elastic/kibana/issues/167421
resolves: #158403
When conflicts are detected while updating alert docs after a rule runs,
we'll try to resolve the conflict by `mget()`'ing the alert documents
again, to get the updated OCC info `_seq_no` and `_primary_term`. We'll
also get the current versions of "ad-hoc" updated fields (which caused
the conflict), like workflow status, case assignments, etc. And then
attempt to update the alert doc again, with that info, which should get
it back up-to-date.
Note that the rule registry was not touched here. During this PR's
development, I added the retry support to it, but then my function tests
were failing because there were never any conflicts happening. Turns out
rule registry mget's the alerts before it updates them, to get the
latest values. So they won't need this fix.
It's also not clear to me if this can be exercised in serverless, since
it requires the use of an alerting framework based AaD implementation
AND the ability to ad-hoc update alerts. I think this can only be done
with Elasticsearch Query and Index Threshold, and only when used in
metrics scope, so it will show up in the metrics UX, which is where you
can add the alerts to the case.
## manual testing
It's hard! I've seen the conflict messages before, but it's quite
difficult to get them to go off whenever you want. The basic idea is to
get a rule that uses alerting framework AAD (not rule registry, which is
not affected the same way with conflicts (they mget alerts right before
updating them), set it to run on a `1s` interval, and probably also
configure TM to run a `1s` interval, via the following configs:
```
xpack.alerting.rules.minimumScheduleInterval.value: "1s"
xpack.task_manager.poll_interval: 1000
```
You want to get the rule to execute often and generate a lot of alerts,
and run for as long as possible. Then while it's running, add the
generated alerts to cases. Here's the EQ rule definition I used:
I selected the alerts from the o11y alerts page, since you can't add
alerts to cases from the stack page. Hmm. :-). Sort the alert list by
low-high duration, so the newest alerts will be at the top. Refresh,
select all the rules (set page to show 100), then add to case from the
`...` menu. If you force a conflict, you should see something like this
in the Kibana logs:
```
[ERROR] [plugins.alerting] Error writing alerts: 168 successful, 100 conflicts, 0 errors:
[INFO ] [plugins.alerting] Retrying bulk update of 100 conflicted alerts
[INFO ] [plugins.alerting] Retried bulk update of 100 conflicted alerts succeeded
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes#167408
This PR reduces the amount of "visualization modifiers" in the specific
case where ALL annotations in a layer are manual.
The `Ignore global filters` will still be shown if at least one
query-based annotation is defined in the layer.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary
This PR introduces the new color mapping feature into Lens.
The color mapping feature is introduced as a standalone sharable
component available from `@kbn/coloring`. The
[README.md](ddd216457d/packages/kbn-coloring/src/shared_components/color_mapping/README.md)
file describes the components and the logic behind it.
The Color Mapping component is also connected to Lens and is available
in the following charts:
- XY (you can specify the mappings from a breakdown dimension
- Partition (you can specify the mappings from the main slice/group by
dimension)
- Tag cloud (you can specify the mappings from the tags dimension)
This MVP feature will be released under the Tech Preview flag.
This PR needs to prove the user experience and the ease of use. UI
styles, design improvements and embellishments will be released in
subsequent PRs.
The current MVP-provided palettes are just a placeholder. I'm
coordinating with @gvnmagni for a final set of palettes.
close https://github.com/elastic/kibana/issues/155037
close https://github.com/elastic/kibana/issues/6480
fix https://github.com/elastic/kibana/issues/28618
fix https://github.com/elastic/kibana/issues/96044
fix https://github.com/elastic/kibana/issues/101942
fix https://github.com/elastic/kibana/issues/112839
fix https://github.com/elastic/kibana/issues/116634
## Release note
This feature introduces the ability to change and map colors to break
down dimensions in Lens. The feature provides an improved way to specify
colors and their association with categories by giving the user a
predefined set of color choices
or customized one that drives the user toward a correct color selection.
It provides ways to pick new colors and generate gradients.
This feature is in Tech Preview and is enabled by default on every new
visualization but can be turned off at will.
closes: https://github.com/elastic/kibana/issues/166428
## Summary
This PR removes code that is no longer needed after replacing the Node
Details View for Host with the Asset Details.
### TSVB
The TSVB files were apparently only used to display charts in the Node
Details view. Due to the Asset Details using Lens to power the charts,
corresponding `host` TSVB formulas and configs are no longer needed.
Therefore, `host*`, `hostK8s*`, and `hostDocker*` (the latter appears to
have never been used) have been removed. Additionally, `aws*` from
`required_metrics` was also removed because it was host-specific.
### FE Components
The main change is in the `useMetadata` hook. I have changed the hook
signature, making `requiredMetrics` optional. This parameter is used to
process additional filtering and is only used in asset types that the
old Node Details page supports. Not passing it is not expected to break
other places that depend on this hook.
### Server
Removing TSVB files has a direct impact on the
`api/metrics/node_details` endpoint. This endpoint is only used to
provide data to the Node Details page. It returns a 400 error if an
invalid metric is passed - which will be the case for hosts
**Example Request:**
```json
POST kbn:api/metrics/node_details
{
"metrics": [
"hostK8sCpuCap",
"hostSystemOverview"
],
"nodeId": "gke-release-oblt-release-oblt-pool-c4163099-bcaj",
"nodeType": "host",
"timerange": {
"from": 1695210522045,
"to": 1695214122045,
"interval": ">=1m"
},
"cloudId": "6106013995483209805",
"sourceId": "default"
}
```
**Response:**
```json
{
"statusCode": 400,
"error": "Bad Request",
"message": "Failed to validate: \n in metrics/0: \"hostK8sCpuCap\" does not match expected type \"podOverview\" | \"podCpuUsage\" | \"podMemoryUsage\" | \"podLogUsage\" | \"podNetworkTraffic\" | \"containerOverview\" | \"containerCpuKernel\" | \"containerCpuUsage\" | \"containerDiskIOOps\" | \"containerDiskIOBytes\" | \"containerMemory\" | \"containerNetworkTraffic\" | \"containerK8sOverview\" | \"containerK8sCpuUsage\" | \"containerK8sMemoryUsage\" | \"nginxHits\" | \"nginxRequestRate\" | \"nginxActiveConnections\" | \"nginxRequestsPerConnection\" | \"awsEC2CpuUtilization\" | \"awsEC2NetworkTraffic\" | \"awsEC2DiskIOBytes\" | \"awsS3TotalRequests\" | \"awsS3NumberOfObjects\" | \"awsS3BucketSize\" | \"awsS3DownloadBytes\" | \"awsS3UploadBytes\" | \"awsRDSCpuTotal\" | \"awsRDSConnections\" | \"awsRDSQueriesExecuted\" | \"awsRDSActiveTransactions\" | \"awsRDSLatency\" | \"awsSQSMessagesVisible\" | \"awsSQSMessagesDelayed\" | \"awsSQSMessagesSent\" | \"awsSQSMessagesEmpty\" | \"awsSQSOldestMessage\" | \"custom\"\n in metrics/1: \"hostSystemOverview\" does not match expected type \"podOverview\" | \"podCpuUsage\" | \"podMemoryUsage\" | \"podLogUsage\" | \"podNetworkTraffic\" | \"containerOverview\" | \"containerCpuKernel\" | \"containerCpuUsage\" | \"containerDiskIOOps\" | \"containerDiskIOBytes\" | \"containerMemory\" | \"containerNetworkTraffic\" | \"containerK8sOverview\" | \"containerK8sCpuUsage\" | \"containerK8sMemoryUsage\" | \"nginxHits\" | \"nginxRequestRate\" | \"nginxActiveConnections\" | \"nginxRequestsPerConnection\" | \"awsEC2CpuUtilization\" | \"awsEC2NetworkTraffic\" | \"awsEC2DiskIOBytes\" | \"awsS3TotalRequests\" | \"awsS3NumberOfObjects\" | \"awsS3BucketSize\" | \"awsS3DownloadBytes\" | \"awsS3UploadBytes\" | \"awsRDSCpuTotal\" | \"awsRDSConnections\" | \"awsRDSQueriesExecuted\" | \"awsRDSActiveTransactions\" | \"awsRDSLatency\" | \"awsSQSMessagesVisible\" | \"awsSQSMessagesDelayed\" | \"awsSQSMessagesSent\" | \"awsSQSMessagesEmpty\" | \"awsSQSOldestMessage\" | \"custom\""
}
```
### How to Test
- Start a local Kibana instance pointing to an oblt cluster.
- Navigate to `Infrastructure`.
- Try different asset types and navigate to the Node Details view.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#166848
This work adds a new tab to navigate Data View from the Log Explorer
selector.
In this first iteration, when the user selects a data view, we move into
discovering preselecting and loading the data view of choice.
**N.B.**: this recording is made on a setup where I have no installed
integrations, so having the no integrations panel is the expected
behaviour.
e8d1f622-86fb-4841-b4cc-4a913067d2cc
## Updated selector state machine
<img width="1492" alt="Screenshot 2023-09-22 at 12 15 44"
src="c563b765-6c6c-41e8-b8cd-769c518932c3">
## New DataViews state machine
<img width="995" alt="Screenshot 2023-09-22 at 12 39 09"
src="e4e43343-c915-42d8-8660-a2ee89f8d595">
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Hopefully
closes#167104closes#167130closes#167100closes#167013closes#166964
Fixing a few issues with login/logout:
1. Failed to login in "before" hook
<img width="1336" alt="Screenshot 2023-09-25 at 12 37 45"
src="e3b2830e-7b0d-4467-9b90-261b385bf71e">
My theory is that we are loading `/login` route too soon while log out
was not completed yet.
When we navigate to `https://localhost:5620/logout` there are multiple
url re-directions with final page being Cloud login form. This PR makes
sure we wait for this form to be displayed + 2500 ms extra to avoid
"immediate" /login navigation
2. Failed login on MKI:
Updating login via UI for serverless to pass password valid for
deployment: currently FTR uses `changeme` for both Kibana CI & MKI.
3. ES activate user profile call returning 500
We saw some login failures that are preceded with the following logs:
```
[00:03:27] │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:03:27] │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:03:27] │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:03:27] │ proc [kibana] [2023-09-19T07:08:26.126+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
[00:03:27] │ info [o.e.x.s.s.SecurityIndexManager] [ftr] security index does not exist, creating [.security-profile-8] with alias [.security-profile]
[00:03:27] │ proc [kibana] [2023-09-19T07:08:26.140+00:00][ERROR][plugins.security.user-profile] Failed to activate user profile: {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [27]/[27] maximum normal shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [27]/[27] maximum normal shards open;"},"status":400}.
[00:03:27] │ proc [kibana] [2023-09-19T07:08:26.140+00:00][ERROR][http] 500 Server Error
[00:03:27] │ warn browser[SEVERE] http://localhost:5620/internal/security/login - Failed to load resource: the server responded with a status of 500 (Internal Server Error)
```
User activation happens during `POST internal/security/login` call to
Kibana server. ~~The only improvement that we can do from FTR
perspective is to call this end-point via API to makes sure user is
activated and only after proceed with UI login.~~
While working on issue #4 and talking to @jeramysoucy I believe retrying
login via UI will work here as well. We are checking if we are still on
login page (similar to incorrect password login), waiting 2500 ms and
pressing login button again.
4. Failed to login with Kibana reporting UNEXPECTED_SESSION_ERROR and
been re-directed to Cloud login page
```
proc [kibana] [2023-09-25T11:35:12.794+00:00][INFO ][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
```
Temporary solution is to retry login from scratch (navigation to Kibana
login page & re-login )
Flaky-test-runner for functional obtl tests 50x
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3215
This PR is not fixing random 401 response when user navigates to some
apps with direct url
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/166636
- use metrics_data_access plugin to dynamically retrieve indices where
needed
- add basic `/assets/hosts` api test
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co>
## Summary
This PR cleaned up the loading and empty state of most components in the
new expandable alerts flyout:
1) Refactor `FlyoutLoading` component to be used across components
2) Updated unit tests to explicitly check no data messages
3) Changed loading spinner in smaller components to be skeleton text
4) Moved error/no data message to table
5) Added loading and/or error messages for:
1) Right panel -> about -> Rule description, note that rule preview is
disabled if no rule is found
2) Right panel -> about -> Alert reason, note that the alert reason
preview is disabled if no reason available
3) Right panel -> Investigation -> Highlighted fields
4) Right panel -> Insights -> host and user preview
5) Right panel -> Visualization -> analyzer preview (to match error in
analyzer graph)
6) Right panel -> About -> show rule summary & show alert reason when
there is an error
- Very uncommon because the button should be disabled if data is not
available, adding as fall back
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
We're breaking #166813 up into smaller PRs in the interest of getting
PRs through sooner for type fixes. These are the changes for
`x-pack/test`.
---------
Co-authored-by: Thomas Watson <watson@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
## Summary
This PR removes the feature flag and enables the new index details page
by default. The index details page was implemented in following PRs:
- https://github.com/elastic/kibana/pull/163521
- https://github.com/elastic/kibana/pull/163955
- https://github.com/elastic/kibana/pull/164741
- https://github.com/elastic/kibana/pull/165027
- https://github.com/elastic/kibana/pull/165038
- https://github.com/elastic/kibana/pull/165456
In this PR we completely remove now obsolete code for the old index
details flyout: react components and corresponding redux code. All
related tests are updated and cleaned up. The config value for Index
Management plugin `xpack.index_management.dev.enableIndexDetailsPage` is
deprecated as unused and can be removed in v9.0.
### How to test
1. Start ES and Kibana with `yarn es snapshot` and `yarn start`
3. Navigate to Index Management and create an index
4. Click the index name in the table and check the tabs of the details
page
### Screenshots
#### Stateful
Overview
<img width="1387" alt="Screenshot 2023-09-27 at 14 41 57"
src="e58b15e7-d10c-4473-873c-d0f128392404">
Mappings
<img width="1392" alt="Screenshot 2023-09-27 at 14 42 05"
src="441157cb-5a26-47c3-8da0-b4df51ebec5d">
Settings
<img width="1385" alt="Screenshot 2023-09-27 at 14 42 13"
src="da66a2eb-1f21-44c1-9356-484c66caab88">
Statistics
<img width="1380" alt="Screenshot 2023-09-27 at 14 42 22"
src="ec93d85c-e754-4c21-88ab-0124dc114fc9">
Error loading data
<img width="1333" alt="Screenshot 2023-09-26 at 19 05 37"
src="fc1804b3-6aa0-4019-bae6-e7bb40113b28">
<img width="1327" alt="Screenshot 2023-09-26 at 19 06 07"
src="ca711697-cc74-4ba8-b17c-ec9b01f3026e">
<img width="1329" alt="Screenshot 2023-09-26 at 19 06 28"
src="0cb46b09-8542-452a-8845-40d060057e95">
<img width="1331" alt="Screenshot 2023-09-26 at 19 06 48"
src="87de8d3d-b6e5-4e8f-b27c-18a1c6e950d8">
Error saving index settings
<img width="1332" alt="Screenshot 2023-09-26 at 19 07 31"
src="e6e4b3d0-c237-4d0a-995a-4562bc78f88e">
### Serverless
Overview
<img width="1336" alt="Screenshot 2023-09-26 at 19 51 47"
src="6c76c23b-4be6-4ab3-ae1d-c7ae751e100d">
Mappings
<img width="1336" alt="Screenshot 2023-09-26 at 19 23 51"
src="625fa703-506f-4389-9df0-86441a655074">
Settings
<img width="1332" alt="Screenshot 2023-09-26 at 19 24 02"
src="c496ab09-f2db-4c1b-9fb6-1e9b64b1c142">
# Release note
Index details can now be viewed on a new index details page in Index
Management.
<img width="1387" alt="Screenshot 2023-09-27 at 14 41 57"
src="b90c706d-8b15-49e4-8f6a-cb66f3ed1822">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
