## Summary
Revives this https://github.com/elastic/kibana/pull/181969
To do so, I had to create a new package `search-types` and move the
types I need there.
The Discovery team can take it from here.
Note: It also does a cleanup on the types I move, some of them were
declared twice.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This pr adds pinned events, notes, and row actions to the unified data
table within timeline. Uses the existing shared hooks and components
from timeline, with only a few casts to make everything work. As with
the other parts of the unified timeline, this is hidden behind the
feature flag 'unifiedComponentsInTimelineEnabled'.
Correlation/EQL tab:
<img width="862" alt="image"
src="5b7facfc-e385-41a2-b14c-e36cf134fe00">
Improved header controls positioning:
<img width="1456" alt="image"
src="a87e39d3-3f53-4266-9a2d-5bc33a37cfdc">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
## Summary
While debugging another issue I've noticed there were few packages
without treeshake enabled who could be optimized for bundle size, so
I've enabled it for a few of them.
This PR focuses only on `shared-ux`, `core` and `ml` packages for now.
it relies on the tests to check if the treeshake broke anything deep.
## Summary
In preparation for the KB UI and ES API to accept `description` for
Roles, Kibana `get`, `getAll`, and `put` Roles routes should handle a
description.
## Testing
Start KB/ES locally
In Dev Tools PUT role:
```
PUT kbn:api/security/role/mytestrole
{
"description": "This is a test role",
"metadata": {
"version": 1
},
"elasticsearch": {
"cluster": [ ],
"indices": [ ]
},
"kibana": [
{
"base": [ ],
"feature": {
"discover": [ "all" ],
"visualize": [ "all" ],
"dashboard": [ "all" ],
"dev_tools": [ "read" ],
"advancedSettings": [ "read" ],
"indexPatterns": [ "read" ],
"graph": [ "all" ],
"apm": [ "read" ],
"maps": [ "read" ],
"canvas": [ "read" ],
"infrastructure": [ "all" ],
"logs": [ "all" ],
"uptime": [ "all" ]
},
"spaces": [ "*" ]
}
]
}
```
This will fail since ES doesn't accept `descriptions` yet
Pull the ES Role Description PR
https://github.com/elastic/elasticsearch/pull/107088
and run `yarn es source` and `yarn start`
Rerun the PUT above, receive 204!
Check the role `description` with a GET:
```
GET kbn:api/security/role/mytestrole
```
<img width="1712" alt="Screenshot 2024-04-30 at 9 43 32 PM"
src="20394086-f223-4be8-8660-eb8d3930116c">
It has a limit of 2048 per the requirements:
<img width="2248" alt="Screenshot 2024-04-30 at 9 45 13 PM"
src="7dc97a8e-1246-49e4-954e-885be433c7c7">
## Summary
This will allow users to view data beyond default values in details page
chart, users will be able to select date range via picker to view
historical SLI, Burn rate or Good/Bad Events chart
48a6a029-ad36-4457-b195-4ab2d739fca0
<img width="1726" alt="image"
src="c8936443-5fe4-4925-9a9f-a22c847f32b7">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/181739
Disable federated view settings in serverless !!
- [ ] Disabled route and settings page in public
- [ ] Uses different put request schema on serverless which doesn't
contains remote cluster settings
## Summary
Prep work for React@18 bump
tl;dr In React@18 `React.FC` doesn't contain `children` anymore, so in
order to make the bump easier I have decided to split the effort in
multiple faces and hopefully this will make it easier for everyone
This PR focuses only on adding explicit `children` declaration either by
using `React.PropsWithChildren` type or by adding `children:
React.ReactNode` to the existing props types
https://github.com/DefinitelyTyped/DefinitelyTyped/issues/46691
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
Co-authored-by: Marco Vettorello <marco.vettorello@elastic.co>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
## [Security Solution] [Attack discovery] Attack discovery
### Summary
This PR renames the _Attack discovery_ Security Solution feature from its original name, [AI Insights](https://github.com/elastic/kibana/pull/180611).
_Above: Attack discovery in the Security Solution_
Attack discovery uses AI to identify active attacks in the environment, without the time (or prior experience) required to manually investigate individual alerts in Elastic Security, identify if they are related, and document the identified attack progression.
While users can ask the Assistant to find these progressions today, Attack discovery is a dedicated UI to identify these progressions and action them accordingly. This feature adds a new page, `Attack discovery`, to the Security Solution's global navigation.
Attack discoveries are generated from Large Language Models (LLMs) to identify attack progressions in alert data, and to correlate and identify related entities and events. When possible, attack progressions are attributed to threat actors.
### Details
Users may generate attack discoveries from a variety of LLMs, configured via [Connectors](https://www.elastic.co/guide/en/kibana/master/action-types.html):
_Above: LLM selection via the connectors popup menu_
Clicking on the title of an attack discovery toggles the discovery between the collapsed and expanded state:
_Above: Collapsing / expanding an attack discovery (animated gif)_
The first three discoveries displayed on the Attack discoveries page are expanded by default. Any additional discoveries that appear after the first three must be expanded manually.
Attack discoveries provide a summary of the entities impacted by an attack. Clicking on an entity, i.e. a hostname or username, displays the entity flyout with the entity's risk summary:
_Above: Clicking on a host in the summary of the attack discovery reveals the host risk summary (animated gif)_
Hover over fields in the discovery's summary or details to reveal pivot actions for investigations:
_Above: Hovering over fields in the details of an attack discovery reveals pivot actions (animated gif)_
Attack discoveries are generated from alerts provided as context to the selected LLM. The alert data provided to the LLM is anonymized automatically. Anonymization is [configured](https://www.elastic.co/guide/en/security/current/security-assistant.html#ai-assistant-anonymization) via the same anonymization settings as the Assistant. Users may override the defaults to allow or deny specific alert fields, and to toggle anonymization on or off for specific fields.
Click the Anonymization toggle to show or hide the actual values sent to the LLM:
_Above: Toggling anonymization to reveal the actual values sent to the LLM (animated gif)_
### Empty prompt
At the start of a session, or when a user selects a connector that doesn't (yet) have any attack discoveries, an [empty prompt](https://eui.elastic.co/#/display/empty-prompt) is displayed.
The animated counter in the empty prompt counts up until it displays the maximum number of alerts that will be sent to the LLM:
_Above: An animated counter displays the maximum number of alerts that will be sent to the LLM (animiated gif)_
The _Settings_ section of this PR details how users configure the number of alerts sent to the LLM. The animated counter in the empty prompt immediately re-animates to the newly-selected number when the setting is updated.
### Take action workflows
The _Take action_ popover displays the following actions:
- `Add to new case`
- `Add to existing case`
- `View in AI Assistant`
_Above: The Take action popover_
#### Add to new case
Clicking the `Add to new` case action displays the `Create case` flyout.
_Above: The `Add to new case` workflow_
An `Alerts were added to <case name>` toast is displayed when the case is created:
_Above: Case creation toast_
A markdown representation of the attack discovery is added to the case:
_Above: A markdown representation of an attack discovery in a case_
The alerts correlated to generate the discovery are attached to the case:
_Above: Attack discovery alerts attached to a case_
#### Add to existing case
Clicking the `Add to existing case` action displays the `Select case` popover.
_Above: The `Select case` popover_
When users select an existing case, a markdown representation of the attack discovery, and the alerts correlated to generate the discovery are attached to the case, as described above in the _Add to new case_ section.
#### View in AI Assistant
The `View in AI Assistant` action in the `Take action` popover, and two additional `View in AI Assistant` affordances that appear in each discovery have the same behavior:
Clicking `View in AI Assistant` opens the assistant and adds the attack discovery as context to the current conversation.
_Above: An attack discovery added as context to the current conversation_
Clicking on the attack discovery in the assistant expands it to reveal a preview of the discovery.
_Above: An expanded attack discovery preview in the assistant_
The expanded attack discovery preview reveals the number of anonymzied fields from the discovery that were made available to the conversation. This feature ensures discoveries are added to a conversation with the anonymized field values.
An attack discovery viewed in the AI assistant doesn't become part of the conversation until the user submits it by asking a question, e.g. `How do I remediate this?`.
Attack discoveries provided as context to a conversation are formatted as markdown when sent to the LLM:
_Above: Attack discoveries provided as context to a conversation are formatted as markdown_
Users may toggle anonymization in the conversation to reveal the original field values.
_Above: Revealing the original field values of an attack discovery added as markdown to a conversation (animated gif)_
#### Alerts tab
The _Alerts_ tab displays the alerts correlated to generate the discovery.
_Above: The alerts correlated to generate the attack discovery in the Alerts tab_
The `View details`, `Investigate in timeline`, and overflow row-level alert actions displayed in the Alerts tab are the same actions available on the Cases's page's Alerts tab:
_Above: Row-level actions are the same as the Cases pages Alert's tab_
#### Investigate in Timeline
Click an attack discovery's `Investigate in Timeline` button to begin an investigation of an discovery's alerts in Timeline. Alert IDs are queried via the `Alert Ids` filter:
_Above: Clicking Investigate in Timeline (animated gif)_
The alerts from the attack discovery are explained via row renderers in Timeline:
_Above: Row rendered attack discovery alerts in Timeline_
### Attack Chain
When alerts are indicative of attack [tactics](https://attack.mitre.org/tactics/enterprise/), those tactics are displayed in the discovery's _Attack Chain_ section:
_Above: An attack discovery with tactics in the Attack chain_
The Attack Chain section will be hidden if an attack discovery is not indicative of specific tactics.
### Mini attack chain
Every attack discovery includes a mini attack chain that visually summarizes the tactics in a discovery. Hovering over the mini attack chain reveals a tooltip with the details:
_Above: The mini attack chain tooltip_
### Storage
The latest attack discoveries generated for each connector are cached in the browser's session storage in the following key:
```
elasticAssistantDefault.attackDiscovery.default.cachedAttackDiscoveries
```
Caching attack discoveries in session storage makes it possible to immediately display the latest when users return to the Attack discoveries page from other pages in the security solution (e.g. Cases).
_Above: Cached attack discoveries from session storage are immediately displayed when users navigate back to Attack discoveries (animated gif)_
While waiting for a connector to generate results, users may view the cached results from other connectors.
Cached attack discoveries are immediately available, even after a full page refresh, as long as the browser session is still active.
### `Approximate time remaining` / `Above average time` counters
Some LLMs may take seconds, or even minutes to generate attack discoveries. To help users anticipate the time it might take to generate new discoveries, the page displays a `Approximate time remaining: mm:ss` countdown timer that counts down to zero from the average time it takes to generate discoveries for the selected LLM:
_Above: The `Approximate time remaining: mm:ss` countdown counter (animated gif)_
If the LLM doesn't generate attack discoveries before the counter reaches zero, the text will change from `Approximate time remaining: mm:ss` to `Above average time: mm:ss`, and start counting up from `00:00` until the attack discoveries are generated:
_Above: The `Above average time: mm:ss` counter (animated gif)_
The first time attack discoveries are generated for a model, the `Approximate time remaining: mm:ss` counter is not displayed.
Average time is calculated over the last 5 generations on the selected connector. This is illustrated by clicking on the (?) information icon next to the timer. The popover displays the average time, and the time in seconds for the last 5 runs:
_Above: Clicking on the (?) information icon displays the average time, and the duration / datetimes for the last 5 generations_
The time and duration of the last 5 generations (for each connector) are persisted in the browser's local storage in the following key:
```
elasticAssistantDefault.attackDiscovery.default.generationIntervals
```
### Errors
When attack discovery generation fails, an error toaster is displayed to explain the failure:
_Above: An error toast explains why attack discovery generation failed_
### Feature flag
The `attackDiscoveryEnabled` feature flag must be enabled to view the `Attack discovery` link in the Security Solution's global navigation.
Add the `attackDiscoveryEnabled` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['attackDiscoveryEnabled']
```
### Settings
The number of alerts sent as context to the LLM is configured by `Knowledge Base` > `Alerts` slider in the screenshot below:
- The slider has a range of `10` - `100` alerts (default: `20`)
Up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- The `kibana.alert.workflow_status` must be `open`
- The alert must have been generated in the last `24 hours`
- The alert must NOT be a `kibana.alert.building_block_type` alert
- The `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### License
An Enterprise license is required to use Attack discovery.
The following empty view is displayed for users who don't have an Enterprise license:
## How it works
- Users navigate to the Attack discovery page: `x-pack/plugins/security_solution/public/attack_discovery/pages/index.tsx`
- When users click the `Generate` button(s) on the Attack discovery page, attack discoveries are fetched via the `useAttackDiscovery` hook in `x-pack/plugins/security_solution/public/attack_discovery/use_attack_discovery/index.tsx`.
- The `fetchAttackDiscoveries` function makes an http `POST` request is made to the `/internal/elastic_assistant/attack_discovery` route. Requests include the following parameters:
- `actionTypeId`, determines temperature and other connector-specific request parameters
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `anonymizationFields`, the user's `Allowed` and (when applicable `Anonymized` ) fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `connectorId`, id of the connector to generate the attack discoveries
- `size`, the maximum number of alerts to generate attack discoveries from. This numeric value is set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- `replacements`, an optional `Record<string, string>` collection of replacements that's always empty in the current implementation. When non-empty, this collection enables new attack discoveries to be generated using existing replacements.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- The `postAttackDiscoveryRoute` function in `x-pack/plugins/elastic_assistant/server/routes/attack_discovery/post_attack_discovery.ts` handles the request.
- The inputs and outputs to/from this route are defined by the [OpenAPI](https://spec.openapis.org/oas/v3.1.0) schema in `x-pack/packages/kbn-elastic-assistant-common/impl/schemas/attack_discovery/post_attack_discovery_route.schema.yaml`.
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
- The `postAttackDiscoveryRoute` route handler function in `x-pack/plugins/elastic_assistant/server/routes/attack_discovery/post_attack_discovery.ts` invokes the `attack-discovery` tool, defined in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/attack_discovery_tool.ts`.
The `attack-discovery` tool is registered by the Security Solution. Note: The `attack-discovery` tool is only used by the attack discovery page. It is not used to generate new attack discoveries from the context of an assistant conversation, but that feature could be enabled in a future release.
- The `attack-discovery` tool uses a LangChain `OutputFixingParser` to create a [prompt sandwich](https://www.elastic.co/blog/crafting-prompt-sandwiches-generative-ai) with the following parts:
```
______________________________________________________
/ \
| Attack discovery JSON formatting instructions | (1)
\ _____________________________________________________/
+-----------------------------------------------------+
| Attack discovery prompt | (2)
+-----------------------------------------------------+
/ \
| Anonymized Alerts | (3)
\_____________________________________________________/
```
- The `Attack discovery JSON formatting instructions` in section `(1)` of the prompt sandwich are defined in the `getOutputParser()` function in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/get_output_parser.ts`. This function creates a LangChain `StructuredOutputParser` from a Zod schema. This parser validates responses from the LLM to ensure they are formatted as JSON representing an attack discovery.
- The `Attack discovery prompt` in section `(2)` of the prompt sandwich is defined in the `getAttackDiscoveryPrompt()` function in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/get_attack_discovery_prompt.ts`. This part of the prompt sandwich includes instructions for correlating alerts, and additional instructions to the LLM for formatting JSON.
- The `Anonymized Alerts` in section `(3)` of the prompt sandwich are returned by the `getAnonymizedAlerts()` function in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/get_anonymized_alerts.ts`. The allow lists configured by the user determine which alert fields will be included and anonymized.
- The `postAttackDiscoveryRoute` route handler returns the attack discoveries generated by the `attack-discovery` tool to the client (browser).
- Attack discoveries are rendered in the browser via the `AttackDiscoveryPanel` component in `x-pack/plugins/security_solution/public/attack_discovery/attack_discovery_panel/index.tsx`
- The `AttackDiscoveryTab` tab in `x-pack/plugins/security_solution/public/attack_discovery/attack_discovery_panel/tabs/attack_discovery_tab/index.tsx` includes the _Summary_ and _Details_ section of the attack discovery.
- The `AttackDiscoveryMarkdownFormatter` in `x-pack/plugins/security_solution/public/attack_discovery/attack_discovery_markdown_formatter/index.tsx` renders hover actions on entities (like hostnames and usernames) and other fields in the attack discovery.
- The `AttackDiscoveryPanel` component makes use of the `useAssistantOverlay` hook in `x-pack/packages/kbn-elastic-assistant/impl/assistant/use_assistant_overlay/index.tsx` to register the attack discovery as context with the assistant. This registration process makes it possible to view discoveries in the assistant, and ask questions like "How do I remediate this?". In this feature, the `useAssistantOverlay` hook was enhanced to accept anonymizaton replacements. This enables an assistant conversation to (re)use replacements originally generated for an attack discovery.
## Summary
BC1 Security Assistant fixes
- View in assistant in Attack discovery now properly propagates the
title and the context of the conversation
- Fixed spacing around the Icon on the Welcome screen
- Fixed clearing the text input after the message was sent
- Fixed showing proper dates for the comments in `isStreaming` mode
- Fixed scrolling to the bottom in `isFlyoutMode`
- Added `Add connector` button when the Connector selector was empty
- Extracted `View in assistant` logic to the separate hook
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/179075
Filters out no data values (-1) for slo preview chart, in order to
ensure that the bounds do not end up spanning all the way to -100%.
Before
<img width="822" alt="Screenshot 2024-04-24 at 1 46 20 PM"
src="30b7cd9e-d4a1-4e9a-b5a0-e5fb8d1fe857">
After
<img width="833" alt="Screenshot 2024-04-24 at 1 42 54 PM"
src="ea63be37-46f8-4212-b0b4-f01038f5b353">
In this PR, we added the following items.
- Add a toast for letting the user know that the underlying model is
being deployed
- Start the deployment of trained model if the model deployment has not
been started
- Add a modal to display the current status of trained model deployment
- Show a link to the model management page.
- Create a inference endpoint for default inference_ids if they are
missing
- Display a badge for inference_endpoint
- Show warning if mappingsDefinition is null
Please be aware that currently, we won't be able to save the mapping
using the 'Save mappings' button because the 'semantic_text'
functionality doesn't support 'inference_id'. However, there is ongoing
parallel work in a GitHub
[branch](https://github.com/elastic/elasticsearch/tree/feature/semantic-text)
to enable 'inference_id' in 'semantic_text' for Elasticsearch.
### How to test the changes locally
- Download the elasticsearch changes from GitHub
[branch](https://github.com/elastic/elasticsearch/tree/feature/semantic-text)
- Run the elasticsearch: `./gradlew :run -Drun.license_type=trial`
- Download the changes of this PR in local kibana and do the following
steps
+ Set isSemanticTextEnabled = true in this
[location](https://github.com/elastic/kibana/pull/180246/files#diff-92f4739f8a4a6917951a1b6e1af21a96d54313eaa2b5ce4c0e0553dd2ee11fcaL80)
+ Run `yarn start`
- Create an index named 'books' using the following command:
<details>
<summary>Click to expand</summary>
```
PUT books
{
"mappings": {
"dynamic_templates": [],
"properties": {
"date_published": {
"type": "date"
},
"price": {
"type": "float"
},
"title": {
"type": "text"
},
"attributes": {
"type": "nested",
"properties": {
"authors": {
"type": "nested",
"properties": {
"author_name": {
"type": "text"
},
"author_birthdate": {
"type": "date"
}
// Add more author attributes as needed
}
},
"genres": {
"type": "nested",
"properties": {
"genre_name": {
"type": "keyword"
},
"genre_description": {
"type": "text"
}
// Add more genre attributes as needed
}
}
}
}
}
}
}
```
</details>
- Follow the steps mentioned in this video:
f1ef71e3-8adf-4bcd-837c-754929fe6f1c
### Screenshots
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sander Philipse <sander.philipse@elastic.co>
## Summary
Fix https://github.com/elastic/kibana/issues/178932
- Introduce the new `userProfile` core service, both on the browser and
server-side.
- Have the security plugin register its API to Core for re-exposition
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part of #172981.
Field caps requests can be heavy calls in larger clusters. For all other
queries for log rate analysis we were applying filters based on the time
range selection. This was missing from the field caps call. The
following parameters were added to improve the call:
- `index_filter`: Adds a range filter to only get field caps from
indices spanning the deviation time range.
- `filters`: `-metadata` was added to not return fields like `_id` and
esp. `_tier`. We previously had a manually check for `_tier` which is
now unnecessary using this option.
- `types`: Previously we fetched all field types and then filtered out
the ones we don't support. This option allows us to pass in the
supported fields right away and not return unsupported ones in the first
place.
----
Here are examples that show how `index_filter` get applied correctly:
Here the deviation selection spans only 1 month and that is reflected in
the response from the field caps call:
<img width="1026" alt="image"
src="50a00e5a-2b59-4ae5-9d50-e2ed766f68f5">
```
{
indices: [ 'gallery-2021-11' ],
fields: {
...
}
}
```
Now the deviation selection covers more months:
<img width="1010" alt="image"
src="0034a19e-b136-4261-9761-6b48fdf45989">
```
{
indices: [
'gallery-2021-09',
'gallery-2021-10',
'gallery-2021-11',
'gallery-2021-12',
'gallery-2022-01'
],
fields: {
...
}
}
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
* Creates a new package - ` @kbn/inference_integration_flyout`
* Above package has implementation for adding new inference endpoint id,
including Elastic, third party integration and instructions to upload
via Eland python client, in a flyout.
* The above package is used for supporting semantic_text feature and is
accessed via`add a new inference endpoint` button
## Screen Recording
dbd36634-bd4a-49f1-b1d5-d7b6c90444bc
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
`v93.6.0` ⏩ `v94.1.0`
> [!important]
> 👋 Hello everyone - this is a special release containing `EuiTable`'s
conversion to Emotion, several long-overdue cleanups and breaking
changes, and one or two fun new features. First, let's address the big
questions:
### Q: I'm listed as a codeowner, how much should I manually QA/review?
Answer: It depends on what exactly in your code changed, but _in
general_ I would strongly suggest at least pulling this branch down and
doing a quick visual smoke test of all tables (_note: **not**
datagrids_) in your apps/plugins. You should not expect to see any major
visual regressions.
If your table contained any kind of custom styling or behavior (e.g.
custom CSS, etc.) I **strongly** recommend taking more time to QA
thoroughly to ensure your table still looks and behaves as expected.
Teams with very manual or specific updates will be flagged below in
comment threads.
### Q: When do I need to review by?
This PR will be merged **after** 8.14FF. Because this upgrade touches so
many files and teams, we're aiming for asking for an admin merge by EOD
4/18 regardless of full approval status.
As always, you're welcome to ping us after merge if you find any issues
later ([see our
FAQ](https://github.com/elastic/eui/blob/main/wiki/eui-team-processes/upgrading-kibana.md#faq-for-kibana-teams)),
as you will have until 8.15FF to catch any bugs.
### Q: What breaking changes were made, and why?
Here's a quick shortlist of all the changes made that affected the
majority of the commits in this PR:
#### <u>Removed several top-level table props</u>
- The `responsive` prop has been removed in favor of the new
`responsiveBreakpoint` prop (same `false` behavior as before)
- The following props were removed from basic and in-memory tables:
- `hasActions`, `isSelectable`, and `isExpandable`
- These props were not used for functionality and were only used for
styling tables in mobile/responsive views, which is not a best practice
pattern we wanted for our APIs. Mobile tables are now styled correctly
without needing consumers to pass these extra props.
- `textOnly`
- This prop was unused and had no meaningful impact on tables or table
content.
#### Removed hidden mobile vs. desktop DOM
Previously, EUI would set classes that applied `display: none` CSS for
content that was hidden for mobile vs. desktop. This is no longer the
case, and content that only displays for mobile or only displays for
desktop will no longer render to the DOM at all if the table is not in
that responsive state.
This is both more performant when rendering large quantities of
cells/content, and simpler to write test assertions for when comparing
what the user actually sees vs. what the DOM ‘sees’.
(c3eeb08441e4b6efe6505e7cddaa87b540ddb259,
78cefcd954a7b46eaccd05e431b5e24dc86071a3)
#### Removed direct usages of table `className`s
EuiTable `classNames` no longer have any styles attached to them, so
some instances where Kibana usages were applying the `className` for
styles have been replaced with direct component usage or removed
entirely (86ce80b61f).
#### Custom table cell styles
Any custom CSS for table cells was previously being applied to the inner
`div.euiTableCellContent` wrapper. As of this latest release, the
`className` and `css` props will now be applied directly to the outer
`td.euiTableRowCell` element. If you were targeting custom styles table
cells, please re-QA your styles to ensure everything still looks as
expected.
---
<details open><summary>Full changelog (click to collapse)</summary>
##
[`v94.1.0-backport.0`](https://github.com/elastic/eui/releases/v94.1.0-backport.0)
**This is a backport release only intended for use by Kibana.**
**Bug fixes**
- Fixed a visual text alignment regression in `EuiTableRowCell`s with
the `row` header scope
([#7681](https://github.com/elastic/eui/pull/7681))
**Accessibility**
- Improved `EuiBasicTable` and `EuiInMemoryTable`'s selection checkboxes
to have unique aria-labels per row
([#7672](https://github.com/elastic/eui/pull/7672))
## [`v94.1.0`](https://github.com/elastic/eui/releases/v94.1.0)
- Updated `EuiTableHeaderCell` to show a subdued `sortable` icon for
columns that are not currently sorted but can be
([#7656](https://github.com/elastic/eui/pull/7656))
- Updated `EuiBasicTable` and `EuiInMemoryTable`'s
`columns[].actions[]`'s to pass back click events to `onClick` callbacks
as the second callback
([#7667](https://github.com/elastic/eui/pull/7667))
## [`v94.0.0`](https://github.com/elastic/eui/releases/v94.0.0)
- Updated `EuiTable`, `EuiBasicTable`, and `EuiInMemoryTable` with a new
`responsiveBreakpoint` prop, which allows customizing the point at which
the table collapses into a mobile-friendly view with cards
([#7625](https://github.com/elastic/eui/pull/7625))
- Updated `EuiProvider`'s `componentDefaults` prop to allow configuring
`EuiTable.responsiveBreakpoint`
([#7625](https://github.com/elastic/eui/pull/7625))
**Bug fixes**
- `EuiBasicTable` & `EuiInMemoryTable` `isPrimary` actions are now
correctly shown on mobile views
([#7640](https://github.com/elastic/eui/pull/7640))
- Table `mobileOptions`:
([#7642](https://github.com/elastic/eui/pull/7642))
- `mobileOptions.align` is now respected instead of all cells being
forced to left alignment
- `textTruncate` and `textOnly` are now respected even if a `render`
function is not passed
**Breaking changes**
- Removed unused `EuiTableHeaderButton` component
([#7621](https://github.com/elastic/eui/pull/7621))
- Removed the `responsive` prop from `EuiTable`, `EuiBasicTable`, and
`EuiInMemoryTable`. Use the new `responsiveBreakpoint` prop instead
([#7625](https://github.com/elastic/eui/pull/7625))
- The following props are no longer needed by `EuiBasicTable` or
`EuiInMemoryTable` for responsive table behavior to work correctly, and
can be removed: ([#7632](https://github.com/elastic/eui/pull/7632))
- `isSelectable`
- `isExpandable`
- `hasActions`
- Removed the `showOnHover` prop from `EuiTableRowCell` /
`EuiBasicTable`/`EuiInMemoryTable`'s `columns` API. Use the new actions
`columns[].actions[].showOnHover` API instead.
([#7640](https://github.com/elastic/eui/pull/7640))
- Removed top-level `textOnly` prop from `EuiBasicTable` and
`EuiInMemoryTable`. Use `columns[].textOnly` instead.
([#7642](https://github.com/elastic/eui/pull/7642))
**DOM changes**
- `EuiTable` mobile headers no longer render in the DOM when not visible
(previously rendered with `display: none`). This may affect DOM testing
assertions. ([#7625](https://github.com/elastic/eui/pull/7625))
- `EuiTableRowCell` now applies passed `className`s to the parent `<td>`
element, instead of to the inner cell content `<div>`.
([#7631](https://github.com/elastic/eui/pull/7631))
- `EuiTableRow`s rendered by basic and memory tables now only render a
`.euiTableRow-isSelectable` className if the selection checkbox is not
disabled ([#7632](https://github.com/elastic/eui/pull/7632))
- `EuiTableRowCell`s with `textOnly` set to `false` will no longer
attempt to apply the `.euiTableCellContent__text` className to child
elements. ([#7641](https://github.com/elastic/eui/pull/7641))
- `EuiTableRowCell` no longer renders mobile headers to the DOM unless
the current table is displaying its responsive view.
([#7642](https://github.com/elastic/eui/pull/7642))
- `EuiTableHeaderCell` and `EuiTableRowCell` will no longer render in
the DOM at all on mobile if their columns' `mobileOptions.show` is set
to `false`. ([#7642](https://github.com/elastic/eui/pull/7642))
- `EuiTableHeaderCell` and `EuiTableRowCell` will no longer render in
the DOM at all on desktop if their columns' `mobileOptions.only` is set
to `true`. ([#7642](https://github.com/elastic/eui/pull/7642))
**CSS-in-JS conversions**
- Converted `EuiTable`, `EuiTableRow`, `EuiTableRowCell`, and all other
table subcomponents to Emotion
([#7654](https://github.com/elastic/eui/pull/7654))
- Removed the following `EuiTable` Sass variables:
([#7654](https://github.com/elastic/eui/pull/7654))
- `$euiTableCellContentPadding`
- `$euiTableCellContentPaddingCompressed`
- `$euiTableCellCheckboxWidth`
- `$euiTableHoverColor`
- `$euiTableSelectedColor`
- `$euiTableHoverSelectedColor`
- `$euiTableActionsBorderColor`
- `$euiTableHoverClickableColor`
- `$euiTableFocusClickableColor`
- Removed the following `EuiTable` Sass mixins:
([#7654](https://github.com/elastic/eui/pull/7654))
- `euiTableActionsBackgroundMobile`
- `euiTableCellCheckbox`
- `euiTableCell`
</details>
## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments
### Summary
This PR enables LangSmith tracing for the [AI Insights](https://github.com/elastic/kibana/pull/180611) feature in cloud deployments.
LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>
### Details
To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:
```
xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']
```
- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here
- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing
After enabling the feature flags above:
1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant
2) Click the settings gear in the assistant
3) Click the `Evaluation` settings category
4) Click `Show Trace Options (for internal use only)`
5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:
## Summary
Part of #172981.
This fixes to not run log rate analysis twice when no spike/dip detected
and a user needs to adapt the initial selection. When a user clicks in
an area of the histogram chart that's not a highlighted change point,
the click will just trigger the baseline/deviation time range selection,
but it will not automatically run the analysis. Instead, an updated
prompt is shown below the chart that explains that the
baseline/deviation can be adjusted via dragging and the analysis can be
run via the button below that description.
Initial view after loading the page:
<img width="1040" alt="image"
src="90e8c390-af2a-45e2-8d11-cfd42285200b">
User clicked in an area that's not covered by the highlighted change
point:
<img width="1026" alt="image"
src="050a07e0-c5e6-4639-a854-83fae10b125b">
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Fixes https://github.com/elastic/kibana/issues/178446
This PR adds the following AI Ops features to the Burn rate alert detail
page:
- Log Rate Analysis
- ML features / AI Assistant
bf00c636-10ab-4cd8-859b-4a8f0471bf15
## Implementation details
Log rate analysis is useful when there is a significant dip/spike in
number of logs. It finds significant differences between the logs in
baseline (before dip/spike) and in deviation (after dip/spike). For the
Log Rate Analysis to work on the Alert details of an SLO burn rate rule
we had to specify a few params, here are the most important ones:
- dataView
- esSearchQuery
- timeRange -> I am looking into this and need a few clarifications
regarding lookBackPeriod
- initialAnalysisStart
For the esQuery here's what we used
```
const finalQuery = {
bool: {
filter: [customTotalFilter, customFilters, customGroupByFilters],
must_not: customGoodFilter,
},
};
```
We add the `total query`, any `optional filters` and any `group by
filters` to the `filter` clause of the ES query, where as we add the
good filter in the `must_not` clause.
## 🔬 How to test
- Add the following lines to your `config/kibana.dev.yaml`:
- `server.basePath: '/kibana'`
- `server.publicBaseUrl: 'http://localhost:5601/kibana'`
- Start with the following command: `node x-pack/scripts/data_forge.js
--events-per-cycle 50 --lookback now-1d --dataset fake_stack
--install-kibana-assets --kibana-url http://localhost:5601/kibana
--event-template good` to start with some good events
- Wait til the log message says `info Waiting 60000ms`
- Create one SLO:
- "Admin Console Availability" using the "Custom Query" SLI with the
`Admin Console` DataView, set the "Good query" to
`http.response.status_code < 500` and the set the "Total query" to
`http.response.status_code: *` using a rolling `7d` time window
- You should have 1 burn rate rule created by default
- Stop the first `data_forge.js` command
- Start `node x-pack/scripts/data_forge.js --events-per-cycle 50
--lookback now --dataset fake_stack --install-kibana-assets --kibana-url
http://localhost:5601/kibana --event-template bad`
- Go to alert details page of the burn rate rule to verify that the Log
Rate Analysis tool detected a spike
Note: you can skip first step, but in this case you should start kibana
with this command instead `yarn start --no-base-path`. This way you
don't have to pass `--kibana-url` when running the `data_forge` command
I included some snapshot tests with different SLOs that could be used
for verifying the generation of the ES query. Here are a few examples:
- Good query (KQL), no filter, no total query
- Good query (filter), no optional filter, no total query
- Good query (kql) optional filter (KQL), no total query
- Good query (kql), optional filter (filter), no total query
- Good query (kql), optional filter (kql), total query (kql)
- Good query (filter), optional filter(filter), total query(kql)
💀 I will need extra help testing this feature, to make sure that
provided params to the `LogRateAnalysisContent` component. Especially
`esSearchQuery` and `timeRange` (timeRange still WIP, but I want some
early feedback on the lookbackDuration and the intervalFactor)
## ✔️ Acceptance criteria
- Render the Log Rate Analysis tool only to platinum users and above
- Render AI assistant contextual insights only to platinum users and
above
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Cauê Marcondes <55978943+cauemarcondes@users.noreply.github.com>
Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
## [Security Solution] [AI Insights] AI Insights
### Summary
This PR introduces _AI Insights_ to the Security Solution:
_Above: AI Insights in the Security Solution_
AI Insights identify active attacks in the environment, without the time
(or prior experience) required to manually investigate individual alerts
in Elastic Security, identify if they are related, and document the
identified attack progression.
While users can ask the Assistant to find these progressions today, AI
Insights is a dedicated UI to identify these progressions and action
them accordingly. This feature adds a new page, `AI Insights`, to the
Security Solution's global navigation.
AI Insights are generated from Large Language Models (LLMs) to identify
attack progressions in alert data, and to correlate and identify related
entities and events. When possible, attack progressions are attributed
to threat actors.
### Details
Users may generate insights from a varetiy of LLMs, configured via
[Connectors](https://www.elastic.co/guide/en/kibana/master/action-types.html):
_Above: LLM selection via the connectors popup menu_
Clicking on the title of an insight toggles the insight between the
collapsed and expanded state:
_Above: Collapsing / expanding an insight (animated gif)_
The first three insights displayed on the AI Insights page are expanded
by default. Any additional insights that appear after the first three
must be expanded manually.
Insights provide a summary of the entities impacted by an attack.
Clicking on an entity, i.e. a hostname or username, displays the entity
flyout with the entity's risk summary:
_Above: Clicking on a host in the summary of the insight reveals the
host risk summary (animated gif)_
Hover over fields in the insight's summary or details to reveal pivot
actions for investigations:
_Above: Hovering over fields in the details of an insight reveals pivot
actions (animated gif)_
Insights are generated from alerts provided as context to the selected
LLM. The alert data provided to the LLM is anonymized automatically.
Anonymization is
[configured](https://www.elastic.co/guide/en/security/current/security-assistant.html#ai-assistant-anonymization)
via the same anonymization settings as the Assistant. Users may override
the defaults to allow or deny specific alert fields, and to toggle
anonymization on or off for specific fields.
Click the Anonymization toggle to show or hide the actual values sent to
the LLM:
_Above: Toggling anonymization to reveal the actual values sent to the
LLM (animated gif)_
### Empty prompt
At the start of a session, or when a user selects a connector that
doesn't (yet) have any insights, an [empty
prompt](https://eui.elastic.co/#/display/empty-prompt) is displayed.
The animated counter in the empty prompt counts up until it displays the
maximum number of alerts that will be sent to the LLM:
_Above: An animated counter displays the maximum number of alerts that
will be sent to the LLM (animiated gif)_
The _Settings_ section of this PR details how users configure the number
of alerts sent to the LLM. The animated counter in the empty prompt
immediately re-animates to the newly-selected number when the setting is
updated.
### Take action workflows
The _Take action_ popover displays the following actions:
- `Add to new case`
- `Add to existing case`
- `View in AI Assistant`
_Above: The Take action popover_
#### Add to new case
Clicking the `Add to new` case action displays the `Create case` flyout.
_Above: The `Add to new case` workflow_
An `Alerts were added to <case name>` toast is displayed when the case
is created:
_Above: Case creation toast_
A markdown representation of the insight is added to the case:
_Above: A markdown representation of an insight in a case_
The alerts correlated to generate the insight are attached to the case:
_Above: Insight alerts attached to a case_
#### Add to existing case
Clicking the `Add to existing case` action displays the `Select case`
popover.
_Above: The `Select case` popover_
When users select an existing case, a markdown representation of the
insight, and the alerts correlated to generate the insight are attached
to the case, as described above in the _Add to new case_ section.
#### View in AI Assistant
The `View in AI Assistant` action in the `Take action` popover, and two
additional `View in AI Assistant` affordances that appear in each
insight have the same behavior:
Clicking `View in AI Assistant` opens the assistant and adds the insight
as context to the current conversation.
_Above: An insight added as context to the current conversation_
Clicking on the insight in the assistant expands it to reveal a preview
of the insight.
_Above: An expanded insight preview in the assistant_
The expanded insight preview reveals the number of anonymzied fields
from the insight that were made available to the conversation. This
feature ensures insights are added to a conversation with the anonymized
field values.
An insight viewed in the AI assistant doesn't become part of the
conversation until the user submits it by asking a question, e.g. `How
do I remediate this?`.
Insights provided as context to a conversation are formatted as markdown
when sent to the LLM:
_Above: Insights provided as context to a conversation are formatted as
markdown_
Users may toggle anonymization in the conversation to reveal the
original field values.
_Above: Revealing the original field values of an insight added as
markdown to a conversation (animated gif)_
#### Alerts tab
The _Alerts_ tab displays the alerts correlated to generate the insight.
_Above: The alerts correlated to generate the insight in the Alerts tab_
The `View details`, `Investigate in timeline`, and overflow row-level
alert actions displayed in the Alerts tab are the same actions available
on the Cases's page's Alerts tab:
_Above: Row-level actions are the same as the Cases pages Alert's tab_
#### Investigate in Timeline
Click an insight's `Investigate in Timeline` button to begin an
investigation of an insights's alerts in Timeline. Alert IDs are queried
via the `Alert Ids` filter:
_Above: Clicking Investigte in Timeline (animated gif)_
The alerts from the insight are explained via row renderers in Timeline:
_Above: Row rendered insight alerts in Timeline_
### Attack Chain
When alerts are indicative of attack
[tactics](https://attack.mitre.org/tactics/enterprise/), those tactics
are displayed in the insights's _Attack Chain_ section:
_Above: An insight with tactics in the Attach chain_
The Attack Chain section will be hidden if an insight is not indicative
of specific tactics.
### Mini attack chain
Every insight includes a mini attack chain that visually summarizes the
tactics in an insight. Hovering over the mini attack chain reveals a
tooltip with the details:
_Above: The mini attack chain tooltip_
### Storage
The latest insights generated for each connector are cached in the
browser's session storage in the following key:
```
elasticAssistantDefault.aiInsights.cachedInsights
```
Caching insights in session storage makes it possible to immediately
display the latest when users return to to the AI insights page from
other pages in the security solution (e.g. Cases).
_Above: Cached insights from sesion storage are immediately displayed
when users navigate back to AI Insights (animated gif)_
While waiting for a connector to generate results, users may view the
cached results from other connectors.
Cached insights are immediately available, even after a full page
refresh, as long as the browser session is still active.
### `Approximate time remaining` / `Above average time` counters
Some LLMs may take seconds, or even minutes to generate insights. To
help users anticipate the time it might take to generate an insight, the
AI insights feature displays a `Approximate time remaining: mm:ss`
countdown timer that counts down to zero from the average time it takes
to generate an insight for the selected LLM:
_Above: The `Approximate time remaining: mm:ss` countdown counter
(animated gif)_
If the LLM doesn't generate insights before the counter reaches zero,
the text will change from `Approximate time remaining: mm:ss` to `Above
average time: mm:ss`, and start counting up from `00:00` until the
insights are generated:
_Above: The `Above average time: mm:ss` counter (animated gif)_
The first time insights are generated for a model, the `Approximate time
remaining: mm:ss` counter is not displayed.
Average time is calculated over the last 5 generations on the selected
connector. This is illustrated by clicking on the (?) information icon
next to the timer. The popover displays the average time, and the time
in seconds for the last 5 runs:
_Above: Clicking on the (?) information icon displays the average time,
and the duration / datetimes for the last 5 generations_
The time and duration of the last 5 generations (for each connector) are
persisted in the browser's local storage in the following key:
```
elasticAssistantDefault.aiInsights.generationIntervals
```
### Errors
When insight generation fails, an error toaster is displayed to explain
the failure:
_Above: An error toaster explains why insights generation failed_
### Feature flag
The `assistantAlertsInsights` feature flag must be enabled to view the
`AI Insights` link in the Security Solution's global navigation.
Add the `assistantAlertsInsights` feature flag to the
`xpack.securitySolution.enableExperimental` setting in
`config/kibana.yml` (or `config/kibana.dev.yml` in local development
environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantAlertsInsights']
```
### Settings
The number of alerts sent as context to the LLM is configured by
`Knowledge Base` > `Alerts` slider in the screenshot below:
- The slider has a range of `10` - `100` alerts (default: `20`)
Up to `n` alerts (as determined by the slider) that meet the following
criteria will be returned:
- The `kibana.alert.workflow_status` must be `open`
- The alert must have been generated in the last `24 hours`
- The alert must NOT be a `kibana.alert.building_block_type` alert
- The `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize
the riskiest alerts
### License
An Enterprise license is required to use AI Insights.
The following AI Insights view is displayed for users who don't have an
Enterprise license:
## How it works
- Users navigate to the AI insights page:
`x-pack/plugins/security_solution/public/ai_insights/pages/index.tsx`
- When users click the `Generate` button(s) on the AI Insights page,
insights are fetched via the `useInsights` hook in
`x-pack/plugins/security_solution/public/ai_insights/use_insights/index.tsx`.
- The `fetchInsights` function makes an http `POST` request is made to
the `/internal/elastic_assistant/insights/alerts` route. include the
following new (optional) parameters:
- `actionTypeId`, determines tempature and other connector-specific
request parameters
- `alertsIndexPattern`, the alerts index for the current Kibana Space,
e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings,
e.g. `["@timestamp", "cloud.availability_zone", "file.name",
"user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the
`Anonymization` settings, e.g. `["cloud.availability_zone", "host.name",
"user.name", ...]`
- `connectorId`, id of the connector to generate the insights
- `replacements`, an optional `Record<string, string>` collection of
replacements that always empty in the current implementation. When
non-empty, this collection enables new insights to be generated using
existing replacements.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the maximum number of alerts to generate insights from. This
numeric value is set by the slider in the user's `Knowledge Base >
Alerts` setting, e.g. `20`
- The `postAlertsInsightsRoute` function in
`x-pack/plugins/elastic_assistant/server/routes/insights/alerts/post_alerts_insights.ts`
handles the request.
- The inputs and outputs to this route are defined by the
[OpenAPI](https://spec.openapis.org/oas/v3.1.0) schema in
`x-pack/packages/kbn-elastic-assistant-common/impl/schemas/insights/alerts/post_alerts_insights_route.schema.yaml`.
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
- The `postAlertsInsightsRoute` route handler function in
`x-pack/plugins/elastic_assistant/server/routes/insights/alerts/post_alerts_insights.ts`
invokes the `insights-tool`, defined in
`x-pack/plugins/security_solution/server/assistant/tools/insights/insights_tool.ts`.
The `insights-tool` is registered by the Security Solution. Note: The
`insights-tool` is only used for generating insights. It is not used to
generate new insights from the context of an assistant conversation, but
that feature could be enabled in a future release.
- The `insights-tool` uses a LangChain `OutputFixingParser` to create a
[prompt
sandwich](https://www.elastic.co/blog/crafting-prompt-sandwiches-generative-ai)
with the following parts:
```
_________________________________________________
/ \
| Insight JSON formatting instructions | (1)
\ _______________________________________________/
+------------------------------------------------+
| Insights prompt | (2)
+------------------------------------------------+
/ \
| Anonymized Alerts | (3)
\_______________________________________________/
```
- The `Insight JSON formatting instructions` in section `(1)` of the prompt sandwich are defined in the `getOutputParser()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_output_parser.ts`. This function creates a LangChain `StructuredOutputParser` from a Zod schema. This parser validates responses from the LLM to ensure they are formatted as JSON representing an insight.
- The `Insights prompt` in section `(2)` of the prompt sandwich is defined in the `getInsightsPrompt()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_insights_prompt.ts`. This part of the prompt sandwich includes instructions for correlating insights, and additional instructions to the LLM for formatting JSON.
- The `Anonymized Alerts` in section `(3)` of the prompt sandwich are returned by the `getAnonymizedAlerts()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_anonymized_alerts.ts`. The allow lists configured by the user determine which alert fields will be included and anonymized.
- The `postAlertsInsightsRoute` route handler returns the insights generated by the `insights-tool` to the client (browser).
- Insights are rendered in the browser via the `Insight` component in `x-pack/plugins/security_solution/public/ai_insights/insight/index.tsx`
- The `AiInsights` tab in `x-pack/plugins/security_solution/public/ai_insights/insight/tabs/ai_insights/index.tsx` includes the _Summary_ and _Details_ section of the Insight.
- The `InsightMarkdownFormatter` in `x-pack/plugins/security_solution/public/ai_insights/insight_markdown_formatter/index.tsx` renders hover actions on entities (like hostnames and usernames) and other fields in the insight.
- The `Insight` component makes use of the `useAssistantOverlay` hook in `x-pack/packages/kbn-elastic-assistant/impl/assistant/use_assistant_overlay/index.tsx` to register the insight as context with the assistant. This registration process makes it possible to view insights in the assistant, and ask questions like "How do I remediate this?". In this PR, the `useAssistantOverlay` hook was enhanced to accept anonymizaton replacements. This enables an assistant conversation to (re)use replacements originally generated for an insight.
Fixes https://github.com/elastic/kibana/issues/179908
Fixes https://github.com/elastic/kibana/issues/1798952924e619-9080-4c80-b5c3-7066a100026c
## UI changes
### SLO Overview configuration
<img width="600" alt="Screenshot 2024-04-15 at 13 34 21"
src="3635c9e7-e5c1-454b-bb60-93b5d637083e">
### Rendered panel with Group of SLOs
<img width="1022" alt="Screenshot 2024-04-03 at 13 08 11"
src="377c5cd0-b6e0-4906-888b-93893a5f1208">
### Updated SLO list group by page
<img width="1316" alt="Screenshot 2024-03-28 at 23 56 12"
src="d6b306a0-77e8-44e5-b103-ce3537b0ceda">
## Responsiveness
This PR makes sure that the UI doesn't look broken in smaller screen
resolutions and fixes following issues. If we want to display things
differently in smaller screen resolutions, we should handle in a follow
up PR (cc @maciejforcone)
### Before
#### SLO list page in smaller screen resolution
<img width="500" alt="Screenshot 2024-04-03 at 10 14 31"
src="f2b11408-3356-4f4b-9c4b-ec085ff494fa">
#### Rendered panel with Group of SLOs in smaller screen resolution
<img width="500" alt="Screenshot 2024-04-03 at 10 13 18"
src="6e37ad61-5a0a-4027-8b90-4f8fad243637">
#### After
#### SLO list page in smaller screen resolution
<img width="500" alt="Screenshot 2024-04-03 at 11 04 35"
src="fa875aa8-bc54-4c6b-9159-4462e585a9cb">
#### Rendered panel with Group of SLOs in smaller screen resolution
I applied a `min-width` of 100px to the SLO name, so that it doesn't
look broken. What is not visible in the screenshot is a horizontal
scrollbar, so user can still see the part with Worst performing that is
not visible.
<img width="500" alt="Screenshot 2024-04-03 at 10 13 47"
src="d9566ca3-42af-4d85-a3f7-6374cc27981d">
## Acceptance criteria
- Overview configuration
- toggles between Single SLO or Grouped SLOs
- Required field for the `Grouped SLOs` view is only the `Group by`
option, the rest filters are optional
- Users can optionally select the groups they want
- Users can use the KQL bar to apply extra filters
- Overview configuration opens in a `Flyout` (modal was causing some
issues with z-index and kqlbar suggestions being displayed below the
overlay)
- Embeddable panel
- The rendered panel shows the selected groups with summary data divided
with a horizontal like and no border around it
- Clicking on a group opens an accordion similar to the SLO list group
by page
- An Edit criteria link is visible only in the Grouped Embeddable panel
- Clicking on the Edit criteria link opens the configuration modal,
showing only the grouped tab and hiding the tabbed button group
- The grouped embeddable should react to Dashboard's Refresh button
- `Add to Dashboard` action should be hidden in Dashboard context
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Cee Chen <549407+cee-chen@users.noreply.github.com>
## Summary
issue: https://github.com/elastic/kibana/issues/180680
Adds the `checkedBy` and `indexPattern` properties to the data quality
dashboard data stream mapping.
- The `checkedBy` information is retrieved from the `currentUser` uuid
in the server API.
- The `indexPattern` is sent from the UI.
This is a preparation PR for the Historical data quality checks page we
plan to introduce in 8.15. We will need these new fields stored in the
results data stream to implement the new page. So we should start
indexing these entries as soon as possible.
The `indexPattern` will be needed to group individual index results, and
the `checkedBy` user information will be displayed in the summary.
### Screenshots
Before:
After:
The `indexPattern` is set according to the pattern used by the UI to
group the indices:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Serverless pending on
https://github.com/elastic/elasticsearch-serverless/pull/1482
1. Adding a data fatcher for metering/stats api - This is for Serverless
Kibana especially.
2. Given that the response for metering/stats api is simpler. I amended
the response of original stats api to align with the metering/stats.
Both apis now return:
e.g.:
```
{
".ds-my-datastream-03-02-2024-00001": {
"name": ".ds-my-datastream-03-02-2024-00001",
"num_docs": 3,
"size_in_bytes": 15785, // ---> null on serverless
"datastream": "my-datastream" // ---> ds name, null on ESS
},
"my-test-index": {
"name": "my-test-index",
"num_docs": 2,
"size_in_bytes": 11462
},
}
```
**Note**:
When an index health is `green` , its primaries.docs.count and
total.docs.count have different values after deployed to the cloud.
However, the doc count of both health yellow and green indices were all
coming from mockStatsGreenIndex[indexName].primaries?.docs?.count. So
when we are normalising the api formate, we should just use
`primaries.docs.count` to retrieve the doc count.
Example:
https://github.com/elastic/kibana/pull/179666/files#diff-49d86730fb4dd8bff8fca0ca72013703759075a4a81694349db881ebb1173188R60
**Steps to verify (serverless):**
https://p.elstc.co/paste/5R+Z497w#oT-1mqcTLxIEwYNtZFnv4BMYclGU7HPS1By+XQAjeIR
0.
x-pack/plugins/security_solution/public/app/solution_navigation/links/app_links.ts
remove L30-33 to reveal the entry of dq dashboard on serverless.
1. Mock the response in
https://github.com/elastic/kibana/pull/179666/files#diff-24b3062cb373e7849bda6cf4be35466545f00333b558d2047fcdaaf272a9ba4cR44
to return
```
Promise.resolve({
_shards: {
total: 4,
successful: 2,
failed: 0,
},
indices: [
{
name: '.ds-my-datastream-03-02-2024-00001',
num_docs: 3,
datastream: 'my-datastream', // ---> ds name
},
{
name: 'my-test-index',
num_docs: 2,
},
],
datastreams: [
{
name: 'my-datastream',
num_docs: 6,
},
],
total: {
num_docs: 8,
},
});
```
2. Create an index with docs containing same-family field
```
PUT my-test-index
PUT my-test-index/_mapping
{
"properties": {
"@timestamp": {
"type": "date"
},
"agent.type": {
"type": "constant_keyword"
},
"event.category": {
"type": "constant_keyword"
}
}
}
POST my-test-index/_doc
{
"@timestamp": "2024-04-08T09:41:49.668Z",
"host": {
"name": "foo"
},
"event": {
"category": "an_invalid_category"
},
"some.field": "this",
"source": {
"port": 90210,
"ip": "10.1.2.3"
}
}
POST my-test-index/_doc
{
"@timestamp": "2024-02-08T09:42:22.123Z",
"host": {
"name": "bar"
},
"event": {
"category": "an_invalid_category"
},
"some.field": "space",
"source": {
"port": 867,
"ip": "10.9.8.7"
}
}
```
3. Add my-test-* into security-default-dataview from stack management.
**Expected result (Serverless)** (with the mock api response as above to
simulate the result):
1. `doc size` is hidden on the Serverless dq dashboard.
2. It shows `doc count` in the treemap for each index on the Serverless
dq dashboard.
<img width="2560" alt="Screenshot 2024-04-15 at 11 28 33"
src="1d3b7d3e-4650-4711-b76b-6215de9e1a59">
70edb22c-15c9-4f2e-b740-5bf2bd630da5
**Expected result (ESS)**: Should remain the same
https://p.elstc.co/paste/yRGlJc2z#mfABwGRSjk7dbxhzQ3bn+jqCtSSBT5UVK4W46CsVoBh
<img width="2560" alt="Screenshot 2024-04-10 at 09 31 58"
src="b6194cc8-8975-4ac4-b157-64d1e39ecb28">
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
Added new sub-feature to AI-Assistant which allows to grant user role
privilege to edit Anonymization fields:
<img width="761" alt="Screenshot 2024-04-11 at 7 29 50 PM"
src="d4358178-d8e9-4c68-b7d4-a19d2befa29b">
How to test:
1. Create user role, which has access to Security and Actions/Connectors
and AI Assistant.
2. Customize sub-feature privileges: Remove checkbox for Update
anonymization fields.
3. Save role.
4. Create/update existing user with the new role.
5. Go to Security AI Assistant settings and open Anonymization tab.
6. For the user role with removed privilege to edit anonymization, all
actionable buttons should be disabled. Button Save will remain enabled,
because it correspond to all Assistant settings.
7. Public API "/api/elastic_assistant/anonymization_fields/_bulk_action"
should return "Forbidden" 403 access error.
How it looks when no privilege for the user role:
<img width="771" alt="Screenshot 2024-04-12 at 10 12 34 AM"
src="7c3b6c92-12cb-46ae-8356-dc687c82726f">
@patrykkopycinski please adopt it after merging this PR to the new UX
---------
Co-authored-by: Garrett Spong <garrett.spong@elastic.co>
## Summary
issue: https://github.com/elastic/kibana/issues/179162
This PR brings the new navigation, with the solution-centric IA
(Information Architecture), to the ESS (stateful) Security Solution.
To do so, the implementation of the `navigationTree$`, which was
previously only implemented in serverless, has been integrated inside
the generic "security_solution" plugin, so now it is available for ESS
and serverless offerings.
In ESS users can still choose the navigation version, so we have to
temporarily keep supporting both, the classic and new navigation
implementations. After the rollout, the classic navigation components
will be removed and the unified links architecture should be reassessed.
The issue for the cleaning:
https://github.com/elastic/kibana/issues/179572
### Rollout
The new solutions navigation will not be available for customers on
8.14, it will only be enabled for internal Elastic users (via
Lauchdarkly), who will have the ability to opt out of it from their
profile menu. We'll collect feedback and telemetry and address any bugs
or improvements (together with the Kibana platform team). The plan is to
start making it available to customers in 8.15.
### Testing
Unless we add the _kibana.yml_ configurations to enable the new
navigation, the regular classic navigation will be displayed, it should
keep working the same way without any change. The new landing pages
(`Assets`, `Investigations`, `Machine Learning`,...), that exist only
when using the new navigation, should not be accessible using the
classic version.
To enable the new navigation add the following _kibana.yml_ configs:
```
xpack.cloud_integrations.experiments.enabled: true
xpack.cloud_integrations.experiments.flag_overrides:
"navigation.solutionNavEnabled": true
xpack.cloud.id: "ftr_fake_cloud_id:aGVsbG8uY29tOjQ0MyRFUzEyM2FiYyRrYm4xMjNhYmM="
xpack.cloud.base_url: "https://cloud.elastic.co"
xpack.cloud.deployment_url: "/deployments/deploymentId"
```
And enable the advanced setting
### Screenshots
The app switcher:
<img width="293" alt="app switcher"
src="0a638b8f-fdc0-4d1a-b8d3-607e487215f4">
---
New Assets landing page:
<img width="549" alt="assets landing"
src="17bc8a94-02b4-4996-b9f5-8731ba81ac43">
---
For `Stack Management` we set the nav panel flyout and the cards landing
page, this is temporary until a decision on how to show Stack Management
links is made:
<img width="954" alt="stack management"
src="27ce6534-0508-4804-b224-8dc409042825">
---
The switch to go back to the classic nav is in the profile menu at the
top-right corner:
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
Co-authored-by: jennypavlova <dzheni.pavlova@elastic.co>
Co-authored-by: Katerina <aikaterini.patticha@elastic.co>
Co-authored-by: Sébastien Loix <sebastien.loix@elastic.co>
Co-authored-by: Kurt <kc13greiner@users.noreply.github.com>
Co-authored-by: Justin Kambic <jk@elastic.co>
Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
Co-authored-by: Nathan Reese <reese.nathan@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
Co-authored-by: Milton Hultgren <milton.hultgren@elastic.co>
## Summary
The `AnomalyExplorerChartsService` was importing the
`SWIM_LANE_LABEL_WIDTH` numerical constant from
`swimlane_container.tsx`. As a result, the _entirety_ of that file was
being included in the async bundle.
`AnomalyExplorerChartsService` is loaded asynchronously on page load by
the embeddable. By relocating the constant to its own file-- as well as
other optimizations (see below)-- we reduce the async page load by 74%,
(in dev mode).
### Before - `351k`
<img width="1220" alt="Screenshot 2024-03-24 at 10 09 31 AM"
src="6cea41f5-fb10-41c3-8951-a4be897174f9">
### After - `93.4k`
<img width="1471" alt="Screenshot 2024-04-05 at 11 45 45 AM"
src="a07e7a19-c1af-4b45-a195-69730fc61b0c">
Unfortunately, this change led to a bloating of async modules, the cause
of which is still unclear. The application async chunk weighed in at 2.2
MB compressed! To get this PR to a shippable state, I refactored
additional code to bring down duplication and bloat.
The result is an `ml` experience that fetches small bundles on demand as
someone interacts with it:
More work can be done to continue to optimize the plugin, of course, but
this set of changes is an excellent start, (and case study on bundle
load).
### Other optimizations in this PR
- Registration of some `start` services are conditional, and contain
their own async calls. I've removed these from the register helper,
(which itself is a brute-force offload of code from the plugin, but is
still loaded every time), and loaded them async if the conditions apply.
- Routing in `ml` use factories to create individual routes. In a lot of
cases, the pages these routes displayed were not asynchronously loaded,
adding tremendous amounts of React code to the root application.
- I made all pages loaded by routes async.
- In some cases, the components themselves were colocated with the route
factory. I moved those to their own files for async import.
- Similarly, some state managers were colocated. Those were moved to
their own files, as well.
- Moved flyouts, modals, expanding rows to `React.lazy` async modules,
(using `dynamic` from `@kbn/shared-ux-utility`.
- Refactored `export * from` directives from `public/shared.ts` to
accurately reflect what is being consumed outside the `ml` plugin, (and
also reduce the size of _that_ bundle.
- Refactored `lodash` imports to submodule imports to enable
tree-shaking, (if ever enabled in webpack).
- Moved `getMlGlobalServices` off of the `app.tsx` file for import by
others, (to avoid depending on the rest of the code there).
- Moved some types to `/common` to avoid importing code, (though,
admittedly, types are compiled away). But it felt cleaner to move them
out.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
PR changes includes migration for anonymization defaults from the local
storage to use the ES data stream instead.
1. Extended `AIAssistantService` with the default setup method
`createDefaultAnonymizationFields` for anonymization fields, which will
be executed once for each space data stream. With that approach we will
have anonymization settings centralized for all the applications which
will use security GenAI functionality.
2. Anonymization fields is not personalizable and all the changes will
be applied to all users in the current space. Only users with the
defined privileges will be able to change anonymization setting.
3. Removed anonymization defaults from the AssistantContext provider.
Now fetch anonymization fields using API
`"/api/elastic_assistant/anonymization_fields/_find"`.
4. Anonymization settings component:
- replaced usage of `defaultAllow: string[];`, `defaultAllowReplacement:
string[];`,
`setUpdatedDefaultAllow:React.Dispatch<React.SetStateAction<string[]>>;`,
`setUpdatedDefaultAllowReplacement:
React.Dispatch<React.SetStateAction<string[]>>;` with
anonymizationFields fetched from the APIs;
- for the anonymize/allow UI operations use bulk actions update
operation;
- all the changes will be send to the server side using bulk API
`"/api/elastic_assistant/anonymization_fields/_bulk_action" ` after Save
settings button clicked or reset to the previous state when clicked
Cancel.
5. Data anonymization editor:
- This component will remain to apply the anonymization changes only in
the message scope. That means that any type of changes will be persisted
in the `.kibana-elastic-ai-assistant-anonymization-fields` and impact
only on the fields replacements in the selectedPromptContext.
6. Changed `.kibana-elastic-ai-assistant-anonymization-fields` data
stream schema to fits the UX:
```
"@timestamp": "2024-04-05T04:10:24.764Z",
"created_at": "2024-04-05T04:10:24.764Z",
"created_by": "",
"field": "_id",
"anonymized": false,
"allowed": true,
"namespace": "default",
"updated_at": "2024-04-05T17:15:00.903Z"
```
Components involved:
<img width="1091" alt="Screenshot 2024-04-05 at 11 35 05 AM" src="807ee596-101f-413c-97d6-8f62c723397e">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Closes#176718
## Summary
In this PR, I will apply the group by information on the history chart
of the following rules:
1. Log threshold
2. APM Latency threshold
3. SLO burn rate
I also update the custom threshold history chart to use instanceId for
filtering alerts instead of queries.
## 🧪 How to test
Check both the number of alerts on the annotation and also the data in
the history chart to match the data for the defined group, in the
following rules.
1. Log threshold
2. APM Latency threshold
3. SLO burn rate
