## Summary
This PR refactors https://github.com/elastic/kibana/pull/179206 to have
each export type be registered in Reporting and then passed into the
share plugin.
This PR is focused on the redesign in terms of the export modals. Test
refactoring will be done in a separate PR.
Partially closes https://github.com/elastic/kibana-team/issues/753
- [x] Need to refactor this PR to include @eokoneyo's general modal
component
- [x] Lens needs to have Export with all three report type options - to
avoid circular dependencies move the Lens CSV stuff into the reporting
plugin vs having it in Lens
- [x] Canvas should not be affected by these changes (so the old
share/reporting code has to stay for canvas)
https://github.com/elastic/kibana/issues/151523 to keep in mind for the
redesign
Failed tests will be covered in this PR
https://github.com/elastic/kibana/pull/180406
### TO TEST
Mark `share.new_version.enabled: true` in your kibana.dev.yml
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
---------
Co-authored-by: Eyo Okon Eyo <eyo.eyo@elastic.co>
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
## Summary
This is needed where users want to generate links to synthetics monitors
using uptime alerts !!
### Example usage
<img width="625" alt="image"
src="4f2f7833-9313-4c3a-8ea0-a383cb2ec857">
<img width="956" alt="image"
src="10541a9c-987b-427c-a55e-4e0ba01891cf">
closes https://github.com/elastic/kibana/issues/178719
A new ES API has been created to support linking APM from the Profiling
UI. It's called `topN/functions`. The new API allows grouping fields. So
we first fetch functions grouping by `service.name` and when the user
opens the APM Transactions we make another request grouping by
`transaction.name`.
A new Advanced setting was created to toggle the old API on (fetch
functions from Stacktraces API): It's turned off by default.
<img width="1235" alt="Screenshot 2024-04-12 at 10 39 36"
src="ee6e7731-2f44-43ca-9793-23ba87e22e6e">
When there are services on the selected function:
*If we cannot find the transaction, we show `N/A`.
<img width="933" alt="Screenshot 2024-04-12 at 10 16 34"
src="2c5dbf60-3a47-4f4c-a46d-8a0984e0e482">
When there are **no** services on the selected function:
*hide the APM transactions section
<img width="921" alt="Screenshot 2024-04-12 at 10 59 14"
src="3fc4c5b1-da62-47c8-97a8-8bcbd1ae1b75">
--
Performance boost:
The new API is faster than the Stacktraces API, especially because
there's no logic on the Kibana side.
Stacktraces API:
<img width="1210" alt="Screenshot 2024-04-12 at 10 50 26"
src="158d73d1-ed91-4652-97c1-c7c3328d5e3d">
TopN/Functions API:
<img width="1195" alt="Screenshot 2024-04-12 at 10 51 20"
src="2de4ef46-eb8a-4557-b7b8-a1c2fed6fd8a">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Adds the Breakdown field to Degraded Docs Visualization.
The PR exposes the `BreakdownFieldSelector` from
**@kbn/unified-histogram-plugin** as a similar visualization behaviors
and actions are needed on the Dataset Quality flyout as with Logs
Explorer.
<img width="1242" alt="Screenshot 2024-04-12 at 04 00 06"
src="0e34397a-ac56-4a2b-8483-94c96dcbf3c2">
## Summary
- Renames `GlobalAlertsPage` to `StackAlertsPage` to better account for
the change in scope
- Renames the `triggerActionsUi.globalAlertsPage` flag to
`triggerActionsUi.stackAlertsPage`
- Toggles the flag on by default
- Updates the tech preview badge to match the other badges used across
triggers_actions_ui
- Fixes the feature cell rendering logic not displaying Observability
alerts correctly
- Other small fixes and improvements
## To verify
1. Navigate to the `Stack management > Alerts` page (the page should now
be visible without a the feature flag turned on manually)
2. Check that the tech preview badge appears correctly and shows the
tooltip on hover
3. Check that Observability alerts show an `Observability` badge in the
`Feature` column
## Bugs Fixed
1. The OpenAI `invokeAI` method did not properly handle `signal`
2. Bedrock did not have a `signal` implementation at all 😳
## Summary
In my [LangChain streaming
PR](https://github.com/elastic/kibana/pull/174126), I poorly implemented
a fix to stop the stream on the server when "Stop generating..." was hit
on the client. I did this by piping through an `AbortSignal` to
`invokeStream`/`invokeAsyncIterator` subactions. However, in the
`invokeAI` subaction I did not properly remove `signal` before
`JSON.strinigfy`ing the body, so the below error was happening in the
security non-streaming implementation. Additionally, for Bedrock I
somehow only implemented `signal` in part of the type and nothing else,
so token tracking would be off when Stop generating button is hit 🤦
<img width="1376" alt="Screenshot 2024-04-15 at 2 00 38 PM"
src="e57241d9-9fd2-4dd3-bb3a-72a7c61a3d4b">
## To test
1. Turn off streaming in the Security AI Assistant and select an OpenAI
connector (LangChain off)
3. Send a message
4. Ensure expected results (prior the above error would occur)
The test of the Bedrock connector will be harder to confirm. Where the
issue would show up would be subtle, in the token counter. Before I
implemented the signal in the Bedrock connector, if you ask Bedrock to
repeat a word 100 times with streaming enabled, and then hit "Stop
generating..." after 10 words, you would see a token count for
`completion_tokens` be equivalent to ~100 tokens as the full response
would have "streamed" on the server. After this bug fix, if you hit
"Stop generating..." after 10 words, you will see a token count for
`completion_tokens` be equivalent to ~15 tokens as it takes a second for
the `abort()` to reach the server. To be clear, this bug would not have
shown in persistent storage because we call abort in
`handleStreamStorage` ASAP instead of relying on axios to complete its
abort.
## Summary
Exclude tiers function wasn't working for alerting and other server side
data view usage. This PR moves the tier exclusion code to a more central
location so it will apply in ALL circumstances, not just REST API
requests for fields.
There are three places that call the IndexPatternsFetcher - the
fields_for_wildcard API, the fields API, and the IndexPatternsApiServer
for server side field loading.
Oh, `x-pack/plugins/alerting/server/routes/suggestions/fields_rules.ts`
calls IndexPatternFetcher too. And some rules, which don't need this
functionality. IndexPatternsFetcher is used in more places than I knew /
intended.
Closes https://github.com/elastic/kibana/issues/180675
## Summary
Changes done in SentinelOne connector in support of Security Solution
`get-file` response action (forthcoming):
- Added `fetchAgentFiles()` sub-action
- Added `downloadAgentFile()` sub-action
- Added `getActivities()` sub-action
- Improved error messages for SentinelOne API failures
- Added `logger.debug()` to Sub-Actions connector `validateResponse()`
to output data that failed validation
## Summary
Fixes a bug with OpenAI (not azure) connectors when used with
`invokeAsyncIterator`:
```
ActionsClientChatOpenAI: an error occurred while running the action - Unexpected API Error: - 404 Invalid URL (POST /v1/chat/completions/chat/completions)
```
Our current default url for OpenAI connectors is
`https://api.openai.com/v1/chat/completions`
When using the `invokeAsyncIterator` subaction, we use the OpenAI node
package. This takes the url as an argument to instatiate. It wants the
URL of OpenAI, not the completions endpoint which is our default.
Looking back, I wish we had made the default url
`https://api.openai.com/v1`. However, because we want to support
existing connectors I think we should keep this the default and remove
the endpoint from the url when it is passed to the OpenAI node package.
### To test
Send a message in Security Assistant with an OpenAI (not azure)
connector with streaming enabled. Observe a successful response.
### Note to docs team
@elastic/security-docs We should call this out. If a user has an open
source OpenAI connector with a chat completions endpoint that does not
have the same structure as OpenAI's ending in `/chat/completions`, their
full endpoint url will be used with LangChain OpenAI Streaming
implementations. For example, if a user had a url like
`https://mycustomllm.com/execute/completions` and tries to use the the
Security Assistant with Knowledge Base on and Streaming on, they may get
this error:
```
ActionsClientChatOpenAI: an error occurred while running the action - Unexpected API Error: - 404 Invalid URL (POST /execute/completions/chat/completions)
```
We should instruct them to follow the same route structure as OpenAI.
Their endpoint needs to end in `/chat/completions`
## Summary
Moves Security's alerts filter controls bar implementation to the
`@kbn/alerts-ui-shared` package for re-use in the Stack alerts (unified)
page and Observability solution (separate PR).
<img width="1339" alt="image"
src="986e984a-0290-4980-859b-9d6e02ef335f">
## To verify
1. Enable the Stack alerts page feature flag:
```yaml
# config/kibana.dev.yml
xpack.trigger_actions_ui.enableExperimental:
- globalAlertsPage
```
2. Create some Stack/O11y rules that fire some alerts
3. Navigate to `Management > Stack management > Alerts`, you should see
the new controls below the KQL bar.
4. Check that the filters apply correctly and reflect on the table
results
5. Check that the filter controls can be customized (`••• > Edit
controls`, add/remove controls, save)
6. Check that the customized configuration persists across reloads
7. Check that the filter controls can be temporarily overridden through
the URL param `filterControls`
8. Check that when the filters are overridden through the URL, a notice
is shown to the user with CTAs to discard/save the temporary filter
configuration
---
1. Navigate to `Security > Alerts > Manage rules`
2. Create one or more security rule that fire alerts
3. Go back to the security alerts page
4. Repeat steps 4-8
Closes#176711
---------
Co-authored-by: Xavier Mouligneau <xavier.mouligneau@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jatin Kathuria <jtn.kathuria@gmail.com>
This adds an API call to the contextual insights component which should
improve the relevance of the responses by providing more context to the
LLM.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Cauê Marcondes <55978943+cauemarcondes@users.noreply.github.com>
## Summary
This pr makes use of a new prop (and some generic memoization fixes) in
2 eui prs merged recently (https://github.com/elastic/eui/pull/7556 and
https://github.com/elastic/eui/pull/7374) to improve the performance of
the alerts table. Mainly, the cellContext prop is now available to
consumers of the triggersActionsUi alerts table to pass in custom props
in a way that allows the renderCellValue/rowCellRender functions of the
various EuiDataGrid prop apis to remain referentially stable across
re-renders. There are also some various changes to various hooks and
props found throughout plugins that use the table to improve
performance. There should be no change in functionality, just a moderate
to substantial reduction in time the app spends rendering the alerts
table in various scenarios. Below will be some react dev tools
performance profiles, main compared to this branch, with the exact same
set of generated data.
Main, switching from 10-100 rows:
This branch 10-100 rows:
Pretty modest gain here, 1 render is slower than any others on main, but
overall less time spent rendering by about a second.
Main, opening the cell popover on 2 cells
This branch, opening cellpopover
Again nothing crazy here, modest improvement.
Main opening timeline and hitting refresh
This branch, opening timeline and hitting refresh
This is the case that brought this about in the first place, as security
solution shares a lot of code between tables, the alerts table
recreating all cells on every render was destroying performance of the
timeline rendering in a flyout/modal while users were on alerts page or
the rule detail page, which is probably the most common use case. 93ms
in this branch vs 2500+ ms in main. This type of performance hit happens
when almost any action is taken in timeline.
Main, selecting 3 alerts 1 at a time
This branch selecting 3 alerts 1 at a time
Pretty substantial improvement here as well, ~2000ms vs 67ms.
Apologies if some of the gifs are cut off, please try the branch vs main
on your own! This was done on a local kibana in dev mode, so things like
emotion are eating up more cpu than they would for a user, but there
still should be improvement, and I can't think of any circumstance where
things will be worse. Also this branch was a little longer lived than I
would have liked, so if you are reviewing and changed something around
these files recently please double check I didn't miss anything.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
- [x] Adds updated warning messaging for trusted apps entries that use
wildcards `*?` with the "IS" operator
- [x] Three different warnings: callout, individual entry item warnings
and a final confirmation modal when the user tries to add an event
filter with ineffective IS / wildcard combination entry.
- [x] Unit tests
- [x] Fixes bug in endpoint exceptions where there is a missing tooltip
icon for the wildcard performance warning
# Screenshots
Bug Fix
## Summary
Adds a bunch of descriptive tooltips to the stats.
<img width="324" alt="Screenshot 2024-04-16 at 15 02 57"
src="c0df028f-ad1d-45c3-b16e-abaa496b55ad">
<img width="283" alt="Screenshot 2024-04-16 at 15 03 11"
src="8569e289-4a21-4df2-9f5c-7d75205a03f2">
<img width="281" alt="Screenshot 2024-04-16 at 15 03 13"
src="f17b6023-0468-46c7-b5b0-c7c62af5c353">
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/178446
This PR adds the following AI Ops features to the Burn rate alert detail
page:
- Log Rate Analysis
- ML features / AI Assistant
bf00c636-10ab-4cd8-859b-4a8f0471bf15
## Implementation details
Log rate analysis is useful when there is a significant dip/spike in
number of logs. It finds significant differences between the logs in
baseline (before dip/spike) and in deviation (after dip/spike). For the
Log Rate Analysis to work on the Alert details of an SLO burn rate rule
we had to specify a few params, here are the most important ones:
- dataView
- esSearchQuery
- timeRange -> I am looking into this and need a few clarifications
regarding lookBackPeriod
- initialAnalysisStart
For the esQuery here's what we used
```
const finalQuery = {
bool: {
filter: [customTotalFilter, customFilters, customGroupByFilters],
must_not: customGoodFilter,
},
};
```
We add the `total query`, any `optional filters` and any `group by
filters` to the `filter` clause of the ES query, where as we add the
good filter in the `must_not` clause.
## 🔬 How to test
- Add the following lines to your `config/kibana.dev.yaml`:
- `server.basePath: '/kibana'`
- `server.publicBaseUrl: 'http://localhost:5601/kibana'`
- Start with the following command: `node x-pack/scripts/data_forge.js
--events-per-cycle 50 --lookback now-1d --dataset fake_stack
--install-kibana-assets --kibana-url http://localhost:5601/kibana
--event-template good` to start with some good events
- Wait til the log message says `info Waiting 60000ms`
- Create one SLO:
- "Admin Console Availability" using the "Custom Query" SLI with the
`Admin Console` DataView, set the "Good query" to
`http.response.status_code < 500` and the set the "Total query" to
`http.response.status_code: *` using a rolling `7d` time window
- You should have 1 burn rate rule created by default
- Stop the first `data_forge.js` command
- Start `node x-pack/scripts/data_forge.js --events-per-cycle 50
--lookback now --dataset fake_stack --install-kibana-assets --kibana-url
http://localhost:5601/kibana --event-template bad`
- Go to alert details page of the burn rate rule to verify that the Log
Rate Analysis tool detected a spike
Note: you can skip first step, but in this case you should start kibana
with this command instead `yarn start --no-base-path`. This way you
don't have to pass `--kibana-url` when running the `data_forge` command
I included some snapshot tests with different SLOs that could be used
for verifying the generation of the ES query. Here are a few examples:
- Good query (KQL), no filter, no total query
- Good query (filter), no optional filter, no total query
- Good query (kql) optional filter (KQL), no total query
- Good query (kql), optional filter (filter), no total query
- Good query (kql), optional filter (kql), total query (kql)
- Good query (filter), optional filter(filter), total query(kql)
💀 I will need extra help testing this feature, to make sure that
provided params to the `LogRateAnalysisContent` component. Especially
`esSearchQuery` and `timeRange` (timeRange still WIP, but I want some
early feedback on the lookbackDuration and the intervalFactor)
## ✔️ Acceptance criteria
- Render the Log Rate Analysis tool only to platinum users and above
- Render AI assistant contextual insights only to platinum users and
above
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Cauê Marcondes <55978943+cauemarcondes@users.noreply.github.com>
Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
## Summary
* [Prettify asset criticality alert
column](38fa0e2b50)
* [Memoize badge to prevent unnecessary
rerender](22599d2f25)
Before | After
:-------------------------:|:-------------------------:
<img width="200"
src="78fb2130-9ce2-4c67-85d7-5a5d78e39773"/>
| <img width="200"
src="91cfcdab-4fa0-4022-8fc4-ac218a75a377"/>
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
This PR enables Session View to support for auditbeat data stream and
adds the `event.action: executed`.
- In the `Security>Explore>Hosts>Events` and
`Security>Explore>Hosts>Session` tab, shows process events from
Auditbeats and should show `event.action: [ fork, executed, end ]`.
- Auditbeat session events now has alerts capabiltiy
<img width="1722" alt="image"
src="3824347f-fc34-4b24-b949-284e530ada19">
1fb7dcc7-184f-4e02-a431-892e62f239f6
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Switching to use retrievers instead of query. This also makes use of RRF
support in retrievers for sparse, dense and bm25 combinations.
# ELSER only
# ELSER + bm25 hybrid
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/172698
The default setting for the header row height is custom, 3 lines (so if
user doesn't modify the header row height). It doesn't visually affect
datatables with short labels.
<img width="765" alt="Screenshot 2024-04-10 at 18 39 51"
src="bdec216b-36e4-4ca4-8135-85008b7f311d">
<img width="739" alt="Screenshot 2024-04-10 at 18 39 25"
src="e5e6c4d5-f9c6-4270-85ba-f0ada2fbea69">
To code reviewers - this PR in itself is quite small but I rewrote all
the datatable tests to testing library and made some code style
improvements.
## Summary
In this PR:
- Address @adcoelho comments regarding documentation.
- Fix @js-jankisalvi bug about unsupported consumers
(https://github.com/elastic/kibana/pull/168369#pullrequestreview-1988654505).
- Address @shanisagiv1 feedback regarding the title and the description.
Specifically:
- The title changed to "<rule_name> - Grouping by <grouping_by_value>
(Auto-created)".
- The description changed to "This case was created by the Case action
in <rule_name_link>. The assigned alerts are grouped by
<grouping_by_key>:<grouping_by_value>".
- Add the grouping key as a tag.
<img width="2289" alt="Screenshot 2024-04-13 at 4 41 36 PM"
src="63e17947-5f39-4437-820b-7c69f42bfbe3">
The issue about the "Unknown" user will be fixed in another PR.
About @adcoelho bug:
c46aa7c4-9d1a-475b-9d07-6bdff3ef00c8
I think it is fine to leave it as it is because a) the value will not be
saved even if they are added b) an error is being shown c) the only way
to do it properly is to validate while the user is typing which is going
to lead to bad UX. If you feel otherwise let me know.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
Related to https://github.com/elastic/kibana/pull/179891
* Display error message in case user doesn't have write access to asset
criticality index
* Display error message if the user navigates to the page when the user
advanced (UI) setting is disabled
* Remove links from menus and global search if advanced (UI) setting is
disabled
Recoding of what happens when you don't refresh the page:
2b67403c-d58e-4d92-b12d-9f9de4c9a213
### Expected behaviour:
* After the setting is enabled, the user needs to refresh the browser to
find the page
* If users disable the setting on a different browser/tab and navigate
to the page without refreshing, they will see an error message on the
page.
* If users disable the flag while the page is already rendered and try
to upload the file, it will display an error on the last step.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes https://github.com/elastic/kibana/issues/180886
## Metric threshold alert
Feature flag for Metric threshold alert details page:
`xpack.observability.unsafe.alertDetails.metrics.enabled`
### When feature flag is OFF
<img width="1470" alt="Screenshot 2024-04-16 at 12 11 01"
src="25c9a97e-e081-4761-b448-e97353ec5573">
### When feature flag is ON
<img width="1476" alt="Screenshot 2024-04-16 at 12 09 32"
src="be295276-f919-4fa7-b411-94c3d1e0c3b5">
## Inventory threshold alert
There is no separate feature flag for Inventory threshold alert details
page and we don't have `AlertDetailsAppSection` defined for this alert,
so I have removed "Technical Preview" badge from it.
<img width="1485" alt="Screenshot 2024-04-16 at 12 10 45"
src="ee23b25f-7fbc-4f4a-871a-18e5ba424afd">
## Summary
It fixes#179039
### Logs:
<img width="1210" alt="Screenshot 2024-04-11 at 18 59 27"
src="1795895a-f759-4bd4-a803-abec32dfed03">
### Infra:
<img width="1215" alt="Screenshot 2024-04-11 at 19 00 05"
src="74fede30-3ca4-4971-8e31-d9534bb66204">
### Release notes:
Warn the user when changing the index pattern while at least one rule
relies on the current one.
---------
Co-authored-by: Bena Kansara <bena.kansara@elastic.co>
We were dropping ES|QL docs about CCQ on the floor, which led to the
Assistant assuming CCQ is not supported. This change includes docs about
CCQ, ES|QL in Kibana, and the query API. Additionally, the function
response now includes some data that is useful for debugging.
Closes https://github.com/elastic/kibana/issues/180699
When we add the `anomalyDetectorTypes` we introduced a breaking change,
not making this new parameter optional.
This PR is meant to make this parameter optional for backwards
compatibility
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Carlos Crespo <carloshenrique.leonelcrespo@elastic.co>
Co-authored-by: Nathan L Smith <nathan.smith@elastic.co>
## 🍒 Summary
This PR adds a rule dependency feature to the SLO Burn Rate rule to
enable rule suppression when one of the dependencies meets the
suppression criteria.
### 📟 Use case
When you add a rule dependency to your SLO Burn Rate rule, you will also
choose which action groups you want to suppress on. For example, if you
have rule `A` which depends on rule `B` and you want to suppress the
actions of rule `A` when rule `B` is triggering `Critical` or `High`,
you'd add rule `B` and pick the action groups `Critical` and `High`.
When rule `B` is triggering either of those action groups, ALL of the
actions for rule `A` will be suppressed.
When an action is suppressed, we will trigger a `Suppressed` action
group an set the context variable `{context.suppressedAction}` to the
action that would have been trigger if they rule wasn't suppressed. This
will allow users to create an "action" for `Suppressed` alerts so they
can still create notification without waking up the team for a
`Critical` or `High` severity alert.
If you have 2 rules that use a group by, then the suppression will
happen on the intersection of the `slo.instanceId`. For example, imagine
we have a Nginx Proxy in front of an Node.js web service and we've
created an availability SLO based on `status_code < 500` for both,
grouped-by `url.domain`. When the Node.js app responds with a `500`, the
Nginx Proxy's SLO will start to degrade because of the Node.js service.
The admins for the Nginx Proxy would like to only receive alerts if the
Node.js web services is "healthy" so they've listed the Node.js burn
rate rule as a dependency to suppress on `Critical` or `High` burn
rates.
When one of the domains, `you-got.mail`, starts to throw 500's and the
burn rate becomes `High`, the rule will suppress the alert for the
`you-got.mail` Nginx Proxy instance. If one of the other domains,
`box.mail`, for Nginx started throwing `502` because of a
mis-configuration, the alert would trigger normally because the
`box.mail` instance of the rule dependency for the Node.js web service
is still healthy (or not triggering `Critical` or `High`).
The suppression between group-by SLOs and non-group-by SLOs works like
this:
- SLO with a group-by depends on a non-grouped-by SLO, all the instances
of the group by will be suppressed.
- SLO without a group-by depends on an SLO with a group-by, the
non-grouped SLO will be suppressed if ANY of the instances of the
group-by are triggering the "suppress on" action groups.
### 💻 Screenshots
Adding a rule dependency for MongoDB to a Node.js web app
<img width="764" alt="image"
src="da2fd411-2a8e-4433-a505-2c4111e115be">
In this scenario, Nginx Proxies to Admin Console which reads data from
MongoDB. The connection between MongoDB and the Admin Console has a
network outage which causes the MongoDB rule to trigger a `Critical`
action group and suppresses the `Critical` action for the Admin Console.
The Admin Console also goes `Critical` which then suppresses the rule
for the Nginx Proxy.
<img width="1784" alt="image"
src="2db75993-8912-4769-83f8-240de811a92f">
### ⚙️ How it works
- Execute the primary rule and evaluate if should trigger any actions
- If the primary rule is triggering, execute each of the dependencies
(in the same process using the same function) and suppress when:
- For group-by SLOs that depend on another SLO with a group by, we
suppress the intersection between the instanceIds.
- For group-by SLOs that depend on a non-group-by SLO, we suppress all
the instanceIds.
- For non-group-by SLO that depends on a group-by SLO, we suppress if
ANY instanceId matches. (not recommended)
### 🔬 How to test
- Add the following lines to your `config/kibana.dev.yaml`:
- `server.basePath: '/kibana'`
- `server.publicBaseUrl: 'http://localhost:5601/kibana'`
- Start with the following command: `node x-pack/scripts/data_forge.js
--events-per-cycle 50 --lookback now-1d --dataset fake_stack
--install-kibana-assets --kibana-url http://localhost:5601/kibana
--event-template good`
- Wait till the log message says `info Waiting 60000ms`
- Create 2 SLOs:
- "Admin Console Availability" using the "Custom Query" SLI with the
`Admin Console` DataView, set the "Good query" to
`http.response.status_code < 500` and the set the "Total query" to
`http.response.status_code: *` using a rolling `7d` time window
- "MongoDB Availability" using the "Custom Query" SLI with the
`Heartbeat` DataView, set the "Good query" to `event.outcome: "success"`
and the set the "Total query" to `event.outcome: *` using a rolling `7d`
time window
- You should have 2 burn rate rules that were created by default
- Open the "Admin Console Availability Burn Rate rule" and add the
"MongoDB Availability Burn Rate rule" as the dependency with `Critical`
and `High` action groups to "Suppress on".
- Save the rule
- Stop the first `data_forge.js` command
- Start `node x-pack/scripts/data_forge.js --events-per-cycle 50
--lookback now --dataset fake_stack --install-kibana-assets --kibana-url
http://localhost:5601/kibana --event-template bad`
Once the Burn Rate rules go `Critical`, you should see the "MongoDB
Availability Burn Rate rule" reason message should start with
`CRITICAL:...` and the "Admin Console Availability Burn Rate rule"
reason message should start with `SUPPRESSED - CRITICAL: ...`
Fixes#173653
---------
Co-authored-by: Panagiota Mitsopoulou <giota85@gmail.com>
Co-authored-by: Dominique Clarke <doclarke71@gmail.com>
Co-authored-by: Kevin Delemme <kdelemme@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co>
We've identified all Cypress tests using `fleet-server` in `8.14.0`
version have [started to fail
recently](https://buildkite.com/elastic/kibana-on-merge/builds/43732#_)
therefore blocking other PRs from merging.
Together with @szwarckonrad we came up with this PR as a quick fix that
hardcodes our tests to use FS 8.13.0 in order to unblock other PRs.
@elastic/fleet I believe this should be investigated on your side too as
to understand what changed in 8.14.0 that FS is unable to register
within ES. I am wondering if this is just our tests not being aligned
with new changes, or is it a **real bug in production?**
cc: @dasansol92
<img width="1017" alt="Zrzut ekranu 2024-04-16 o 11 31 43"
src="c0d27c30-83cd-4c6a-905d-6584e89c86a4">
```
│ERROR Error: Timed out waiting for host [dev-fleet-server.8212.1n5b] to show up in Fleet. Waited 120 seconds
2024-04-16 05:56:03 │ at waitForHostToEnroll (fleet_services.ts:221:7)
2024-04-16 05:56:03 │ at fleet_server_services.ts:330:11
2024-04-16 05:56:03 │ at tooling_log.ts:84:18
2024-04-16 05:56:03 │ at startFleetServerWithDocker (fleet_server_services.ts:395:35)
2024-04-16 05:56:03 │ at fleet_server_services.ts:142:32
2024-04-16 05:56:03 │ at tooling_log.ts:84:18
2024-04-16 05:56:03 │ at parallel.ts:344:31
2024-04-16 05:56:03 │ at withProcRunner (with_proc_runner.ts:29:5)
2024-04-16 05:56:03 │ at concurrency (parallel.ts:217:13)
2024-04-16 05:56:03 │ at /opt/local-ssd/buildkite/builds/kb-n2-4-virt-c91969ffbad5cc2b/elastic/kibana-on-merge/kibana/node_modules/p-map/index.js:57:22 {
2024-04-16 05:56:03 │ agentId: '732018e1-53f1-4d41-84a8-bf92127abb57',
2024-04-16 05:56:03 │ hostname: 'dev-fleet-server.8212.1n5b'
2024-04-16 05:56:03 │ }
2024-04-16 05:56:03 info [kibana] exited with null after 189.0 seconds
2024-04-16 05:56:03 ERROR UNHANDLED ERROR
```
## [Security Solution] [AI Insights] AI Insights
### Summary
This PR introduces _AI Insights_ to the Security Solution:
_Above: AI Insights in the Security Solution_
AI Insights identify active attacks in the environment, without the time
(or prior experience) required to manually investigate individual alerts
in Elastic Security, identify if they are related, and document the
identified attack progression.
While users can ask the Assistant to find these progressions today, AI
Insights is a dedicated UI to identify these progressions and action
them accordingly. This feature adds a new page, `AI Insights`, to the
Security Solution's global navigation.
AI Insights are generated from Large Language Models (LLMs) to identify
attack progressions in alert data, and to correlate and identify related
entities and events. When possible, attack progressions are attributed
to threat actors.
### Details
Users may generate insights from a varetiy of LLMs, configured via
[Connectors](https://www.elastic.co/guide/en/kibana/master/action-types.html):
_Above: LLM selection via the connectors popup menu_
Clicking on the title of an insight toggles the insight between the
collapsed and expanded state:
_Above: Collapsing / expanding an insight (animated gif)_
The first three insights displayed on the AI Insights page are expanded
by default. Any additional insights that appear after the first three
must be expanded manually.
Insights provide a summary of the entities impacted by an attack.
Clicking on an entity, i.e. a hostname or username, displays the entity
flyout with the entity's risk summary:
_Above: Clicking on a host in the summary of the insight reveals the
host risk summary (animated gif)_
Hover over fields in the insight's summary or details to reveal pivot
actions for investigations:
_Above: Hovering over fields in the details of an insight reveals pivot
actions (animated gif)_
Insights are generated from alerts provided as context to the selected
LLM. The alert data provided to the LLM is anonymized automatically.
Anonymization is
[configured](https://www.elastic.co/guide/en/security/current/security-assistant.html#ai-assistant-anonymization)
via the same anonymization settings as the Assistant. Users may override
the defaults to allow or deny specific alert fields, and to toggle
anonymization on or off for specific fields.
Click the Anonymization toggle to show or hide the actual values sent to
the LLM:
_Above: Toggling anonymization to reveal the actual values sent to the
LLM (animated gif)_
### Empty prompt
At the start of a session, or when a user selects a connector that
doesn't (yet) have any insights, an [empty
prompt](https://eui.elastic.co/#/display/empty-prompt) is displayed.
The animated counter in the empty prompt counts up until it displays the
maximum number of alerts that will be sent to the LLM:
_Above: An animated counter displays the maximum number of alerts that
will be sent to the LLM (animiated gif)_
The _Settings_ section of this PR details how users configure the number
of alerts sent to the LLM. The animated counter in the empty prompt
immediately re-animates to the newly-selected number when the setting is
updated.
### Take action workflows
The _Take action_ popover displays the following actions:
- `Add to new case`
- `Add to existing case`
- `View in AI Assistant`
_Above: The Take action popover_
#### Add to new case
Clicking the `Add to new` case action displays the `Create case` flyout.
_Above: The `Add to new case` workflow_
An `Alerts were added to <case name>` toast is displayed when the case
is created:
_Above: Case creation toast_
A markdown representation of the insight is added to the case:
_Above: A markdown representation of an insight in a case_
The alerts correlated to generate the insight are attached to the case:
_Above: Insight alerts attached to a case_
#### Add to existing case
Clicking the `Add to existing case` action displays the `Select case`
popover.
_Above: The `Select case` popover_
When users select an existing case, a markdown representation of the
insight, and the alerts correlated to generate the insight are attached
to the case, as described above in the _Add to new case_ section.
#### View in AI Assistant
The `View in AI Assistant` action in the `Take action` popover, and two
additional `View in AI Assistant` affordances that appear in each
insight have the same behavior:
Clicking `View in AI Assistant` opens the assistant and adds the insight
as context to the current conversation.
_Above: An insight added as context to the current conversation_
Clicking on the insight in the assistant expands it to reveal a preview
of the insight.
_Above: An expanded insight preview in the assistant_
The expanded insight preview reveals the number of anonymzied fields
from the insight that were made available to the conversation. This
feature ensures insights are added to a conversation with the anonymized
field values.
An insight viewed in the AI assistant doesn't become part of the
conversation until the user submits it by asking a question, e.g. `How
do I remediate this?`.
Insights provided as context to a conversation are formatted as markdown
when sent to the LLM:
_Above: Insights provided as context to a conversation are formatted as
markdown_
Users may toggle anonymization in the conversation to reveal the
original field values.
_Above: Revealing the original field values of an insight added as
markdown to a conversation (animated gif)_
#### Alerts tab
The _Alerts_ tab displays the alerts correlated to generate the insight.
_Above: The alerts correlated to generate the insight in the Alerts tab_
The `View details`, `Investigate in timeline`, and overflow row-level
alert actions displayed in the Alerts tab are the same actions available
on the Cases's page's Alerts tab:
_Above: Row-level actions are the same as the Cases pages Alert's tab_
#### Investigate in Timeline
Click an insight's `Investigate in Timeline` button to begin an
investigation of an insights's alerts in Timeline. Alert IDs are queried
via the `Alert Ids` filter:
_Above: Clicking Investigte in Timeline (animated gif)_
The alerts from the insight are explained via row renderers in Timeline:
_Above: Row rendered insight alerts in Timeline_
### Attack Chain
When alerts are indicative of attack
[tactics](https://attack.mitre.org/tactics/enterprise/), those tactics
are displayed in the insights's _Attack Chain_ section:
_Above: An insight with tactics in the Attach chain_
The Attack Chain section will be hidden if an insight is not indicative
of specific tactics.
### Mini attack chain
Every insight includes a mini attack chain that visually summarizes the
tactics in an insight. Hovering over the mini attack chain reveals a
tooltip with the details:
_Above: The mini attack chain tooltip_
### Storage
The latest insights generated for each connector are cached in the
browser's session storage in the following key:
```
elasticAssistantDefault.aiInsights.cachedInsights
```
Caching insights in session storage makes it possible to immediately
display the latest when users return to to the AI insights page from
other pages in the security solution (e.g. Cases).
_Above: Cached insights from sesion storage are immediately displayed
when users navigate back to AI Insights (animated gif)_
While waiting for a connector to generate results, users may view the
cached results from other connectors.
Cached insights are immediately available, even after a full page
refresh, as long as the browser session is still active.
### `Approximate time remaining` / `Above average time` counters
Some LLMs may take seconds, or even minutes to generate insights. To
help users anticipate the time it might take to generate an insight, the
AI insights feature displays a `Approximate time remaining: mm:ss`
countdown timer that counts down to zero from the average time it takes
to generate an insight for the selected LLM:
_Above: The `Approximate time remaining: mm:ss` countdown counter
(animated gif)_
If the LLM doesn't generate insights before the counter reaches zero,
the text will change from `Approximate time remaining: mm:ss` to `Above
average time: mm:ss`, and start counting up from `00:00` until the
insights are generated:
_Above: The `Above average time: mm:ss` counter (animated gif)_
The first time insights are generated for a model, the `Approximate time
remaining: mm:ss` counter is not displayed.
Average time is calculated over the last 5 generations on the selected
connector. This is illustrated by clicking on the (?) information icon
next to the timer. The popover displays the average time, and the time
in seconds for the last 5 runs:
_Above: Clicking on the (?) information icon displays the average time,
and the duration / datetimes for the last 5 generations_
The time and duration of the last 5 generations (for each connector) are
persisted in the browser's local storage in the following key:
```
elasticAssistantDefault.aiInsights.generationIntervals
```
### Errors
When insight generation fails, an error toaster is displayed to explain
the failure:
_Above: An error toaster explains why insights generation failed_
### Feature flag
The `assistantAlertsInsights` feature flag must be enabled to view the
`AI Insights` link in the Security Solution's global navigation.
Add the `assistantAlertsInsights` feature flag to the
`xpack.securitySolution.enableExperimental` setting in
`config/kibana.yml` (or `config/kibana.dev.yml` in local development
environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantAlertsInsights']
```
### Settings
The number of alerts sent as context to the LLM is configured by
`Knowledge Base` > `Alerts` slider in the screenshot below:
- The slider has a range of `10` - `100` alerts (default: `20`)
Up to `n` alerts (as determined by the slider) that meet the following
criteria will be returned:
- The `kibana.alert.workflow_status` must be `open`
- The alert must have been generated in the last `24 hours`
- The alert must NOT be a `kibana.alert.building_block_type` alert
- The `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize
the riskiest alerts
### License
An Enterprise license is required to use AI Insights.
The following AI Insights view is displayed for users who don't have an
Enterprise license:
## How it works
- Users navigate to the AI insights page:
`x-pack/plugins/security_solution/public/ai_insights/pages/index.tsx`
- When users click the `Generate` button(s) on the AI Insights page,
insights are fetched via the `useInsights` hook in
`x-pack/plugins/security_solution/public/ai_insights/use_insights/index.tsx`.
- The `fetchInsights` function makes an http `POST` request is made to
the `/internal/elastic_assistant/insights/alerts` route. include the
following new (optional) parameters:
- `actionTypeId`, determines tempature and other connector-specific
request parameters
- `alertsIndexPattern`, the alerts index for the current Kibana Space,
e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings,
e.g. `["@timestamp", "cloud.availability_zone", "file.name",
"user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the
`Anonymization` settings, e.g. `["cloud.availability_zone", "host.name",
"user.name", ...]`
- `connectorId`, id of the connector to generate the insights
- `replacements`, an optional `Record<string, string>` collection of
replacements that always empty in the current implementation. When
non-empty, this collection enables new insights to be generated using
existing replacements.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the maximum number of alerts to generate insights from. This
numeric value is set by the slider in the user's `Knowledge Base >
Alerts` setting, e.g. `20`
- The `postAlertsInsightsRoute` function in
`x-pack/plugins/elastic_assistant/server/routes/insights/alerts/post_alerts_insights.ts`
handles the request.
- The inputs and outputs to this route are defined by the
[OpenAPI](https://spec.openapis.org/oas/v3.1.0) schema in
`x-pack/packages/kbn-elastic-assistant-common/impl/schemas/insights/alerts/post_alerts_insights_route.schema.yaml`.
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
- The `postAlertsInsightsRoute` route handler function in
`x-pack/plugins/elastic_assistant/server/routes/insights/alerts/post_alerts_insights.ts`
invokes the `insights-tool`, defined in
`x-pack/plugins/security_solution/server/assistant/tools/insights/insights_tool.ts`.
The `insights-tool` is registered by the Security Solution. Note: The
`insights-tool` is only used for generating insights. It is not used to
generate new insights from the context of an assistant conversation, but
that feature could be enabled in a future release.
- The `insights-tool` uses a LangChain `OutputFixingParser` to create a
[prompt
sandwich](https://www.elastic.co/blog/crafting-prompt-sandwiches-generative-ai)
with the following parts:
```
_________________________________________________
/ \
| Insight JSON formatting instructions | (1)
\ _______________________________________________/
+------------------------------------------------+
| Insights prompt | (2)
+------------------------------------------------+
/ \
| Anonymized Alerts | (3)
\_______________________________________________/
```
- The `Insight JSON formatting instructions` in section `(1)` of the prompt sandwich are defined in the `getOutputParser()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_output_parser.ts`. This function creates a LangChain `StructuredOutputParser` from a Zod schema. This parser validates responses from the LLM to ensure they are formatted as JSON representing an insight.
- The `Insights prompt` in section `(2)` of the prompt sandwich is defined in the `getInsightsPrompt()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_insights_prompt.ts`. This part of the prompt sandwich includes instructions for correlating insights, and additional instructions to the LLM for formatting JSON.
- The `Anonymized Alerts` in section `(3)` of the prompt sandwich are returned by the `getAnonymizedAlerts()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_anonymized_alerts.ts`. The allow lists configured by the user determine which alert fields will be included and anonymized.
- The `postAlertsInsightsRoute` route handler returns the insights generated by the `insights-tool` to the client (browser).
- Insights are rendered in the browser via the `Insight` component in `x-pack/plugins/security_solution/public/ai_insights/insight/index.tsx`
- The `AiInsights` tab in `x-pack/plugins/security_solution/public/ai_insights/insight/tabs/ai_insights/index.tsx` includes the _Summary_ and _Details_ section of the Insight.
- The `InsightMarkdownFormatter` in `x-pack/plugins/security_solution/public/ai_insights/insight_markdown_formatter/index.tsx` renders hover actions on entities (like hostnames and usernames) and other fields in the insight.
- The `Insight` component makes use of the `useAssistantOverlay` hook in `x-pack/packages/kbn-elastic-assistant/impl/assistant/use_assistant_overlay/index.tsx` to register the insight as context with the assistant. This registration process makes it possible to view insights in the assistant, and ask questions like "How do I remediate this?". In this PR, the `useAssistantOverlay` hook was enhanced to accept anonymizaton replacements. This enables an assistant conversation to (re)use replacements originally generated for an insight.
## Summary
Fixes#178181.
This updates the transform list to fetch stats only with the option
`basic=true`. Only for certain tabs in the expanded rows we'll still
fetch the full stats.
If the full stats for a transform fail to load, we show a callout in the
affected tabs in the expanded row like this:
<img width="1067" alt="image"
src="e1d0eeff-7c97-4418-93eb-33832b070a17">
To reproduce this for testing, throw an error in the file
`x-pack/plugins/transform/server/routes/api/transforms_stats_single/route_handler.ts`
just like this:
```ts
export const routeHandler: RequestHandler<
TransformIdParamSchema,
GetTransformStatsQuerySchema,
undefined,
TransformRequestHandlerContext
> = async (ctx, req, res) => {
const { transformId } = req.params;
try {
throw new Error('just testing error handling...');
const basic = req.query.basic ?? false;
const esClient = (await ctx.core).elasticsearch.client;
const body = await esClient.asCurrentUser.transform.getTransformStats(
{
...
```
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
