Resolves https://github.com/elastic/kibana/issues/173009
## Summary
This PR:
- Changes the field name from `notification_delay` to `alert_delay`
- Updates the alerts client and rule registry to index new alert docs on
a delay
- Updates the framework code to delay the creation of an alert
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
- Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to
create a rule with the `alertDelay`
```
POST kbn:/api/alerting/rule
{
"params": {
"searchType": "esQuery",
"timeWindowSize": 5,
"timeWindowUnit": "m",
"threshold": [
-1
],
"thresholdComparator": ">",
"size": 100,
"esQuery": """{
"query":{
"match_all" : {}
}
}""",
"aggType": "count",
"groupBy": "all",
"termSize": 5,
"excludeHitsFromPreviousRun": false,
"sourceFields": [],
"index": [
".kibana-event-log*"
],
"timeField": "@timestamp"
},
"consumer": "stackAlerts",
"schedule": {
"interval": "1m"
},
"tags": [],
"name": "test",
"rule_type_id": ".es-query",
"actions": [
{
"group": "query matched",
"id": "${ACTION_ID}",
"params": {
"level": "info",
"message": """Elasticsearch query rule '{{rule.name}}' is active:
- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
},
"frequency": {
"notify_when": "onActionGroupChange",
"throttle": null,
"summary": false
}
}
],
"alert_delay": {
"active": 3
}
}
```
- Verify that the alert will not be created until it has matched the
delay threshold.
- Verify that the delay does not affect recovered alerts
## Summary
follow-up of https://github.com/elastic/kibana/pull/173185
This PR enables the persistence layer implemented in the previous PR,
applying the following changes:
- Update the mapping to store unitary index results instead of storing
the whole pattern with the results in each document.
- Change the query to get the stored results by aggregating documents by
indexName. The authorized indexNames derived from the `pattern`
parameter are retrieved using the `indices.get` request.
- A bug involving a race condition with the initialization and the
retrieval of stored results, resulting in an unintended reset of the
results in the UI, has been fixed.
0598606b-c5f4-42b3-901c-f86a3cac65e4
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
With this pull request, we implement Protection Updates gating on the
Security Essentials tier. The changes include:
1. Addition of an upselling component on the Protection Updates tab.
2. Extension of the package policy create/update API callback to verify
the protection updates app feature before committing changes to
global_manifest_version.
3. Extension of the turn_off_policy_protections plugin callback to
inspect the protection updates app feature on server start. If no app
feature is present, it will roll down global_manifest_version to the
default of 'latest'.
> [!IMPORTANT]
> This PR is a reintroduction of
https://github.com/elastic/kibana/pull/175338, which was
[reverted](https://github.com/elastic/kibana/pull/175338#issuecomment-1919803575)
due to sporadic jest failures within the `security_solution` plugin.
Root cause was identified and detailed in
https://github.com/elastic/kibana/pull/176005#issuecomment-1919959743.
## Summary
In https://github.com/elastic/kibana/pull/174317 we added support for
OpenAPI codegen, this PR builds on that functionality by migrating the
`Post Evaluate` route `/internal/elastic_assistant/evaluate` to be
backed by an OAS, and adds a basic `Get Evaluate` route for rounding out
the enhancements outlined in
https://github.com/elastic/security-team/issues/8167 (to be in a
subsequent PR).
Changes include:
* Migration of `Post Evaluate` route to OAS
* Migration of `Post Evaluate` route to use versioned router
* Extracted `evaluate` API calls from
`x-pack/packages/kbn-elastic-assistant/impl/assistant/api/api.tsx` to
`x-pack/packages/kbn-elastic-assistant/impl/assistant/api/evaluate/evaluate.tsx`
* Co-located relevant `use_perform_evaluation` hook
* Adds `Get Evaluate` route, and corresponding `use_evaluation_data`
hook. Currently only returns `agentExecutors` to be selected for
evaluation.
* API versioning constants added to
`x-pack/packages/kbn-elastic-assistant-common/impl/schemas/index.ts`
* Adds new `buildRouteValidationWithZod` function to
`x-pack/plugins/elastic_assistant/server/schemas/common.ts` for
validating routes against OAS generated zod schemas.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR is part of an effort to limit EDR Workflow features to the
Endpoint Complete tier on serverless and focuses on UI part of gating
Agent Tamper Protection.
Related PRs:
- [Agent Tamper Protection
API](https://github.com/elastic/kibana/pull/174400)
- [Protection updates](https://github.com/elastic/kibana/pull/175129)
Currently, the Agent tamper protection switch is in Fleet's agent policy
settings. Right now (for ESS and Serverless), we only control access to
this field with a license check (platinum). This PR adds an extra check
for AppFeature, which is included in the Complete tier PLI config. We
decided to stick with the existing Fleet privileges for this component,
and no extra changes are needed RBAC wise (confirmed with
@roxana-gheorghe).
Changes:
1. Added `endpointAgentTamperProtection` appFeature and linked it to the
endpoint:complete tier.
2. Made an upselling component and registered it with the Upselling
Service.
3. Passed the upselling component to Fleet using UIExtension.
4. Added Cypress end-to-end coverage for Essentials showing the
upselling component and Complete showing the form component.
## Summary
See https://github.com/elastic/eui/pull/7321
EUI will shortly be removing this deprecated ref `setSelection` method
in favor of the new controlled `selection.selected` prop. This PR
converts basic usages of controlled selection, which should not suffer
any UI/UX regressions.
**Please help us QA your affected tables to confirm that your table
selection still works as before!**
## Summary
This PR adds support for ES|QL queries in Data visualizer.
<img width="1695" alt="Screenshot 2024-01-26 at 17 07 59"
src="8a54b859-60d6-4c47-b3dd-e5f3ed43b6b0">
<img width="1695" alt="Screenshot 2024-01-26 at 17 12 39"
src="32fd08e4-7f3b-43e6-81a7-7ec4e777bac0">
a3f540e9-461d-4ebc-bd69-de4ffa2bc554
### Changes:
- Add a new card from the Data visualizer main page
- Add a link from the ML navigation
<img width="1717" alt="Screenshot 2024-01-08 at 18 03 50"
src="832f7890-4ce6-44c1-ab87-cde01f4bf1c0">
- Added a new button to Use ES|QL
<img width="1714" alt="Screenshot 2024-01-09 at 11 23 09"
src="a38a9360-6691-4f3b-a824-8481ab543250">
- Support for **keyword**, **text**, **numeric**, **boolean**, **date**,
and **ip** fields
<img width="1714" alt="Screenshot 2024-01-09 at 11 24 38"
src="b122ee5c-1500-4e2b-9434-e64b0b6ea3be">
<img width="1441" alt="Screenshot 2024-01-09 at 11 25 25"
src="eb35ee78-8a34-467e-84da-2026b01fcda1">
<img width="969" alt="Screenshot 2024-01-09 at 11 44 02"
src="d0f9947d-2b2c-4c14-89ba-9fc5d0a2bf64">
<img width="981" alt="Screenshot 2024-01-10 at 12 01 42"
src="aa5a8d44-7447-41fc-a544-d1b626bf8bce">
- Default to user's fieldFormats for fields that are dynamic generated
by ES|QL, else use Data view's format
- Default to Data view's setting (e.g. type `bytes` in this case for
field `bytes_normal_counter`)
<img width="1037" alt="Screenshot 2024-01-10 at 12 10 38"
src="9fb7e31c-f397-4209-a463-e1a43fe27ffd">
- Default to user's fieldFormats formatting for dynamically generated
fields (e.g. type `number` in this case for field `avg_price`)
<img width="1283" alt="Screenshot 2024-01-10 at 12 01 03"
src="acc25358-50bb-4237-9476-86067ef0badf">
- Add a new UI control to allow users to limit analysis to 5,000 -
10,000 - 100,000 - 1,000,000, rows. This speeds up fetching of the stats
for big data sets and avoid potential circuit breaking exceptions.
- Break overall stats request into smaller parallel requests (which
prevent time out or payload too big due by too many fields), at 10
requests at a time
- Break field stats for individual fields into more efficient batches
(which prevent time out or payload too big due by too many fields), at
10 requests at at ime
- Improve error handling by propagating up the error AND the ES|QL
request in both the UI and the developer's console (for better
debugging)
- Improve error handling in field stats rows: If one field, or a group
of fields, say 'keyword' fields fail to fetch for some reasons, it will
show error for that field but not affect all other fields.
<img width="1690" alt="Screenshot 2024-01-26 at 16 04 28"
src="6e240e12-76b4-42d6-b3be-c05342d76df9">
- Add deep linking in the top search bar
<img width="1185" alt="Screenshot 2024-01-26 at 16 56 49"
src="4f24df68-edc5-41c5-b2ed-d6150ba1e20b">
- More robust support for keyword fields with geo data
<img width="1438" alt="Screenshot 2024-01-26 at 16 55 01"
src="3b97925b-ca28-4952-8082-8d3242e3cb3f">
### Todos:
- [x] Add earliest/latest for date time fields -> Current blocker:
escape special characters in esql variable names
- [x] Fix formatting of numbers for dynamic query, where we don't know
the formatting based on the data view
- [x] Fix date time 'Update' not updating until Refresh is clicked
- [x] Better optimization to not fetch distribution & expanded row
content for pages that are not visible
### Good to have:
- [ ] Investigate bringing back the +/- filter buttons (either by
modifying the ES|QL query directly or by adding separate DSL filters?)
------------
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
In https://github.com/elastic/kibana/pull/174317 we added support for
OpenAPI codegen, this PR builds on that functionality by migrating the
`Post Evaluate` route `/internal/elastic_assistant/evaluate` to be
backed by an OAS, and adds a basic `Get Evaluate` route for rounding out
the enhancements outlined in
https://github.com/elastic/security-team/issues/8167 (to be in a
subsequent PR).
Changes include:
* Migration of `Post Evaluate` route to OAS
* Migration of `Post Evaluate` route to use versioned router
* Extracted `evaluate` API calls from
`x-pack/packages/kbn-elastic-assistant/impl/assistant/api/api.tsx` to
`x-pack/packages/kbn-elastic-assistant/impl/assistant/api/evaluate/evaluate.tsx`
* Co-located relevant `use_perform_evaluation` hook
* Adds `Get Evaluate` route, and corresponding `use_evaluation_data`
hook. Currently only returns `agentExecutors` to be selected for
evaluation.
* API versioning constants added to
`x-pack/packages/kbn-elastic-assistant-common/impl/schemas/index.ts`
* Adds new `buildRouteValidationWithZod` function to
`x-pack/plugins/elastic_assistant/server/schemas/common.ts` for
validating routes against OAS generated zod schemas.
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds support for the Timeslice Metric visualization on the SLO
Detail page.
Fixes#170135
<img width="1756" alt="image"
src="56599b91-8827-4c6a-9df1-ccd80c5ab097">
## Summary
This PR fixes#175407 by increasing the rule lookback to 5 minutes to
try and avoid picking up 2 buckets since we can't control the exact time
of the rule execution to ensure accuracy and consistency. 😦Fixes#175360
[Flaky Test Runner
Results](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4959)
---------
Co-authored-by: Maryam Saeidi <maryam.saeidi@elastic.co>
## Summary
issue https://github.com/elastic/security-team/issues/7382
### Data Stream Adapter
This PR introduces the `@kbn/data-stream-adapter` package, which is a
utility library to facilitate Data Stream creation and maintenance in
Kibana, it was inspired by the data stream implementation in the Alerts
plugin.
The library has two exports:
- `DataStreamSpacesAdapter`: to manage space data streams. It uses the
`name-of-the-data-stream-<spaceId>` naming pattern.
- `DataStreamAdapter`: to manage single (not space-aware) data streams.
Usage examples in the package
[README](450be0369d/packages/kbn-data-stream-adapter/README.md)
### Data Quality Dashboard
The `DataStreamSpacesAdapter` has been integrated into the data quality
dashboard to store all the quality checks users perform. The information
stored is the metadata (also used for telemetry) and the actual data
rendered in the tables.
FieldMap definition
[here](450be0369d/x-pack/plugins/ecs_data_quality_dashboard/server/lib/data_stream/results_field_map.ts)
### Demo
311a0bf5-004b-46d7-8140-52a233361c91
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Philippe Oberti <philippe.oberti@elastic.co>
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Efe Gürkan YALAMAN <efeguerkan.yalaman@elastic.co>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Sander Philipse <94373878+sphilipse@users.noreply.github.com>
Co-authored-by: JD Kurma <JDKurma@gmail.com>
Co-authored-by: Jan Monschke <jan.monschke@elastic.co>
Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>
Co-authored-by: Khristinin Nikita <nikita.khristinin@elastic.co>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Davis McPhee <davis.mcphee@elastic.co>
Co-authored-by: Eyo O. Eyo <7893459+eokoneyo@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Marta Bondyra <4283304+mbondyra@users.noreply.github.com>
Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
Co-authored-by: Candace Park <56409205+parkiino@users.noreply.github.com>
## Summary
This PR adds the [High Cardinality
Indexer](https://github.com/elastic/high-cardinality-cluster) to Kibana
as a new package called `kbn-data-forge`. It also replaces
`kbn-infra-forge` usage in the test and is the preferred way to generate
data for Observability use cases, specifically for SLO testing.
### Todo
- [x] Replace `kbn-infra-forge` usage
- [x] Create convenience functions for testing (`generate` and
`cleanup`)
- [x] Make the logger (`LoggingTool`) configurable as an injected
dependency
- [x] Make the Elasticsearch client (`Client`) configurable as an
injected dependency
- [x] Fix the ECS Generate commands
- [x] Add CLI options via Commander
### CLI Help Screen
```
Usage: data_forge.js [options]
A data generation tool that will create realistic data with different scenarios.
Options:
--config <filepath> The YAML config file
--lookback <datemath> When to start the indexing (default: "now-15m")
--events-per-cycle <number> The number of events per cycle (default: 1)
--payload-size <number> The size of the ES bulk payload (default: 10000)
--concurrency <number> The number of concurrent connections to Elasticsearch (default: 5)
--index-interval <milliseconds> The interval of the data in milliseconds (default: 60000)
--dataset <dataset> The name of the dataset to use. Valid options: "fake_logs", "fake_hosts", "fake_stack" (default: "fake_logs")
--scenario <scenerio> The scenario to label the events with (default: "good")
--elasticsearch-host <address> The address to the Elasticsearch cluster (default: "http://localhost:9200")
--elasticsearch-username <username> The username to for the Elasticsearch cluster (default: "elastic")
--elasticsearch-password <password> The password for the Elasticsearch cluster (default: "changeme")
--elasticsearch-api-key <key> The API key to connect to the Elasticsearch cluster
--kibana-url <address> The address to the Kibana server (default: "http://localhost:5601")
--kibana-username <username> The username for the Kibana server (default: "elastic")
--kibana-password <password> The password for the Kibana server (default: "changeme")
--install-kibana-assets This will install index patterns, visualizations, and dashboards for the dataset
--event-template <template> The name of the event template (default: "good")
--reduce-weekend-traffic-by <ratio> This will reduce the traffic on the weekends by the specified amount. Example: 0.5 will reduce the traffic by half (default: 0)
--ephemeral-project-ids <number> The number of ephemeral projects to create. This is only enabled for the "fake_stack" dataset. It will create project IDs that will last 5 to 12 hours. (default: 0)
-h, --help output usage information
```
### Testing an Example
Run the following command against a clean Kibana development enviroment:
```
node x-pack/scripts/data_forge.js --events-per-cycle 200 --lookback now-1h --install-kibana-assets --ephemeral-project-ids 10 --dataset fake_stack
```
This should install a handful of DataViews (Admin Console, Message
Processor, Nginx Logs, Mongodb Logs) along with a few dashboards and
visualizations.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Related to https://github.com/elastic/kibana/issues/173009
## Summary
This is the first of two PRs and only focuses on the backend
implementation. This PR adds a new `notificationDelay` field to the
`Rule` object. With the delay the rule will run X times and has to match
the threshold X times before triggering actions. It won't affect the
alert recovery, but it can be expanded on easily if we want to include
recovered alerts in the future.
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To verify
- Use [Dev Tools](http://localhost:5601/app/dev_tools#/console) to
create a rule with the `notificationDelay`
```
POST kbn:/api/alerting/rule
{
"params": {
"searchType": "esQuery",
"timeWindowSize": 5,
"timeWindowUnit": "m",
"threshold": [
-1
],
"thresholdComparator": ">",
"size": 100,
"esQuery": """{
"query":{
"match_all" : {}
}
}""",
"aggType": "count",
"groupBy": "all",
"termSize": 5,
"excludeHitsFromPreviousRun": false,
"sourceFields": [],
"index": [
".kibana-event-log*"
],
"timeField": "@timestamp"
},
"consumer": "stackAlerts",
"schedule": {
"interval": "1m"
},
"tags": [],
"name": "test",
"rule_type_id": ".es-query",
"actions": [
{
"group": "query matched",
"id": "${ACTION_ID}",
"params": {
"level": "info",
"message": """Elasticsearch query rule '{{rule.name}}' is active:
- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
},
"frequency": {
"notify_when": "onActionGroupChange",
"throttle": null,
"summary": false
}
}
],
"notification_delay": {
"active": 3
}
}
```
- Verify that the rule will not trigger actions until it has matched the
delay threshold. It might be helpful to look at rule details page and
add the Triggered actions column to easily see the action was triggered
after X consecutive active alerts
<img width="1420" alt="Screen Shot 2024-01-16 at 1 18 52 PM"
src="85d8ceef-042c-4a52-950e-24492dc0e79f">
- Verify that the delay does not affect recovered alerts
## Summary
Fixes https://github.com/elastic/kibana/issues/173601
Implement Unified Search in SLO List view
1. SLO title has been removed to make it more consistent with other
kibana apps
2. Feedback button is moved up
3. Auto refresh is removed
<img width="1720" alt="image"
src="0ff6fd83-98ac-4737-bf4f-aa087739f110">
<img width="1728" alt="image"
src="aa71e2e7-3bc8-4be7-afb3-f4b7efffc953">
Filters
<img width="1727" alt="image"
src="d3bbf3d3-d2b3-4574-ae7f-bb8258016930">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Follow up PR from https://github.com/elastic/kibana/pull/174317 with the
following fixes/enhancements to `kbn-openapi-generator`:
* Fix extraneous `.`'s in paths causing generated files to not be
written to disk
* Updates `README.md` for latest method of adding CI actions
* Adds `info` details to generated metadata comment for more easily
tracing back to source schema
* Moves assistant `*.schema.yaml` files from `elastic_assistant` plugin
to `kbn-elastic-assistant-common` package
> [!NOTE]
> This PR includes a manual run of the `kbn-elastic-assistant-common`
package `yarn openapi:generate` script as a reference example. Since
this PR also updates the generation template to include the `info`
metadata, CI will run the generator for the other consumers
(`security_solution` & `osquery`) automatically, and commit those
updates to this PR. <img width="16"
src="https://user-images.githubusercontent.com/2946766/160040365-b1b8bb8a-d2d7-4187-b9b9-04817f8e2ae5.gif"
/>
### Test instructions
You can test against the `kbn-elastic-assistant-common` package using
either the main CLI script from kibana root:
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
or via the yarn command:
```
cd x-pack/packages/kbn-elastic-assistant-common/
yarn openapi-generate
```
### Checklist
Delete any items that are not applicable to this PR.
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds tests for basic SLO api routes, including:
1. Find slos
2. Get slo by id
3. Get slo definitions
4. Get slo instances
5. Create slo
6. Delete slo
7. Update slo
8. Reset slo
The create slo tests include some basic assertions that the resulting
calculated SLO is correct.
These tests do not cover:
1. SLOs in spaces
2. SLO permissions model
Passed flaky test runner for 200 iterations:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4595#_
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Kevin Delemme <kdelemme@gmail.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/172509
Adds ability to register feature capabilities through the assistant
server so they no longer need to be plumbed through the
`ElasticAssistantProvider`, which now also makes them available server
side.
Adds new `/internal/elastic_assistant/capabilities` route and
`useCapabilities()` UI hook for fetching capabilities.
### OpenAPI Codegen
Implemented using the new OpenAPI codegen and bundle packages:
* Includes OpenAPI codegen script and CI action as detailed in:
https://github.com/elastic/kibana/pull/166269
* Includes OpenAPI docs bundling script as detailed in:
https://github.com/elastic/kibana/pull/171526
To run codegen/bundling locally, cd to
`x-pack/plugins/elastic_assistant/` and run any of the following
commands:
```bash
yarn openapi:generate
yarn openapi:generate:debug
yarn openapi:bundle
```
> [!NOTE]
> At the moment `yarn openapi:bundle` will output an empty bundled
schema since `get_capabilities_route` is an internal route, this is to
be expected. Also, if you don't see the file in your IDE, it's probably
because `target` directories are ignored, so you may need to manually
find/open the bundled schema at it's target location:
`/x-pack/plugins/elastic_assistant/target/openapi/elastic_assistant.bundled.schema.yaml`
### Registering Capabilities
To register a capability on plugin start, add the following in the
consuming plugin's `start()`:
```ts
plugins.elasticAssistant.registerFeatures(APP_UI_ID, {
assistantModelEvaluation: config.experimentalFeatures.assistantModelEvaluation,
assistantStreamingEnabled: config.experimentalFeatures.assistantStreamingEnabled,
});
```
### Declaring Feature Capabilities
Feature capabilities are declared in
`x-pack/packages/kbn-elastic-assistant-common/impl/capabilities/index.ts`:
```ts
/**
* Interfaces for features available to the elastic assistant
*/
export type AssistantFeatures = { [K in keyof typeof assistantFeatures]: boolean };
export const assistantFeatures = Object.freeze({
assistantModelEvaluation: false,
assistantStreamingEnabled: false,
});
```
### Using Capabilities Client Side
And can be fetched client side using the `useCapabilities()` hook ala:
```ts
// Fetch assistant capabilities
const { data: capabilities } = useCapabilities({ http, toasts });
const { assistantModelEvaluation: modelEvaluatorEnabled, assistantStreamingEnabled } = capabilities ?? assistantFeatures;
```
### Using Capabilities Server Side
Or server side within a route (or elsewhere) via the `assistantContext`:
```ts
const assistantContext = await context.elasticAssistant;
const pluginName = getPluginNameFromRequest({ request, logger });
const registeredFeatures = assistantContext.getRegisteredFeatures(pluginName);
if (!registeredFeatures.assistantModelEvaluation) {
return response.notFound();
}
```
> [!NOTE]
> Note, just as with [registering arbitrary
tools](https://github.com/elastic/kibana/pull/172234), features are
registered for a specific plugin, where the plugin name that corresponds
to your application is defined in the `x-kbn-context` header of requests
made from your application, which may be different than your plugin's
registered `APP_ID`.
Perhaps this separation of concerns from one plugin to another isn't
necessary, but it was easy to add matching the behavior of registering
arbitrary tools. We can remove this granularity in favor of global
features if desired.
### Test Steps
* Verify `/internal/elastic_assistant/capabilities` route is called on
security solution page load in dev tools, and that by default the
`Evaluation` UI in setting does is not displayed and `404`'s if manually
called.
* Set the below experimental feature flag in your `kibana.dev.yml` and
observe the feature being enabled by inspecting the capabilities api
response, and that the evaluation feature becomes available:
```
xpack.securitySolution.enableExperimental: [ 'assistantModelEvaluation']
```
* Run the `yarn openapi:*` codegen scripts above and ensure they execute
as expected (code is generated/bundled)
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
In this PR, I'm renaming `Generative AI` to `Generative AI for Security`
in the connectors comatibility list so we have a split on Gen AI for
Security and Observability (follow up from
https://github.com/elastic/kibana/pull/173826).
## Screenshots
<img width="419" alt="Screenshot 2024-01-03 at 11 53 00 AM"
src="cb53c304-c96e-42c9-bce2-94b130040907">
<img width="542" alt="Screenshot 2024-01-03 at 11 53 32 AM"
src="6185010a-4b99-4dc7-bf62-9915c7b75a88">
<img width="1008" alt="Screenshot 2024-01-03 at 11 53 39 AM"
src="26301ee6-a50f-40ac-b898-91bf3e67c719">
## To verify
**Connectors**
1. Startup Kibana in trial mode
2. Open the create Bedrock connector flyout from the connectors page
3. Notice the compatibility is only for Security
4. Create a Bedrock connector (input random text in all fields to pass
validation)
5. Open the create OpenAI connector from the connectors page
6. Notice the compatibility is for Security and Observability
7. Create an OpenAI connector (input random text in all fields to pass
validation)
**Security Solution**
9. Navigate to the Security solution (`/app/security/get_started`)
10. Open the AI Assistant on the top right
11. Open the `Conversation Settings`
12. See OpenAI and Bedrock connectors displaying
**Observability**
13. Navigate to the Observability app (`/app/observability/overview`)
14. Open the AI Assistant on the top right
15. Select the actions menu on the top right of the flyout and open `AI
Assistant Settings`
16. Open the default connector dropdown
17. Notice only OpenAI connectors displaying
## Summary
It will show all the associated configs at one place in json form,
configuration, ingest pipeline config, roll up transform and summary
transform config !!
Motivation is to understand things while onboarding devs to slo and
during normal development.
a22ad292-ba59-4145-989e-80803b6a1e3e
## [Security Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented Generation (RAG) for Alerts_ Feature Flag
This PR deletes the `assistantRagOnAlerts` feature flag introduced in [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542](https://github.com/elastic/kibana/pull/172542).
Deleting the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle available in the assistant settings, per the screenshot below:
This PR should not be merged until the docs describing the feature in <https://github.com/elastic/security-docs/issues/4456> have been merged.
This PR also includes @benironside improvements to the Alerts setting in the video below:
73ea2717-ad2a-4998-afe2-cc154d8d19a9
### Desk testing
To desk test this change:
1) Delete the following `assistantRagOnAlerts` feature flag from your local `config/kibana.dev.yml`:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Start Kibana
3) Generate alerts with a variety of severity (e.g. `low`, `medium`, `high`, and `critical`)
4) Navigate to Security > Alerts
5) Click the `AI Assistant` button to open the assistant
6) Click the `X` button to clear the conversation
7) Click the assistant's `Settings` gear
8) Click the `Knowledge Base` category
**Expected result**
- The `Alerts` toggle shown in the screenshot below is available
9) Click the `Alerts` toggle to enable the feature
10) Click the `Save` button to close settings
11) Enter the following prompt:
```
How many open alerts do I have?
```
**Expected result**
- A response with alert counts grouped by workflow status will be returned, similar to the example below:
```
You currently have 48 open alerts in your system. These are categorized by severity as following: 19 of them are low severity, 16 are high severity, 12 are of medium severity and 1 is of critical severity. There is also 1 critical severity alert which is acknowledged.
```
12) Enter the following prompt:
```
Which alerts should I look at first?
```
**Expected result**
A response with alert details, similar to the following is returned:
```
Based on the latest information, the alerts to prioritize first are those related to a mimikatz process starting on the hosts, which have a critical severity and the highest risk score of 99. There are also a series of alerts related to an EQL process sequence with a high severity and risk scores of 73. There is one alert about an Elastic Endpoint Security alert with a medium severity and risk score of 47.
```
13) Once again, click the assistant's `Settings` gear
14) Click the `Knowledge Base` category
15) Click the `Alerts` toggle to disable the feature
16) Click the `Save` button to close settings
17) Once again, enter the following prompt:
```
How many open alerts do I have?
```
**Expected result**
- The assistant does NOT respond with a breakdown of alerts by severity. Instead it replies with something like the following example response:
```
I'm sorry for any confusion, but as an AI, I don't have real-time access to your data or system to provide the number of your current open alerts. You can check your Elastic Security dashboard or use the appropriate querying commands to get the updated count of your open alerts.
```
18) One more time, enter the following prompt:
```
Which alerts should I look at first?
```
**Expected result**
- The assistant does NOT respond with alert details. Instead it replies with something like the following example response:
```
As an AI model, I don't have the capability to access real-time data. However, when it comes to managing alerts in Elastic Security, it's generally recommended to first look at the ones with the highest severity and risk score. Alerts related to malware, unauthorized access attempts, and abnormal data transfers or process activities, for example, may need immediate attention due to their potential high impact.
```
## Summary
Part of https://github.com/elastic/kibana/issues/171831
Replaces the single boolean button with two ever-present buttons that
allow the user to "Add line breaks on pipes" and "Remove line breaks on
pipes"
<img width="435" alt="image"
src="a7042e15-f5b4-4a24-aa68-a7b7ca980895">
### Note
I had to use the TooltipWrapper and realized we are using this in many
places and every time we are duplicating the code. I moved it to
visualization-utils and changed the occurences.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Several enhancements to the error toast notifications in the ML plugin
to improve error reporting.
The bulk of the changes are to add 'See the full error' buttons to the
toasts allowing the user to see further details on the error that has
occurred. Also makes minor edits to some of the error messages to
improve clarity.
Also closes#171839 by changes to
`x-pack/plugins/ml/public/application/jobs/jobs_list/components/utils.js`
so that the the 'Jobs started successfully' toast is only shown if 1 or
more jobs have been started successfully.
Fixes#171839
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
`v90.0.0`⏩`v91.0.0-backport.0`
⚠️ While this upgrade pings many teams and has a large code diff, **the
majority of the changes are snapshots or tests-related** and do not
touch source code, so should theoretically only need a code review and
not dedicated QA.
The changes in EUI that required a large swathe of these updates are:
- **EuiPopover** removed an extra unnecessary `<div>` wrapper on its
anchors, which affected many snapshots and a few CSS overrides, which
should have been updated
- **EuiButtonGroup** now renders `<button>` elements instead of `<input
type="radio">` elements for single selection, which affected both
snapshots and E2E tests
- **EuiSuperDatePicker**'s absolute date input now requires an `Enter`
keypress when parsing dates (affected E2E tests)
- **EuiComboBox**, when rendered with `singleSelection={{ plainText:
'true' }}`, no longer renders a pill (i.e. text). This combobox type now
behaves more like an `EuiFieldText`, where the selection is rendered via
input `value` instead. This affected a high amount of E2E tests (both
FTR and Cypress), both in terms of updating assertions and changing
selections, but should **not** significantly affect user experience -
see https://github.com/elastic/eui/pull/7332 for more.
---
##
[`v91.0.0-backport.0`](https://github.com/elastic/eui/tree/v91.0.0-backport.0)
**This is a backport release only intended for use by Kibana.**
- Added `esqlVis`, `pipeBreaks`, and `pipeNoBreaks` icon glyphs.
- `EuiSelectable` now allows configurable text truncation via
`listProps.truncationProps`
([#7388](https://github.com/elastic/eui/pull/7388))
- `EuiTextTruncate` now supports a new `calculationDelayMs` prop for
working around font loading or layout shifting scenarios
([#7388](https://github.com/elastic/eui/pull/7388))
**Bug fixes**
- Fixed a bug with `EuiSelectable`s with custom `truncationProps`, where
scrollbar widths were not being accounted for
([#7392](https://github.com/elastic/eui/pull/7392))
## [`91.0.0`](https://github.com/elastic/eui/tree/v91.0.0)
- Updated the background color of `EuiPopover`s in dark mode to increase
visibility & contrast against other page/panel backgrounds
([#7310](https://github.com/elastic/eui/pull/7310))
- Memoized `EuiDataGrid` to prevent unneeded re-renders
([#7324](https://github.com/elastic/eui/pull/7324))
- Added a configurable `role` prop to `EuiAccordion`
([#7326](https://github.com/elastic/eui/pull/7326))
- Added a configurable `role` prop to `EuiGlobalToastList`
([#7328](https://github.com/elastic/eui/pull/7328))
- For greater flexibility, `EuiSuperDatePicker` now allows users to
paste ISO 8601, RFC 2822, and Unix timestamps in the `Absolute` tab
input, in addition to timestamps in the `dateFormat` prop
([#7331](https://github.com/elastic/eui/pull/7331))
- Plain text `EuiComboBox`es now behave more like a normal text
field/input. Backspacing will no longer delete the entire value, and
selected values can now be double clicked and copied.
([#7332](https://github.com/elastic/eui/pull/7332))
- `EuiDataGrid`'s display settings popover now allows users to clear the
"Lines per row" input before typing in a new number
([#7338](https://github.com/elastic/eui/pull/7338))
- Improved the UX of `EuiSuperDatePicker`'s Absolute tab for users
manually typing in timestamps
([#7341](https://github.com/elastic/eui/pull/7341))
- Updated `EuiI18n`s with multiple `tokens` to accept dynamic `values`
([#7341](https://github.com/elastic/eui/pull/7341))
**Bug fixes**
- Fixed `EuiComboBox`'s `onSearchChange` callback to pass the correct
`hasMatchingOptions` value
([#7334](https://github.com/elastic/eui/pull/7334))
- Fixed an `EuiSelectableTemplateSitewide` bug where the `popoverButton`
behavior would break if passed a non-DOM React wrapper
([#7339](https://github.com/elastic/eui/pull/7339))
**Deprecations**
- `EuiPopover`: deprecated `anchorClassName`. Use `className` instead
([#7311](https://github.com/elastic/eui/pull/7311))
- `EuiPopover`: deprecated `buttonRef`. Use `popoverRef` instead
([#7311](https://github.com/elastic/eui/pull/7311))
- `EuiPopover`: removed extra `.euiPopover__anchor` div wrapper. Target
`.euiPopover` instead if necessary
([#7311](https://github.com/elastic/eui/pull/7311))
- Deprecated `EuiButtonGroup`'s `name` prop. This can safely be removed.
([#7325](https://github.com/elastic/eui/pull/7325))
**Breaking changes**
- Removed deprecated `euiPaletteComplimentary` - use
`euiPaletteComplementary` Instead
([#7333](https://github.com/elastic/eui/pull/7333))
**Accessibility**
- Updated `type="single"` `EuiButtonGroup`s to render standard buttons
instead of radio buttons under the hood, per recent a11y recommendations
([#7325](https://github.com/elastic/eui/pull/7325))
- `EuiAccordion` now defaults to a less screenreader-noisy `group` role
instead of `region`. If your accordion contains significant enough
content to be a document landmark role, you may re-configure it back to
`region`. ([#7326](https://github.com/elastic/eui/pull/7326))
- Reduced screen reader noisiness when sorting `EuiDataGrid` columns via
toolbar ([#7327](https://github.com/elastic/eui/pull/7327))
- `EuiGlobalToastList` now defaults to a `log` role. If your toasts will
always require immediate user action, consider (with caution) using the
`alert` role instead.
([#7328](https://github.com/elastic/eui/pull/7328))
**CSS-in-JS conversions**
- Updated `$euiFontFamily` and `$euiCodeFontFamily` to match Emotion
fonts ([#7332](https://github.com/elastic/eui/pull/7332))
---------
Co-authored-by: Cee Chen <constance.chen@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Cee Chen <549407+cee-chen@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## [Security Solution] [Elastic AI Assistant] Include `acknowledged` alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts)
This PR updates the query used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542](https://github.com/elastic/kibana/pull/172542) to include alerts with a `kibana.alert.workflow_status` value of `acknowledged`.
The query previously only returned alerts with a status of `open`. This change ensures both `open` and `acknowledged` alerts are provided as context to the LLM.
### Updated Anonymization defaults
Three fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts.
For example, the LLM can refer to specific alerts by ID when the `_id` field is provided.
This PR makes the following additive changes to the Assistant's `Anonymization` defaults:
| Field | Allow by default | Anonymize by default | Value add |
|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs). |
| `kibana.alert.risk_score` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query sorts alerts by `kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses. |
| `kibana.alert.workflow_status` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query filters alerts by `kibana.alert.workflow_status` to ensure only `open` and `acknowledged` alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses. |
- Clicking the `Reset` button shown in the screenshot below will reset the user's `Anonymization` defaults, such that they include the additive changes in the table above:
### Updated settings text
The text in the settings below was also updated:
### Desk testing
To desk test this change:
- Enable the `assistantRagOnAlerts` feature flag described in [#172542](https://github.com/elastic/kibana/pull/172542) must be enabled, per the following example:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
- The `Alerts` feature must be enabled in the assistant settings, per the screenshot below:
1) Navigate to Security > Alerts
2) Click the `AI Assistant` button to open the assistant
3) Click the `Settings` gear to open the assistant settings
4) Click the `Anonymization` category
5) Click the `Reset` button, shown in the screenshot below
**Expected results**
- `65` fields are allowed by default, per the screenshot above
- `12` fields are anonymized by default, per the screenshot above
- The `_id` field is allowed by default, per the screenshot above
- The `_id` field is anonymized by default, per the screenshot above
6) Type `kibana.alert.risk` in the search box
**Expected result**
- The `kibana.alert.risk_score` field is allowed by default
7) Type `kibana.alert.workflow` in the search box
**Expected result**
- The `kibana.alert.workflow_status` field is allowed by default
8) Click `Save`
9) Click the `X` button to clear the conversation
10) Close the assistant
11) Add the following two fields as columns to the Alerts page table:
- `kibana.alert.workflow_status`
- `_id`
12) Sort the table, first by `kibana.alert.risk_score` from high to low, and then by `@timestamp` from new to old, per the screenshot below:
13) Filter the alerts page to only show `open` and `acknowledged` alerts
**Expected result**
- The alerts page has custom columns, sorting, and filtering, per the screenshot below:
14) Click the `AI Assistant` button to open the assistant
15) Ask the assistant:
```
What is the workflow status of my alerts?
```
**Expected result**
- The assistant will report on the workflow status of alerts, per the example response below:
```
The workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system.
```
16) Close the assistant
17) Change the workflow status of an alert in the Alerts table from `open` to `acknowledged`
**Expected result**
- The alerts table shows the updated alert, per the screenshot below:
18) Once again, open the assistant
19) Once again, ask the (same) question:
```
What is the workflow status of my alerts?
```
**Expected result**
- The response from the assistant makes reference to the alert who's workflow status was changed from `open` to `acknowledged`, per the example response below:
```
Based on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure.
```
20) Ask the assistant for details about the acknowledged alerts:
```
What are the details of the acknowledged alerts?
```
**Expected result**
The assistant for details about the acknowledged alert that, for example, includes the `kibana.alert.risk_score`, per the example response below:
```
In response to your previous question, here are the details of the acknowledged alerts:
1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'.
2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'.
3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'.
Please, take appropriate action to address these alerts.
```
21) Ask the assistant for the `_id` of the acknowledged alert:
```
What is the id of the acknowledged alert?
```
**Expected results**
- The response from the assistant contains the `_id` of the `acknowledged` alert, per the example response below:
```
The id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'.
```
- The `_id` shown in the assistant is the same `_id` of the acknowledged alert on the alerts page, per the screeenshot below:
22) Click the `Show anonymized` toggle in the assistant
**Expected result**
- The `_id` shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below:
- Closes https://github.com/elastic/kibana/issues/172570
## Summary
This PR creates a new package `@kbn/visualization-utils` and moves
`getTimeZone` helper into it. Also the PR removes duplicates of other
similar helpers.
And the histogram in the field popover has now the same time zone
configuration as the the main hits histogram:
<img width="500" alt="Screenshot 2023-12-06 at 18 46 25"
src="2d350c91-1a41-419a-9d80-f203f1c90327">
## For testing
Change `dateFormat:tz` in Advanced Settings and check if histograms are
rendered accordingly.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:
- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_
### More context
Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:
- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
- Only fields allowed by the user will be sent as context to the LLM
- Users may enable or disable anonymization for specific fields (via settings)
- Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
### Settings
This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)
When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### Feature flag
To use this feature:
1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:
## How it works
- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
- `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)
- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:
```typescript
const tools: Tool[] = getApplicableTools({
allow,
allowReplacement,
alertsIndexPattern,
assistantLangChain,
chain,
esClient,
modelExists,
onNewReplacements,
replacements,
request,
size,
});
```
- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.
- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details
- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.
- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared
- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
- The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
## Summary
Fixes https://github.com/elastic/kibana/issues/171243. This PR adds
field `_tier` to the list of omit fields to not show or display. This is
especially relevant when `_tier` is added in the list of meta fields in
Kibana.
Steps to reproduce:
1. In Advanced settings, add `_tier` to the list of meta fields. This
will show _tier as a field across Kibana if data has a tier applied.
<img width="976" alt="image"
src="86ecbbba-c574-42f6-97cf-c465ec334d7e">
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#171613
## Summary
This PR adds the viewInApp URL to the custom threshold rule type. This
URL will send the user to the log explorer with the selected data view
and the rule's query filter. If there is only one document aggregation,
then the filter related to this aggregation will be added as shown
below:
|Rule|Discover with pre-fill data|
|---|---|
||
For the ad-hoc data view, you should be able to see the selected index
pattern in discover similar to this:
<img
src="046493ae-ba59-46b7-a40f-68d1836d43f1"
width=400 />
### 🧪 How to test
- Check the viewInApp URL both in action variables and the alert table
for the following scenarios:
- A rule with a persisted data view
- A rule with an ad-hoc data view
- A rule with count aggregation and filter
- A rule with an optional query filter
- A rule with non-count aggregation
In all the above scenarios, the starting time in the Discover should be
before the alert's start time.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
With this PR we introduce a new Alert User Assignment feature:
- It is possible to assign a user/s to alert/s
- There is a new "Assignees" column in the alerts table which displays
avatars of assigned users
- There is a bulk action to update assignees for multiple alerts
- It is possible to see and update assignees inside the alert details
flyout component
- There is an "Assignees" filter button on the Alerts page which allows
to filter alerts by assignees
We decided to develop this feature on a separate branch. This gives us
ability to make sure that it is thoroughly tested and we did not break
anything in production. Since there is a data scheme changes involved we
decided that it will be a better approach. cc @yctercero
## Testing notes
In order to test assignments you need to create a few users. Then for
users to appear in user profiles dropdown menu you need to activate them
by login into those account at least once.
8eeb13f3-2d16-4fba-acdf-755024a59fc2
Main ticket https://github.com/elastic/security-team/issues/2504
## Bugfixes
- [x] https://github.com/elastic/security-team/issues/8028
- [x] https://github.com/elastic/security-team/issues/8034
- [x] https://github.com/elastic/security-team/issues/8006
- [x] https://github.com/elastic/security-team/issues/8025
## Enhancements
- [x] https://github.com/elastic/security-team/issues/8033
### Checklist
- [x] Functional changes are hidden behind a feature flag. If not
hidden, the PR explains why these changes are being implemented in a
long-living feature branch.
- [x] Functional changes are covered with a test plan and automated
tests.
- [x] https://github.com/elastic/kibana/issues/171306
- [x] https://github.com/elastic/kibana/issues/171307
- [x] Stability of new and changed tests is verified using the [Flaky
Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner).
- [x]
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4091
- [x] Comprehensive manual testing is done by two engineers: the PR
author and one of the PR reviewers. Changes are tested in both ESS and
Serverless.
- [x] Mapping changes are accompanied by a technical design document. It
can be a GitHub issue or an RFC explaining the changes. The design
document is shared with and approved by the appropriate teams and
individual stakeholders.
* https://github.com/elastic/security-team/issues/7647
- [x] Functional changes are communicated to the Docs team. A ticket or
PR is opened in https://github.com/elastic/security-docs. The following
information is included: any feature flags used, affected environments
(Serverless, ESS, or both). **NOTE: as discussed we will wait until docs
are ready to merge this PR**.
* https://github.com/elastic/security-docs/issues/4226
* https://github.com/elastic/staging-serverless-security-docs/pull/232
---------
Co-authored-by: Marshall Main <marshall.main@elastic.co>
Co-authored-by: Xavier Mouligneau <xavier.mouligneau@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
Consolidates UI elements and backend code to create/delete data views
and destination indices related to transforms and data frame analytics.
We ended up with two different approaches for creating data views in the
wizards for transforms and data frame analytics, the original reason was
we were not aware of the `allowNoIndex: true` setting and worked around
that in different ways.
This PR aligns UI workflows and moves related code to a new package
`@kbn/ml-data-view-utils` for data views and
`@kbn/ml-creation-wizard-utils` for the destination index form. The
latter might be used for other shared components across wizard..
In Data Frame Analytics, the checkbox to create a data view was removed
from the last "Create" step, instead the option to create a data view
was moved to the "Details" step.
In Transforms, the UI component to create the destination index was
brought over from DFA where there is a switch option to automatically
use the job ID as the name for the destination index by default.
