This PR consists of the following changes:
- An option to delete an existing inference endpoint
- Filtering the endpoints based on 'provider' and 'type'
- Search option
- Display the trained models deployment status
- Display additional 3rd party providers (Mistral, Azure OpenAI, Azure
AI Studio)
- Add licensing for gating enterprise licensed users
### Stack Management
### Serverless
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
## Summary
Closes - https://github.com/elastic/kibana/issues/172272
The PR adds the degraded Field Table in the Logs Flyout. The accordion
is kept closed by default. For demo purposes below screenshot will show
it expanded
This PR will also fix a very simply Flaky Test -
https://github.com/elastic/kibana/issues/186244
## Pending Items
- [x] Add Locator for Dataset Quality Page
- [x] Add tests
## Demo
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introducing the `search_homepage` plugin along with integration into
`enterprise_search` and `serverless_search` behind a feature flag. This
will allow implementing the feature gated behind the feature flag.
To test these changes you can enable the feature flag with the Kibana
Dev Console using the following command:
```
POST kbn:/internal/kibana/settings/searchHomepage:homepageEnabled
{"value": true}
```
You can then disable the feature flag with the following command:
```
DELETE kbn:/internal/kibana/settings/searchHomepage:homepageEnabled
```
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Description
In this PR, we implemented a view for managing inference endpoints. The
changes include the following items for both **Serverless** and
**Stack**.
- A blank page will be displayed if no inference endpoints are
available.
- A page displaying a list of inference endpoints. The user can view
various details about each endpoint, such as the endpoint itself, the
provider, and the type. The table supports pagination and sorting.
- Users can add a new inference endpoint using Elasticsearch models and
third-party APIs, including Hugging Face, Cohere, and OpenAI.
To keep the changes in this PR manageable, the following items are **out
of scope** but will be added in subsequent PRs
- Option to delete an inference endpoint
- Filtering and Search bar
- Information about allocations, thread.
- Icons for **Provider**
- Deployment status of underlying trained models
## Empty page in Stack Management
e2064ee8-3623-457f-8a04-19603e97e815
## Page with all inference endpoints in Stack Management
89bec450-1569-4425-b013-5058b577b95a
## Inference Endpoints Management in Serverless
bd8b6b71-0e09-49f4-aa9a-19338a1da225
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves#179543.
This patch will update the deep linking used by `Add data` links
throughout Observability to pre-select the proper experience when
navigating. This will streamline the process for users to help them more
quickly ingest the data they need, allowing them to get value out of
solution pages with fewer clicks.
The changes (from the parent issue):
- [x] Allow o11y Onboarding Locator to take query params
4a024a09c7
- [x] 'Add data' on the top right of Logs Explorer page should link to
the new Add data UX where the use case 'collect and analyze logs' is
pre-selected.
1cef1c68d5
- [x] 'Add data' on the top right of Logs -> Stream page should link to
the new Add data UX where the use case 'collect and analyze logs' is
pre-selected.
75615b34dd
- [x] 'Add data' on the top right of Logs -> Anomalies page should link
to the new Add data UX where the use case 'collect and analyze logs' is
pre-selected.
75615b34dd
- [x] 'Add data' on the top right of Logs -> Categories page should link
to the new Add data UX where the use case 'collect and analyze logs' is
pre-selected.
75615b34dd
- [x] 'Add data' on the top right of Infrastructure -> Inventory page
should link to the new Add data UX where the use case 'monitor
infrastructure' is pre-selected.
07fac0f8b5
- [x] 'Add data' on the top right of Infrastructure -> Metrics Explorer
page should link to the new Add data UX where the use case 'monitor
infrastructure' is pre-selected.
07fac0f8b5
- [x] 'Add data' on the top right of Infrastructure -> Hosts page should
link to the new Add data UX where the use case 'monitor infrastructure'
is pre-selected.
07fac0f8b5
### Demo
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Relates to https://github.com/elastic/kibana/issues/183406.
## 📝 Summary
This PR creates a new plugin `data_quality` in order to register dataset
quality as a Stack management page under data section. For now there is
no reference to this new page in the sideNav in stateful or serverless.
In order to navigate to this new page you can use the url
`/app/management/data/data_quality`
Changes included in this PR:
- New plugin created
- Plugin registered in stack management, data section
- Dataset quality plugin is instantiated and the state is in sync with
URL
- Removed references to dataset quality in Logs explorer
## 🎥 Demo
501c9c47-4a1b-4f91-9be6-d022a821e88e
## 🙅🏼 Missing
- Dataset quality locator
- There are still references to logs explorer (table and flyout) that
will be handled in a follow up PR.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [Attack discovery] Attack discovery
### Summary
This PR renames the _Attack discovery_ Security Solution feature from its original name, [AI Insights](https://github.com/elastic/kibana/pull/180611).
_Above: Attack discovery in the Security Solution_
Attack discovery uses AI to identify active attacks in the environment, without the time (or prior experience) required to manually investigate individual alerts in Elastic Security, identify if they are related, and document the identified attack progression.
While users can ask the Assistant to find these progressions today, Attack discovery is a dedicated UI to identify these progressions and action them accordingly. This feature adds a new page, `Attack discovery`, to the Security Solution's global navigation.
Attack discoveries are generated from Large Language Models (LLMs) to identify attack progressions in alert data, and to correlate and identify related entities and events. When possible, attack progressions are attributed to threat actors.
### Details
Users may generate attack discoveries from a variety of LLMs, configured via [Connectors](https://www.elastic.co/guide/en/kibana/master/action-types.html):
_Above: LLM selection via the connectors popup menu_
Clicking on the title of an attack discovery toggles the discovery between the collapsed and expanded state:
_Above: Collapsing / expanding an attack discovery (animated gif)_
The first three discoveries displayed on the Attack discoveries page are expanded by default. Any additional discoveries that appear after the first three must be expanded manually.
Attack discoveries provide a summary of the entities impacted by an attack. Clicking on an entity, i.e. a hostname or username, displays the entity flyout with the entity's risk summary:
_Above: Clicking on a host in the summary of the attack discovery reveals the host risk summary (animated gif)_
Hover over fields in the discovery's summary or details to reveal pivot actions for investigations:
_Above: Hovering over fields in the details of an attack discovery reveals pivot actions (animated gif)_
Attack discoveries are generated from alerts provided as context to the selected LLM. The alert data provided to the LLM is anonymized automatically. Anonymization is [configured](https://www.elastic.co/guide/en/security/current/security-assistant.html#ai-assistant-anonymization) via the same anonymization settings as the Assistant. Users may override the defaults to allow or deny specific alert fields, and to toggle anonymization on or off for specific fields.
Click the Anonymization toggle to show or hide the actual values sent to the LLM:
_Above: Toggling anonymization to reveal the actual values sent to the LLM (animated gif)_
### Empty prompt
At the start of a session, or when a user selects a connector that doesn't (yet) have any attack discoveries, an [empty prompt](https://eui.elastic.co/#/display/empty-prompt) is displayed.
The animated counter in the empty prompt counts up until it displays the maximum number of alerts that will be sent to the LLM:
_Above: An animated counter displays the maximum number of alerts that will be sent to the LLM (animiated gif)_
The _Settings_ section of this PR details how users configure the number of alerts sent to the LLM. The animated counter in the empty prompt immediately re-animates to the newly-selected number when the setting is updated.
### Take action workflows
The _Take action_ popover displays the following actions:
- `Add to new case`
- `Add to existing case`
- `View in AI Assistant`
_Above: The Take action popover_
#### Add to new case
Clicking the `Add to new` case action displays the `Create case` flyout.
_Above: The `Add to new case` workflow_
An `Alerts were added to <case name>` toast is displayed when the case is created:
_Above: Case creation toast_
A markdown representation of the attack discovery is added to the case:
_Above: A markdown representation of an attack discovery in a case_
The alerts correlated to generate the discovery are attached to the case:
_Above: Attack discovery alerts attached to a case_
#### Add to existing case
Clicking the `Add to existing case` action displays the `Select case` popover.
_Above: The `Select case` popover_
When users select an existing case, a markdown representation of the attack discovery, and the alerts correlated to generate the discovery are attached to the case, as described above in the _Add to new case_ section.
#### View in AI Assistant
The `View in AI Assistant` action in the `Take action` popover, and two additional `View in AI Assistant` affordances that appear in each discovery have the same behavior:
Clicking `View in AI Assistant` opens the assistant and adds the attack discovery as context to the current conversation.
_Above: An attack discovery added as context to the current conversation_
Clicking on the attack discovery in the assistant expands it to reveal a preview of the discovery.
_Above: An expanded attack discovery preview in the assistant_
The expanded attack discovery preview reveals the number of anonymzied fields from the discovery that were made available to the conversation. This feature ensures discoveries are added to a conversation with the anonymized field values.
An attack discovery viewed in the AI assistant doesn't become part of the conversation until the user submits it by asking a question, e.g. `How do I remediate this?`.
Attack discoveries provided as context to a conversation are formatted as markdown when sent to the LLM:
_Above: Attack discoveries provided as context to a conversation are formatted as markdown_
Users may toggle anonymization in the conversation to reveal the original field values.
_Above: Revealing the original field values of an attack discovery added as markdown to a conversation (animated gif)_
#### Alerts tab
The _Alerts_ tab displays the alerts correlated to generate the discovery.
_Above: The alerts correlated to generate the attack discovery in the Alerts tab_
The `View details`, `Investigate in timeline`, and overflow row-level alert actions displayed in the Alerts tab are the same actions available on the Cases's page's Alerts tab:
_Above: Row-level actions are the same as the Cases pages Alert's tab_
#### Investigate in Timeline
Click an attack discovery's `Investigate in Timeline` button to begin an investigation of an discovery's alerts in Timeline. Alert IDs are queried via the `Alert Ids` filter:
_Above: Clicking Investigate in Timeline (animated gif)_
The alerts from the attack discovery are explained via row renderers in Timeline:
_Above: Row rendered attack discovery alerts in Timeline_
### Attack Chain
When alerts are indicative of attack [tactics](https://attack.mitre.org/tactics/enterprise/), those tactics are displayed in the discovery's _Attack Chain_ section:
_Above: An attack discovery with tactics in the Attack chain_
The Attack Chain section will be hidden if an attack discovery is not indicative of specific tactics.
### Mini attack chain
Every attack discovery includes a mini attack chain that visually summarizes the tactics in a discovery. Hovering over the mini attack chain reveals a tooltip with the details:
_Above: The mini attack chain tooltip_
### Storage
The latest attack discoveries generated for each connector are cached in the browser's session storage in the following key:
```
elasticAssistantDefault.attackDiscovery.default.cachedAttackDiscoveries
```
Caching attack discoveries in session storage makes it possible to immediately display the latest when users return to the Attack discoveries page from other pages in the security solution (e.g. Cases).
_Above: Cached attack discoveries from session storage are immediately displayed when users navigate back to Attack discoveries (animated gif)_
While waiting for a connector to generate results, users may view the cached results from other connectors.
Cached attack discoveries are immediately available, even after a full page refresh, as long as the browser session is still active.
### `Approximate time remaining` / `Above average time` counters
Some LLMs may take seconds, or even minutes to generate attack discoveries. To help users anticipate the time it might take to generate new discoveries, the page displays a `Approximate time remaining: mm:ss` countdown timer that counts down to zero from the average time it takes to generate discoveries for the selected LLM:
_Above: The `Approximate time remaining: mm:ss` countdown counter (animated gif)_
If the LLM doesn't generate attack discoveries before the counter reaches zero, the text will change from `Approximate time remaining: mm:ss` to `Above average time: mm:ss`, and start counting up from `00:00` until the attack discoveries are generated:
_Above: The `Above average time: mm:ss` counter (animated gif)_
The first time attack discoveries are generated for a model, the `Approximate time remaining: mm:ss` counter is not displayed.
Average time is calculated over the last 5 generations on the selected connector. This is illustrated by clicking on the (?) information icon next to the timer. The popover displays the average time, and the time in seconds for the last 5 runs:
_Above: Clicking on the (?) information icon displays the average time, and the duration / datetimes for the last 5 generations_
The time and duration of the last 5 generations (for each connector) are persisted in the browser's local storage in the following key:
```
elasticAssistantDefault.attackDiscovery.default.generationIntervals
```
### Errors
When attack discovery generation fails, an error toaster is displayed to explain the failure:
_Above: An error toast explains why attack discovery generation failed_
### Feature flag
The `attackDiscoveryEnabled` feature flag must be enabled to view the `Attack discovery` link in the Security Solution's global navigation.
Add the `attackDiscoveryEnabled` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['attackDiscoveryEnabled']
```
### Settings
The number of alerts sent as context to the LLM is configured by `Knowledge Base` > `Alerts` slider in the screenshot below:
- The slider has a range of `10` - `100` alerts (default: `20`)
Up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- The `kibana.alert.workflow_status` must be `open`
- The alert must have been generated in the last `24 hours`
- The alert must NOT be a `kibana.alert.building_block_type` alert
- The `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### License
An Enterprise license is required to use Attack discovery.
The following empty view is displayed for users who don't have an Enterprise license:
## How it works
- Users navigate to the Attack discovery page: `x-pack/plugins/security_solution/public/attack_discovery/pages/index.tsx`
- When users click the `Generate` button(s) on the Attack discovery page, attack discoveries are fetched via the `useAttackDiscovery` hook in `x-pack/plugins/security_solution/public/attack_discovery/use_attack_discovery/index.tsx`.
- The `fetchAttackDiscoveries` function makes an http `POST` request is made to the `/internal/elastic_assistant/attack_discovery` route. Requests include the following parameters:
- `actionTypeId`, determines temperature and other connector-specific request parameters
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `anonymizationFields`, the user's `Allowed` and (when applicable `Anonymized` ) fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `connectorId`, id of the connector to generate the attack discoveries
- `size`, the maximum number of alerts to generate attack discoveries from. This numeric value is set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- `replacements`, an optional `Record<string, string>` collection of replacements that's always empty in the current implementation. When non-empty, this collection enables new attack discoveries to be generated using existing replacements.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- The `postAttackDiscoveryRoute` function in `x-pack/plugins/elastic_assistant/server/routes/attack_discovery/post_attack_discovery.ts` handles the request.
- The inputs and outputs to/from this route are defined by the [OpenAPI](https://spec.openapis.org/oas/v3.1.0) schema in `x-pack/packages/kbn-elastic-assistant-common/impl/schemas/attack_discovery/post_attack_discovery_route.schema.yaml`.
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
- The `postAttackDiscoveryRoute` route handler function in `x-pack/plugins/elastic_assistant/server/routes/attack_discovery/post_attack_discovery.ts` invokes the `attack-discovery` tool, defined in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/attack_discovery_tool.ts`.
The `attack-discovery` tool is registered by the Security Solution. Note: The `attack-discovery` tool is only used by the attack discovery page. It is not used to generate new attack discoveries from the context of an assistant conversation, but that feature could be enabled in a future release.
- The `attack-discovery` tool uses a LangChain `OutputFixingParser` to create a [prompt sandwich](https://www.elastic.co/blog/crafting-prompt-sandwiches-generative-ai) with the following parts:
```
______________________________________________________
/ \
| Attack discovery JSON formatting instructions | (1)
\ _____________________________________________________/
+-----------------------------------------------------+
| Attack discovery prompt | (2)
+-----------------------------------------------------+
/ \
| Anonymized Alerts | (3)
\_____________________________________________________/
```
- The `Attack discovery JSON formatting instructions` in section `(1)` of the prompt sandwich are defined in the `getOutputParser()` function in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/get_output_parser.ts`. This function creates a LangChain `StructuredOutputParser` from a Zod schema. This parser validates responses from the LLM to ensure they are formatted as JSON representing an attack discovery.
- The `Attack discovery prompt` in section `(2)` of the prompt sandwich is defined in the `getAttackDiscoveryPrompt()` function in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/get_attack_discovery_prompt.ts`. This part of the prompt sandwich includes instructions for correlating alerts, and additional instructions to the LLM for formatting JSON.
- The `Anonymized Alerts` in section `(3)` of the prompt sandwich are returned by the `getAnonymizedAlerts()` function in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/get_anonymized_alerts.ts`. The allow lists configured by the user determine which alert fields will be included and anonymized.
- The `postAttackDiscoveryRoute` route handler returns the attack discoveries generated by the `attack-discovery` tool to the client (browser).
- Attack discoveries are rendered in the browser via the `AttackDiscoveryPanel` component in `x-pack/plugins/security_solution/public/attack_discovery/attack_discovery_panel/index.tsx`
- The `AttackDiscoveryTab` tab in `x-pack/plugins/security_solution/public/attack_discovery/attack_discovery_panel/tabs/attack_discovery_tab/index.tsx` includes the _Summary_ and _Details_ section of the attack discovery.
- The `AttackDiscoveryMarkdownFormatter` in `x-pack/plugins/security_solution/public/attack_discovery/attack_discovery_markdown_formatter/index.tsx` renders hover actions on entities (like hostnames and usernames) and other fields in the attack discovery.
- The `AttackDiscoveryPanel` component makes use of the `useAssistantOverlay` hook in `x-pack/packages/kbn-elastic-assistant/impl/assistant/use_assistant_overlay/index.tsx` to register the attack discovery as context with the assistant. This registration process makes it possible to view discoveries in the assistant, and ask questions like "How do I remediate this?". In this feature, the `useAssistantOverlay` hook was enhanced to accept anonymizaton replacements. This enables an assistant conversation to (re)use replacements originally generated for an attack discovery.
## Summary
The PR adds the `breakdownField` param in `LogsExplorerNavigationParams`
so that when "Explorer data in Logs Explorer" is clicked on Degraded
Docs chart on Dataset Quality flyout while the chart has a breakdown
field selected, the field is passed over to Logs Explorer.
b380ac85-e40e-451b-983f-41c68f87ed7b
Update Search nav to build and move playground from content to build
update the kibana side nav to feature Playground. This routes from
application to playground.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## [Security Solution] [AI Insights] AI Insights
### Summary
This PR introduces _AI Insights_ to the Security Solution:
_Above: AI Insights in the Security Solution_
AI Insights identify active attacks in the environment, without the time
(or prior experience) required to manually investigate individual alerts
in Elastic Security, identify if they are related, and document the
identified attack progression.
While users can ask the Assistant to find these progressions today, AI
Insights is a dedicated UI to identify these progressions and action
them accordingly. This feature adds a new page, `AI Insights`, to the
Security Solution's global navigation.
AI Insights are generated from Large Language Models (LLMs) to identify
attack progressions in alert data, and to correlate and identify related
entities and events. When possible, attack progressions are attributed
to threat actors.
### Details
Users may generate insights from a varetiy of LLMs, configured via
[Connectors](https://www.elastic.co/guide/en/kibana/master/action-types.html):
_Above: LLM selection via the connectors popup menu_
Clicking on the title of an insight toggles the insight between the
collapsed and expanded state:
_Above: Collapsing / expanding an insight (animated gif)_
The first three insights displayed on the AI Insights page are expanded
by default. Any additional insights that appear after the first three
must be expanded manually.
Insights provide a summary of the entities impacted by an attack.
Clicking on an entity, i.e. a hostname or username, displays the entity
flyout with the entity's risk summary:
_Above: Clicking on a host in the summary of the insight reveals the
host risk summary (animated gif)_
Hover over fields in the insight's summary or details to reveal pivot
actions for investigations:
_Above: Hovering over fields in the details of an insight reveals pivot
actions (animated gif)_
Insights are generated from alerts provided as context to the selected
LLM. The alert data provided to the LLM is anonymized automatically.
Anonymization is
[configured](https://www.elastic.co/guide/en/security/current/security-assistant.html#ai-assistant-anonymization)
via the same anonymization settings as the Assistant. Users may override
the defaults to allow or deny specific alert fields, and to toggle
anonymization on or off for specific fields.
Click the Anonymization toggle to show or hide the actual values sent to
the LLM:
_Above: Toggling anonymization to reveal the actual values sent to the
LLM (animated gif)_
### Empty prompt
At the start of a session, or when a user selects a connector that
doesn't (yet) have any insights, an [empty
prompt](https://eui.elastic.co/#/display/empty-prompt) is displayed.
The animated counter in the empty prompt counts up until it displays the
maximum number of alerts that will be sent to the LLM:
_Above: An animated counter displays the maximum number of alerts that
will be sent to the LLM (animiated gif)_
The _Settings_ section of this PR details how users configure the number
of alerts sent to the LLM. The animated counter in the empty prompt
immediately re-animates to the newly-selected number when the setting is
updated.
### Take action workflows
The _Take action_ popover displays the following actions:
- `Add to new case`
- `Add to existing case`
- `View in AI Assistant`
_Above: The Take action popover_
#### Add to new case
Clicking the `Add to new` case action displays the `Create case` flyout.
_Above: The `Add to new case` workflow_
An `Alerts were added to <case name>` toast is displayed when the case
is created:
_Above: Case creation toast_
A markdown representation of the insight is added to the case:
_Above: A markdown representation of an insight in a case_
The alerts correlated to generate the insight are attached to the case:
_Above: Insight alerts attached to a case_
#### Add to existing case
Clicking the `Add to existing case` action displays the `Select case`
popover.
_Above: The `Select case` popover_
When users select an existing case, a markdown representation of the
insight, and the alerts correlated to generate the insight are attached
to the case, as described above in the _Add to new case_ section.
#### View in AI Assistant
The `View in AI Assistant` action in the `Take action` popover, and two
additional `View in AI Assistant` affordances that appear in each
insight have the same behavior:
Clicking `View in AI Assistant` opens the assistant and adds the insight
as context to the current conversation.
_Above: An insight added as context to the current conversation_
Clicking on the insight in the assistant expands it to reveal a preview
of the insight.
_Above: An expanded insight preview in the assistant_
The expanded insight preview reveals the number of anonymzied fields
from the insight that were made available to the conversation. This
feature ensures insights are added to a conversation with the anonymized
field values.
An insight viewed in the AI assistant doesn't become part of the
conversation until the user submits it by asking a question, e.g. `How
do I remediate this?`.
Insights provided as context to a conversation are formatted as markdown
when sent to the LLM:
_Above: Insights provided as context to a conversation are formatted as
markdown_
Users may toggle anonymization in the conversation to reveal the
original field values.
_Above: Revealing the original field values of an insight added as
markdown to a conversation (animated gif)_
#### Alerts tab
The _Alerts_ tab displays the alerts correlated to generate the insight.
_Above: The alerts correlated to generate the insight in the Alerts tab_
The `View details`, `Investigate in timeline`, and overflow row-level
alert actions displayed in the Alerts tab are the same actions available
on the Cases's page's Alerts tab:
_Above: Row-level actions are the same as the Cases pages Alert's tab_
#### Investigate in Timeline
Click an insight's `Investigate in Timeline` button to begin an
investigation of an insights's alerts in Timeline. Alert IDs are queried
via the `Alert Ids` filter:
_Above: Clicking Investigte in Timeline (animated gif)_
The alerts from the insight are explained via row renderers in Timeline:
_Above: Row rendered insight alerts in Timeline_
### Attack Chain
When alerts are indicative of attack
[tactics](https://attack.mitre.org/tactics/enterprise/), those tactics
are displayed in the insights's _Attack Chain_ section:
_Above: An insight with tactics in the Attach chain_
The Attack Chain section will be hidden if an insight is not indicative
of specific tactics.
### Mini attack chain
Every insight includes a mini attack chain that visually summarizes the
tactics in an insight. Hovering over the mini attack chain reveals a
tooltip with the details:
_Above: The mini attack chain tooltip_
### Storage
The latest insights generated for each connector are cached in the
browser's session storage in the following key:
```
elasticAssistantDefault.aiInsights.cachedInsights
```
Caching insights in session storage makes it possible to immediately
display the latest when users return to to the AI insights page from
other pages in the security solution (e.g. Cases).
_Above: Cached insights from sesion storage are immediately displayed
when users navigate back to AI Insights (animated gif)_
While waiting for a connector to generate results, users may view the
cached results from other connectors.
Cached insights are immediately available, even after a full page
refresh, as long as the browser session is still active.
### `Approximate time remaining` / `Above average time` counters
Some LLMs may take seconds, or even minutes to generate insights. To
help users anticipate the time it might take to generate an insight, the
AI insights feature displays a `Approximate time remaining: mm:ss`
countdown timer that counts down to zero from the average time it takes
to generate an insight for the selected LLM:
_Above: The `Approximate time remaining: mm:ss` countdown counter
(animated gif)_
If the LLM doesn't generate insights before the counter reaches zero,
the text will change from `Approximate time remaining: mm:ss` to `Above
average time: mm:ss`, and start counting up from `00:00` until the
insights are generated:
_Above: The `Above average time: mm:ss` counter (animated gif)_
The first time insights are generated for a model, the `Approximate time
remaining: mm:ss` counter is not displayed.
Average time is calculated over the last 5 generations on the selected
connector. This is illustrated by clicking on the (?) information icon
next to the timer. The popover displays the average time, and the time
in seconds for the last 5 runs:
_Above: Clicking on the (?) information icon displays the average time,
and the duration / datetimes for the last 5 generations_
The time and duration of the last 5 generations (for each connector) are
persisted in the browser's local storage in the following key:
```
elasticAssistantDefault.aiInsights.generationIntervals
```
### Errors
When insight generation fails, an error toaster is displayed to explain
the failure:
_Above: An error toaster explains why insights generation failed_
### Feature flag
The `assistantAlertsInsights` feature flag must be enabled to view the
`AI Insights` link in the Security Solution's global navigation.
Add the `assistantAlertsInsights` feature flag to the
`xpack.securitySolution.enableExperimental` setting in
`config/kibana.yml` (or `config/kibana.dev.yml` in local development
environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantAlertsInsights']
```
### Settings
The number of alerts sent as context to the LLM is configured by
`Knowledge Base` > `Alerts` slider in the screenshot below:
- The slider has a range of `10` - `100` alerts (default: `20`)
Up to `n` alerts (as determined by the slider) that meet the following
criteria will be returned:
- The `kibana.alert.workflow_status` must be `open`
- The alert must have been generated in the last `24 hours`
- The alert must NOT be a `kibana.alert.building_block_type` alert
- The `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize
the riskiest alerts
### License
An Enterprise license is required to use AI Insights.
The following AI Insights view is displayed for users who don't have an
Enterprise license:
## How it works
- Users navigate to the AI insights page:
`x-pack/plugins/security_solution/public/ai_insights/pages/index.tsx`
- When users click the `Generate` button(s) on the AI Insights page,
insights are fetched via the `useInsights` hook in
`x-pack/plugins/security_solution/public/ai_insights/use_insights/index.tsx`.
- The `fetchInsights` function makes an http `POST` request is made to
the `/internal/elastic_assistant/insights/alerts` route. include the
following new (optional) parameters:
- `actionTypeId`, determines tempature and other connector-specific
request parameters
- `alertsIndexPattern`, the alerts index for the current Kibana Space,
e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings,
e.g. `["@timestamp", "cloud.availability_zone", "file.name",
"user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the
`Anonymization` settings, e.g. `["cloud.availability_zone", "host.name",
"user.name", ...]`
- `connectorId`, id of the connector to generate the insights
- `replacements`, an optional `Record<string, string>` collection of
replacements that always empty in the current implementation. When
non-empty, this collection enables new insights to be generated using
existing replacements.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the maximum number of alerts to generate insights from. This
numeric value is set by the slider in the user's `Knowledge Base >
Alerts` setting, e.g. `20`
- The `postAlertsInsightsRoute` function in
`x-pack/plugins/elastic_assistant/server/routes/insights/alerts/post_alerts_insights.ts`
handles the request.
- The inputs and outputs to this route are defined by the
[OpenAPI](https://spec.openapis.org/oas/v3.1.0) schema in
`x-pack/packages/kbn-elastic-assistant-common/impl/schemas/insights/alerts/post_alerts_insights_route.schema.yaml`.
```
node scripts/generate_openapi --rootDir ./x-pack/packages/kbn-elastic-assistant-common
```
- The `postAlertsInsightsRoute` route handler function in
`x-pack/plugins/elastic_assistant/server/routes/insights/alerts/post_alerts_insights.ts`
invokes the `insights-tool`, defined in
`x-pack/plugins/security_solution/server/assistant/tools/insights/insights_tool.ts`.
The `insights-tool` is registered by the Security Solution. Note: The
`insights-tool` is only used for generating insights. It is not used to
generate new insights from the context of an assistant conversation, but
that feature could be enabled in a future release.
- The `insights-tool` uses a LangChain `OutputFixingParser` to create a
[prompt
sandwich](https://www.elastic.co/blog/crafting-prompt-sandwiches-generative-ai)
with the following parts:
```
_________________________________________________
/ \
| Insight JSON formatting instructions | (1)
\ _______________________________________________/
+------------------------------------------------+
| Insights prompt | (2)
+------------------------------------------------+
/ \
| Anonymized Alerts | (3)
\_______________________________________________/
```
- The `Insight JSON formatting instructions` in section `(1)` of the prompt sandwich are defined in the `getOutputParser()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_output_parser.ts`. This function creates a LangChain `StructuredOutputParser` from a Zod schema. This parser validates responses from the LLM to ensure they are formatted as JSON representing an insight.
- The `Insights prompt` in section `(2)` of the prompt sandwich is defined in the `getInsightsPrompt()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_insights_prompt.ts`. This part of the prompt sandwich includes instructions for correlating insights, and additional instructions to the LLM for formatting JSON.
- The `Anonymized Alerts` in section `(3)` of the prompt sandwich are returned by the `getAnonymizedAlerts()` function in `x-pack/plugins/security_solution/server/assistant/tools/insights/get_anonymized_alerts.ts`. The allow lists configured by the user determine which alert fields will be included and anonymized.
- The `postAlertsInsightsRoute` route handler returns the insights generated by the `insights-tool` to the client (browser).
- Insights are rendered in the browser via the `Insight` component in `x-pack/plugins/security_solution/public/ai_insights/insight/index.tsx`
- The `AiInsights` tab in `x-pack/plugins/security_solution/public/ai_insights/insight/tabs/ai_insights/index.tsx` includes the _Summary_ and _Details_ section of the Insight.
- The `InsightMarkdownFormatter` in `x-pack/plugins/security_solution/public/ai_insights/insight_markdown_formatter/index.tsx` renders hover actions on entities (like hostnames and usernames) and other fields in the insight.
- The `Insight` component makes use of the `useAssistantOverlay` hook in `x-pack/packages/kbn-elastic-assistant/impl/assistant/use_assistant_overlay/index.tsx` to register the insight as context with the assistant. This registration process makes it possible to view insights in the assistant, and ask questions like "How do I remediate this?". In this PR, the `useAssistantOverlay` hook was enhanced to accept anonymizaton replacements. This enables an assistant conversation to (re)use replacements originally generated for an insight.
## Summary
Create a new Asset Criticality page for updating asset criticality by
file upload.
Flaky test runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5662
Server side PR: https://github.com/elastic/kibana/pull/179930f524b5e8-8efa-40c7-8e43-45cf43decefb
The new page has three steps. You can access the page by going to
Security -> Manage -> Asset Criticality.
<img
src="080a51bf-20e9-4f4b-84b2-13fe1cfdc1d5"
width="400" />
### File picker Step:
<img
src="e3aea4b8-2083-49a4-b4bf-dbb645fb463b"
width="400" />
### File validation step
<img
src="54b3018e-ef0e-4ac4-93b2-67ae02743eb8"
width="400" />
### Result step
<img
src="aa47a7af-1108-4ad6-8dc0-f728e0187026"
width="400" />
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
a-docker)
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
## How to test it?
* Open the page
* Upload a valid CSV file
* Check if everything is ok on the validation step
* Click Assign
* Check if the success message is displayed
* Open the alert flyout for an updated asset and check if it has the new
value
## What is not included?
* Serverless
* Disable the feature when asset criticality advanced setting is
disabled
## Code owners files:
<details>
<summary>elastic/docs</summary>
* packages/kbn-doc-links/src/get_doc_links.ts
* packages/kbn-doc-links/src/types.ts
</details>
<details>
<summary>elastic/security-defend-workflows</summary>
* x-pack/plugins/security_solution/public/management/links.ts
</details>
<details>
<summary>elastic/security-detection-engine</summary>
* x-pack/test/security_solution_cypress/cypress/urls/navigation.ts
</details>
<details>
<summary>elastic/security-detections-response</summary>
*
x-pack/test/security_solution_cypress/cypress/fixtures/asset_criticality.csv
</details>
<details>
<summary>elastic/security-engineering-productivity</summary>
*
x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/asset_criticality_upload_page.cy.ts
*
x-pack/test/security_solution_cypress/cypress/fixtures/asset_criticality.csv
*
x-pack/test/security_solution_cypress/cypress/screens/asset_criticality.ts
*
x-pack/test/security_solution_cypress/cypress/tasks/asset_criticality.ts
* x-pack/test/security_solution_cypress/cypress/urls/navigation.ts
</details>
<details>
<summary>elastic/security-threat-hunting</summary>
*
x-pack/test/security_solution_cypress/cypress/fixtures/asset_criticality.csv
</details>
<details>
<summary>elastic/security-threat-hunting-investigations</summary>
*
x-pack/plugins/security_solution/public/resolver/view/panels/node_list.tsx
* x-pack/test/security_solution_cypress/cypress/urls/navigation.ts
</details>
---------
Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>
closes https://github.com/elastic/kibana/issues/178843
## 📝 Summary
This PR adds actions to the integration section in the dataset quality
flyout.
These actions navigate to different integration-related pages.
The Dashboards action is only visible if the integration does have
dashboard assets installed, otherwise its hidden.
## 🎥 Demo
91c417e6-be7d-45eb-91dc-2f5b29e7aeb5
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR moves the AI Assistant Management plugin into x-pack to
co-locate it with the other assistant plugins and to make it possible to
statically import from the other assistant plugins. This is not
currently possible because the Management plugin is in OSS and the other
plugins are in xpack.
Fixes https://github.com/elastic/kibana/issues/176420
## 🍒 Summary
This PR copies the SLO code that was inside the Observability app into
its own app under `observability-solution/slo` folder.
4f6b8dfb-9612-4d30-ad50-4ee5c55a9c32
## ✔️ Acceptance criteria
- URL of new app: `app/slos`
- Design and functionality are not changed.
- Git history has been retained for all files in
`x-pack/plugins/observability_solution/slo`.
- SLO should appear on server less
- SLO code inside `observability_solution/observability` code has been
removed. A new clean up round might be needed though for possible
leftovers.
- Burn rate rule is registered within the new slo app
- SLO embeddables are moved inside the new slo app
- overview
- alerts embeddable
- error budget burn down
- Alerts table configuration registration for slo details page and
alerts table embeddable is still done in the observability app. Response
Ops team is working on removing the need to register the alert table
anyway
- Slo app is wrapped into `ApplicationUsageTrackingProvider` which will
send slo `Application usage` information tracked by the `slo` appId
- Redirect old `app/observability/slos` route to `app/slos`
- Rename old `xpack.observability.slo` keys to `xpack.slo` in the
translation files
## 🌮 How to test
Design and functionality didn't change, so simply navigate to existing
slo pages and try to break it
- Slo list page
- group by
- unified search
- toggle buttons
- actions
- Slo creation
- try group by as well
- Slo detail page
- Actions on top
- navigate to overview and alerts tabs
- Create SLO flyout in Logs Explorer
- Create burn rate rules and verify they appear on rules page
- Verify SLO alerts appear on Alerts page and slo details page
- Embeddables
- Through the dashboard app
- Using the attach to dashboard action on the slo card item on slo list
page and the error budget burn down chart on the slo detail page
- SLOs only for platinum users
- Permissions
- Spaces
## TODO
- [x] Move slo stuff from observability folder to new slo plugin
- [x] Remove old slo stuff from observability folder
- [x] Update references
- [x] Fix typescript and eslint errors
- [x] Paths
- [x] Locators
- [x] Burn rate rule registration
- [x] Embeddable Alerts table configuration registration
- [x] Embeddables
- [x] Translations
- [x] Verify plugin.ts files contain all registration logic
- [x] public
- [x] server
- [x] Final cleanup for observability folder
- [x] Run tests
- [x] Application Usage (Telemetry)
- [x] Permissions
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: shahzad31 <shahzad31comp@gmail.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
The AppEx Management team was recently renamed on GitHub from
`platform-deployment-management` to `kibana-management`. This PR updates
the Codeowners file and all references to the team name.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/synthetics-dev/issues/289.
The Synthetics plugin now consumes the Serverless breadcrumbs API when
Kibana is running in Serverless mode.
This patch will re-use all the Synthetics plugin's existing breadcrumb
logic, with some minor modifications. At plugin start time, Synthetics
will detect if Kibana is stateful or stateless, and either assign the
`serverless.setBreadcrumbs` or `core.chrome.setBreadcrumbs` function to
the props that get propagated to the `SyntheticsSettingsContext`. The
breadcrumb hooks in the React code will now reference this field, rather
than directly pulling `chrome.setBreadrumbs` from the Kibana services
object as it did before.
This patch also introduces a new deep link for the Settings page, and
adds an associated object to the Observability project nav tree.
One other thing to note is that this patch will also require Synthetics
to add the `serverless` plugin as a dependency in its `kibana.json`
file.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
closes [176602](https://github.com/elastic/kibana/issues/176602)
## Summary
This PR fixes a few small issues that became more evident in the
serverless offering
### Serverless
- Breadcrumbs
<img width="1727" alt="image"
src="5808ccd2-8733-406a-8a98-7aedb3e21e8a">
<img width="1727" alt="image"
src="bac99d0c-8146-4946-acc8-7b52133d79fb">
<img width="1727" alt="image"
src="34442e1f-f7ac-425f-9712-ad0a2188cace">
- Asset Details Page Template
| before | after |
| --- | --- |
|<img width="1220" alt="image"
src="4bb92ff0-5e27-4ca9-b177-ba2715996648">|<img
width="1227" alt="image"
src="3d6f1783-01a1-4413-8acf-640b0d6af7f1">
|
The page now uses the `PageTemplate` from `observability-shared`, which
is what other pages in observability use.
### Other fixes
- Spacing between header and unified search in the Hosts View
| before | after |
| --- | --- |
|<img width="884" alt="image"
src="49727fc5-0f9b-4ee4-b560-b489c175b1ba">|<img
width="885" alt="image"
src="858e6930-6210-42a1-8414-bb6e5d60933c">
|
The default spacing is 24px. For some reason, the hosts view had a 12px
space between the 2 components.
- Breadcrumb (still works as expected)
<img width="1714" alt="image"
src="ef019d40-2a88-4920-a5dc-ad72a8485536">
### How to test
- Start a serverless Kibana, ES instances and run metricbeat with system
module enabled
- run `yarn es serverless --projectType=oblt` and `yarn serverless-oblt`
- Navigate to Inventory pages and APM Settings
- Check the changes described above
- Start a stateful Kibana instance
- Navigate to inventory pages and APM Settings
- Check the changes described above
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#175770
This PR introduces support for a data view locator which allows reaching
the Observability Logs Explorer app and accessing an existing data view.
To make this possible, it was required to introduce a look-up step on
our state machine that, whenever a data view selection is not resolved
yet, tries to retrieve it by the passed `id`.
In case the data view we try to access does not exist, we'll fall back
to the default `All logs` selection. The look-up step also considers
whether the pointed data view is of logs type or not, redirecting to
Discover in case the data view does not match our criteria.
## Usage
As we do for the existing locators for the Observability Logs Explorer
app, this new locator also supports the default parameters to set the
`timeRange`, `refreshInterval`, `query`, `columns`, `filters` and
`filterControls` options.
```ts
const obsLogsExplorerDataViewLocator = share.url.locators.get<ObsLogsExplorerDataViewLocatorParams>(OBS_LOGS_EXPLORER_DATA_VIEW_LOCATOR_ID);
const urlToLogsExplorer = obsLogsExplorerDataViewLocator?.getRedirectUrl({
id: 'data-view-unique-id',
timeRange: {
from: 'now-1d',
to: 'now',
},
integration: dataStreamStat.integration?.name,
filterControls: {
namespace: {
mode: 'include',
values: [dataStreamStat.namespace],
},
},
});
```
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
> Derived from https://github.com/elastic/kibana/pull/173068
> Further addresses https://github.com/elastic/kibana/issues/169873
## Summary
- We're adding a link to the No Data Prompt to send a person to explore
their data using ES|QL if there exists any ingested data but no data
view.
- This PR populates the query bar with the first available index (some
special handling for logs index.
2a9edec678)
- All consumers of the prompt/no data view are updated in this PR.
- [x] ~There's an issue where, if you're in Discover, clicking the link
won't refresh the page. .~ This is fixed on Discover side by
reinitializing the state container when user clicks the "try es|ql" link
and URL state updates.
- [x] ~There is an issue that you can save the es|ql chart from
Discover, but Dashboard's empty screen blocks the navigation because
data views don't exist
https://github.com/elastic/kibana/pull/174316#issuecomment-1914722657~.
This is fixed by allowing the dashboard to work without the default data
view. Hopefully, this won't lead to major issues
- [x] ~ES|QL panels can't be created without the default data views~
this is fixed by trying to fallback to an ad-hoc dataview, plan to move
that code to the utils introduced here
https://github.com/elastic/kibana/pull/174736/
- [x] fix circular deps
- [x] Add functional tests
## Visuals
af3592c1-f4c8-43bb-a128-3268b7761367
### Storybook Stories
#### Can access ES|QL
#### Cannot access (e.g. preview is unavailable - _not implemented_)
---------
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Rachel Shen <rshen@elastic.co>
Co-authored-by: Anton Dosov <anton.dosov@elastic.co>
## Summary
This PR adds support for ES|QL queries in Data visualizer.
<img width="1695" alt="Screenshot 2024-01-26 at 17 07 59"
src="8a54b859-60d6-4c47-b3dd-e5f3ed43b6b0">
<img width="1695" alt="Screenshot 2024-01-26 at 17 12 39"
src="32fd08e4-7f3b-43e6-81a7-7ec4e777bac0">
a3f540e9-461d-4ebc-bd69-de4ffa2bc554
### Changes:
- Add a new card from the Data visualizer main page
- Add a link from the ML navigation
<img width="1717" alt="Screenshot 2024-01-08 at 18 03 50"
src="832f7890-4ce6-44c1-ab87-cde01f4bf1c0">
- Added a new button to Use ES|QL
<img width="1714" alt="Screenshot 2024-01-09 at 11 23 09"
src="a38a9360-6691-4f3b-a824-8481ab543250">
- Support for **keyword**, **text**, **numeric**, **boolean**, **date**,
and **ip** fields
<img width="1714" alt="Screenshot 2024-01-09 at 11 24 38"
src="b122ee5c-1500-4e2b-9434-e64b0b6ea3be">
<img width="1441" alt="Screenshot 2024-01-09 at 11 25 25"
src="eb35ee78-8a34-467e-84da-2026b01fcda1">
<img width="969" alt="Screenshot 2024-01-09 at 11 44 02"
src="d0f9947d-2b2c-4c14-89ba-9fc5d0a2bf64">
<img width="981" alt="Screenshot 2024-01-10 at 12 01 42"
src="aa5a8d44-7447-41fc-a544-d1b626bf8bce">
- Default to user's fieldFormats for fields that are dynamic generated
by ES|QL, else use Data view's format
- Default to Data view's setting (e.g. type `bytes` in this case for
field `bytes_normal_counter`)
<img width="1037" alt="Screenshot 2024-01-10 at 12 10 38"
src="9fb7e31c-f397-4209-a463-e1a43fe27ffd">
- Default to user's fieldFormats formatting for dynamically generated
fields (e.g. type `number` in this case for field `avg_price`)
<img width="1283" alt="Screenshot 2024-01-10 at 12 01 03"
src="acc25358-50bb-4237-9476-86067ef0badf">
- Add a new UI control to allow users to limit analysis to 5,000 -
10,000 - 100,000 - 1,000,000, rows. This speeds up fetching of the stats
for big data sets and avoid potential circuit breaking exceptions.
- Break overall stats request into smaller parallel requests (which
prevent time out or payload too big due by too many fields), at 10
requests at a time
- Break field stats for individual fields into more efficient batches
(which prevent time out or payload too big due by too many fields), at
10 requests at at ime
- Improve error handling by propagating up the error AND the ES|QL
request in both the UI and the developer's console (for better
debugging)
- Improve error handling in field stats rows: If one field, or a group
of fields, say 'keyword' fields fail to fetch for some reasons, it will
show error for that field but not affect all other fields.
<img width="1690" alt="Screenshot 2024-01-26 at 16 04 28"
src="6e240e12-76b4-42d6-b3be-c05342d76df9">
- Add deep linking in the top search bar
<img width="1185" alt="Screenshot 2024-01-26 at 16 56 49"
src="4f24df68-edc5-41c5-b2ed-d6150ba1e20b">
- More robust support for keyword fields with geo data
<img width="1438" alt="Screenshot 2024-01-26 at 16 55 01"
src="3b97925b-ca28-4952-8082-8d3242e3cb3f">
### Todos:
- [x] Add earliest/latest for date time fields -> Current blocker:
escape special characters in esql variable names
- [x] Fix formatting of numbers for dynamic query, where we don't know
the formatting based on the data view
- [x] Fix date time 'Update' not updating until Refresh is clicked
- [x] Better optimization to not fetch distribution & expanded row
content for pages that are not visible
### Good to have:
- [ ] Investigate bringing back the +/- filter buttons (either by
modifying the ES|QL query directly or by adding separate DSL filters?)
------------
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Renames test subjects and page objects
- Renames test folders from `observability_log_explorer` to
`observability_logs_explorer`
- Changes app url from `observability-log-explorer` to
`observability-logs-explorer` and adds another app for redirects
Related to https://github.com/elastic/kibana/issues/171991
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/synthetics-dev/issues/290.
~Creating this PR as a place to experiment with Synthetics running
against the Serverless platform. Main goals are to find areas of the
codebase that require revision and get as much of Synthetics functioning
as possible without access to public locations.~
This adds the necessary config and other features to make Synthetics
workable as part of the oblt serverless project. Notably, we aren't
including the `xpack.uptime.enabled` flag here, because we are not ready
to expose the plugin to production users yet. We're going to enable the
plugin on a per-env basis using other means to start, and when we are
ready to expose the plugin generally in prod, we will add that flag.
Also adds nav and fixes a few other things that were broken for
serverless specifically.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Resolves#172283
## Summary
Fixes breadcrumbs and document title for Log Explorer app.
## Screenshot
<img width="970" alt="Screenshot 2023-12-05 at 17 29 53"
src="d80fc9de-fbad-4c84-a61d-8ff8c5ad7271">
## Summary
This PR adds a bunch of plugins to help manage AI Assistant Management
settings.
It offers a 'selection' plugin inside Stack Management where a user can
select which AI Assistant she wants to manage.
The Security team can hook into this one, so settings for both AI
Assistants can be accessed from inside one place inside Stack
Management.
This PR also adds the plugin to manage settings for the AI Assistant for
Observability, including Knowledge Base management. This plugin is
available both in Stack Management (stateful) and Project Settings
(serverless).
## What it looks like
51392ec5-05c9-4947-9bf2-810d8d0b7525
## Detailed
1. **Adds a Stack Management plugin**
(`/src/plugins/ai_assistant_management/selection`). Its primary function
is to render a selection screen to help users navigate to the settings
plugin for the AI Assistant for a specific solution. This plugin is
displayed in Stack Management, which is only available in stateful
versions of Kibana.
2. **Adds a AI Assistant for Observability Settings plugin**
(`/src/plugins/ai_assistant_management/observability`). This plugin
allows management of specific Observability AI Assistant settings. It is
available in stateful versions of Kibana (via the aforementioned Stack
Management plugin) or in serverless versions via Project Management.
3. **Knowledge Base management for Observability AI Assistant**: The AI
Assistant for Observability Settings plugin has a Knowledge Base tab,
which allows users to add / read / update / delete and bulk import
entries into the Knowledge Base of the Observability AI Assistant.
4. **Moving of KB endpoints in Observability AI Assistant plugin**: KB
endpoints and functions were located in the same folder. As this PR adds
new endpoints for the KB for CRUD operations, it also moves the existing
ones from the function folder into a dedicated one so there's a clearer
distinction between kb and functions.
5. **Adding of GenAI Connector inside Chat Flyout**: If the user has
admin rights, it is possible to set up a GenAI connector from within the
Observability AI Assistant Chat Flyout. This provides a faster and more
seamless onboarding experience. If the user does not, she will be
redirected to the Settings page.
## Bug fixes
* Fixes chat item styling issues (padding, background color).
## How to test
* Check if the Stack Management plugin works on stateful
* Check if the AI Assistant Settings plugin works on stateful +
serverless
* Check if CRUD operations on KB work
* Check if searching on KB entries work
* Check if its possible to navigate to KB tab directly
(`app/management/kibana/aiAssistantManagementObservability?tab=knowledge_base`)
## Todo
- [x] Add sorting to getEntries
- [x] Add params for tab routing
- [x] Add unit tests
- [ ] Add API tests
- [ ] Add fallback for already indexed entries when searching
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Updates new teams as codeowners for Observability team changes.
Also took the opportunity to:
- Delete some paths that no longer exist
- Split infra code ownership between teams (from #168992)
## Summary
Closes https://github.com/elastic/kibana/issues/166968
Adds a toast message asking for (optional) feedback when navigating from
Observability Onboarding > Observability Log Explorer (via the Explore
Logs button) after three minutes. The origin is attached to the history
location state as part of the Locator.
## State machine
A lightweight state machine handles the origin interpreting. We will
very likely have more origins in the future.
## Reviewer hints
- Start the flow at `/app/observabilityOnboarding/customLogs`, continue
to the last step of the wizard, click the "Explore logs" button to
navigate to the Observability Log Explorer.
- You can alter the `FEEDBACK_DELAY` constant for easier testing.
- **Only** the onboarding origin should trigger the feedback toast.
- Moves the feedback link to Observability shared to avoid cyclic
dependency issues.
## Screenshot
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani01@gmail.com>
## Summary
This PR adds the `Infrastructure` item to the serverless side-nav
_infra enabled_
<img width="1200" alt="image"
src="baaf95ae-2960-4cdc-8d90-c95018d2fa24">
_infra disabled_
<img width="1200" alt="image"
src="386635b3-d095-4f62-9831-a37a723b69c6">
Stateful Kibana isn't affected by this change.
### How to test
- Start a local es instance: `yarn es serverless --kill --clean
--license trial --ssl`
- Enable `infra` in the `serverless.oblt.dev.yml` file:
- `xpack.infra.enabled: true`
- Start a local kibana instance: `yarn serverless-oblt --ssl` and see if
the side nav contains the Infrastructure item
- Disable infra and check if the side nav does not contain the
infrastructure item.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR adds data drift detection workflow from Trained models to Data
comparison view. It also renames Data comparison to Data Drift.
**From the new map view in Trained model list:**
- Clicking on the index icon in the map view will give an option/action
to Analyze data drift
a68163ab-8a83-4378-8cf3-ea49f4480a06
- If model has detected related indices, it will also give an option to
Analyze data drift in the Transform actions
**From the data comparison/drift page:**
- Default screen with list of available data views and saved search will
be shown
<img width="1470" alt="Screen Shot 2023-09-07 at 00 22 01"
src="db13b8b7-9d90-4220-b03e-9f9d12ab53e9">
- But can also customize index patterns for the data sets to analyze.
Upon 'analyzing', a new data view will be created if needed (either
permanently or temporarily).
<img width="1271" alt="Screen Shot 2023-08-29 at 16 56 57"
src="e000e920-162b-4369-8762-70b6244e50e7">
<img width="1470" alt="Screen Shot 2023-09-07 at 00 22 49"
src="6577a530-c3b0-4ab9-95e4-d1d8fd1c9f0a">
- If there exists a data view with exact combination of index patterns
and time field, it will use that data view
- If there exists a data view with the same index patterns but different
time field, it will create a new data view with name
`{referencePattern},{comparisonPattern}-{timeField}`
- If no data view exists that matches, it will create a new data view
with name `{referencePattern},{comparisonPattern}`
## For reviewers:
- **appex-sharedux**: [Small change in the exported type interface for
BaseSavedObjectFinder](https://github.com/elastic/kibana/pull/162853/files#diff-5e2e62df8aba5ac9445962bfa00eee933a386110d0a24dfe6ac0f300a796ccc3)
to correctly list `children` as an accepted prop. This prop which is
used for the `toolsRight`.
- **security-solution**: Renaming of `Data comparison` to `Data Drift`
## Tests:
[Flaky test suite runner with Data Drift
test](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3216#018accc2-d33b-4cd6-a178-589e6698b675)
... successful after 50 runs✅
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
closes https://github.com/elastic/kibana/issues/164995
closes https://github.com/elastic/kibana/issues/165618
closes https://github.com/elastic/kibana/issues/166596
## 📝 Summary
### Observability Log Explorer Locators:
This PR adds 2 new customized locators to the Observability log explorer
profile. At the moment we implemented:
1- Single dataset selector locator
2- All dataset selector locator
With more locators to come in the future depending on the use cases.
### Log Explorer Locators:
We also added a log explorer locator that navigates to discover, this
can be used in case the **Observability Log Explorer** plugin is
disabled.
### Logs Onboarding:
The PR also replaces the temp navigation to the default discover we
implemented for[ 8.10
here](https://github.com/elastic/kibana/pull/163218) with the above new
Observability Log Explorer locators.
### APM:
After [disabling infra plugin in serverless
projects](https://github.com/elastic/kibana/pull/165289), APM links to
infra locators in serverless have been replaced to use the above
locators.
### Observability Landing Page:
The landing page now redirects to the Log Explorer if `logs-*-*` has
data in it, otherwise the flow continues as before.
### Necessary Refactoring:
To avoid the circular dependency between `ObservabilityLogExplorer` &
`ObservabilityOnboarding` after each one using the other's locator and
importing the necessary types, I moved the type definition for all
locators in the `deeplinks` package.
## ✅ Testing
- Onboarding Wizard in Serverless and Stateful
1. Navigate to the onboarding flow `/app/observabilityOnboarding/`
2. Choose either System logs or Stream log files
3. Go through the onboarding wizard
4. Click the Explore logs button at the end
5. You should be redirected to observability log explorer with the
integration and dataset preselected.
- APM links in Serverless
1. Navigate to APM and click on the logs links as shown in the Demos
below
2. All links should navigate to Observability Log Explorer with the
queries set in the search bar.
## 🎥 Demos
- APM Serverless
7161364e-333f-4ac4-87d5-7f1ffec699b3
- APM Stateful
058c9587-b766-4d4f-a73d-50fd381be4bb
- Onboarding Serverless
ee1cab42-f91c-4558-aa5f-4fa7e8963427
- Onboarding Stateful
a376a12b-499b-4488-a75a-d06e81f8e21d
- Observability Landing Page
c1c084ca-b1b1-4c4b-a4e6-ae8e157dcf57
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani01@gmail.com>
This adds an empty connectors page to the Elasticsearch project. The
Enterprise Search plugin exports relevant connectors data and functions,
the Search plugin simply consumes these.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
