# Backport
This will backport the following commits from `main` to `8.12`:
- [[DOCS] Add new sub feature privilege to prevent access to the cases
settings (#174223)](https://github.com/elastic/kibana/pull/174223)
<!--- Backport version: 9.4.3 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Lisa
Cawley","email":"lcawley@elastic.co"},"sourceCommit":{"committedDate":"2024-01-08T15:58:38Z","message":"[DOCS]
Add new sub feature privilege to prevent access to the cases settings
(#174223)","sha":"ee0cb0b5418ce83ccc7c8681e2da6d0d24534ec6","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","docs","Feature:Cases","Team:obs-ux-management","v8.12.1","v8.13.0"],"title":"[DOCS]
Add new sub feature privilege to prevent access to the cases
settings","number":174223,"url":"https://github.com/elastic/kibana/pull/174223","mergeCommit":{"message":"[DOCS]
Add new sub feature privilege to prevent access to the cases settings
(#174223)","sha":"ee0cb0b5418ce83ccc7c8681e2da6d0d24534ec6"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/174223","number":174223,"mergeCommit":{"message":"[DOCS]
Add new sub feature privilege to prevent access to the cases settings
(#174223)","sha":"ee0cb0b5418ce83ccc7c8681e2da6d0d24534ec6"}}]}]
BACKPORT-->
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
# Backport
This will backport the following commits from `main` to `8.12`:
- [[Security Solution] [Elastic AI Assistant] Delete the _Retrieval
Augmented Generation (RAG) for Alerts_ Feature Flag
(#173809)](https://github.com/elastic/kibana/pull/173809)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Andrew
Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2023-12-21T23:01:15Z","message":"[Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag (#173809)\n\n## [Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag\r\n\r\nThis PR deletes the
`assistantRagOnAlerts` feature flag introduced in [[Security Solution]
[Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542).\r\n\r\nDeleting
the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle
available in the assistant settings, per the screenshot
below:\r\n\r\n\r\n\r\nThis
PR should not be merged until the docs describing the feature in
<https://github.com/elastic/security-docs/issues/4456> have been
merged.\r\n\r\nThis PR also includes @benironside improvements to the
Alerts setting in the video
below:\r\n\r\n73ea2717-ad2a-4998-afe2-cc154d8d19a9\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n1) Delete the
following `assistantRagOnAlerts` feature flag from your local
`config/kibana.dev.yml`:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n2) Start Kibana\r\n\r\n3)
Generate alerts with a variety of severity (e.g. `low`, `medium`,
`high`, and `critical`)\r\n\r\n4) Navigate to Security >
Alerts\r\n\r\n5) Click the `AI Assistant` button to open the
assistant\r\n\r\n6) Click the `X` button to clear the
conversation\r\n\r\n7) Click the assistant's `Settings` gear\r\n\r\n8)
Click the `Knowledge Base` category\r\n\r\n**Expected result**\r\n\r\n-
The `Alerts` toggle shown in the screenshot below is
available\r\n\r\n\r\n\r\n9)
Click the `Alerts` toggle to enable the feature\r\n\r\n10) Click the
`Save` button to close settings\r\n\r\n11) Enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- A response with alert
counts grouped by workflow status will be returned, similar to the
example below:\r\n\r\n```\r\nYou currently have 48 open alerts in your
system. These are categorized by severity as following: 19 of them are
low severity, 16 are high severity, 12 are of medium severity and 1 is
of critical severity. There is also 1 critical severity alert which is
acknowledged.\r\n```\r\n\r\n12) Enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\nA response with alert
details, similar to the following is returned:\r\n\r\n```\r\nBased on
the latest information, the alerts to prioritize first are those related
to a mimikatz process starting on the hosts, which have a critical
severity and the highest risk score of 99. There are also a series of
alerts related to an EQL process sequence with a high severity and risk
scores of 73. There is one alert about an Elastic Endpoint Security
alert with a medium severity and risk score of 47.\r\n```\r\n\r\n13)
Once again, click the assistant's `Settings` gear\r\n\r\n14) Click the
`Knowledge Base` category\r\n\r\n15) Click the `Alerts` toggle to
disable the feature\r\n\r\n16) Click the `Save` button to close
settings\r\n\r\n17) Once again, enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with a breakdown of alerts by severity. Instead it replies with
something like the following example response:\r\n\r\n```\r\nI'm sorry
for any confusion, but as an AI, I don't have real-time access to your
data or system to provide the number of your current open alerts. You
can check your Elastic Security dashboard or use the appropriate
querying commands to get the updated count of your open
alerts.\r\n```\r\n\r\n18) One more time, enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with alert details. Instead it replies with something like the
following example response:\r\n\r\n```\r\nAs an AI model, I don't have
the capability to access real-time data. However, when it comes to
managing alerts in Elastic Security, it's generally recommended to first
look at the ones with the highest severity and risk score. Alerts
related to malware, unauthorized access attempts, and abnormal data
transfers or process activities, for example, may need immediate
attention due to their potential high
impact.\r\n```","sha":"ec05dd7afddaef353d27f0bcbc7046ff09c0a5d6","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","Team:Threat Hunting:Investigations","Feature:Elastic
AI
Assistant","v8.12.0","v8.13.0"],"number":173809,"url":"https://github.com/elastic/kibana/pull/173809","mergeCommit":{"message":"[Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag (#173809)\n\n## [Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag\r\n\r\nThis PR deletes the
`assistantRagOnAlerts` feature flag introduced in [[Security Solution]
[Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542).\r\n\r\nDeleting
the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle
available in the assistant settings, per the screenshot
below:\r\n\r\n\r\n\r\nThis
PR should not be merged until the docs describing the feature in
<https://github.com/elastic/security-docs/issues/4456> have been
merged.\r\n\r\nThis PR also includes @benironside improvements to the
Alerts setting in the video
below:\r\n\r\n73ea2717-ad2a-4998-afe2-cc154d8d19a9\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n1) Delete the
following `assistantRagOnAlerts` feature flag from your local
`config/kibana.dev.yml`:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n2) Start Kibana\r\n\r\n3)
Generate alerts with a variety of severity (e.g. `low`, `medium`,
`high`, and `critical`)\r\n\r\n4) Navigate to Security >
Alerts\r\n\r\n5) Click the `AI Assistant` button to open the
assistant\r\n\r\n6) Click the `X` button to clear the
conversation\r\n\r\n7) Click the assistant's `Settings` gear\r\n\r\n8)
Click the `Knowledge Base` category\r\n\r\n**Expected result**\r\n\r\n-
The `Alerts` toggle shown in the screenshot below is
available\r\n\r\n\r\n\r\n9)
Click the `Alerts` toggle to enable the feature\r\n\r\n10) Click the
`Save` button to close settings\r\n\r\n11) Enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- A response with alert
counts grouped by workflow status will be returned, similar to the
example below:\r\n\r\n```\r\nYou currently have 48 open alerts in your
system. These are categorized by severity as following: 19 of them are
low severity, 16 are high severity, 12 are of medium severity and 1 is
of critical severity. There is also 1 critical severity alert which is
acknowledged.\r\n```\r\n\r\n12) Enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\nA response with alert
details, similar to the following is returned:\r\n\r\n```\r\nBased on
the latest information, the alerts to prioritize first are those related
to a mimikatz process starting on the hosts, which have a critical
severity and the highest risk score of 99. There are also a series of
alerts related to an EQL process sequence with a high severity and risk
scores of 73. There is one alert about an Elastic Endpoint Security
alert with a medium severity and risk score of 47.\r\n```\r\n\r\n13)
Once again, click the assistant's `Settings` gear\r\n\r\n14) Click the
`Knowledge Base` category\r\n\r\n15) Click the `Alerts` toggle to
disable the feature\r\n\r\n16) Click the `Save` button to close
settings\r\n\r\n17) Once again, enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with a breakdown of alerts by severity. Instead it replies with
something like the following example response:\r\n\r\n```\r\nI'm sorry
for any confusion, but as an AI, I don't have real-time access to your
data or system to provide the number of your current open alerts. You
can check your Elastic Security dashboard or use the appropriate
querying commands to get the updated count of your open
alerts.\r\n```\r\n\r\n18) One more time, enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with alert details. Instead it replies with something like the
following example response:\r\n\r\n```\r\nAs an AI model, I don't have
the capability to access real-time data. However, when it comes to
managing alerts in Elastic Security, it's generally recommended to first
look at the ones with the highest severity and risk score. Alerts
related to malware, unauthorized access attempts, and abnormal data
transfers or process activities, for example, may need immediate
attention due to their potential high
impact.\r\n```","sha":"ec05dd7afddaef353d27f0bcbc7046ff09c0a5d6"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173809","number":173809,"mergeCommit":{"message":"[Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag (#173809)\n\n## [Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag\r\n\r\nThis PR deletes the
`assistantRagOnAlerts` feature flag introduced in [[Security Solution]
[Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542).\r\n\r\nDeleting
the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle
available in the assistant settings, per the screenshot
below:\r\n\r\n\r\n\r\nThis
PR should not be merged until the docs describing the feature in
<https://github.com/elastic/security-docs/issues/4456> have been
merged.\r\n\r\nThis PR also includes @benironside improvements to the
Alerts setting in the video
below:\r\n\r\n73ea2717-ad2a-4998-afe2-cc154d8d19a9\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n1) Delete the
following `assistantRagOnAlerts` feature flag from your local
`config/kibana.dev.yml`:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n2) Start Kibana\r\n\r\n3)
Generate alerts with a variety of severity (e.g. `low`, `medium`,
`high`, and `critical`)\r\n\r\n4) Navigate to Security >
Alerts\r\n\r\n5) Click the `AI Assistant` button to open the
assistant\r\n\r\n6) Click the `X` button to clear the
conversation\r\n\r\n7) Click the assistant's `Settings` gear\r\n\r\n8)
Click the `Knowledge Base` category\r\n\r\n**Expected result**\r\n\r\n-
The `Alerts` toggle shown in the screenshot below is
available\r\n\r\n\r\n\r\n9)
Click the `Alerts` toggle to enable the feature\r\n\r\n10) Click the
`Save` button to close settings\r\n\r\n11) Enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- A response with alert
counts grouped by workflow status will be returned, similar to the
example below:\r\n\r\n```\r\nYou currently have 48 open alerts in your
system. These are categorized by severity as following: 19 of them are
low severity, 16 are high severity, 12 are of medium severity and 1 is
of critical severity. There is also 1 critical severity alert which is
acknowledged.\r\n```\r\n\r\n12) Enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\nA response with alert
details, similar to the following is returned:\r\n\r\n```\r\nBased on
the latest information, the alerts to prioritize first are those related
to a mimikatz process starting on the hosts, which have a critical
severity and the highest risk score of 99. There are also a series of
alerts related to an EQL process sequence with a high severity and risk
scores of 73. There is one alert about an Elastic Endpoint Security
alert with a medium severity and risk score of 47.\r\n```\r\n\r\n13)
Once again, click the assistant's `Settings` gear\r\n\r\n14) Click the
`Knowledge Base` category\r\n\r\n15) Click the `Alerts` toggle to
disable the feature\r\n\r\n16) Click the `Save` button to close
settings\r\n\r\n17) Once again, enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with a breakdown of alerts by severity. Instead it replies with
something like the following example response:\r\n\r\n```\r\nI'm sorry
for any confusion, but as an AI, I don't have real-time access to your
data or system to provide the number of your current open alerts. You
can check your Elastic Security dashboard or use the appropriate
querying commands to get the updated count of your open
alerts.\r\n```\r\n\r\n18) One more time, enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with alert details. Instead it replies with something like the
following example response:\r\n\r\n```\r\nAs an AI model, I don't have
the capability to access real-time data. However, when it comes to
managing alerts in Elastic Security, it's generally recommended to first
look at the ones with the highest severity and risk score. Alerts
related to malware, unauthorized access attempts, and abnormal data
transfers or process activities, for example, may need immediate
attention due to their potential high
impact.\r\n```","sha":"ec05dd7afddaef353d27f0bcbc7046ff09c0a5d6"}}]}]
BACKPORT-->
Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
# Backport
This will backport the following commits from `main` to `8.12`:
- [[Security Solution] [Elastic AI Assistant] Include acknowledged
alerts in the context sent to the LLM (Retrieval Augmented Generation
(RAG) for Alerts)
(#173121)](https://github.com/elastic/kibana/pull/173121)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Andrew
Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2023-12-13T17:39:08Z","message":"[Security
Solution] [Elastic AI Assistant] Include acknowledged alerts in the
context sent to the LLM (Retrieval Augmented Generation (RAG) for
Alerts) (#173121)\n\n## [Security Solution] [Elastic AI Assistant]
Include `acknowledged` alerts in the context sent to the LLM (Retrieval
Augmented Generation (RAG) for Alerts)\r\n\r\nThis PR updates the query
used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented
Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542) to include
alerts with a `kibana.alert.workflow_status` value of
`acknowledged`.\r\n\r\nThe query previously only returned alerts with a
status of `open`. This change ensures both `open` and `acknowledged`
alerts are provided as context to the LLM.\r\n\r\n### Updated
Anonymization defaults\r\n\r\nThree fields, detailed below, were added
as anonymization defaults because they improve the quality of responses
from the LLM when it answers questions about alerts.\r\n\r\nFor example,
the LLM can refer to specific alerts by ID when the `_id` field is
provided.\r\n\r\nThis PR makes the following additive changes to the
Assistant's `Anonymization` defaults:\r\n\r\n| Field | Allow by default
| Anonymize by default | Value add
|\r\n|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n|
`_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM
to refer to specific documents (but doesn't provide it the actual
document IDs). |\r\n| `kibana.alert.risk_score` | ✅ | ❌ | The
`getOpenAndAcknowledgedAlertsQuery` query sorts alerts by
`kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing
this field (by default) enables the LLM to include actual alert risk
scores in responses. |\r\n| `kibana.alert.workflow_status` | ✅ | ❌ | The
`getOpenAndAcknowledgedAlertsQuery` query filters alerts by
`kibana.alert.workflow_status` to ensure only `open` and `acknowledged`
alerts are provided as context to the LLM. Allowing this field (by
default) enables the LLM answer questions about workflow status, and
echo the workflow status of alerts in responses. |\r\n\r\n- Clicking the
`Reset` button shown in the screenshot below will reset the user's
`Anonymization` defaults, such that they include the additive changes in
the table
above:\r\n\r\n\r\n\r\n###
Updated settings text\r\n\r\nThe text in the settings below was also
updated:\r\n\r\n\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n- Enable the
`assistantRagOnAlerts` feature flag described in
[#172542](https://github.com/elastic/kibana/pull/172542) must be
enabled, per the following
example:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n- The `Alerts` feature must be
enabled in the assistant settings, per the screenshot below:\r\n\r\n
\r\n\r\n1)
Navigate to Security > Alerts\r\n\r\n2) Click the `AI Assistant` button
to open the assistant\r\n\r\n3) Click the `Settings` gear to open the
assistant settings\r\n\r\n4) Click the `Anonymization`
category\r\n\r\n5) Click the `Reset` button, shown in the screenshot
below\r\n\r\n\r\n\r\n**Expected
results**\r\n\r\n- `65` fields are allowed by default, per the
screenshot above\r\n- `12` fields are anonymized by default, per the
screenshot above\r\n- The `_id` field is allowed by default, per the
screenshot above\r\n- The `_id` field is anonymized by default, per the
screenshot above\r\n\r\n6) Type `kibana.alert.risk` in the search
box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.risk_score`
field is allowed by default\r\n\r\n7) Type `kibana.alert.workflow` in
the search box\r\n\r\n**Expected result**\r\n\r\n- The
`kibana.alert.workflow_status` field is allowed by default\r\n\r\n8)
Click `Save`\r\n\r\n9) Click the `X` button to clear the
conversation\r\n\r\n10) Close the assistant\r\n\r\n11) Add the following
two fields as columns to the Alerts page table:\r\n\r\n-
`kibana.alert.workflow_status`\r\n- `_id`\r\n\r\n12) Sort the table,
first by `kibana.alert.risk_score` from high to low, and then by
`@timestamp` from new to old, per the screenshot
below:\r\n\r\n\r\n\r\n13)
Filter the alerts page to only show `open` and `acknowledged`
alerts\r\n\r\n**Expected result**\r\n\r\n- The alerts page has custom
columns, sorting, and filtering, per the screenshot
below:\r\n\r\n\r\n\r\n14)
Click the `AI Assistant` button to open the assistant\r\n\r\n15) Ask the
assistant:\r\n\r\n```\r\nWhat is the workflow status of my
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant will
report on the workflow status of alerts, per the example response
below:\r\n\r\n```\r\nThe workflow status for your alerts is currently
'open'. This status was observed on alerts related to processes started
by Mimikatz, a known tool used in many cyberattacks, and sequences of
processes that are often indicative of malicious activity. The severity
of most of these alerts is 'high' or 'critical'. You may want to
investigate these alerts further to ensure there's no ongoing threat to
your
system.\r\n```\r\n\r\n\r\n\r\n16)
Close the assistant\r\n\r\n17) Change the workflow status of an alert in
the Alerts table from `open` to `acknowledged`\r\n\r\n**Expected
result**\r\n\r\n- The alerts table shows the updated alert, per the
screenshot
below:\r\n\r\n\r\n\r\n18)
Once again, open the assistant\r\n\r\n19) Once again, ask the (same)
question:\r\n\r\n```\r\nWhat is the workflow status of my
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The response from the
assistant makes reference to the alert who's workflow status was changed
from `open` to `acknowledged`, per the example response
below:\r\n\r\n```\r\nBased on the latest information, your alerts mainly
show 'open' status, indicating that they have not been resolved yet.
Some alerts have been acknowledged. Most of these unaddressed alerts
have a critical severity rating and are primarily triggered by a
Mimikatz process start and an EQL process sequence. You may want to
prioritize these if the severity of the threat they pose is truly high
or critical. It's also noteworthy that some alerts have a high severity
rating. You should review all of these alerts as soon as possible to
ensure your systems are
secure.\r\n```\r\n\r\n\r\n\r\n20)
Ask the assistant for details about the acknowledged
alerts:\r\n\r\n```\r\nWhat are the details of the acknowledged
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\nThe assistant for
details about the acknowledged alert that, for example, includes the
`kibana.alert.risk_score`, per the example response
below:\r\n\r\n```\r\nIn response to your previous question, here are the
details of the acknowledged alerts:\r\n\r\n1. There is a 'mimikatz
process started' alert, which is of 'critical' severity and
'acknowledged' status. It has a high risk score of 99. Its threat tactic
is 'Command and Control'. The process involved was 'mimikatz.exe'
running with arguments '--fo1'.\r\n\r\n2. A 'Threshold rule' alert of
'critical' severity and 'open' status has also been detected with a risk
score of 99 and threat tactic 'Collection'.\r\n\r\n3. Lastly, there are
several 'EQL process sequence' alerts of 'high' severity with 'open'
status. These alerts involve execution of various processes including
'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these
alerts is 73 and the threat tactic involved is
'Execution'.\r\n\r\nPlease, take appropriate action to address these
alerts.\r\n```\r\n\r\n\r\n\r\n21)
Ask the assistant for the `_id` of the acknowledged
alert:\r\n\r\n```\r\nWhat is the id of the acknowledged
alert?\r\n```\r\n\r\n**Expected results**\r\n\r\n- The response from the
assistant contains the `_id` of the `acknowledged` alert, per the
example response below:\r\n\r\n```\r\nThe id of the acknowledged alert
is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'.
This alert was for a 'mimikatz process started' event with a severity
rating of 'critical' and a risk score of 99. The host name associated
with this alert is
'Host-terkvbzvtj'.\r\n```\r\n\r\n\r\n\r\n-
The `_id` shown in the assistant is the same `_id` of the acknowledged
alert on the alerts page, per the screeenshot
below:\r\n\r\n\r\n\r\n22)
Click the `Show anonymized` toggle in the assistant\r\n\r\n**Expected
result**\r\n\r\n- The `_id` shown in the latest result is replaced with
the actual anonymized value that was sent to the LLM, per the example
screenshot
below:\r\n\r\n","sha":"0d9c261530b102e7a7a87f4d5dcdf423cdd17a2c","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","Team:Threat Hunting:Investigations","Feature:Elastic
AI
Assistant","v8.12.0","v8.13.0"],"number":173121,"url":"https://github.com/elastic/kibana/pull/173121","mergeCommit":{"message":"[Security
Solution] [Elastic AI Assistant] Include acknowledged alerts in the
context sent to the LLM (Retrieval Augmented Generation (RAG) for
Alerts) (#173121)\n\n## [Security Solution] [Elastic AI Assistant]
Include `acknowledged` alerts in the context sent to the LLM (Retrieval
Augmented Generation (RAG) for Alerts)\r\n\r\nThis PR updates the query
used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented
Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542) to include
alerts with a `kibana.alert.workflow_status` value of
`acknowledged`.\r\n\r\nThe query previously only returned alerts with a
status of `open`. This change ensures both `open` and `acknowledged`
alerts are provided as context to the LLM.\r\n\r\n### Updated
Anonymization defaults\r\n\r\nThree fields, detailed below, were added
as anonymization defaults because they improve the quality of responses
from the LLM when it answers questions about alerts.\r\n\r\nFor example,
the LLM can refer to specific alerts by ID when the `_id` field is
provided.\r\n\r\nThis PR makes the following additive changes to the
Assistant's `Anonymization` defaults:\r\n\r\n| Field | Allow by default
| Anonymize by default | Value add
|\r\n|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n|
`_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM
to refer to specific documents (but doesn't provide it the actual
document IDs). |\r\n| `kibana.alert.risk_score` | ✅ | ❌ | The
`getOpenAndAcknowledgedAlertsQuery` query sorts alerts by
`kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing
this field (by default) enables the LLM to include actual alert risk
scores in responses. |\r\n| `kibana.alert.workflow_status` | ✅ | ❌ | The
`getOpenAndAcknowledgedAlertsQuery` query filters alerts by
`kibana.alert.workflow_status` to ensure only `open` and `acknowledged`
alerts are provided as context to the LLM. Allowing this field (by
default) enables the LLM answer questions about workflow status, and
echo the workflow status of alerts in responses. |\r\n\r\n- Clicking the
`Reset` button shown in the screenshot below will reset the user's
`Anonymization` defaults, such that they include the additive changes in
the table
above:\r\n\r\n\r\n\r\n###
Updated settings text\r\n\r\nThe text in the settings below was also
updated:\r\n\r\n\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n- Enable the
`assistantRagOnAlerts` feature flag described in
[#172542](https://github.com/elastic/kibana/pull/172542) must be
enabled, per the following
example:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n- The `Alerts` feature must be
enabled in the assistant settings, per the screenshot below:\r\n\r\n
\r\n\r\n1)
Navigate to Security > Alerts\r\n\r\n2) Click the `AI Assistant` button
to open the assistant\r\n\r\n3) Click the `Settings` gear to open the
assistant settings\r\n\r\n4) Click the `Anonymization`
category\r\n\r\n5) Click the `Reset` button, shown in the screenshot
below\r\n\r\n\r\n\r\n**Expected
results**\r\n\r\n- `65` fields are allowed by default, per the
screenshot above\r\n- `12` fields are anonymized by default, per the
screenshot above\r\n- The `_id` field is allowed by default, per the
screenshot above\r\n- The `_id` field is anonymized by default, per the
screenshot above\r\n\r\n6) Type `kibana.alert.risk` in the search
box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.risk_score`
field is allowed by default\r\n\r\n7) Type `kibana.alert.workflow` in
the search box\r\n\r\n**Expected result**\r\n\r\n- The
`kibana.alert.workflow_status` field is allowed by default\r\n\r\n8)
Click `Save`\r\n\r\n9) Click the `X` button to clear the
conversation\r\n\r\n10) Close the assistant\r\n\r\n11) Add the following
two fields as columns to the Alerts page table:\r\n\r\n-
`kibana.alert.workflow_status`\r\n- `_id`\r\n\r\n12) Sort the table,
first by `kibana.alert.risk_score` from high to low, and then by
`@timestamp` from new to old, per the screenshot
below:\r\n\r\n\r\n\r\n13)
Filter the alerts page to only show `open` and `acknowledged`
alerts\r\n\r\n**Expected result**\r\n\r\n- The alerts page has custom
columns, sorting, and filtering, per the screenshot
below:\r\n\r\n\r\n\r\n14)
Click the `AI Assistant` button to open the assistant\r\n\r\n15) Ask the
assistant:\r\n\r\n```\r\nWhat is the workflow status of my
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant will
report on the workflow status of alerts, per the example response
below:\r\n\r\n```\r\nThe workflow status for your alerts is currently
'open'. This status was observed on alerts related to processes started
by Mimikatz, a known tool used in many cyberattacks, and sequences of
processes that are often indicative of malicious activity. The severity
of most of these alerts is 'high' or 'critical'. You may want to
investigate these alerts further to ensure there's no ongoing threat to
your
system.\r\n```\r\n\r\n\r\n\r\n16)
Close the assistant\r\n\r\n17) Change the workflow status of an alert in
the Alerts table from `open` to `acknowledged`\r\n\r\n**Expected
result**\r\n\r\n- The alerts table shows the updated alert, per the
screenshot
below:\r\n\r\n\r\n\r\n18)
Once again, open the assistant\r\n\r\n19) Once again, ask the (same)
question:\r\n\r\n```\r\nWhat is the workflow status of my
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The response from the
assistant makes reference to the alert who's workflow status was changed
from `open` to `acknowledged`, per the example response
below:\r\n\r\n```\r\nBased on the latest information, your alerts mainly
show 'open' status, indicating that they have not been resolved yet.
Some alerts have been acknowledged. Most of these unaddressed alerts
have a critical severity rating and are primarily triggered by a
Mimikatz process start and an EQL process sequence. You may want to
prioritize these if the severity of the threat they pose is truly high
or critical. It's also noteworthy that some alerts have a high severity
rating. You should review all of these alerts as soon as possible to
ensure your systems are
secure.\r\n```\r\n\r\n\r\n\r\n20)
Ask the assistant for details about the acknowledged
alerts:\r\n\r\n```\r\nWhat are the details of the acknowledged
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\nThe assistant for
details about the acknowledged alert that, for example, includes the
`kibana.alert.risk_score`, per the example response
below:\r\n\r\n```\r\nIn response to your previous question, here are the
details of the acknowledged alerts:\r\n\r\n1. There is a 'mimikatz
process started' alert, which is of 'critical' severity and
'acknowledged' status. It has a high risk score of 99. Its threat tactic
is 'Command and Control'. The process involved was 'mimikatz.exe'
running with arguments '--fo1'.\r\n\r\n2. A 'Threshold rule' alert of
'critical' severity and 'open' status has also been detected with a risk
score of 99 and threat tactic 'Collection'.\r\n\r\n3. Lastly, there are
several 'EQL process sequence' alerts of 'high' severity with 'open'
status. These alerts involve execution of various processes including
'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these
alerts is 73 and the threat tactic involved is
'Execution'.\r\n\r\nPlease, take appropriate action to address these
alerts.\r\n```\r\n\r\n\r\n\r\n21)
Ask the assistant for the `_id` of the acknowledged
alert:\r\n\r\n```\r\nWhat is the id of the acknowledged
alert?\r\n```\r\n\r\n**Expected results**\r\n\r\n- The response from the
assistant contains the `_id` of the `acknowledged` alert, per the
example response below:\r\n\r\n```\r\nThe id of the acknowledged alert
is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'.
This alert was for a 'mimikatz process started' event with a severity
rating of 'critical' and a risk score of 99. The host name associated
with this alert is
'Host-terkvbzvtj'.\r\n```\r\n\r\n\r\n\r\n-
The `_id` shown in the assistant is the same `_id` of the acknowledged
alert on the alerts page, per the screeenshot
below:\r\n\r\n\r\n\r\n22)
Click the `Show anonymized` toggle in the assistant\r\n\r\n**Expected
result**\r\n\r\n- The `_id` shown in the latest result is replaced with
the actual anonymized value that was sent to the LLM, per the example
screenshot
below:\r\n\r\n","sha":"0d9c261530b102e7a7a87f4d5dcdf423cdd17a2c"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173121","number":173121,"mergeCommit":{"message":"[Security
Solution] [Elastic AI Assistant] Include acknowledged alerts in the
context sent to the LLM (Retrieval Augmented Generation (RAG) for
Alerts) (#173121)\n\n## [Security Solution] [Elastic AI Assistant]
Include `acknowledged` alerts in the context sent to the LLM (Retrieval
Augmented Generation (RAG) for Alerts)\r\n\r\nThis PR updates the query
used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented
Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542) to include
alerts with a `kibana.alert.workflow_status` value of
`acknowledged`.\r\n\r\nThe query previously only returned alerts with a
status of `open`. This change ensures both `open` and `acknowledged`
alerts are provided as context to the LLM.\r\n\r\n### Updated
Anonymization defaults\r\n\r\nThree fields, detailed below, were added
as anonymization defaults because they improve the quality of responses
from the LLM when it answers questions about alerts.\r\n\r\nFor example,
the LLM can refer to specific alerts by ID when the `_id` field is
provided.\r\n\r\nThis PR makes the following additive changes to the
Assistant's `Anonymization` defaults:\r\n\r\n| Field | Allow by default
| Anonymize by default | Value add
|\r\n|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n|
`_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM
to refer to specific documents (but doesn't provide it the actual
document IDs). |\r\n| `kibana.alert.risk_score` | ✅ | ❌ | The
`getOpenAndAcknowledgedAlertsQuery` query sorts alerts by
`kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing
this field (by default) enables the LLM to include actual alert risk
scores in responses. |\r\n| `kibana.alert.workflow_status` | ✅ | ❌ | The
`getOpenAndAcknowledgedAlertsQuery` query filters alerts by
`kibana.alert.workflow_status` to ensure only `open` and `acknowledged`
alerts are provided as context to the LLM. Allowing this field (by
default) enables the LLM answer questions about workflow status, and
echo the workflow status of alerts in responses. |\r\n\r\n- Clicking the
`Reset` button shown in the screenshot below will reset the user's
`Anonymization` defaults, such that they include the additive changes in
the table
above:\r\n\r\n\r\n\r\n###
Updated settings text\r\n\r\nThe text in the settings below was also
updated:\r\n\r\n\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n- Enable the
`assistantRagOnAlerts` feature flag described in
[#172542](https://github.com/elastic/kibana/pull/172542) must be
enabled, per the following
example:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n- The `Alerts` feature must be
enabled in the assistant settings, per the screenshot below:\r\n\r\n
\r\n\r\n1)
Navigate to Security > Alerts\r\n\r\n2) Click the `AI Assistant` button
to open the assistant\r\n\r\n3) Click the `Settings` gear to open the
assistant settings\r\n\r\n4) Click the `Anonymization`
category\r\n\r\n5) Click the `Reset` button, shown in the screenshot
below\r\n\r\n\r\n\r\n**Expected
results**\r\n\r\n- `65` fields are allowed by default, per the
screenshot above\r\n- `12` fields are anonymized by default, per the
screenshot above\r\n- The `_id` field is allowed by default, per the
screenshot above\r\n- The `_id` field is anonymized by default, per the
screenshot above\r\n\r\n6) Type `kibana.alert.risk` in the search
box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.risk_score`
field is allowed by default\r\n\r\n7) Type `kibana.alert.workflow` in
the search box\r\n\r\n**Expected result**\r\n\r\n- The
`kibana.alert.workflow_status` field is allowed by default\r\n\r\n8)
Click `Save`\r\n\r\n9) Click the `X` button to clear the
conversation\r\n\r\n10) Close the assistant\r\n\r\n11) Add the following
two fields as columns to the Alerts page table:\r\n\r\n-
`kibana.alert.workflow_status`\r\n- `_id`\r\n\r\n12) Sort the table,
first by `kibana.alert.risk_score` from high to low, and then by
`@timestamp` from new to old, per the screenshot
below:\r\n\r\n\r\n\r\n13)
Filter the alerts page to only show `open` and `acknowledged`
alerts\r\n\r\n**Expected result**\r\n\r\n- The alerts page has custom
columns, sorting, and filtering, per the screenshot
below:\r\n\r\n\r\n\r\n14)
Click the `AI Assistant` button to open the assistant\r\n\r\n15) Ask the
assistant:\r\n\r\n```\r\nWhat is the workflow status of my
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant will
report on the workflow status of alerts, per the example response
below:\r\n\r\n```\r\nThe workflow status for your alerts is currently
'open'. This status was observed on alerts related to processes started
by Mimikatz, a known tool used in many cyberattacks, and sequences of
processes that are often indicative of malicious activity. The severity
of most of these alerts is 'high' or 'critical'. You may want to
investigate these alerts further to ensure there's no ongoing threat to
your
system.\r\n```\r\n\r\n\r\n\r\n16)
Close the assistant\r\n\r\n17) Change the workflow status of an alert in
the Alerts table from `open` to `acknowledged`\r\n\r\n**Expected
result**\r\n\r\n- The alerts table shows the updated alert, per the
screenshot
below:\r\n\r\n\r\n\r\n18)
Once again, open the assistant\r\n\r\n19) Once again, ask the (same)
question:\r\n\r\n```\r\nWhat is the workflow status of my
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The response from the
assistant makes reference to the alert who's workflow status was changed
from `open` to `acknowledged`, per the example response
below:\r\n\r\n```\r\nBased on the latest information, your alerts mainly
show 'open' status, indicating that they have not been resolved yet.
Some alerts have been acknowledged. Most of these unaddressed alerts
have a critical severity rating and are primarily triggered by a
Mimikatz process start and an EQL process sequence. You may want to
prioritize these if the severity of the threat they pose is truly high
or critical. It's also noteworthy that some alerts have a high severity
rating. You should review all of these alerts as soon as possible to
ensure your systems are
secure.\r\n```\r\n\r\n\r\n\r\n20)
Ask the assistant for details about the acknowledged
alerts:\r\n\r\n```\r\nWhat are the details of the acknowledged
alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\nThe assistant for
details about the acknowledged alert that, for example, includes the
`kibana.alert.risk_score`, per the example response
below:\r\n\r\n```\r\nIn response to your previous question, here are the
details of the acknowledged alerts:\r\n\r\n1. There is a 'mimikatz
process started' alert, which is of 'critical' severity and
'acknowledged' status. It has a high risk score of 99. Its threat tactic
is 'Command and Control'. The process involved was 'mimikatz.exe'
running with arguments '--fo1'.\r\n\r\n2. A 'Threshold rule' alert of
'critical' severity and 'open' status has also been detected with a risk
score of 99 and threat tactic 'Collection'.\r\n\r\n3. Lastly, there are
several 'EQL process sequence' alerts of 'high' severity with 'open'
status. These alerts involve execution of various processes including
'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these
alerts is 73 and the threat tactic involved is
'Execution'.\r\n\r\nPlease, take appropriate action to address these
alerts.\r\n```\r\n\r\n\r\n\r\n21)
Ask the assistant for the `_id` of the acknowledged
alert:\r\n\r\n```\r\nWhat is the id of the acknowledged
alert?\r\n```\r\n\r\n**Expected results**\r\n\r\n- The response from the
assistant contains the `_id` of the `acknowledged` alert, per the
example response below:\r\n\r\n```\r\nThe id of the acknowledged alert
is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'.
This alert was for a 'mimikatz process started' event with a severity
rating of 'critical' and a risk score of 99. The host name associated
with this alert is
'Host-terkvbzvtj'.\r\n```\r\n\r\n\r\n\r\n-
The `_id` shown in the assistant is the same `_id` of the acknowledged
alert on the alerts page, per the screeenshot
below:\r\n\r\n\r\n\r\n22)
Click the `Show anonymized` toggle in the assistant\r\n\r\n**Expected
result**\r\n\r\n- The `_id` shown in the latest result is replaced with
the actual anonymized value that was sent to the LLM, per the example
screenshot
below:\r\n\r\n","sha":"0d9c261530b102e7a7a87f4d5dcdf423cdd17a2c"}}]}]
BACKPORT-->
Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
# Backport
This will backport the following commits from `main` to `8.12`:
- [feat(slo): new slo architecture
(#172224)](https://github.com/elastic/kibana/pull/172224)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Kevin
Delemme","email":"kevin.delemme@elastic.co"},"sourceCommit":{"committedDate":"2023-12-12T13:45:12Z","message":"feat(slo):
new slo architecture
(#172224)","sha":"b51304f3f3c3e8510c44a235d0fc65c44fcce225","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:breaking","backport:prev-minor","ci:build-serverless-image","Feature:SLO","v8.12.0","Team:obs-ux-management","v8.13.0"],"number":172224,"url":"https://github.com/elastic/kibana/pull/172224","mergeCommit":{"message":"feat(slo):
new slo architecture
(#172224)","sha":"b51304f3f3c3e8510c44a235d0fc65c44fcce225"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/172224","number":172224,"mergeCommit":{"message":"feat(slo):
new slo architecture
(#172224)","sha":"b51304f3f3c3e8510c44a235d0fc65c44fcce225"}}]}]
BACKPORT-->
Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co>
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:
- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_
### More context
Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:
- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
- Only fields allowed by the user will be sent as context to the LLM
- Users may enable or disable anonymization for specific fields (via settings)
- Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
### Settings
This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)
When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:
- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts
### Feature flag
To use this feature:
1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:
```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```
2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:
## How it works
- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
- `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
- `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
- `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
- `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.
```json
"replacements": {
"e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
"cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
"7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
"f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
// ...
},
```
- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`
- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)
- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:
```typescript
const tools: Tool[] = getApplicableTools({
allow,
allowReplacement,
alertsIndexPattern,
assistantLangChain,
chain,
esClient,
modelExists,
onNewReplacements,
replacements,
request,
size,
});
```
- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.
- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details
- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.
- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared
- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
- The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
## Summary
Fixes https://github.com/elastic/kibana/issues/171243. This PR adds
field `_tier` to the list of omit fields to not show or display. This is
especially relevant when `_tier` is added in the list of meta fields in
Kibana.
Steps to reproduce:
1. In Advanced settings, add `_tier` to the list of meta fields. This
will show _tier as a field across Kibana if data has a tier applied.
<img width="976" alt="image"
src="86ecbbba-c574-42f6-97cf-c465ec334d7e">
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#171613
## Summary
This PR adds the viewInApp URL to the custom threshold rule type. This
URL will send the user to the log explorer with the selected data view
and the rule's query filter. If there is only one document aggregation,
then the filter related to this aggregation will be added as shown
below:
|Rule|Discover with pre-fill data|
|---|---|
||
For the ad-hoc data view, you should be able to see the selected index
pattern in discover similar to this:
<img
src="046493ae-ba59-46b7-a40f-68d1836d43f1"
width=400 />
### 🧪 How to test
- Check the viewInApp URL both in action variables and the alert table
for the following scenarios:
- A rule with a persisted data view
- A rule with an ad-hoc data view
- A rule with count aggregation and filter
- A rule with an optional query filter
- A rule with non-count aggregation
In all the above scenarios, the starting time in the Discover should be
before the alert's start time.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
With this PR we introduce a new Alert User Assignment feature:
- It is possible to assign a user/s to alert/s
- There is a new "Assignees" column in the alerts table which displays
avatars of assigned users
- There is a bulk action to update assignees for multiple alerts
- It is possible to see and update assignees inside the alert details
flyout component
- There is an "Assignees" filter button on the Alerts page which allows
to filter alerts by assignees
We decided to develop this feature on a separate branch. This gives us
ability to make sure that it is thoroughly tested and we did not break
anything in production. Since there is a data scheme changes involved we
decided that it will be a better approach. cc @yctercero
## Testing notes
In order to test assignments you need to create a few users. Then for
users to appear in user profiles dropdown menu you need to activate them
by login into those account at least once.
8eeb13f3-2d16-4fba-acdf-755024a59fc2
Main ticket https://github.com/elastic/security-team/issues/2504
## Bugfixes
- [x] https://github.com/elastic/security-team/issues/8028
- [x] https://github.com/elastic/security-team/issues/8034
- [x] https://github.com/elastic/security-team/issues/8006
- [x] https://github.com/elastic/security-team/issues/8025
## Enhancements
- [x] https://github.com/elastic/security-team/issues/8033
### Checklist
- [x] Functional changes are hidden behind a feature flag. If not
hidden, the PR explains why these changes are being implemented in a
long-living feature branch.
- [x] Functional changes are covered with a test plan and automated
tests.
- [x] https://github.com/elastic/kibana/issues/171306
- [x] https://github.com/elastic/kibana/issues/171307
- [x] Stability of new and changed tests is verified using the [Flaky
Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner).
- [x]
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4091
- [x] Comprehensive manual testing is done by two engineers: the PR
author and one of the PR reviewers. Changes are tested in both ESS and
Serverless.
- [x] Mapping changes are accompanied by a technical design document. It
can be a GitHub issue or an RFC explaining the changes. The design
document is shared with and approved by the appropriate teams and
individual stakeholders.
* https://github.com/elastic/security-team/issues/7647
- [x] Functional changes are communicated to the Docs team. A ticket or
PR is opened in https://github.com/elastic/security-docs. The following
information is included: any feature flags used, affected environments
(Serverless, ESS, or both). **NOTE: as discussed we will wait until docs
are ready to merge this PR**.
* https://github.com/elastic/security-docs/issues/4226
* https://github.com/elastic/staging-serverless-security-docs/pull/232
---------
Co-authored-by: Marshall Main <marshall.main@elastic.co>
Co-authored-by: Xavier Mouligneau <xavier.mouligneau@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
Consolidates UI elements and backend code to create/delete data views
and destination indices related to transforms and data frame analytics.
We ended up with two different approaches for creating data views in the
wizards for transforms and data frame analytics, the original reason was
we were not aware of the `allowNoIndex: true` setting and worked around
that in different ways.
This PR aligns UI workflows and moves related code to a new package
`@kbn/ml-data-view-utils` for data views and
`@kbn/ml-creation-wizard-utils` for the destination index form. The
latter might be used for other shared components across wizard..
In Data Frame Analytics, the checkbox to create a data view was removed
from the last "Create" step, instead the option to create a data view
was moved to the "Details" step.
In Transforms, the UI component to create the destination index was
brought over from DFA where there is a switch option to automatically
use the job ID as the name for the destination index by default.
## Summary
This PR instruments the Elastic AI Assistant with the Kibana APM Agent
enabling the tracing of retrievers, llms, chains, and tools which can
then be viewed within the Observability app. This PR also improves the
Assistant Model Evaluation tooling by enabling support for pulling and
running test datasets from LangSmith.
If the `assistantModelEvaluation` experimental feature flag is enabled,
and an APM server is configured, messages that have a corresponding
trace will have an additional `View APM trace` action:
<p align="center">
<img width="500"
src="e0b372ee-139a-4eed-8b09-f01dd88c72b0"
/>
</p>
Viewing the trace you can see a breakdown of the time spent in each
retriever, llm, chain, and tool:
<p align="center">
<img width="500"
src="f7cbd4bc-207c-4c88-a032-70a8de4f9b9a"
/>
</p>
Additionally the Evaluation interface has been updated to support adding
additional metadata like `Project Name`, `Run Name`, and pulling test
datasets from LangSmith. Predictions can now also be run without having
to run an Evaluation, so datasets can quickly be run for manual
analysis.
<p align="center">
<img width="500"
src="acebf719-29fd-4fcc-aef1-99fd00ca800a"
/>
</p>
<p align="center">
<img width="500"
src="7081d993-cbe0-4465-a734-ff9be14d7d0d"
/>
</p>
## Testing
### Configuring APM
First, enable the `assistantModelEvaluation` experimental feature flag
by adding the following to your `kibana.dev.yml`:
```
xpack.securitySolution.enableExperimental: [ 'assistantModelEvaluation' ]
```
Next, you'll need an APM server to collect the traces. You can either
[follow the documentation for
installing](https://www.elastic.co/guide/en/apm/guide/current/installing.html)
the released artifact, or [run from
source](https://github.com/elastic/apm-server#apm-server-development)
and set up using the [quickstart guide
provided](https://www.elastic.co/guide/en/apm/guide/current/apm-quick-start.html)
(be sure to install the APM Server integration to ensure the necessary
indices are created!). Once your APM server is running, add your APM
server configuration to your `kibana.dev.yml` as well using the
following:
```
# APM
elastic.apm:
active: true
environment: 'SpongBox5002c™'
serverUrl: 'http://localhost:8200'
transactionSampleRate: 1.0
breakdownMetrics: true
spanStackTraceMinDuration: 10ms
# Disables Kibana RUM
servicesOverrides.kibana-frontend.active: false
```
> [!NOTE]
> If connecting to a cloud APM server (like our [ai-assistant apm
deployment](https://ai-assistant-apm-do-not-delete.kb.us-central1.gcp.cloud.es.io/)),
follow [these
steps](https://www.elastic.co/guide/en/apm/guide/current/api-key.html#create-an-api-key)
to create an API key, and then set it via `apiKey` and also set your
`serverUrl` as shown in the APM Integration details within fleet. Note
that the `View APM trace` button within the UI will link to your local
instance, not the cloud instance.
> [!NOTE]
> If you're an Elastic developer running Kibana from source, you can
just enable APM as above, and _not_ include a `serverUrl`, and your
traces will be sent to the https://kibana-cloud-apm.elastic.dev cluster.
Note that the `View APM trace` button within the UI will link to your
local instance, not the cloud instance.
### Configuring LangSmith
If wanting to push traces to LangSmith, or leverage any datasets that
you may have hosted in a project, all you need to do is configure a few
environment variables, and then start the kibana server. See the
[LangSmith Traces
documentation](https://docs.smith.langchain.com/tracing) for details, or
just add the below env variables to enable:
```
# LangChain LangSmith
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
export LANGCHAIN_API_KEY=""
export LANGCHAIN_PROJECT="8.12 ESQL Query Generation"
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
In this PR, I'm relocating all Kibana Security types (along with a few
schemas necessary for some of these types, unfortunately) that are part
of public contracts to separate packages. This change will enable any
plugin to utilize Security APIs via "static" or
["runtime"](https://github.com/elastic/kibana/pull/167113) dependencies,
regardless of whether Kibana Security already relies on these plugins or
not.
__NOTE TO REVIEWERS:__ I tried to minimize changes as much as I could
via moving only necessary types. I also didn't move deprecated parts of
the Setup/Start contracts to these new packages.
__Triggered by:__ https://github.com/elastic/kibana/pull/168910
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Moves the categorize field uiAction trigger and action and related items
to the AIOps/ML uiActions package.
ML and AIOps are adding more and more uiActions, and so it's nicer to
have them all in one package.
Also cleans up the registration of the uiActions in the AIOps plugin
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Renames references to index patterns to data views in function and
variable names.
- Some inconsistent naming of schemas for data frame analytics was
cleaned up as part of this PR.
- Note this doesn't cover the whole ml owned codebase but just code
related to data frame analytics.
Closes#159340
## Summary
Since the Custom threshold is the default aggregation type, we no longer
need to pass `aggType: CUSTOM_AGGREGATOR` to the rule creation API. I
added this as optional in the schema so the previous rules with this
field will not throw a schema validation error.
## How to test
- Everything should work as before in the custom threshold; the only
difference is that there is no need to provide aggType at the top level.
- Providing `aggType: custom` through API should be OK, and the rule
should work as expected with or without this field.
Support to restore baseline/deviation time ranges from url state on full
page refresh. Also updates functional tests to include a full page refresh after the
first analysis run for each dataset.
## Summary
Fixes https://github.com/elastic/kibana/issues/168194
Under some circumstance, when navigating to the timelines page, we would
get a runtime exception for `state.tableById[action.id]` not being
defined. When that happened, the redux store would be in a broken state.
This PR makes the responsible destructuring assignment more save.
Adds the ability to quickly create a categorisation anomaly detection
job from the pattern analysis flyout.
Adds a new `created_by` ID `categorization-wizard-from-pattern-analysis`
which can be picked up by telemetry.
Creates a new package for sharing our AIOPs ui actions IDs. I think we
should move the pattern analysis ID to this package too, but that can be
done in a separate PR.
51349f93-f072-4983-85f0-98741902fb5a
6e618581-8916-4e63-930f-945c96c25e6c
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Update new user details flyout to be consistent with Expandable Alerts
Flyout. The previous user details flyout implementation was hidden
behind a flag and never went live.
### What is included
* Update new user details flyout to use the expandable flyout component
* Update UI components according to the new design
* Keep the feature hidden behind newUserDetailsFlyout flag
* Supporting alert risk inputs
### What is NOT included
* Supporting multiple categories of risk inputs
* Host details flyout
* User and host pages
* Asset integrations (okta and azure)
* Update the flyout on the timeline (It is currently a technical
restriction of the expandable flyout, but the team is working to fix it)
### How to test it?
* Enable experimental flag `newUserDetailsFlyout`
`xpack.securitySolution.enableExperimental: ['newUserDetailsFlyout']`
* Create alerts and open alerts page
* Click on a username
- [x] Test edge cases
- [x] No cases permissions (it hides cases actions)
- [x] Basic license (it hides the risk score summary)
- [x] No risk score data for a user (It hides the risk score summary)
<img width="434" alt="Screenshot 2023-11-13 at 15 56 33"
src="4fc13042-cd3d-487b-9982-bfbf02f003b4">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
## Summary
Thanks @spong for the speedy assistance with getting this code-complete!
Utilizing the Security Assistant to provide some suggested mediation
steps for rule errors could help customers to better self-diagnose rule
errors. Thus, enhancing their experience with the Security Solution and
potentially reducing new support tickets.
Error on rule details page:
<img width="1462" alt="threshold_rule_exception_error"
src="9f31fad5-f1e5-46b2-accf-2739ac3b83dd">
Response from security assistant:
<img width="1454" alt="threshold_rule_exception_assistant_resolved"
src="5fbd8ea5-8a5d-47ea-8f24-6698b298f023">
Available for warnings too:
<img width="1205" alt="assistant_error_help_warning"
src="e93bb870-9688-4d87-a6db-59a552ab9af9">
Includes the rule name and data sources for pre-built rules for
additional information to generate a slightly more helpful response:
<img width="1958" alt="pre_built_rule_name_data_source"
src="d6e797c8-e014-4cb0-be95-fcce02568121">
---------
Co-authored-by: Garrett Spong <garrett.spong@elastic.co>
This refactors the route handler of the log rate analysis API endpoint.
So far this route handler contained a lot of logic and was growing past
900+ lines with every new feature we worked on. This PR changes it so
the route handler can walk through the analysis steps on a higher level.
`define_route.ts:defineRoute()` is the outer most wrapper that's used to
define the route and its versions. It calls
`route_handler_factory:routeHandlerFactory()` for each version.
The route handler sets up
`response_stream_factory:responseStreamFactory()` to create the response
stream and then walks through the steps of the analysis.
The response stream factory acts as a wrapper to set up the stream
itself, the stream state (for example to set if it's running etc.), some
custom actions on the stream as well as analysis handlers that fetch
data from ES and pass it on to the stream.
Part of #159340Closes#169364
## Summary
This PR:
1. Removes preFill logic
In this PR, I removed the logic about prefilling custom threshold rule
params as it was originally for other rule types (not custom equation)
and to be used in the Metric threshold rule and the code related to this
logic was super confusing, and I wasn't even sure if it works as
expected since we haven't used this logic anywhere. I created a
[ticket](https://github.com/elastic/kibana/issues/170295) to bring back
this feature properly later, specifically for the custom equation, and
integrate it in one of the apps, such as Infra. We also need to be able
to preFill data view information (both adHoc and persisted data view)
2. Renames types and file names
- From `metricThreshold` to `customThreshold`
- From `metricExplorer` to `expression`
3. Removes unused types
4. Remove logic related to aggregations other than the custom equation
at the top level
Also, the fields that end with `pct` now have the `%` after the related
value: (The reason message was fixed in another PR)
<img
src="83694d3b-2ee2-4e95-afe9-5a959c76c3c7"
width=400 />
## 🧪 How to test
- Nothing has changed related to functionality, so please make sure the
custom threshold rule is working as before for
- Creating a new rule with multiple conditions
- Adding groups
- Editing a rule and checking the charts are shown as before
- Test both adHoc and persisted data view
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Faisal Kanout <faisal.kanout@elastic.co>
Log rate analysis now supports both keywords and log patterns derived
from text fields. The type `SignificantTerm` originally was used for the
results of the `significant_terms` agg to get the p-values for keyword
fields. Since it's now used for both cases (keyword fields and log
patterns) this PR renames the type and related variables etc. to
`SignificantItem` (we used the wording `item` already in some cases in
the context of groups).
This PR uses [conditional
types](https://www.typescriptlang.org/docs/handbook/2/conditional-types.html)
to allow the handling of both multiple API versions within one route handler. The more tricky bit turned out
to be not the updated request body, but the response since it is an
NDJSON stream where some messages were updated. In this case also the
functions that create these messages were updated with conditional types
to be able to create a message that fits the definition of the API
version.
The API integration tests originally had these message identifiers in
the `expected` section of their `testData`. I changed that to use helper
functions that retrieve the expected messages from the stream according
to the expected version. All API integration tests are run on both
versions. The functional tests are run only on the newer version since
the UI is expected to work with version `2` only.
## 🍒 Summary
This PR fixes#170905 by adding the aggregation menu to the Custom
Metric indicator to allow the user to pick either `doc_count` or `sum`
for the aggregation.
<img width="1152" alt="image"
src="35aea8bd-d21c-4780-bad6-1efe5fc8902b">
## Summary
Removes `testing-library/dom` from dependencies. As all the utilities
from`dom` are available already in `testing-library/react`, there's no
need to have both `dom` and `react` libraries available in our
package.json.
Following the [@testing-library/react
documentation:](https://testing-library.com/docs/react-testing-library/intro)
> [React Testing
Library](https://github.com/testing-library/react-testing-library)
builds on top of DOM Testing Library by adding APIs for working with
React components.
Let's just import everything from `testing-library/react`, this way we
won't need to worry about inconsistencies between `testing-library/dom`
we have in our `package.json` and the one that is
`testing-library/react` dependency.
Updates new teams as codeowners for Observability team changes.
Also took the opportunity to:
- Delete some paths that no longer exist
- Split infra code ownership between teams (from #168992)
## Summary
1. Added `sameFamilyFields`, `numberOfSameFamily` field to `Data Quality
Index Checked`
<img width="2546" alt="Screenshot 2023-11-03 at 18 23 27"
src="a56a74fd-13a1-4489-9e29-367ff3754cbe">
2. Added `numberOfSameFamily` field to `Data Quality Check All
Completed`
<img width="2557" alt="Screenshot 2023-11-03 at 18 24 39"
src="e1ff3497-3dd0-4352-a8c0-92c0ed588863">
## Summary
After testing in the latest BC these three tweaks were identified:
- [X] Set k value to 10 so more documents are returned (since ESQL KB
docs are broken up by function and can be small)
- [X] Adds Quick Prompt to aid in ESQL Query Generation that only shows
if KB is enabled
Improves the `flushFix` behaviour for Log Rate Analysis. Previously the
setting would add a 4KB size additional dummy payload to each object
returned as ndjson. For the dataset used for testing this, this would
result in an overall response payload of ˜900Kbytes. For comparison,
without `flushFix` the response size would be ˜40Kbytes in this case.
This PR changes the behaviour to only send a dummy payload every 500ms
if the real data sent in the last 500ms wasn't bigger than 4Kbytes.
Depending on the speed of the response, this can bring down the overall
response payload to ˜300Kbytes (Cloud uncached), ˜150Kbytes (Cloud
cached) or even ˜70Kbytes (local cluster) for the same dataset.
## Summary
Fixes `ES|QL` codeblocks from not being able to be sent to Timeline.
<p align="center">
<img width="500"
src="fdc3b2a0-5b4b-4584-b304-c4d24de1917c"
/>
</p>
## Test instructions
Either request the assistant to generate an ESQL query or just paste
this codeblock into the conversation to test the action directly from
the user message. Be sure to declare the codeblock language as `esql` or
include one of the [string match
patterns](https://github.com/elastic/kibana/pull/169478/files#diff-f70f0b96568e024e53bfbb62adcca72051f0a2e824d4ab22664eed0e149be248R38)
above the code block so the action can be recognized.
````
Below is an `Elasticsearch Query Language` query:
```esql
FROM logs-endpoint*
| WHERE event.category == \"process\"
| STATS proc_count = COUNT(process.name) BY host.name
| KEEP host.name, proc_count
```
````
Note: The `send to timeline` actions appear to only reliably show up
when using the Assistant instance within Timeline. Now that we have a
more reliable way of attaching actions via markdown plugins/parsers, we
should refactor this code to use that method as opposed to the code
block/dom inspection route that is used currently. In the meantime I
will see if there is a low-impact fix that can be made here.
## Summary
Implements the `panelContentProvider` for the `DefaultNavigation`
component, so the content of the panels when open is provided by the
Security Solution plugin.
In order to test it, the experimental flag needs to be enabled.
In `config/serverless.security.yml` add:
```
xpack.securitySolutionServerless.enableExperimental: ['platformNavEnabled']
```
## Screenshot
<img width="1718" alt="Captura de pantalla 2023-10-18 a les 18 38 04"
src="5022a7d9-c619-4dbb-87cf-ee3ed0090853">
(The vertical separation of the main nav links is still not implemented
by the DefaultNavigation)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This fixes the Knowledge Base UX a bit by throwing an error if somehow
ELSER has been disabled in the background, and instructs the user on how
to resolve or to disable the Knowledge Base to continue.
Additionally, if ELSER is not available, we prevent the enabling of the
Knowledge Base as to not provide a degraded experience when ELSER and
the ES|QL documentation is not available.
<p align="center">
<img width="500"
src="e4d326fa-c996-43ad-9d1c-d76f7d16f916"
/>
</p>
> [!NOTE]
> `isModelInstalled` logic has been updated to not just check the model
`definition_status`, but to actually ensure that it's deployed by
checking to see that it is `started` and `fully_allocated`. This better
guards ELSER availability as the previous check would return true if the
model was just downloaded and not actually deployed.
Also resolves: https://github.com/elastic/kibana/issues/169403
## Test Instructions
After enabling the KB, disable the ELSER deployment in the `Trained
Models` ML UI and then try using the assistant.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
`v89.0.0`⏩`v89.1.0`
This upgrade also contains an EuiDataGrid refactor
(https://github.com/elastic/eui/pull/7255) not listed in the changelog
(as no end-user functionality or props changed as a result of the
refactor). The unlisted changes should only affect DOM and `className`
usages in Kibana (primarily CSS overrides and test selectors).
---
## [`89.1.0`](https://github.com/elastic/eui/tree/v89.1.0)
- Added `tokenVectorSparse` token and updated `tokenDenseVector` token
(now named `tokenVectorDense`).
([#7282](https://github.com/elastic/eui/pull/7282))
**CSS-in-JS conversions**
- Reduced default CSS prefixes generated by Emotion to only browsers
supported by EUI (latest evergreen browsers). This can be customized by
passing your own Emotion cache to `EuiProvider`.
([#7272](https://github.com/elastic/eui/pull/7272))
