mirror of
https://github.com/elastic/kibana.git
synced 2025-04-23 01:13:23 -04:00
[8.12] [Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121) (#173310)
# Backport This will backport the following commits from `main` to `8.12`: - [[Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121)](https://github.com/elastic/kibana/pull/173121) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Andrew Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2023-12-13T17:39:08Z","message":"[Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121)\n\n## [Security Solution] [Elastic AI Assistant] Include `acknowledged` alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts)\r\n\r\nThis PR updates the query used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542](https://github.com/elastic/kibana/pull/172542) to include alerts with a `kibana.alert.workflow_status` value of `acknowledged`.\r\n\r\nThe query previously only returned alerts with a status of `open`. This change ensures both `open` and `acknowledged` alerts are provided as context to the LLM.\r\n\r\n### Updated Anonymization defaults\r\n\r\nThree fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts.\r\n\r\nFor example, the LLM can refer to specific alerts by ID when the `_id` field is provided.\r\n\r\nThis PR makes the following additive changes to the Assistant's `Anonymization` defaults:\r\n\r\n| Field | Allow by default | Anonymize by default | Value add |\r\n|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n| `_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs). |\r\n| `kibana.alert.risk_score` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query sorts alerts by `kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses. |\r\n| `kibana.alert.workflow_status` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query filters alerts by `kibana.alert.workflow_status` to ensure only `open` and `acknowledged` alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses. |\r\n\r\n- Clicking the `Reset` button shown in the screenshot below will reset the user's `Anonymization` defaults, such that they include the additive changes in the table above:\r\n\r\n\r\n\r\n### Updated settings text\r\n\r\nThe text in the settings below was also updated:\r\n\r\n\r\n\r\n### Desk testing\r\n\r\nTo desk test this change:\r\n\r\n- Enable the `assistantRagOnAlerts` feature flag described in [#172542](https://github.com/elastic/kibana/pull/172542) must be enabled, per the following example:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']\r\n```\r\n\r\n- The `Alerts` feature must be enabled in the assistant settings, per the screenshot below:\r\n\r\n \r\n\r\n1) Navigate to Security > Alerts\r\n\r\n2) Click the `AI Assistant` button to open the assistant\r\n\r\n3) Click the `Settings` gear to open the assistant settings\r\n\r\n4) Click the `Anonymization` category\r\n\r\n5) Click the `Reset` button, shown in the screenshot below\r\n\r\n\r\n\r\n**Expected results**\r\n\r\n- `65` fields are allowed by default, per the screenshot above\r\n- `12` fields are anonymized by default, per the screenshot above\r\n- The `_id` field is allowed by default, per the screenshot above\r\n- The `_id` field is anonymized by default, per the screenshot above\r\n\r\n6) Type `kibana.alert.risk` in the search box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.risk_score` field is allowed by default\r\n\r\n7) Type `kibana.alert.workflow` in the search box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.workflow_status` field is allowed by default\r\n\r\n8) Click `Save`\r\n\r\n9) Click the `X` button to clear the conversation\r\n\r\n10) Close the assistant\r\n\r\n11) Add the following two fields as columns to the Alerts page table:\r\n\r\n- `kibana.alert.workflow_status`\r\n- `_id`\r\n\r\n12) Sort the table, first by `kibana.alert.risk_score` from high to low, and then by `@timestamp` from new to old, per the screenshot below:\r\n\r\n\r\n\r\n13) Filter the alerts page to only show `open` and `acknowledged` alerts\r\n\r\n**Expected result**\r\n\r\n- The alerts page has custom columns, sorting, and filtering, per the screenshot below:\r\n\r\n\r\n\r\n14) Click the `AI Assistant` button to open the assistant\r\n\r\n15) Ask the assistant:\r\n\r\n```\r\nWhat is the workflow status of my alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant will report on the workflow status of alerts, per the example response below:\r\n\r\n```\r\nThe workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system.\r\n```\r\n\r\n\r\n\r\n16) Close the assistant\r\n\r\n17) Change the workflow status of an alert in the Alerts table from `open` to `acknowledged`\r\n\r\n**Expected result**\r\n\r\n- The alerts table shows the updated alert, per the screenshot below:\r\n\r\n\r\n\r\n18) Once again, open the assistant\r\n\r\n19) Once again, ask the (same) question:\r\n\r\n```\r\nWhat is the workflow status of my alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The response from the assistant makes reference to the alert who's workflow status was changed from `open` to `acknowledged`, per the example response below:\r\n\r\n```\r\nBased on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure.\r\n```\r\n\r\n\r\n\r\n20) Ask the assistant for details about the acknowledged alerts:\r\n\r\n```\r\nWhat are the details of the acknowledged alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\nThe assistant for details about the acknowledged alert that, for example, includes the `kibana.alert.risk_score`, per the example response below:\r\n\r\n```\r\nIn response to your previous question, here are the details of the acknowledged alerts:\r\n\r\n1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'.\r\n\r\n2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'.\r\n\r\n3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'.\r\n\r\nPlease, take appropriate action to address these alerts.\r\n```\r\n\r\n\r\n\r\n21) Ask the assistant for the `_id` of the acknowledged alert:\r\n\r\n```\r\nWhat is the id of the acknowledged alert?\r\n```\r\n\r\n**Expected results**\r\n\r\n- The response from the assistant contains the `_id` of the `acknowledged` alert, per the example response below:\r\n\r\n```\r\nThe id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'.\r\n```\r\n\r\n\r\n\r\n- The `_id` shown in the assistant is the same `_id` of the acknowledged alert on the alerts page, per the screeenshot below:\r\n\r\n\r\n\r\n22) Click the `Show anonymized` toggle in the assistant\r\n\r\n**Expected result**\r\n\r\n- The `_id` shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below:\r\n\r\n","sha":"0d9c261530b102e7a7a87f4d5dcdf423cdd17a2c","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Threat Hunting:Investigations","Feature:Elastic AI Assistant","v8.12.0","v8.13.0"],"number":173121,"url":"https://github.com/elastic/kibana/pull/173121","mergeCommit":{"message":"[Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121)\n\n## [Security Solution] [Elastic AI Assistant] Include `acknowledged` alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts)\r\n\r\nThis PR updates the query used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542](https://github.com/elastic/kibana/pull/172542) to include alerts with a `kibana.alert.workflow_status` value of `acknowledged`.\r\n\r\nThe query previously only returned alerts with a status of `open`. This change ensures both `open` and `acknowledged` alerts are provided as context to the LLM.\r\n\r\n### Updated Anonymization defaults\r\n\r\nThree fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts.\r\n\r\nFor example, the LLM can refer to specific alerts by ID when the `_id` field is provided.\r\n\r\nThis PR makes the following additive changes to the Assistant's `Anonymization` defaults:\r\n\r\n| Field | Allow by default | Anonymize by default | Value add |\r\n|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n| `_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs). |\r\n| `kibana.alert.risk_score` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query sorts alerts by `kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses. |\r\n| `kibana.alert.workflow_status` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query filters alerts by `kibana.alert.workflow_status` to ensure only `open` and `acknowledged` alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses. |\r\n\r\n- Clicking the `Reset` button shown in the screenshot below will reset the user's `Anonymization` defaults, such that they include the additive changes in the table above:\r\n\r\n\r\n\r\n### Updated settings text\r\n\r\nThe text in the settings below was also updated:\r\n\r\n\r\n\r\n### Desk testing\r\n\r\nTo desk test this change:\r\n\r\n- Enable the `assistantRagOnAlerts` feature flag described in [#172542](https://github.com/elastic/kibana/pull/172542) must be enabled, per the following example:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']\r\n```\r\n\r\n- The `Alerts` feature must be enabled in the assistant settings, per the screenshot below:\r\n\r\n \r\n\r\n1) Navigate to Security > Alerts\r\n\r\n2) Click the `AI Assistant` button to open the assistant\r\n\r\n3) Click the `Settings` gear to open the assistant settings\r\n\r\n4) Click the `Anonymization` category\r\n\r\n5) Click the `Reset` button, shown in the screenshot below\r\n\r\n\r\n\r\n**Expected results**\r\n\r\n- `65` fields are allowed by default, per the screenshot above\r\n- `12` fields are anonymized by default, per the screenshot above\r\n- The `_id` field is allowed by default, per the screenshot above\r\n- The `_id` field is anonymized by default, per the screenshot above\r\n\r\n6) Type `kibana.alert.risk` in the search box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.risk_score` field is allowed by default\r\n\r\n7) Type `kibana.alert.workflow` in the search box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.workflow_status` field is allowed by default\r\n\r\n8) Click `Save`\r\n\r\n9) Click the `X` button to clear the conversation\r\n\r\n10) Close the assistant\r\n\r\n11) Add the following two fields as columns to the Alerts page table:\r\n\r\n- `kibana.alert.workflow_status`\r\n- `_id`\r\n\r\n12) Sort the table, first by `kibana.alert.risk_score` from high to low, and then by `@timestamp` from new to old, per the screenshot below:\r\n\r\n\r\n\r\n13) Filter the alerts page to only show `open` and `acknowledged` alerts\r\n\r\n**Expected result**\r\n\r\n- The alerts page has custom columns, sorting, and filtering, per the screenshot below:\r\n\r\n\r\n\r\n14) Click the `AI Assistant` button to open the assistant\r\n\r\n15) Ask the assistant:\r\n\r\n```\r\nWhat is the workflow status of my alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant will report on the workflow status of alerts, per the example response below:\r\n\r\n```\r\nThe workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system.\r\n```\r\n\r\n\r\n\r\n16) Close the assistant\r\n\r\n17) Change the workflow status of an alert in the Alerts table from `open` to `acknowledged`\r\n\r\n**Expected result**\r\n\r\n- The alerts table shows the updated alert, per the screenshot below:\r\n\r\n\r\n\r\n18) Once again, open the assistant\r\n\r\n19) Once again, ask the (same) question:\r\n\r\n```\r\nWhat is the workflow status of my alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The response from the assistant makes reference to the alert who's workflow status was changed from `open` to `acknowledged`, per the example response below:\r\n\r\n```\r\nBased on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure.\r\n```\r\n\r\n\r\n\r\n20) Ask the assistant for details about the acknowledged alerts:\r\n\r\n```\r\nWhat are the details of the acknowledged alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\nThe assistant for details about the acknowledged alert that, for example, includes the `kibana.alert.risk_score`, per the example response below:\r\n\r\n```\r\nIn response to your previous question, here are the details of the acknowledged alerts:\r\n\r\n1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'.\r\n\r\n2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'.\r\n\r\n3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'.\r\n\r\nPlease, take appropriate action to address these alerts.\r\n```\r\n\r\n\r\n\r\n21) Ask the assistant for the `_id` of the acknowledged alert:\r\n\r\n```\r\nWhat is the id of the acknowledged alert?\r\n```\r\n\r\n**Expected results**\r\n\r\n- The response from the assistant contains the `_id` of the `acknowledged` alert, per the example response below:\r\n\r\n```\r\nThe id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'.\r\n```\r\n\r\n\r\n\r\n- The `_id` shown in the assistant is the same `_id` of the acknowledged alert on the alerts page, per the screeenshot below:\r\n\r\n\r\n\r\n22) Click the `Show anonymized` toggle in the assistant\r\n\r\n**Expected result**\r\n\r\n- The `_id` shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below:\r\n\r\n","sha":"0d9c261530b102e7a7a87f4d5dcdf423cdd17a2c"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173121","number":173121,"mergeCommit":{"message":"[Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121)\n\n## [Security Solution] [Elastic AI Assistant] Include `acknowledged` alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts)\r\n\r\nThis PR updates the query used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542](https://github.com/elastic/kibana/pull/172542) to include alerts with a `kibana.alert.workflow_status` value of `acknowledged`.\r\n\r\nThe query previously only returned alerts with a status of `open`. This change ensures both `open` and `acknowledged` alerts are provided as context to the LLM.\r\n\r\n### Updated Anonymization defaults\r\n\r\nThree fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts.\r\n\r\nFor example, the LLM can refer to specific alerts by ID when the `_id` field is provided.\r\n\r\nThis PR makes the following additive changes to the Assistant's `Anonymization` defaults:\r\n\r\n| Field | Allow by default | Anonymize by default | Value add |\r\n|--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n| `_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs). |\r\n| `kibana.alert.risk_score` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query sorts alerts by `kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses. |\r\n| `kibana.alert.workflow_status` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query filters alerts by `kibana.alert.workflow_status` to ensure only `open` and `acknowledged` alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses. |\r\n\r\n- Clicking the `Reset` button shown in the screenshot below will reset the user's `Anonymization` defaults, such that they include the additive changes in the table above:\r\n\r\n\r\n\r\n### Updated settings text\r\n\r\nThe text in the settings below was also updated:\r\n\r\n\r\n\r\n### Desk testing\r\n\r\nTo desk test this change:\r\n\r\n- Enable the `assistantRagOnAlerts` feature flag described in [#172542](https://github.com/elastic/kibana/pull/172542) must be enabled, per the following example:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']\r\n```\r\n\r\n- The `Alerts` feature must be enabled in the assistant settings, per the screenshot below:\r\n\r\n \r\n\r\n1) Navigate to Security > Alerts\r\n\r\n2) Click the `AI Assistant` button to open the assistant\r\n\r\n3) Click the `Settings` gear to open the assistant settings\r\n\r\n4) Click the `Anonymization` category\r\n\r\n5) Click the `Reset` button, shown in the screenshot below\r\n\r\n\r\n\r\n**Expected results**\r\n\r\n- `65` fields are allowed by default, per the screenshot above\r\n- `12` fields are anonymized by default, per the screenshot above\r\n- The `_id` field is allowed by default, per the screenshot above\r\n- The `_id` field is anonymized by default, per the screenshot above\r\n\r\n6) Type `kibana.alert.risk` in the search box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.risk_score` field is allowed by default\r\n\r\n7) Type `kibana.alert.workflow` in the search box\r\n\r\n**Expected result**\r\n\r\n- The `kibana.alert.workflow_status` field is allowed by default\r\n\r\n8) Click `Save`\r\n\r\n9) Click the `X` button to clear the conversation\r\n\r\n10) Close the assistant\r\n\r\n11) Add the following two fields as columns to the Alerts page table:\r\n\r\n- `kibana.alert.workflow_status`\r\n- `_id`\r\n\r\n12) Sort the table, first by `kibana.alert.risk_score` from high to low, and then by `@timestamp` from new to old, per the screenshot below:\r\n\r\n\r\n\r\n13) Filter the alerts page to only show `open` and `acknowledged` alerts\r\n\r\n**Expected result**\r\n\r\n- The alerts page has custom columns, sorting, and filtering, per the screenshot below:\r\n\r\n\r\n\r\n14) Click the `AI Assistant` button to open the assistant\r\n\r\n15) Ask the assistant:\r\n\r\n```\r\nWhat is the workflow status of my alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant will report on the workflow status of alerts, per the example response below:\r\n\r\n```\r\nThe workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system.\r\n```\r\n\r\n\r\n\r\n16) Close the assistant\r\n\r\n17) Change the workflow status of an alert in the Alerts table from `open` to `acknowledged`\r\n\r\n**Expected result**\r\n\r\n- The alerts table shows the updated alert, per the screenshot below:\r\n\r\n\r\n\r\n18) Once again, open the assistant\r\n\r\n19) Once again, ask the (same) question:\r\n\r\n```\r\nWhat is the workflow status of my alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The response from the assistant makes reference to the alert who's workflow status was changed from `open` to `acknowledged`, per the example response below:\r\n\r\n```\r\nBased on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure.\r\n```\r\n\r\n\r\n\r\n20) Ask the assistant for details about the acknowledged alerts:\r\n\r\n```\r\nWhat are the details of the acknowledged alerts?\r\n```\r\n\r\n**Expected result**\r\n\r\nThe assistant for details about the acknowledged alert that, for example, includes the `kibana.alert.risk_score`, per the example response below:\r\n\r\n```\r\nIn response to your previous question, here are the details of the acknowledged alerts:\r\n\r\n1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'.\r\n\r\n2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'.\r\n\r\n3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'.\r\n\r\nPlease, take appropriate action to address these alerts.\r\n```\r\n\r\n\r\n\r\n21) Ask the assistant for the `_id` of the acknowledged alert:\r\n\r\n```\r\nWhat is the id of the acknowledged alert?\r\n```\r\n\r\n**Expected results**\r\n\r\n- The response from the assistant contains the `_id` of the `acknowledged` alert, per the example response below:\r\n\r\n```\r\nThe id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'.\r\n```\r\n\r\n\r\n\r\n- The `_id` shown in the assistant is the same `_id` of the acknowledged alert on the alerts page, per the screeenshot below:\r\n\r\n\r\n\r\n22) Click the `Show anonymized` toggle in the assistant\r\n\r\n**Expected result**\r\n\r\n- The `_id` shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below:\r\n\r\n","sha":"0d9c261530b102e7a7a87f4d5dcdf423cdd17a2c"}}]}] BACKPORT--> Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
This commit is contained in:
Kibana Machine
GitHub
parent
5716fb945f
commit
825217b784
10 changed files with 92 additions and 26 deletions
|
|
@ -109,6 +109,18 @@ const AlertsSettingsComponent = ({ knowledgeBase, setUpdatedKnowledgeBaseSetting
|
|||
<span>{i18n.LATEST_AND_RISKIEST_OPEN_ALERTS}</span>
|
||||
</EuiText>
|
||||
</EuiFlexItem>
|
||||
|
||||
<EuiFlexItem grow={false}>
|
||||
<EuiText color="subdued" size="xs">
|
||||
<span>{i18n.YOUR_ANONYMIZATION_SETTINGS}</span>
|
||||
</EuiText>
|
||||
</EuiFlexItem>
|
||||
|
||||
<EuiFlexItem grow={false}>
|
||||
<EuiText color="subdued" size="xs">
|
||||
<span>{i18n.SELECT_FEWER_ALERTS}</span>
|
||||
</EuiText>
|
||||
</EuiFlexItem>
|
||||
</EuiFlexGroup>
|
||||
</>
|
||||
);
|
||||
|
|
|
|||
|
|
@ -23,9 +23,23 @@ export const ASK_QUESTIONS_ABOUT = i18n.translate(
|
|||
|
||||
export const LATEST_AND_RISKIEST_OPEN_ALERTS = i18n.translate(
|
||||
'xpack.elasticAssistant.assistant.settings.knowledgeBaseSettings.latestAndRiskiestOpenAlertsLabel',
|
||||
{
|
||||
defaultMessage: 'latest and riskiest open and acknowledged alerts in your environment.',
|
||||
}
|
||||
);
|
||||
|
||||
export const YOUR_ANONYMIZATION_SETTINGS = i18n.translate(
|
||||
'xpack.elasticAssistant.assistant.settings.knowledgeBaseSettings.yourAnonymizationSettingsLabel',
|
||||
{
|
||||
defaultMessage: 'Your Anonymization settings will be applied to the alerts.',
|
||||
}
|
||||
);
|
||||
|
||||
export const SELECT_FEWER_ALERTS = i18n.translate(
|
||||
'xpack.elasticAssistant.assistant.settings.knowledgeBaseSettings.selectFewerAlertsLabel',
|
||||
{
|
||||
defaultMessage:
|
||||
'latest and riskiest open alerts in your environment. Your Anonymization settings will be applied to the alerts',
|
||||
"Select fewer alerts if the model's maximum context length is frequently exceeded.",
|
||||
}
|
||||
);
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ import { Tool } from 'langchain/tools';
|
|||
|
||||
import { getAlertCountsTool } from './alert_counts/get_alert_counts_tool';
|
||||
import { getEsqlLanguageKnowledgeBaseTool } from './esql_language_knowledge_base/get_esql_language_knowledge_base_tool';
|
||||
import { getOpenAlertsTool } from './open_alerts/get_open_alerts_tool';
|
||||
import { getOpenAndAcknowledgedAlertsTool } from './open_and_acknowledged_alerts/get_open_and_acknowledged_alerts_tool';
|
||||
import type { RequestBody } from '../types';
|
||||
|
||||
export interface GetApplicableTools {
|
||||
|
|
@ -50,7 +50,7 @@ export const getApplicableTools = ({
|
|||
replacements,
|
||||
request,
|
||||
}) ?? [],
|
||||
getOpenAlertsTool({
|
||||
getOpenAndAcknowledgedAlertsTool({
|
||||
alertsIndexPattern,
|
||||
allow,
|
||||
allowReplacement,
|
||||
|
|
|
|||
|
|
@ -5,15 +5,15 @@
|
|||
* 2.0.
|
||||
*/
|
||||
|
||||
import { getOpenAlertsQuery } from './get_open_alerts_query';
|
||||
import { getOpenAndAcknowledgedAlertsQuery } from './get_open_and_acknowledged_alerts_query';
|
||||
|
||||
describe('getOpenAlertsQuery', () => {
|
||||
describe('getOpenAndAcknowledgedAlertsQuery', () => {
|
||||
it('returns the expected query', () => {
|
||||
const alertsIndexPattern = 'alerts-*';
|
||||
const allow = ['field1', 'field2'];
|
||||
const size = 10;
|
||||
|
||||
const query = getOpenAlertsQuery({ alertsIndexPattern, allow, size });
|
||||
const query = getOpenAndAcknowledgedAlertsQuery({ alertsIndexPattern, allow, size });
|
||||
|
||||
expect(query).toEqual({
|
||||
allow_no_indices: true,
|
||||
|
|
@ -30,8 +30,20 @@ describe('getOpenAlertsQuery', () => {
|
|||
must: [],
|
||||
filter: [
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'open',
|
||||
bool: {
|
||||
should: [
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'open',
|
||||
},
|
||||
},
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'acknowledged',
|
||||
},
|
||||
},
|
||||
],
|
||||
minimum_should_match: 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
|
|
@ -5,7 +5,7 @@
|
|||
* 2.0.
|
||||
*/
|
||||
|
||||
export const getOpenAlertsQuery = ({
|
||||
export const getOpenAndAcknowledgedAlertsQuery = ({
|
||||
alertsIndexPattern,
|
||||
allow,
|
||||
size,
|
||||
|
|
@ -28,8 +28,20 @@ export const getOpenAlertsQuery = ({
|
|||
must: [],
|
||||
filter: [
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'open',
|
||||
bool: {
|
||||
should: [
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'open',
|
||||
},
|
||||
},
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'acknowledged',
|
||||
},
|
||||
},
|
||||
],
|
||||
minimum_should_match: 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
|
|
@ -10,12 +10,12 @@ import type { KibanaRequest } from '@kbn/core-http-server';
|
|||
import { DynamicTool } from 'langchain/tools';
|
||||
import { omit } from 'lodash/fp';
|
||||
|
||||
import { getOpenAlertsTool } from './get_open_alerts_tool';
|
||||
import { getOpenAndAcknowledgedAlertsTool } from './get_open_and_acknowledged_alerts_tool';
|
||||
import { mockAlertsFieldsApi } from '../../../../__mocks__/alerts';
|
||||
import type { RequestBody } from '../../types';
|
||||
import { MAX_SIZE } from './helpers';
|
||||
|
||||
describe('getOpenAlertsTool', () => {
|
||||
describe('getOpenAndAcknowledgedAlertsTool', () => {
|
||||
const alertsIndexPattern = 'alerts-index';
|
||||
const esClient = {
|
||||
search: jest.fn().mockResolvedValue(mockAlertsFieldsApi),
|
||||
|
|
@ -37,7 +37,7 @@ describe('getOpenAlertsTool', () => {
|
|||
});
|
||||
|
||||
it('returns a `DynamicTool` with a `func` that calls `esClient.search()` with the expected query', async () => {
|
||||
const tool: DynamicTool = getOpenAlertsTool({
|
||||
const tool: DynamicTool = getOpenAndAcknowledgedAlertsTool({
|
||||
alertsIndexPattern,
|
||||
allow: request.body.allow,
|
||||
allowReplacement: request.body.allowReplacement,
|
||||
|
|
@ -75,8 +75,20 @@ describe('getOpenAlertsTool', () => {
|
|||
bool: {
|
||||
filter: [
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'open',
|
||||
bool: {
|
||||
should: [
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'open',
|
||||
},
|
||||
},
|
||||
{
|
||||
match_phrase: {
|
||||
'kibana.alert.workflow_status': 'acknowledged',
|
||||
},
|
||||
},
|
||||
],
|
||||
minimum_should_match: 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
|
|
@ -130,7 +142,7 @@ describe('getOpenAlertsTool', () => {
|
|||
RequestBody
|
||||
>;
|
||||
|
||||
const tool = getOpenAlertsTool({
|
||||
const tool = getOpenAndAcknowledgedAlertsTool({
|
||||
alertsIndexPattern,
|
||||
allow: requestWithMissingParams.body.allow,
|
||||
allowReplacement: requestWithMissingParams.body.allowReplacement,
|
||||
|
|
@ -145,7 +157,7 @@ describe('getOpenAlertsTool', () => {
|
|||
});
|
||||
|
||||
it('returns null when alertsIndexPattern is undefined', () => {
|
||||
const tool = getOpenAlertsTool({
|
||||
const tool = getOpenAndAcknowledgedAlertsTool({
|
||||
// alertsIndexPattern is undefined
|
||||
allow: request.body.allow,
|
||||
allowReplacement: request.body.allowReplacement,
|
||||
|
|
@ -160,7 +172,7 @@ describe('getOpenAlertsTool', () => {
|
|||
});
|
||||
|
||||
it('returns null when size is undefined', () => {
|
||||
const tool = getOpenAlertsTool({
|
||||
const tool = getOpenAndAcknowledgedAlertsTool({
|
||||
alertsIndexPattern,
|
||||
allow: request.body.allow,
|
||||
allowReplacement: request.body.allowReplacement,
|
||||
|
|
@ -175,7 +187,7 @@ describe('getOpenAlertsTool', () => {
|
|||
});
|
||||
|
||||
it('returns null when size out of range', () => {
|
||||
const tool = getOpenAlertsTool({
|
||||
const tool = getOpenAndAcknowledgedAlertsTool({
|
||||
alertsIndexPattern,
|
||||
allow: request.body.allow,
|
||||
allowReplacement: request.body.allowReplacement,
|
||||
|
|
@ -190,7 +202,7 @@ describe('getOpenAlertsTool', () => {
|
|||
});
|
||||
|
||||
it('returns a tool instance with the expected tags', () => {
|
||||
const tool = getOpenAlertsTool({
|
||||
const tool = getOpenAndAcknowledgedAlertsTool({
|
||||
alertsIndexPattern,
|
||||
allow: request.body.allow,
|
||||
allowReplacement: request.body.allowReplacement,
|
||||
|
|
@ -13,17 +13,17 @@ import { DynamicTool, Tool } from 'langchain/tools';
|
|||
import { requestHasRequiredAnonymizationParams } from '../../helpers';
|
||||
import { RequestBody } from '../../types';
|
||||
|
||||
import { getOpenAlertsQuery } from './get_open_alerts_query';
|
||||
import { getOpenAndAcknowledgedAlertsQuery } from './get_open_and_acknowledged_alerts_query';
|
||||
import { getRawDataOrDefault, sizeIsOutOfRange } from './helpers';
|
||||
|
||||
export const OPEN_ALERTS_TOOL_DESCRIPTION =
|
||||
'Call this for knowledge about the latest n open alerts (sorted by `kibana.alert.risk_score`) in the environment, or when answering questions about open alerts';
|
||||
|
||||
/**
|
||||
* Returns a tool for querying open alerts, or null if the request
|
||||
* doesn't have all the required parameters.
|
||||
* Returns a tool for querying open and acknowledged alerts, or null if the
|
||||
* request doesn't have all the required parameters.
|
||||
*/
|
||||
export const getOpenAlertsTool = ({
|
||||
export const getOpenAndAcknowledgedAlertsTool = ({
|
||||
alertsIndexPattern,
|
||||
allow,
|
||||
allowReplacement,
|
||||
|
|
@ -55,7 +55,7 @@ export const getOpenAlertsTool = ({
|
|||
name: 'open-alerts',
|
||||
description: OPEN_ALERTS_TOOL_DESCRIPTION,
|
||||
func: async () => {
|
||||
const query = getOpenAlertsQuery({
|
||||
const query = getOpenAndAcknowledgedAlertsQuery({
|
||||
alertsIndexPattern,
|
||||
allow: allow ?? [],
|
||||
size,
|
||||
|
|
@ -7,6 +7,7 @@
|
|||
|
||||
/** By default, these fields are allowed to be sent to the assistant */
|
||||
export const DEFAULT_ALLOW = [
|
||||
'_id',
|
||||
'@timestamp',
|
||||
'cloud.availability_zone',
|
||||
'cloud.provider',
|
||||
|
|
@ -28,6 +29,7 @@ export const DEFAULT_ALLOW = [
|
|||
'host.risk.calculated_level',
|
||||
'host.risk.calculated_score_norm',
|
||||
'kibana.alert.last_detected',
|
||||
'kibana.alert.risk_score',
|
||||
'kibana.alert.rule.description',
|
||||
'kibana.alert.rule.name',
|
||||
'kibana.alert.rule.references',
|
||||
|
|
@ -42,6 +44,7 @@ export const DEFAULT_ALLOW = [
|
|||
'kibana.alert.rule.threat.technique.subtechnique.name',
|
||||
'kibana.alert.rule.threat.technique.subtechnique.reference',
|
||||
'kibana.alert.severity',
|
||||
'kibana.alert.workflow_status',
|
||||
'process.args',
|
||||
'process.command_line',
|
||||
'process.executable',
|
||||
|
|
@ -73,6 +76,7 @@ export const DEFAULT_ALLOW = [
|
|||
|
||||
/** By default, these fields will be anonymized */
|
||||
export const DEFAULT_ALLOW_REPLACEMENT = [
|
||||
'_id', // the document's _id is replaced with an anonymized value
|
||||
'cloud.availability_zone',
|
||||
'cloud.provider',
|
||||
'cloud.region',
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue