[8.18] [Security Solution] [Attack discovery] Fixes intermittent refinement step error (#215816) (#215965)

# Backport

This will backport the following commits from `main` to `8.18`:
- [[Security Solution] [Attack discovery] Fixes intermittent refinement
step error (#215816)](https://github.com/elastic/kibana/pull/215816)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Andrew
Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2025-03-25T22:10:58Z","message":"[Security
Solution] [Attack discovery] Fixes intermittent refinement step error
(#215816)\n\n## [Security Solution] [Attack discovery] Fixes
intermittent refinement step error\n\nThis PR updates the refine prompt
to fix the following intermittent error, which sometimes occurs during
the refine step in the Attack discovery Langchain graph:\n\n```\nrefine
node is unable to parse (gemini) response from attempt 1; (this may be
an incomplete response from the model): [\n {\n \"code\":
\"invalid_type\",\n \"expected\": \"object\",\n \"received\":
\"array\",\n \"path\": [],\n \"message\": \"Expected object, received
array\"\n }\n]\n```\n\nThe fix wraps the input to the refine prompt with
an opening / closing `json` codeblock, in an object with an `insights`
key:\n\n````\n\"\"\"\n```json\n{\n \"insights\": [\n // ...\n
]\n}\n```\n\"\"\"\n````\n\n### Desk testing\n\n1. Navigate to Security >
Attack discovery\n\n2. Click the `Generate` button to generate Attack
discoveries\n\n3. When generation completes, open the entry for the
completed run in LangGraph\n\n4. In the LangGraph waterfall, click on
the `ActionsClientLlm` entry for the `refine` step\n\n**Expected
result**\n\nThe input to the refine prompt is wrapped with an opening /
closing `json` codeblock, in an object with an `insights` key, as
illustrated by the following screenshot and
example:\n\n![langgraph](https://github.com/user-attachments/assets/d1ec75f9-4201-4ade-a876-170fab41f89b)\n\n````\n//
...\n- Conform exactly to the JSON schema defined earlier\n- Do not
include explanatory text outside the JSON\n\n\n\"\"\"\n```json\n{\n
\"insights\": [\n {\n \"alertIds\": [\n
\"086469904a1ba57f4114466af23bbe2d0c62dde193a2fd4afd4ba3c4b4fc079f\",\n
\"21ca4e4f082fd68ae2ad9a953fb5cfc9395a1769602011684750e95b36a79a99\",\n
\"7a816e5db9464fcea1ba44ad28f4256e1fce079336bd9c32c9933c12fcdeb901\",\n
\"986503ca78da6496646564a467e5aee9bf7fbb347bf0b017f3a57475f3546fa3\"\n
],\n \"detailsMarkdown\": \"- A malicious OneNote file was opened on {{
host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}\\n- This triggered the
execution of a suspicious Go application: {{ process.name My Go
Application.app }}\\n- The Go application then launched a malicious
binary {{ file.name unix1 }} located at {{ file.path /Users/james/unix1
}}\\n- The malicious binary attempted to access the user's keychain at
{{ process.command_line /Users/james/unix1
/Users/james/library/Keychains/login.keychain-db TempTemp1234!! }}\\n-
Multiple alerts were generated for this malware execution chain\",\n
\"mitreAttackTactics\": [\n \"Initial Access\",\n \"Execution\",\n
\"Credential Access\"\n ],\n \"summaryMarkdown\": \"A malicious OneNote
attachment was opened, leading to the execution of malware on {{
host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}. The malware was
detected as it attempted to access sensitive system files.\",\n
\"title\": \"Malware Execution from OneNote Attachment\",\n
\"timestamp\": \"2025-03-25T03:16:20.526Z\"\n },\n //
...\n]\n}\n```\n\"\"\"\n````","sha":"1d457e4b1bdc444f956f7f8feec6cc3415a4a605","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","ci:cloud-deploy","ci:cloud-persist-deployment","Team:Security
Generative
AI","backport:version","v9.1.0","v8.19.0","v8.18.1","v9.0.1"],"title":"[Security
Solution] [Attack discovery] Fixes intermittent refinement step
error","number":215816,"url":"https://github.com/elastic/kibana/pull/215816","mergeCommit":{"message":"[Security
Solution] [Attack discovery] Fixes intermittent refinement step error
(#215816)\n\n## [Security Solution] [Attack discovery] Fixes
intermittent refinement step error\n\nThis PR updates the refine prompt
to fix the following intermittent error, which sometimes occurs during
the refine step in the Attack discovery Langchain graph:\n\n```\nrefine
node is unable to parse (gemini) response from attempt 1; (this may be
an incomplete response from the model): [\n {\n \"code\":
\"invalid_type\",\n \"expected\": \"object\",\n \"received\":
\"array\",\n \"path\": [],\n \"message\": \"Expected object, received
array\"\n }\n]\n```\n\nThe fix wraps the input to the refine prompt with
an opening / closing `json` codeblock, in an object with an `insights`
key:\n\n````\n\"\"\"\n```json\n{\n \"insights\": [\n // ...\n
]\n}\n```\n\"\"\"\n````\n\n### Desk testing\n\n1. Navigate to Security >
Attack discovery\n\n2. Click the `Generate` button to generate Attack
discoveries\n\n3. When generation completes, open the entry for the
completed run in LangGraph\n\n4. In the LangGraph waterfall, click on
the `ActionsClientLlm` entry for the `refine` step\n\n**Expected
result**\n\nThe input to the refine prompt is wrapped with an opening /
closing `json` codeblock, in an object with an `insights` key, as
illustrated by the following screenshot and
example:\n\n![langgraph](https://github.com/user-attachments/assets/d1ec75f9-4201-4ade-a876-170fab41f89b)\n\n````\n//
...\n- Conform exactly to the JSON schema defined earlier\n- Do not
include explanatory text outside the JSON\n\n\n\"\"\"\n```json\n{\n
\"insights\": [\n {\n \"alertIds\": [\n
\"086469904a1ba57f4114466af23bbe2d0c62dde193a2fd4afd4ba3c4b4fc079f\",\n
\"21ca4e4f082fd68ae2ad9a953fb5cfc9395a1769602011684750e95b36a79a99\",\n
\"7a816e5db9464fcea1ba44ad28f4256e1fce079336bd9c32c9933c12fcdeb901\",\n
\"986503ca78da6496646564a467e5aee9bf7fbb347bf0b017f3a57475f3546fa3\"\n
],\n \"detailsMarkdown\": \"- A malicious OneNote file was opened on {{
host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}\\n- This triggered the
execution of a suspicious Go application: {{ process.name My Go
Application.app }}\\n- The Go application then launched a malicious
binary {{ file.name unix1 }} located at {{ file.path /Users/james/unix1
}}\\n- The malicious binary attempted to access the user's keychain at
{{ process.command_line /Users/james/unix1
/Users/james/library/Keychains/login.keychain-db TempTemp1234!! }}\\n-
Multiple alerts were generated for this malware execution chain\",\n
\"mitreAttackTactics\": [\n \"Initial Access\",\n \"Execution\",\n
\"Credential Access\"\n ],\n \"summaryMarkdown\": \"A malicious OneNote
attachment was opened, leading to the execution of malware on {{
host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}. The malware was
detected as it attempted to access sensitive system files.\",\n
\"title\": \"Malware Execution from OneNote Attachment\",\n
\"timestamp\": \"2025-03-25T03:16:20.526Z\"\n },\n //
...\n]\n}\n```\n\"\"\"\n````","sha":"1d457e4b1bdc444f956f7f8feec6cc3415a4a605"}},"sourceBranch":"main","suggestedTargetBranches":["8.x","8.18","9.0"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/215816","number":215816,"mergeCommit":{"message":"[Security
Solution] [Attack discovery] Fixes intermittent refinement step error
(#215816)\n\n## [Security Solution] [Attack discovery] Fixes
intermittent refinement step error\n\nThis PR updates the refine prompt
to fix the following intermittent error, which sometimes occurs during
the refine step in the Attack discovery Langchain graph:\n\n```\nrefine
node is unable to parse (gemini) response from attempt 1; (this may be
an incomplete response from the model): [\n {\n \"code\":
\"invalid_type\",\n \"expected\": \"object\",\n \"received\":
\"array\",\n \"path\": [],\n \"message\": \"Expected object, received
array\"\n }\n]\n```\n\nThe fix wraps the input to the refine prompt with
an opening / closing `json` codeblock, in an object with an `insights`
key:\n\n````\n\"\"\"\n```json\n{\n \"insights\": [\n // ...\n
]\n}\n```\n\"\"\"\n````\n\n### Desk testing\n\n1. Navigate to Security >
Attack discovery\n\n2. Click the `Generate` button to generate Attack
discoveries\n\n3. When generation completes, open the entry for the
completed run in LangGraph\n\n4. In the LangGraph waterfall, click on
the `ActionsClientLlm` entry for the `refine` step\n\n**Expected
result**\n\nThe input to the refine prompt is wrapped with an opening /
closing `json` codeblock, in an object with an `insights` key, as
illustrated by the following screenshot and
example:\n\n![langgraph](https://github.com/user-attachments/assets/d1ec75f9-4201-4ade-a876-170fab41f89b)\n\n````\n//
...\n- Conform exactly to the JSON schema defined earlier\n- Do not
include explanatory text outside the JSON\n\n\n\"\"\"\n```json\n{\n
\"insights\": [\n {\n \"alertIds\": [\n
\"086469904a1ba57f4114466af23bbe2d0c62dde193a2fd4afd4ba3c4b4fc079f\",\n
\"21ca4e4f082fd68ae2ad9a953fb5cfc9395a1769602011684750e95b36a79a99\",\n
\"7a816e5db9464fcea1ba44ad28f4256e1fce079336bd9c32c9933c12fcdeb901\",\n
\"986503ca78da6496646564a467e5aee9bf7fbb347bf0b017f3a57475f3546fa3\"\n
],\n \"detailsMarkdown\": \"- A malicious OneNote file was opened on {{
host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}\\n- This triggered the
execution of a suspicious Go application: {{ process.name My Go
Application.app }}\\n- The Go application then launched a malicious
binary {{ file.name unix1 }} located at {{ file.path /Users/james/unix1
}}\\n- The malicious binary attempted to access the user's keychain at
{{ process.command_line /Users/james/unix1
/Users/james/library/Keychains/login.keychain-db TempTemp1234!! }}\\n-
Multiple alerts were generated for this malware execution chain\",\n
\"mitreAttackTactics\": [\n \"Initial Access\",\n \"Execution\",\n
\"Credential Access\"\n ],\n \"summaryMarkdown\": \"A malicious OneNote
attachment was opened, leading to the execution of malware on {{
host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}. The malware was
detected as it attempted to access sensitive system files.\",\n
\"title\": \"Malware Execution from OneNote Attachment\",\n
\"timestamp\": \"2025-03-25T03:16:20.526Z\"\n },\n //
...\n]\n}\n```\n\"\"\"\n````","sha":"1d457e4b1bdc444f956f7f8feec6cc3415a4a605"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.0","label":"v9.0.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Andrew Macri <andrew.macri@elastic.co>

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/nodes/refine/helpers/get_combined_refine_prompt/index.test.ts

@@ -24,7 +24,11 @@ describe('getCombinedRefinePrompt', () => {
 Refine prompt
 
 """
-${JSON.stringify(mockAttackDiscoveries, null, 2)}
+\`\`\`json
+{
+  "insights": ${JSON.stringify(mockAttackDiscoveries, null, 2)}
+}
+\`\`\`
 """
 
 `);
@@ -44,7 +48,11 @@ ${JSON.stringify(mockAttackDiscoveries, null, 2)}
 Refine prompt
 
 """
-${JSON.stringify(mockAttackDiscoveries, null, 2)}
+\`\`\`json
+{
+  "insights": ${JSON.stringify(mockAttackDiscoveries, null, 2)}
+}
+\`\`\`
 """
 
 
@@ -72,7 +80,11 @@ Combined refinements
 Refine prompt
 
 """
-null
+\`\`\`json
+{
+  "insights": null
+}
+\`\`\`
 """
 
 `);

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/nodes/refine/helpers/get_combined_refine_prompt/index.ts

@@ -29,7 +29,11 @@ export const getCombinedRefinePrompt = ({
 ${refinePrompt}
 
 """
-${JSON.stringify(unrefinedResults, null, 2)}
+\`\`\`json
+{
+  "insights": ${JSON.stringify(unrefinedResults, null, 2)}
+}
+\`\`\`
 """
 
 `;

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/nodes/refine/index.test.ts

@@ -142,7 +142,11 @@ describe('getRefineNode', () => {
 ${ATTACK_DISCOVERY_REFINE}
 
 \"\"\"
-${JSON.stringify(initialGraphState.unrefinedResults, null, 2)}
+\`\`\`json
+{
+  "insights": ${JSON.stringify(initialGraphState.unrefinedResults, null, 2)}
+}
+\`\`\`
 \"\"\"
 
 `,

x-pack/solutions/security/plugins/elastic_assistant/server/lib/prompt/prompts.ts

@@ -95,7 +95,7 @@ Analysis Process:
 Output Requirements:
 - Provide a narrative summary for each identified attack chain
 - Explain connections between events with concrete evidence
-- Use the special {{ field.name fieldValue }} syntax to reference source data fields`;
+- Use the special {{ field.name fieldValue }} syntax to reference source data fields. IMPORTANT - LIMIT the details markdown to 2750 characters and summary to 200 characters! This is to prevent hitting output context limits.`;
 
 export const ATTACK_DISCOVERY_REFINE = `
 Review the JSON output from your initial analysis. Your task is to refine the attack chains by: