kibana/x-pack/packages/kbn-elastic-assistant/impl/assistant/helpers.test.ts
Andrew Macri 3f0fa7d245
[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts (#172542)
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts

This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example:

- _How many alerts are currently open?_
- _Which alerts should I look at first?_
- _Did we have any alerts with suspicious activity on Windows machines?_

### More context

Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature:

- Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097)
- Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts
  - Only fields allowed by the user will be sent as context to the LLM
  - Users may enable or disable anonymization for specific fields (via settings)
  - Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:
  ![show_anonymized](7db85f69-9352-4422-adbf-c97248ccb3dd)

### Settings

This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:
![rag_on_alerts_setting](9161b6d4-b7c3-4f37-bcde-f032f5a02966)

- The `Alerts` toggle enables or disables the feature
- The slider has a range of `10` - `100` alerts (default: `20`)

When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned:

- the `kibana.alert.workflow_status` must be `open`
- the alert must have been generated in the last `24 hours`
- the alert must NOT be a `kibana.alert.building_block_type` alert
- the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts

### Feature flag

To use this feature:

1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below:

```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```

2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:

![alerts_toggle](07f241ea-af4a-43a4-bd19-0dc6337db167)

## How it works

- When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters:
  - `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default`
  - `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g.  `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]`
  - `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]`
  - `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g.

```json
"replacements": {
    "e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm",
    "cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia",
    "7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s",
    "f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc",
    // ...
},
```

- `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20`

- The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.)

- The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function:

```typescript
  const tools: Tool[] = getApplicableTools({
    allow,
    allowReplacement,
    alertsIndexPattern,
    assistantLangChain,
    chain,
    esClient,
    modelExists,
    onNewReplacements,
    replacements,
    request,
    size,
  });
```

- The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned.

- The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details

- The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours  (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details.

- On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared

- Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common`
  - The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
2023-12-06 00:56:04 -05:00

338 lines
11 KiB
TypeScript

/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import {
getBlockBotConversation,
getDefaultConnector,
getFormattedMessageContent,
getOptionalRequestParams,
hasParsableResponse,
} from './helpers';
import { enterpriseMessaging } from './use_conversation/sample_conversations';
import { ActionConnector } from '@kbn/triggers-actions-ui-plugin/public';
describe('getBlockBotConversation', () => {
describe('isAssistantEnabled = false', () => {
const isAssistantEnabled = false;
it('When no conversation history, return only enterprise messaging', () => {
const conversation = {
id: 'conversation_id',
theme: {},
messages: [],
apiConfig: {},
};
const result = getBlockBotConversation(conversation, isAssistantEnabled);
expect(result.messages).toEqual(enterpriseMessaging);
expect(result.messages.length).toEqual(1);
});
it('When conversation history and the last message is not enterprise messaging, appends enterprise messaging to conversation', () => {
const conversation = {
id: 'conversation_id',
theme: {},
messages: [
{
role: 'user' as const,
content: 'Hello',
timestamp: '',
presentation: {
delay: 0,
stream: false,
},
},
],
apiConfig: {},
};
const result = getBlockBotConversation(conversation, isAssistantEnabled);
expect(result.messages.length).toEqual(2);
});
it('returns the conversation without changes when the last message is enterprise messaging', () => {
const conversation = {
id: 'conversation_id',
theme: {},
messages: enterpriseMessaging,
apiConfig: {},
};
const result = getBlockBotConversation(conversation, isAssistantEnabled);
expect(result.messages.length).toEqual(1);
expect(result.messages).toEqual(enterpriseMessaging);
});
it('returns the conversation with new enterprise message when conversation has enterprise messaging, but not as the last message', () => {
const conversation = {
id: 'conversation_id',
theme: {},
messages: [
...enterpriseMessaging,
{
role: 'user' as const,
content: 'Hello',
timestamp: '',
presentation: {
delay: 0,
stream: false,
},
},
],
apiConfig: {},
};
const result = getBlockBotConversation(conversation, isAssistantEnabled);
expect(result.messages.length).toEqual(3);
});
});
describe('isAssistantEnabled = true', () => {
const isAssistantEnabled = true;
it('when no conversation history, returns the welcome conversation', () => {
const conversation = {
id: 'conversation_id',
theme: {},
messages: [],
apiConfig: {},
};
const result = getBlockBotConversation(conversation, isAssistantEnabled);
expect(result.messages.length).toEqual(3);
});
it('returns a conversation history with the welcome conversation appended', () => {
const conversation = {
id: 'conversation_id',
theme: {},
messages: [
{
role: 'user' as const,
content: 'Hello',
timestamp: '',
presentation: {
delay: 0,
stream: false,
},
},
],
apiConfig: {},
};
const result = getBlockBotConversation(conversation, isAssistantEnabled);
expect(result.messages.length).toEqual(4);
});
});
describe('getDefaultConnector', () => {
it('should return undefined if connectors array is undefined', () => {
const connectors = undefined;
const result = getDefaultConnector(connectors);
expect(result).toBeUndefined();
});
it('should return undefined if connectors array is empty', () => {
const connectors: Array<ActionConnector<Record<string, unknown>, Record<string, unknown>>> =
[];
const result = getDefaultConnector(connectors);
expect(result).toBeUndefined();
});
it('should return the connector id if there is only one connector', () => {
const connectors: Array<ActionConnector<Record<string, unknown>, Record<string, unknown>>> = [
{
actionTypeId: '.gen-ai',
isPreconfigured: false,
isDeprecated: false,
referencedByCount: 0,
isMissingSecrets: false,
isSystemAction: false,
secrets: {},
id: 'c5f91dc0-2197-11ee-aded-897192c5d6f5',
name: 'OpenAI',
config: {
apiProvider: 'OpenAI',
apiUrl: 'https://api.openai.com/v1/chat/completions',
},
},
];
const result = getDefaultConnector(connectors);
expect(result).toBe(connectors[0]);
});
it('should return undefined if there are multiple connectors', () => {
const connectors: Array<ActionConnector<Record<string, unknown>, Record<string, unknown>>> = [
{
actionTypeId: '.gen-ai',
isPreconfigured: false,
isDeprecated: false,
referencedByCount: 0,
isMissingSecrets: false,
isSystemAction: false,
secrets: {},
id: 'c5f91dc0-2197-11ee-aded-897192c5d6f5',
name: 'OpenAI',
config: {
apiProvider: 'OpenAI 1',
apiUrl: 'https://api.openai.com/v1/chat/completions',
},
},
{
actionTypeId: '.gen-ai',
isPreconfigured: false,
isDeprecated: false,
referencedByCount: 0,
isMissingSecrets: false,
isSystemAction: false,
secrets: {},
id: 'c7f91dc0-2197-11ee-aded-897192c5d633',
name: 'OpenAI',
config: {
apiProvider: 'OpenAI 2',
apiUrl: 'https://api.openai.com/v1/chat/completions',
},
},
];
const result = getDefaultConnector(connectors);
expect(result).toBeUndefined();
});
});
describe('getFormattedMessageContent', () => {
it('returns the value of the action_input property when `content` has properly prefixed and suffixed JSON with the action_input property', () => {
const content = '```json\n{"action_input": "value from action_input"}\n```';
expect(getFormattedMessageContent(content)).toBe('value from action_input');
});
it('returns the original content when `content` has properly formatted JSON WITHOUT the action_input property', () => {
const content = '```json\n{"some_key": "some value"}\n```';
expect(getFormattedMessageContent(content)).toBe(content);
});
it('returns the original content when `content` has improperly formatted JSON', () => {
const content = '```json\n{"action_input": "value from action_input",}\n```'; // <-- the trailing comma makes it invalid
expect(getFormattedMessageContent(content)).toBe(content);
});
it('returns the original content when `content` is missing the prefix', () => {
const content = '{"action_input": "value from action_input"}\n```'; // <-- missing prefix
expect(getFormattedMessageContent(content)).toBe(content);
});
it('returns the original content when `content` is missing the suffix', () => {
const content = '```json\n{"action_input": "value from action_input"}'; // <-- missing suffix
expect(getFormattedMessageContent(content)).toBe(content);
});
it('returns the original content when `content` does NOT contain a JSON string', () => {
const content = 'plain text content';
expect(getFormattedMessageContent(content)).toBe(content);
});
});
describe('getOptionalRequestParams', () => {
it('should return an empty object when ragOnAlerts is false', () => {
const params = {
alerts: true,
alertsIndexPattern: 'indexPattern',
allow: ['a', 'b', 'c'],
allowReplacement: ['b', 'c'],
ragOnAlerts: false, // <-- false
replacements: { key: 'value' },
size: 10,
};
const result = getOptionalRequestParams(params);
expect(result).toEqual({});
});
it('should return an empty object when alerts is false', () => {
const params = {
alerts: false, // <-- false
alertsIndexPattern: 'indexPattern',
allow: ['a', 'b', 'c'],
allowReplacement: ['b', 'c'],
ragOnAlerts: true,
replacements: { key: 'value' },
size: 10,
};
const result = getOptionalRequestParams(params);
expect(result).toEqual({});
});
it('should return the optional request params when ragOnAlerts is true and alerts is true', () => {
const params = {
alerts: true,
alertsIndexPattern: 'indexPattern',
allow: ['a', 'b', 'c'],
allowReplacement: ['b', 'c'],
ragOnAlerts: true,
replacements: { key: 'value' },
size: 10,
};
const result = getOptionalRequestParams(params);
expect(result).toEqual({
alertsIndexPattern: 'indexPattern',
allow: ['a', 'b', 'c'],
allowReplacement: ['b', 'c'],
replacements: { key: 'value' },
size: 10,
});
});
it('should return (only) the optional request params that are defined when some optional params are not provided', () => {
const params = {
alerts: true,
ragOnAlerts: true,
allow: ['a', 'b', 'c'], // all the others are undefined
};
const result = getOptionalRequestParams(params);
expect(result).toEqual({
allow: ['a', 'b', 'c'],
});
});
});
describe('hasParsableResponse', () => {
it('returns true when assistantLangChain is true', () => {
const result = hasParsableResponse({
alerts: false,
assistantLangChain: true,
ragOnAlerts: false,
});
expect(result).toBe(true);
});
it('returns true when ragOnAlerts is true and alerts is true', () => {
const result = hasParsableResponse({
alerts: true,
assistantLangChain: false,
ragOnAlerts: true,
});
expect(result).toBe(true);
});
it('returns false when assistantLangChain, ragOnAlerts, and alerts are all false', () => {
const result = hasParsableResponse({
alerts: false,
assistantLangChain: false,
ragOnAlerts: false,
});
expect(result).toBe(false);
});
});
});