mirror of
https://github.com/elastic/kibana.git
synced 2025-06-28 11:05:39 -04:00
15 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Andrew Macri
|
3f0fa7d245
|
[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts (#172542)
## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example: - _How many alerts are currently open?_ - _Which alerts should I look at first?_ - _Did we have any alerts with suspicious activity on Windows machines?_ ### More context Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature: - Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](https://github.com/elastic/kibana/pull/167097) - Applies the user's [anonymization](https://github.com/elastic/kibana/pull/159857) settings to those alerts - Only fields allowed by the user will be sent as context to the LLM - Users may enable or disable anonymization for specific fields (via settings) - Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM: 
Steph Milovic
|
39caf945fa
|
[Security solution] Add feature flag for AI streaming (#172505) | ||
Garrett Spong
|
82173b5c6e
|
[Security Solution] [Elastic AI Assistant] Adds APM instrumentation and LangSmith test data integration (#171153)
## Summary This PR instruments the Elastic AI Assistant with the Kibana APM Agent enabling the tracing of retrievers, llms, chains, and tools which can then be viewed within the Observability app. This PR also improves the Assistant Model Evaluation tooling by enabling support for pulling and running test datasets from LangSmith. If the `assistantModelEvaluation` experimental feature flag is enabled, and an APM server is configured, messages that have a corresponding trace will have an additional `View APM trace` action: <p align="center"> <img width="500" src=" |
||
|
Steph Milovic
|
d20161030d
|
[Security solution] Bedrock streaming and token tracking (#170815) | ||
|
Steph Milovic
|
2a52a361ab
|
[Security solution] Elastic Assistant, OpenAI stream (#169729) | ||
|
Garrett Spong
|
60bb1f8da1
|
[Security Solution] [Elastic AI Assistant] Throw error if Knowledge Base is enabled but ELSER is unavailable (#169330)
## Summary
This fixes the Knowledge Base UX a bit by throwing an error if somehow
ELSER has been disabled in the background, and instructs the user on how
to resolve or to disable the Knowledge Base to continue.
Additionally, if ELSER is not available, we prevent the enabling of the
Knowledge Base as to not provide a degraded experience when ELSER and
the ES|QL documentation is not available.
<p align="center">
<img width="500"
src="
|
||
|
Steph Milovic
|
8284398023
|
[Security solution] Fix OpenAI token reporting (#169156) | ||
|
Steph Milovic
|
2174a95807
|
[Security solution] Improve AI connector error handling (#167674) | ||
|
Garrett Spong
|
3ba0f32952
|
[Security Solution][Elastic AI Assistant] Adds Model Evaluation Tooling (#167220)
## Summary This PR introduces a new `internal/elastic_assistant/evaluate` route and `Evaluation` Advanced Setting within the Assistant for benchmarking and testing models, agents, and other aspects of the Assistant configuration. Enable via the `assistantModelEvaluation` experimental feature in your `kibana.dev.yml` (and better add `discoverInTimeline` for good measure as well! :) > xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'discoverInTimeline'] Then access from within the `Advanced Settings` modal in the Assistant. To use, first select your Connectors/Models, then corresponding Agent configurations, then what model you would like to use for final evaluation, the evaluation type, and if `custom`, you can specify the evaluation prompt that is sent off to the evaluator model. Finally, specify the `dataset`, and `output index` that the results should be written to, then click `Perform evaluation`. Sample datasets can be found in `x-pack/plugins/elastic_assistant/server/lib/model_evaluator/datasets`, and include: * `esql_dataset.json` * `query_dataset.json` * `security_labs.json` * `security_questions_dataset.json` <p align="center"> <img width="500" src=" |
||
|
Steph Milovic
|
bacebd27e0
|
[Security solution] AWS Bedrock connector (#166662) | ||
|
Andrew Macri
|
077be69de1
|
[Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER (#167097)
## [Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER
This PR integrates [LangChain](https://www.langchain.com/) [Agents](https://js.langchain.com/docs/modules/agents/) and [Tools](https://js.langchain.com/docs/modules/agents/tools/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant).
These abstractions enable the LLM to dynamically choose whether or not to query, via [ELSER](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html), an [ES|QL](https://www.elastic.co/blog/elasticsearch-query-language-esql) knowledge base. Context from the knowledge base is used to generate `ES|QL` queries, or answer questions about `ES|QL`.
Registration of the tool occurs in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
```typescript
const tools: Tool[] = [
new ChainTool({
name: 'esql-language-knowledge-base',
description:
'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
chain,
}),
];
```
The `tools` array above may be updated in future PRs to include, for example, an `ES|QL` query validator endpoint.
### Details
The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:
1. Creates a `RetrievalQAChain` from an `ELSER` backed `ElasticsearchStore`, which serves as a knowledge base for `ES|QL`:
```typescript
// ELSER backed ElasticsearchStore for Knowledge Base
const esStore = new ElasticsearchStore(esClient, KNOWLEDGE_BASE_INDEX_PATTERN, logger);
const chain = RetrievalQAChain.fromLLM(llm, esStore.asRetriever());
```
2. Registers the chain as a tool, which may be invoked by the LLM based on its description:
```typescript
const tools: Tool[] = [
new ChainTool({
name: 'esql-language-knowledge-base',
description:
'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
chain,
}),
];
```
3. Creates an Agent executor that combines the `tools` above, the `ActionsClientLlm` (an abstraction that calls `actionsClient.execute`), and memory of the previous messages in the conversation:
```typescript
const executor = await initializeAgentExecutorWithOptions(tools, llm, {
agentType: 'chat-conversational-react-description',
memory,
verbose: false,
});
```
Note: Set `verbose` above to `true` to for detailed debugging output from LangChain.
4. Calls the `executor`, kicking it off with `latestMessage`:
```typescript
await executor.call({ input: latestMessage[0].content });
```
### Changes to `x-pack/packages/kbn-elastic-assistant`
A client side change was required to the assistant, because the response returned from the agent executor is JSON. This response is parsed on the client in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx`:
```typescript
return assistantLangChain ? getFormattedMessageContent(result) : result;
```
Client-side parsing of the response only happens when then `assistantLangChain` feature flag is `true`.
## Desk testing
Set
```typescript
assistantLangChain={true}
```
in `x-pack/plugins/security_solution/public/assistant/provider.tsx` to enable this experimental feature in development environments.
Also (optionally) set `verbose` to `true` in the following code in ``x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts``:
```typescript
const executor = await initializeAgentExecutorWithOptions(tools, llm, {
agentType: 'chat-conversational-react-description',
memory,
verbose: true,
});
```
After setting the feature flag and optionally enabling verbose debugging output, you may ask the assistant to generate an `ES|QL` query, per the example in the next section.
### Example output
When the Elastic AI Assistant is asked:
```
From employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. "September 2019". Only show the query
```
it replies:
```
Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:
FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5
```
Per the screenshot below:
## Summary Advanced Settings to enable the Knowledge Base is currently behind the same code toggle introduced in https://github.com/elastic/kibana/pull/164908, please modify `assistantLangChain` to be `true` to enable the Advanced Settings and Knowledge Base options: |
||
|
Andrew Macri
|
3935548f36
|
[Security Solution] [Elastic AI Assistant] LangChain integration (experimental) (#164908)
## [Security Solution] [Elastic AI Assistant] LangChain integration (experimental) This PR integrates [LangChain](https://www.langchain.com/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant) as an experimental, alternative execution path. ### How it works - There are virtually no client side changes to the assistant, apart from a new branch in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx` that chooses a path based on the value of the `assistantLangChain` flag: ```typescript const path = assistantLangChain ? `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` : `/api/actions/connector/${apiConfig?.connectorId}/_execute`; ``` Execution of the LangChain chain happens server-side. The new route still executes the request via the `connectorId` in the route, but the connector won't execute the request exactly as it was sent by the client. Instead, the connector will execute one (or more) prompts that are generated by LangChain. Requests routed to `/internal/elastic_assistant/actions/connector/${apiConfig?.connectorId}/_execute` will be processed by a new Kibana plugin located in: ``` x-pack/plugins/elastic_assistant ``` - Requests are processed in the `postActionsConnectorExecuteRoute` handler in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts`. The `postActionsConnectorExecuteRoute` route handler: 1. Extracts the chat messages sent by the assistant 2. Converts the extracted messages to the format expected by LangChain 3. Passes the converted messages to `executeCustomLlmChain` - The `executeCustomLlmChain` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`: 1. Splits the messages into `pastMessages` and `latestMessage`, where the latter contains only the last message sent by the user 2. Wraps the conversation history in the `BufferMemory` LangChain abstraction 3. Executes the chain, kicking it off with `latestMessage` ```typescript const llm = new ActionsClientLlm({ actions, connectorId, request }); const pastMessages = langchainMessages.slice(0, -1); // all but the last message const latestMessage = langchainMessages.slice(-1); // the last message const memory = new BufferMemory({ chatHistory: new ChatMessageHistory(pastMessages), }); const chain = new ConversationChain({ llm, memory }); await chain.call({ input: latestMessage[0].content }); // kick off the chain with the last message }; ``` - When LangChain executes the chain, it will invoke `ActionsClientLlm`'s `_call` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/llm/actions_client_llm.ts` one or more times. The `_call` function's signature is defined by LangChain: ``` async _call(prompt: string): Promise<string> ``` - The contents of `prompt` are completely determined by LangChain. - The string returned by the promise is the "answer" from the LLM The `ActionsClientLlm` extends LangChain's LLM interface: ```typescript export class ActionsClientLlm extends LLM ``` This let's us do additional "work" in the `_call` function: 1. Create a new assistant message using the contents of the `prompt` (`string`) argument to `_call` 2. Create a request body in the format expected by the connector 3. Create an actions client from the authenticated request context 4. Execute the actions client with the request body 5. Save the raw response from the connector, because that's what the assistant expects 6. Return the result as a plain string, as per the contact of `_call` ## Desk testing This experimental LangChain integration may NOT be enabled via a feature flag (yet). Set ```typescript assistantLangChain={true} ``` in `x-pack/plugins/security_solution/public/app/app.tsx` to enable this experimental feature in development environments. |
||
|
Garrett Spong
|
b323923e65
|
[Security Solution] [Elastic AI Assistant] Consolidates settings into a single modal (#160468)
## Summary This PR fixes the disjointed settings across the assistant by combining them all into a single settings modal. It also resolves the Connector `Model` configuration not being available when using the `OpenAI` variant of the GenAI Connector. Additional issues resolved: - [x] Clearing conversation doesn't restore default system prompt - [X] Double repeated welcome prompt - [X] Clicking skip button broken Resolves: https://github.com/elastic/security-team/issues/7110 Resolves: https://github.com/elastic/kibana/pull/161039#pullrequestreview-1517129764 Resolves: https://github.com/elastic/kibana/pull/161027#pullrequestreview-1523018176 #### Conversations <p align="center"> <img width="500" src=" |
||
|
Garrett Spong
|
4e38817a4d
|
[Security Solution] Elastic Security Assistant (#156933)
## [Security Solution] Elastic Security Assistant The _Elastic Security Assistant_ has entered the chat, integrating generative AI and large language models (LLMs) into the workflows of Elastic Security users. Bring your alerts, events, rules, and data quality checks into the conversation. < |